| 37 37 37 37 37 36 41 41 41 41 41 41 41 41 39 36 35 35 39 39 39 39 39 40 41 41 40 40 39 40 39 41 41 39 41 41 40 40 41 3 3 8 8 1 1 38 37 2 2 2 2 37 37 37 37 37 37 37 37 37 160 160 160 160 140 47 47 47 160 160 41 160 160 47 140 140 140 140 140 47 140 47 160 161 161 161 161 161 161 48 140 140 161 39 41 41 41 41 41 39 40 2 2 8 8 8 1 8 8 8 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 197 198 199 200 201 202 203 204 205 206 207 208 209 210 211 212 213 214 215 216 217 218 219 220 221 222 223 224 225 226 227 228 229 230 231 232 233 234 235 236 237 238 239 240 241 242 243 244 245 246 247 248 249 250 251 252 253 254 255 256 257 258 259 260 261 262 263 264 265 266 267 268 269 270 271 272 273 274 275 276 277 278 279 280 281 282 283 284 285 286 287 288 289 290 291 292 293 294 295 296 297 298 299 300 301 302 303 304 305 306 307 308 309 310 311 312 313 314 315 316 317 318 319 320 321 322 323 324 325 326 327 328 329 330 331 332 333 334 335 336 337 338 339 340 341 342 343 344 345 346 347 348 349 350 351 352 353 354 355 356 357 358 359 360 361 362 363 364 365 366 367 368 369 370 371 372 373 374 375 376 377 378 379 380 381 382 383 384 385 386 387 388 389 390 391 392 393 394 395 396 397 398 399 400 401 402 403 404 405 406 407 408 409 410 411 412 413 414 415 416 417 418 419 420 421 422 423 424 425 426 427 428 429 430 431 432 433 434 435 436 437 438 439 440 441 442 443 444 445 446 447 448 449 450 451 452 453 454 455 456 457 458 459 460 461 462 463 464 465 466 467 468 469 470 471 472 473 474 475 476 477 478 479 480 481 482 483 484 485 486 487 488 489 490 491 492 493 494 495 496 497 498 499 500 501 502 503 504 505 506 507 508 509 510 511 512 513 514 515 516 517 518 519 520 521 522 523 524 525 526 527 528 529 530 531 532 533 534 535 536 537 538 539 540 541 542 543 544 545 546 547 548 549 550 551 552 553 554 555 556 557 558 559 560 561 562 563 564 565 566 567 568 569 570 571 572 573 574 575 576 577 578 579 580 581 582 583 584 585 586 587 588 589 590 591 592 593 594 595 596 597 598 599 600 601 602 603 604 605 606 607 608 609 610 611 612 613 614 615 616 617 618 619 620 621 622 623 624 625 626 627 628 629 630 631 632 633 634 635 636 637 638 639 640 641 642 643 644 645 646 647 648 649 650 651 652 653 654 655 656 657 658 659 660 661 662 663 664 665 666 667 668 669 670 671 672 673 674 675 676 677 678 679 680 681 682 683 684 685 686 687 688 689 690 691 692 693 694 695 696 697 698 699 700 701 702 703 704 705 706 707 708 709 710 711 712 713 714 715 716 717 718 719 720 721 722 723 724 725 726 727 728 729 730 731 732 733 734 735 736 737 738 739 740 741 742 743 744 745 746 747 748 749 750 751 752 753 754 755 756 757 758 759 760 761 762 763 764 765 766 767 768 769 770 771 772 773 774 775 776 777 778 779 780 781 782 783 784 785 786 787 788 789 790 791 792 793 794 795 796 797 798 799 800 801 802 803 804 805 806 807 808 809 810 811 812 813 814 815 816 817 818 819 820 821 822 823 824 825 826 827 828 829 830 831 832 833 834 835 836 837 838 839 840 841 842 843 844 845 846 847 848 849 850 851 852 853 854 855 856 857 858 859 860 861 862 863 864 865 866 867 868 869 870 871 872 873 874 875 876 877 878 879 880 881 882 883 884 885 886 887 888 889 890 891 892 893 894 895 896 897 898 899 900 901 902 903 904 905 906 907 908 909 910 911 912 913 914 915 916 917 918 919 920 921 922 923 924 925 926 927 928 929 930 931 932 933 934 935 936 937 938 939 940 941 942 943 944 945 946 947 948 949 950 951 952 953 954 955 956 957 958 959 960 961 962 963 964 965 966 967 968 969 970 971 972 973 974 975 976 977 978 979 980 981 982 983 984 985 986 987 988 989 990 991 992 993 994 995 996 997 998 999 1000 1001 1002 1003 1004 1005 1006 1007 1008 1009 1010 1011 1012 1013 1014 1015 1016 1017 1018 1019 1020 1021 1022 1023 1024 1025 1026 1027 1028 1029 1030 1031 1032 1033 1034 1035 1036 1037 1038 1039 1040 1041 1042 1043 1044 1045 1046 1047 1048 1049 1050 1051 1052 1053 1054 1055 1056 1057 1058 1059 1060 1061 1062 1063 1064 1065 1066 1067 1068 1069 1070 1071 1072 1073 1074 1075 1076 1077 1078 1079 1080 1081 1082 1083 1084 1085 1086 1087 1088 1089 1090 1091 1092 1093 1094 1095 1096 1097 1098 1099 1100 1101 1102 1103 1104 1105 1106 1107 1108 1109 1110 1111 1112 1113 1114 1115 1116 1117 1118 1119 1120 1121 1122 1123 1124 1125 1126 1127 1128 1129 1130 1131 1132 1133 1134 1135 1136 1137 1138 1139 1140 1141 1142 1143 1144 1145 1146 1147 1148 1149 1150 1151 1152 1153 1154 1155 1156 1157 1158 1159 1160 1161 1162 1163 1164 1165 1166 1167 1168 1169 1170 1171 1172 1173 1174 1175 1176 1177 1178 1179 1180 1181 1182 1183 1184 1185 1186 1187 1188 1189 1190 1191 1192 1193 1194 1195 1196 1197 1198 1199 1200 1201 1202 1203 1204 1205 1206 1207 1208 1209 1210 1211 1212 1213 1214 1215 1216 1217 1218 1219 1220 1221 1222 1223 1224 1225 1226 1227 1228 1229 1230 1231 1232 1233 1234 1235 1236 1237 1238 1239 1240 1241 1242 1243 1244 1245 1246 1247 1248 1249 1250 1251 1252 1253 1254 1255 1256 1257 1258 1259 1260 1261 1262 1263 1264 1265 1266 1267 1268 1269 1270 1271 1272 1273 1274 1275 1276 1277 1278 1279 1280 1281 1282 1283 1284 1285 1286 1287 1288 1289 1290 1291 1292 1293 1294 1295 1296 1297 1298 1299 1300 1301 1302 1303 1304 1305 1306 1307 1308 1309 1310 1311 1312 1313 1314 1315 1316 1317 1318 1319 1320 1321 1322 1323 1324 1325 1326 1327 1328 1329 1330 1331 1332 1333 1334 1335 1336 1337 1338 1339 1340 1341 1342 1343 1344 1345 1346 1347 1348 1349 1350 1351 1352 1353 1354 1355 1356 1357 1358 1359 1360 1361 1362 1363 1364 1365 1366 1367 1368 1369 1370 1371 1372 1373 1374 1375 1376 1377 1378 1379 1380 1381 1382 1383 1384 1385 1386 1387 1388 1389 1390 1391 1392 1393 1394 1395 1396 1397 1398 1399 1400 1401 1402 1403 1404 1405 1406 1407 1408 1409 1410 1411 1412 1413 1414 1415 1416 1417 1418 1419 1420 1421 1422 1423 1424 1425 1426 1427 1428 1429 1430 1431 1432 1433 1434 1435 1436 1437 1438 1439 1440 1441 1442 1443 1444 1445 1446 1447 1448 1449 1450 1451 1452 1453 1454 1455 1456 1457 1458 1459 1460 1461 1462 1463 1464 1465 1466 1467 1468 1469 1470 1471 1472 1473 1474 1475 1476 1477 1478 1479 1480 1481 1482 1483 1484 1485 1486 1487 1488 1489 1490 1491 1492 1493 1494 1495 1496 1497 1498 1499 1500 1501 1502 1503 1504 1505 1506 1507 1508 1509 1510 1511 1512 1513 1514 1515 1516 1517 1518 1519 1520 1521 1522 1523 1524 1525 1526 1527 1528 1529 1530 1531 1532 1533 1534 1535 1536 1537 1538 1539 1540 1541 1542 1543 1544 1545 1546 1547 1548 1549 1550 1551 1552 1553 1554 1555 1556 1557 1558 1559 1560 1561 1562 1563 1564 1565 1566 1567 1568 1569 1570 1571 1572 1573 1574 1575 1576 1577 1578 1579 1580 1581 1582 1583 1584 1585 1586 1587 1588 1589 1590 1591 1592 1593 1594 1595 1596 1597 1598 1599 1600 1601 1602 1603 1604 1605 1606 1607 1608 1609 1610 1611 1612 1613 1614 1615 1616 1617 1618 1619 1620 1621 1622 1623 1624 1625 1626 1627 1628 1629 1630 1631 1632 1633 1634 1635 1636 1637 1638 1639 1640 1641 1642 1643 1644 1645 1646 1647 1648 1649 1650 1651 1652 1653 1654 1655 1656 1657 1658 1659 1660 1661 1662 1663 1664 1665 1666 1667 1668 1669 1670 1671 1672 1673 1674 1675 1676 1677 1678 1679 1680 1681 1682 1683 1684 1685 1686 1687 1688 1689 1690 1691 1692 1693 1694 1695 1696 1697 1698 1699 1700 1701 1702 1703 1704 1705 1706 1707 1708 1709 1710 1711 1712 1713 1714 1715 1716 1717 1718 1719 1720 1721 1722 1723 1724 1725 1726 1727 1728 1729 1730 1731 1732 1733 1734 1735 1736 1737 1738 1739 1740 1741 1742 1743 1744 1745 1746 1747 1748 1749 1750 1751 1752 1753 1754 1755 1756 1757 1758 1759 1760 1761 1762 1763 1764 1765 1766 1767 1768 1769 1770 1771 1772 1773 1774 1775 1776 1777 1778 1779 1780 1781 1782 1783 1784 1785 1786 1787 1788 1789 1790 1791 1792 1793 1794 1795 1796 1797 1798 1799 1800 1801 1802 1803 1804 1805 1806 1807 1808 1809 1810 1811 1812 1813 1814 1815 1816 1817 1818 1819 1820 1821 1822 1823 1824 1825 1826 1827 1828 1829 1830 1831 1832 1833 1834 1835 1836 1837 1838 1839 1840 1841 1842 1843 1844 1845 1846 1847 1848 1849 1850 1851 1852 1853 1854 1855 1856 1857 1858 1859 1860 1861 1862 1863 1864 1865 1866 1867 1868 1869 1870 1871 1872 | /* $NetBSD: usbdi.c,v 1.243 2022/08/20 11:32:20 riastradh Exp $ */ /* * Copyright (c) 1998, 2012, 2015 The NetBSD Foundation, Inc. * All rights reserved. * * This code is derived from software contributed to The NetBSD Foundation * by Lennart Augustsson (lennart@augustsson.net) at * Carlstedt Research & Technology, Matthew R. Green (mrg@eterna.com.au), * and Nick Hudson. * * Redistribution and use in source and binary forms, with or without * modification, are permitted provided that the following conditions * are met: * 1. Redistributions of source code must retain the above copyright * notice, this list of conditions and the following disclaimer. * 2. Redistributions in binary form must reproduce the above copyright * notice, this list of conditions and the following disclaimer in the * documentation and/or other materials provided with the distribution. * * THIS SOFTWARE IS PROVIDED BY THE NETBSD FOUNDATION, INC. AND CONTRIBUTORS * ``AS IS'' AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED * TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR * PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE FOUNDATION OR CONTRIBUTORS * BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR * CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF * SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS * INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN * CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) * ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE * POSSIBILITY OF SUCH DAMAGE. */ #include <sys/cdefs.h> __KERNEL_RCSID(0, "$NetBSD: usbdi.c,v 1.243 2022/08/20 11:32:20 riastradh Exp $"); #ifdef _KERNEL_OPT #include "opt_usb.h" #include "opt_compat_netbsd.h" #include "usb_dma.h" #endif #include <sys/param.h> #include <sys/systm.h> #include <sys/kernel.h> #include <sys/device.h> #include <sys/kmem.h> #include <sys/proc.h> #include <sys/bus.h> #include <sys/cpu.h> #include <dev/usb/usb.h> #include <dev/usb/usbdi.h> #include <dev/usb/usbdi_util.h> #include <dev/usb/usbdivar.h> #include <dev/usb/usb_mem.h> #include <dev/usb/usb_quirks.h> #include <dev/usb/usb_sdt.h> #include <dev/usb/usbhist.h> /* UTF-8 encoding stuff */ #include <fs/unicode.h> SDT_PROBE_DEFINE5(usb, device, pipe, open, "struct usbd_interface *"/*iface*/, "uint8_t"/*address*/, "uint8_t"/*flags*/, "int"/*ival*/, "struct usbd_pipe *"/*pipe*/); SDT_PROBE_DEFINE7(usb, device, pipe, open__intr, "struct usbd_interface *"/*iface*/, "uint8_t"/*address*/, "uint8_t"/*flags*/, "int"/*ival*/, "usbd_callback"/*cb*/, "void *"/*cookie*/, "struct usbd_pipe *"/*pipe*/); SDT_PROBE_DEFINE2(usb, device, pipe, transfer__start, "struct usbd_pipe *"/*pipe*/, "struct usbd_xfer *"/*xfer*/); SDT_PROBE_DEFINE3(usb, device, pipe, transfer__done, "struct usbd_pipe *"/*pipe*/, "struct usbd_xfer *"/*xfer*/, "usbd_status"/*err*/); SDT_PROBE_DEFINE2(usb, device, pipe, start, "struct usbd_pipe *"/*pipe*/, "struct usbd_xfer *"/*xfer*/); SDT_PROBE_DEFINE1(usb, device, pipe, close, "struct usbd_pipe *"/*pipe*/); SDT_PROBE_DEFINE1(usb, device, pipe, abort__start, "struct usbd_pipe *"/*pipe*/); SDT_PROBE_DEFINE1(usb, device, pipe, abort__done, "struct usbd_pipe *"/*pipe*/); SDT_PROBE_DEFINE1(usb, device, pipe, clear__endpoint__stall, "struct usbd_pipe *"/*pipe*/); SDT_PROBE_DEFINE1(usb, device, pipe, clear__endpoint__toggle, "struct usbd_pipe *"/*pipe*/); SDT_PROBE_DEFINE5(usb, device, xfer, create, "struct usbd_xfer *"/*xfer*/, "struct usbd_pipe *"/*pipe*/, "size_t"/*len*/, "unsigned int"/*flags*/, "unsigned int"/*nframes*/); SDT_PROBE_DEFINE1(usb, device, xfer, start, "struct usbd_xfer *"/*xfer*/); SDT_PROBE_DEFINE1(usb, device, xfer, preabort, "struct usbd_xfer *"/*xfer*/); SDT_PROBE_DEFINE1(usb, device, xfer, abort, "struct usbd_xfer *"/*xfer*/); SDT_PROBE_DEFINE1(usb, device, xfer, timeout, "struct usbd_xfer *"/*xfer*/); SDT_PROBE_DEFINE2(usb, device, xfer, done, "struct usbd_xfer *"/*xfer*/, "usbd_status"/*status*/); SDT_PROBE_DEFINE1(usb, device, xfer, destroy, "struct usbd_xfer *"/*xfer*/); SDT_PROBE_DEFINE5(usb, device, request, start, "struct usbd_device *"/*dev*/, "usb_device_request_t *"/*req*/, "size_t"/*len*/, "int"/*flags*/, "uint32_t"/*timeout*/); SDT_PROBE_DEFINE7(usb, device, request, done, "struct usbd_device *"/*dev*/, "usb_device_request_t *"/*req*/, "size_t"/*actlen*/, "int"/*flags*/, "uint32_t"/*timeout*/, "void *"/*data*/, "usbd_status"/*status*/); Static void usbd_ar_pipe(struct usbd_pipe *); Static void usbd_start_next(struct usbd_pipe *); Static usbd_status usbd_open_pipe_ival (struct usbd_interface *, uint8_t, uint8_t, struct usbd_pipe **, int); static void *usbd_alloc_buffer(struct usbd_xfer *, uint32_t); static void usbd_free_buffer(struct usbd_xfer *); static struct usbd_xfer *usbd_alloc_xfer(struct usbd_device *, unsigned int); static void usbd_free_xfer(struct usbd_xfer *); static void usbd_xfer_timeout(void *); static void usbd_xfer_timeout_task(void *); static bool usbd_xfer_probe_timeout(struct usbd_xfer *); static void usbd_xfer_cancel_timeout_async(struct usbd_xfer *); #if defined(USB_DEBUG) void usbd_dump_iface(struct usbd_interface *iface) { USBHIST_FUNC(); USBHIST_CALLARGS(usbdebug, "iface %#jx", (uintptr_t)iface, 0, 0, 0); if (iface == NULL) return; USBHIST_LOG(usbdebug, " device = %#jx idesc = %#jx index = %jd", (uintptr_t)iface->ui_dev, (uintptr_t)iface->ui_idesc, iface->ui_index, 0); USBHIST_LOG(usbdebug, " altindex=%jd", iface->ui_altindex, 0, 0, 0); } void usbd_dump_device(struct usbd_device *dev) { USBHIST_FUNC(); USBHIST_CALLARGS(usbdebug, "dev = %#jx", (uintptr_t)dev, 0, 0, 0); if (dev == NULL) return; USBHIST_LOG(usbdebug, " bus = %#jx default_pipe = %#jx", (uintptr_t)dev->ud_bus, (uintptr_t)dev->ud_pipe0, 0, 0); USBHIST_LOG(usbdebug, " address = %jd config = %jd depth = %jd ", dev->ud_addr, dev->ud_config, dev->ud_depth, 0); USBHIST_LOG(usbdebug, " speed = %jd self_powered = %jd " "power = %jd langid = %jd", dev->ud_speed, dev->ud_selfpowered, dev->ud_power, dev->ud_langid); } void usbd_dump_endpoint(struct usbd_endpoint *endp) { USBHIST_FUNC(); USBHIST_CALLARGS(usbdebug, "endp = %#jx", (uintptr_t)endp, 0, 0, 0); if (endp == NULL) return; USBHIST_LOG(usbdebug, " edesc = %#jx refcnt = %jd", (uintptr_t)endp->ue_edesc, endp->ue_refcnt, 0, 0); if (endp->ue_edesc) USBHIST_LOG(usbdebug, " bEndpointAddress=0x%02jx", endp->ue_edesc->bEndpointAddress, 0, 0, 0); } void usbd_dump_queue(struct usbd_pipe *pipe) { struct usbd_xfer *xfer; USBHIST_FUNC(); USBHIST_CALLARGS(usbdebug, "pipe = %#jx", (uintptr_t)pipe, 0, 0, 0); SIMPLEQ_FOREACH(xfer, &pipe->up_queue, ux_next) { USBHIST_LOG(usbdebug, " xfer = %#jx", (uintptr_t)xfer, 0, 0, 0); } } void usbd_dump_pipe(struct usbd_pipe *pipe) { USBHIST_FUNC(); USBHIST_CALLARGS(usbdebug, "pipe = %#jx", (uintptr_t)pipe, 0, 0, 0); if (pipe == NULL) return; usbd_dump_iface(pipe->up_iface); usbd_dump_device(pipe->up_dev); usbd_dump_endpoint(pipe->up_endpoint); USBHIST_LOG(usbdebug, "(usbd_dump_pipe)", 0, 0, 0, 0); USBHIST_LOG(usbdebug, " running = %jd aborting = %jd", pipe->up_running, pipe->up_aborting, 0, 0); USBHIST_LOG(usbdebug, " intrxfer = %#jx, repeat = %jd, " "interval = %jd", (uintptr_t)pipe->up_intrxfer, pipe->up_repeat, pipe->up_interval, 0); } #endif usbd_status usbd_open_pipe(struct usbd_interface *iface, uint8_t address, uint8_t flags, struct usbd_pipe **pipe) { return (usbd_open_pipe_ival(iface, address, flags, pipe, USBD_DEFAULT_INTERVAL)); } usbd_status usbd_open_pipe_ival(struct usbd_interface *iface, uint8_t address, uint8_t flags, struct usbd_pipe **pipe, int ival) { struct usbd_pipe *p = NULL; struct usbd_endpoint *ep = NULL /* XXXGCC */; bool piperef = false; usbd_status err; int i; USBHIST_FUNC(); USBHIST_CALLARGS(usbdebug, "iface = %#jx address = %#jx flags = %#jx", (uintptr_t)iface, address, flags, 0); /* * Block usbd_set_interface so we have a snapshot of the * interface endpoints. They will remain stable until we drop * the reference in usbd_close_pipe (or on failure here). */ err = usbd_iface_piperef(iface); if (err) goto out; piperef = true; /* Find the endpoint at this address. */ for (i = 0; i < iface->ui_idesc->bNumEndpoints; i++) { ep = &iface->ui_endpoints[i]; if (ep->ue_edesc == NULL) { err = USBD_IOERROR; goto out; } if (ep->ue_edesc->bEndpointAddress == address) break; } if (i == iface->ui_idesc->bNumEndpoints) { err = USBD_BAD_ADDRESS; goto out; } /* Set up the pipe with this endpoint. */ err = usbd_setup_pipe_flags(iface->ui_dev, iface, ep, ival, &p, flags); if (err) goto out; /* Success! */ *pipe = p; p = NULL; /* handed off to caller */ piperef = false; /* handed off to pipe */ SDT_PROBE5(usb, device, pipe, open, iface, address, flags, ival, p); err = USBD_NORMAL_COMPLETION; out: if (p) usbd_close_pipe(p); if (piperef) usbd_iface_pipeunref(iface); return err; } usbd_status usbd_open_pipe_intr(struct usbd_interface *iface, uint8_t address, uint8_t flags, struct usbd_pipe **pipe, void *priv, void *buffer, uint32_t len, usbd_callback cb, int ival) { usbd_status err; struct usbd_xfer *xfer; struct usbd_pipe *ipipe; USBHIST_FUNC(); USBHIST_CALLARGS(usbdebug, "address = %#jx flags = %#jx len = %jd", address, flags, len, 0); err = usbd_open_pipe_ival(iface, address, USBD_EXCLUSIVE_USE | (flags & USBD_MPSAFE), &ipipe, ival); if (err) return err; err = usbd_create_xfer(ipipe, len, flags, 0, &xfer); if (err) goto bad1; usbd_setup_xfer(xfer, priv, buffer, len, flags, USBD_NO_TIMEOUT, cb); ipipe->up_intrxfer = xfer; ipipe->up_repeat = 1; err = usbd_transfer(xfer); *pipe = ipipe; if (err != USBD_IN_PROGRESS) goto bad3; SDT_PROBE7(usb, device, pipe, open__intr, iface, address, flags, ival, cb, priv, ipipe); return USBD_NORMAL_COMPLETION; bad3: ipipe->up_intrxfer = NULL; ipipe->up_repeat = 0; usbd_destroy_xfer(xfer); bad1: usbd_close_pipe(ipipe); return err; } void usbd_close_pipe(struct usbd_pipe *pipe) { USBHIST_FUNC(); USBHIST_CALLED(usbdebug); KASSERT(pipe != NULL); usbd_lock_pipe(pipe); SDT_PROBE1(usb, device, pipe, close, pipe); if (!SIMPLEQ_EMPTY(&pipe->up_queue)) { printf("WARNING: pipe closed with active xfers on addr %d\n", pipe->up_dev->ud_addr); usbd_ar_pipe(pipe); } KASSERT(SIMPLEQ_EMPTY(&pipe->up_queue)); pipe->up_methods->upm_close(pipe); usbd_unlock_pipe(pipe); cv_destroy(&pipe->up_callingcv); if (pipe->up_intrxfer) usbd_destroy_xfer(pipe->up_intrxfer); usb_rem_task_wait(pipe->up_dev, &pipe->up_async_task, USB_TASKQ_DRIVER, NULL); usbd_endpoint_release(pipe->up_dev, pipe->up_endpoint); if (pipe->up_iface) usbd_iface_pipeunref(pipe->up_iface); kmem_free(pipe, pipe->up_dev->ud_bus->ub_pipesize); } usbd_status usbd_transfer(struct usbd_xfer *xfer) { struct usbd_pipe *pipe = xfer->ux_pipe; usbd_status err; unsigned int size, flags; USBHIST_FUNC(); USBHIST_CALLARGS(usbdebug, "xfer = %#jx, flags = %#jx, pipe = %#jx, running = %jd", (uintptr_t)xfer, xfer->ux_flags, (uintptr_t)pipe, pipe->up_running); KASSERT(xfer->ux_status == USBD_NOT_STARTED); SDT_PROBE1(usb, device, xfer, start, xfer); #ifdef USB_DEBUG if (usbdebug > 5) usbd_dump_queue(pipe); #endif xfer->ux_done = 0; KASSERT(xfer->ux_length == 0 || xfer->ux_buf != NULL); size = xfer->ux_length; flags = xfer->ux_flags; if (size != 0) { /* * Use the xfer buffer if none specified in transfer setup. * isoc transfers always use the xfer buffer, i.e. * ux_buffer is always NULL for isoc. */ if (xfer->ux_buffer == NULL) { xfer->ux_buffer = xfer->ux_buf; } /* * If not using the xfer buffer copy data to the * xfer buffer for OUT transfers of >0 length */ if (xfer->ux_buffer != xfer->ux_buf) { KASSERT(xfer->ux_buf); if (!usbd_xfer_isread(xfer)) { memcpy(xfer->ux_buf, xfer->ux_buffer, size); } } } usbd_lock_pipe(pipe); if (pipe->up_aborting) { /* * XXX For synchronous transfers this is fine. What to * do for asynchronous transfers? The callback is * never run, not even with status USBD_CANCELLED. */ usbd_unlock_pipe(pipe); USBHIST_LOG(usbdebug, "<- done xfer %#jx, aborting", (uintptr_t)xfer, 0, 0, 0); SDT_PROBE2(usb, device, xfer, done, xfer, USBD_CANCELLED); return USBD_CANCELLED; } /* xfer is not valid after the transfer method unless synchronous */ SDT_PROBE2(usb, device, pipe, transfer__start, pipe, xfer); do { #ifdef DIAGNOSTIC xfer->ux_state = XFER_ONQU; #endif SIMPLEQ_INSERT_TAIL(&pipe->up_queue, xfer, ux_next); if (pipe->up_running && pipe->up_serialise) { err = USBD_IN_PROGRESS; } else { pipe->up_running = 1; err = USBD_NORMAL_COMPLETION; } if (err) break; err = pipe->up_methods->upm_transfer(xfer); } while (0); SDT_PROBE3(usb, device, pipe, transfer__done, pipe, xfer, err); usbd_unlock_pipe(pipe); if (err != USBD_IN_PROGRESS && err) { /* * The transfer made it onto the pipe queue, but didn't get * accepted by the HCD for some reason. It needs removing * from the pipe queue. */ USBHIST_LOG(usbdebug, "xfer failed: %jd, reinserting", err, 0, 0, 0); usbd_lock_pipe(pipe); SDT_PROBE1(usb, device, xfer, preabort, xfer); #ifdef DIAGNOSTIC xfer->ux_state = XFER_BUSY; #endif SIMPLEQ_REMOVE_HEAD(&pipe->up_queue, ux_next); if (pipe->up_serialise) usbd_start_next(pipe); usbd_unlock_pipe(pipe); } if (!(flags & USBD_SYNCHRONOUS)) { USBHIST_LOG(usbdebug, "<- done xfer %#jx, not sync (err %jd)", (uintptr_t)xfer, err, 0, 0); KASSERTMSG(err != USBD_NORMAL_COMPLETION, "asynchronous xfer %p completed synchronously", xfer); return err; } if (err != USBD_IN_PROGRESS) { USBHIST_LOG(usbdebug, "<- done xfer %#jx, sync (err %jd)", (uintptr_t)xfer, err, 0, 0); SDT_PROBE2(usb, device, xfer, done, xfer, err); return err; } /* Sync transfer, wait for completion. */ usbd_lock_pipe(pipe); while (!xfer->ux_done) { if (pipe->up_dev->ud_bus->ub_usepolling) panic("usbd_transfer: not done"); USBHIST_LOG(usbdebug, "<- sleeping on xfer %#jx", (uintptr_t)xfer, 0, 0, 0); err = 0; if ((flags & USBD_SYNCHRONOUS_SIG) != 0) { err = cv_wait_sig(&xfer->ux_cv, pipe->up_dev->ud_bus->ub_lock); } else { cv_wait(&xfer->ux_cv, pipe->up_dev->ud_bus->ub_lock); } if (err) { if (!xfer->ux_done) { SDT_PROBE1(usb, device, xfer, abort, xfer); pipe->up_methods->upm_abort(xfer); } break; } } SDT_PROBE2(usb, device, xfer, done, xfer, xfer->ux_status); /* XXX Race to read xfer->ux_status? */ usbd_unlock_pipe(pipe); return xfer->ux_status; } /* Like usbd_transfer(), but waits for completion. */ usbd_status usbd_sync_transfer(struct usbd_xfer *xfer) { xfer->ux_flags |= USBD_SYNCHRONOUS; return usbd_transfer(xfer); } /* Like usbd_transfer(), but waits for completion and listens for signals. */ usbd_status usbd_sync_transfer_sig(struct usbd_xfer *xfer) { xfer->ux_flags |= USBD_SYNCHRONOUS | USBD_SYNCHRONOUS_SIG; return usbd_transfer(xfer); } static void * usbd_alloc_buffer(struct usbd_xfer *xfer, uint32_t size) { KASSERT(xfer->ux_buf == NULL); KASSERT(size != 0); xfer->ux_bufsize = 0; #if NUSB_DMA > 0 struct usbd_bus *bus = xfer->ux_bus; if (bus->ub_usedma) { usb_dma_t *dmap = &xfer->ux_dmabuf; KASSERT((bus->ub_dmaflags & USBMALLOC_COHERENT) == 0); int err = usb_allocmem(bus->ub_dmatag, size, 0, bus->ub_dmaflags, dmap); if (err) { return NULL; } xfer->ux_buf = KERNADDR(&xfer->ux_dmabuf, 0); xfer->ux_bufsize = size; return xfer->ux_buf; } #endif KASSERT(xfer->ux_bus->ub_usedma == false); xfer->ux_buf = kmem_alloc(size, KM_SLEEP); xfer->ux_bufsize = size; return xfer->ux_buf; } static void usbd_free_buffer(struct usbd_xfer *xfer) { KASSERT(xfer->ux_buf != NULL); KASSERT(xfer->ux_bufsize != 0); void *buf = xfer->ux_buf; uint32_t size = xfer->ux_bufsize; xfer->ux_buf = NULL; xfer->ux_bufsize = 0; #if NUSB_DMA > 0 struct usbd_bus *bus = xfer->ux_bus; if (bus->ub_usedma) { usb_dma_t *dmap = &xfer->ux_dmabuf; usb_freemem(dmap); return; } #endif KASSERT(xfer->ux_bus->ub_usedma == false); kmem_free(buf, size); } void * usbd_get_buffer(struct usbd_xfer *xfer) { return xfer->ux_buf; } struct usbd_pipe * usbd_get_pipe0(struct usbd_device *dev) { return dev->ud_pipe0; } static struct usbd_xfer * usbd_alloc_xfer(struct usbd_device *dev, unsigned int nframes) { struct usbd_xfer *xfer; USBHIST_FUNC(); ASSERT_SLEEPABLE(); xfer = dev->ud_bus->ub_methods->ubm_allocx(dev->ud_bus, nframes); if (xfer == NULL) goto out; xfer->ux_bus = dev->ud_bus; callout_init(&xfer->ux_callout, CALLOUT_MPSAFE); callout_setfunc(&xfer->ux_callout, usbd_xfer_timeout, xfer); cv_init(&xfer->ux_cv, "usbxfer"); usb_init_task(&xfer->ux_aborttask, usbd_xfer_timeout_task, xfer, USB_TASKQ_MPSAFE); out: USBHIST_CALLARGS(usbdebug, "returns %#jx", (uintptr_t)xfer, 0, 0, 0); return xfer; } static void usbd_free_xfer(struct usbd_xfer *xfer) { USBHIST_FUNC(); USBHIST_CALLARGS(usbdebug, "%#jx", (uintptr_t)xfer, 0, 0, 0); if (xfer->ux_buf) { usbd_free_buffer(xfer); } /* Wait for any straggling timeout to complete. */ mutex_enter(xfer->ux_bus->ub_lock); xfer->ux_timeout_reset = false; /* do not resuscitate */ callout_halt(&xfer->ux_callout, xfer->ux_bus->ub_lock); usb_rem_task_wait(xfer->ux_pipe->up_dev, &xfer->ux_aborttask, USB_TASKQ_HC, xfer->ux_bus->ub_lock); mutex_exit(xfer->ux_bus->ub_lock); cv_destroy(&xfer->ux_cv); xfer->ux_bus->ub_methods->ubm_freex(xfer->ux_bus, xfer); } int usbd_create_xfer(struct usbd_pipe *pipe, size_t len, unsigned int flags, unsigned int nframes, struct usbd_xfer **xp) { KASSERT(xp != NULL); void *buf = NULL; struct usbd_xfer *xfer = usbd_alloc_xfer(pipe->up_dev, nframes); if (xfer == NULL) return ENOMEM; xfer->ux_pipe = pipe; xfer->ux_flags = flags; xfer->ux_nframes = nframes; xfer->ux_methods = pipe->up_methods; if (len) { buf = usbd_alloc_buffer(xfer, len); if (!buf) { usbd_free_xfer(xfer); return ENOMEM; } } if (xfer->ux_methods->upm_init) { int err = xfer->ux_methods->upm_init(xfer); if (err) { usbd_free_xfer(xfer); return err; } } *xp = xfer; SDT_PROBE5(usb, device, xfer, create, xfer, pipe, len, flags, nframes); return 0; } void usbd_destroy_xfer(struct usbd_xfer *xfer) { SDT_PROBE1(usb, device, xfer, destroy, xfer); if (xfer->ux_methods->upm_fini) xfer->ux_methods->upm_fini(xfer); usbd_free_xfer(xfer); } void usbd_setup_xfer(struct usbd_xfer *xfer, void *priv, void *buffer, uint32_t length, uint16_t flags, uint32_t timeout, usbd_callback callback) { KASSERT(xfer->ux_pipe); xfer->ux_priv = priv; xfer->ux_buffer = buffer; xfer->ux_length = length; xfer->ux_actlen = 0; xfer->ux_flags = flags; xfer->ux_timeout = timeout; xfer->ux_status = USBD_NOT_STARTED; xfer->ux_callback = callback; xfer->ux_rqflags &= ~URQ_REQUEST; xfer->ux_nframes = 0; } void usbd_setup_default_xfer(struct usbd_xfer *xfer, struct usbd_device *dev, void *priv, uint32_t timeout, usb_device_request_t *req, void *buffer, uint32_t length, uint16_t flags, usbd_callback callback) { KASSERT(xfer->ux_pipe == dev->ud_pipe0); xfer->ux_priv = priv; xfer->ux_buffer = buffer; xfer->ux_length = length; xfer->ux_actlen = 0; xfer->ux_flags = flags; xfer->ux_timeout = timeout; xfer->ux_status = USBD_NOT_STARTED; xfer->ux_callback = callback; xfer->ux_request = *req; xfer->ux_rqflags |= URQ_REQUEST; xfer->ux_nframes = 0; } void usbd_setup_isoc_xfer(struct usbd_xfer *xfer, void *priv, uint16_t *frlengths, uint32_t nframes, uint16_t flags, usbd_callback callback) { xfer->ux_priv = priv; xfer->ux_buffer = NULL; xfer->ux_length = 0; xfer->ux_actlen = 0; xfer->ux_flags = flags; xfer->ux_timeout = USBD_NO_TIMEOUT; xfer->ux_status = USBD_NOT_STARTED; xfer->ux_callback = callback; xfer->ux_rqflags &= ~URQ_REQUEST; xfer->ux_frlengths = frlengths; xfer->ux_nframes = nframes; for (size_t i = 0; i < xfer->ux_nframes; i++) xfer->ux_length += xfer->ux_frlengths[i]; } void usbd_get_xfer_status(struct usbd_xfer *xfer, void **priv, void **buffer, uint32_t *count, usbd_status *status) { if (priv != NULL) *priv = xfer->ux_priv; if (buffer != NULL) *buffer = xfer->ux_buffer; if (count != NULL) *count = xfer->ux_actlen; if (status != NULL) *status = xfer->ux_status; } usb_config_descriptor_t * usbd_get_config_descriptor(struct usbd_device *dev) { KASSERT(dev != NULL); return dev->ud_cdesc; } usb_interface_descriptor_t * usbd_get_interface_descriptor(struct usbd_interface *iface) { KASSERT(iface != NULL); return iface->ui_idesc; } usb_device_descriptor_t * usbd_get_device_descriptor(struct usbd_device *dev) { KASSERT(dev != NULL); return &dev->ud_ddesc; } usb_endpoint_descriptor_t * usbd_interface2endpoint_descriptor(struct usbd_interface *iface, uint8_t index) { if (index >= iface->ui_idesc->bNumEndpoints) return NULL; return iface->ui_endpoints[index].ue_edesc; } /* Some drivers may wish to abort requests on the default pipe, * * but there is no mechanism for getting a handle on it. */ void usbd_abort_default_pipe(struct usbd_device *device) { usbd_abort_pipe(device->ud_pipe0); } void usbd_abort_pipe(struct usbd_pipe *pipe) { usbd_suspend_pipe(pipe); usbd_resume_pipe(pipe); } void usbd_suspend_pipe(struct usbd_pipe *pipe) { usbd_lock_pipe(pipe); usbd_ar_pipe(pipe); usbd_unlock_pipe(pipe); } void usbd_resume_pipe(struct usbd_pipe *pipe) { usbd_lock_pipe(pipe); KASSERT(SIMPLEQ_EMPTY(&pipe->up_queue)); pipe->up_aborting = 0; usbd_unlock_pipe(pipe); } usbd_status usbd_clear_endpoint_stall(struct usbd_pipe *pipe) { struct usbd_device *dev = pipe->up_dev; usbd_status err; USBHIST_FUNC(); USBHIST_CALLED(usbdebug); SDT_PROBE1(usb, device, pipe, clear__endpoint__stall, pipe); /* * Clearing en endpoint stall resets the endpoint toggle, so * do the same to the HC toggle. */ SDT_PROBE1(usb, device, pipe, clear__endpoint__toggle, pipe); pipe->up_methods->upm_cleartoggle(pipe); err = usbd_clear_endpoint_feature(dev, pipe->up_endpoint->ue_edesc->bEndpointAddress, UF_ENDPOINT_HALT); #if 0 XXX should we do this? if (!err) { pipe->state = USBD_PIPE_ACTIVE; /* XXX activate pipe */ } #endif return err; } void usbd_clear_endpoint_stall_task(void *arg) { struct usbd_pipe *pipe = arg; struct usbd_device *dev = pipe->up_dev; SDT_PROBE1(usb, device, pipe, clear__endpoint__stall, pipe); SDT_PROBE1(usb, device, pipe, clear__endpoint__toggle, pipe); pipe->up_methods->upm_cleartoggle(pipe); (void)usbd_clear_endpoint_feature(dev, pipe->up_endpoint->ue_edesc->bEndpointAddress, UF_ENDPOINT_HALT); } void usbd_clear_endpoint_stall_async(struct usbd_pipe *pipe) { usb_add_task(pipe->up_dev, &pipe->up_async_task, USB_TASKQ_DRIVER); } void usbd_clear_endpoint_toggle(struct usbd_pipe *pipe) { SDT_PROBE1(usb, device, pipe, clear__endpoint__toggle, pipe); pipe->up_methods->upm_cleartoggle(pipe); } usbd_status usbd_endpoint_count(struct usbd_interface *iface, uint8_t *count) { KASSERT(iface != NULL); KASSERT(iface->ui_idesc != NULL); *count = iface->ui_idesc->bNumEndpoints; return USBD_NORMAL_COMPLETION; } usbd_status usbd_interface_count(struct usbd_device *dev, uint8_t *count) { if (dev->ud_cdesc == NULL) return USBD_NOT_CONFIGURED; *count = dev->ud_cdesc->bNumInterface; return USBD_NORMAL_COMPLETION; } void usbd_interface2device_handle(struct usbd_interface *iface, struct usbd_device **dev) { *dev = iface->ui_dev; } usbd_status usbd_device2interface_handle(struct usbd_device *dev, uint8_t ifaceno, struct usbd_interface **iface) { if (dev->ud_cdesc == NULL) return USBD_NOT_CONFIGURED; if (ifaceno >= dev->ud_cdesc->bNumInterface) return USBD_INVAL; *iface = &dev->ud_ifaces[ifaceno]; return USBD_NORMAL_COMPLETION; } struct usbd_device * usbd_pipe2device_handle(struct usbd_pipe *pipe) { KASSERT(pipe != NULL); return pipe->up_dev; } /* XXXX use altno */ usbd_status usbd_set_interface(struct usbd_interface *iface, int altidx) { bool locked = false; usb_device_request_t req; usbd_status err; USBHIST_FUNC(); USBHIST_CALLARGS(usbdebug, "iface %#jx", (uintptr_t)iface, 0, 0, 0); err = usbd_iface_lock(iface); if (err) goto out; locked = true; err = usbd_fill_iface_data(iface->ui_dev, iface->ui_index, altidx); if (err) goto out; req.bmRequestType = UT_WRITE_INTERFACE; req.bRequest = UR_SET_INTERFACE; USETW(req.wValue, iface->ui_idesc->bAlternateSetting); USETW(req.wIndex, iface->ui_idesc->bInterfaceNumber); USETW(req.wLength, 0); err = usbd_do_request(iface->ui_dev, &req, 0); out: /* XXX back out iface data? */ if (locked) usbd_iface_unlock(iface); return err; } int usbd_get_no_alts(usb_config_descriptor_t *cdesc, int ifaceno) { char *p = (char *)cdesc; char *end = p + UGETW(cdesc->wTotalLength); usb_descriptor_t *desc; usb_interface_descriptor_t *idesc; int n; for (n = 0; end - p >= sizeof(*desc); p += desc->bLength) { desc = (usb_descriptor_t *)p; if (desc->bLength < sizeof(*desc) || desc->bLength > end - p) break; if (desc->bDescriptorType != UDESC_INTERFACE) continue; if (desc->bLength < sizeof(*idesc)) break; idesc = (usb_interface_descriptor_t *)desc; if (idesc->bInterfaceNumber == ifaceno) { n++; if (n == INT_MAX) break; } } return n; } int usbd_get_interface_altindex(struct usbd_interface *iface) { return iface->ui_altindex; } usbd_status usbd_get_interface(struct usbd_interface *iface, uint8_t *aiface) { usb_device_request_t req; req.bmRequestType = UT_READ_INTERFACE; req.bRequest = UR_GET_INTERFACE; USETW(req.wValue, 0); USETW(req.wIndex, iface->ui_idesc->bInterfaceNumber); USETW(req.wLength, 1); return usbd_do_request(iface->ui_dev, &req, aiface); } /*** Internal routines ***/ /* Dequeue all pipe operations, called with bus lock held. */ Static void usbd_ar_pipe(struct usbd_pipe *pipe) { struct usbd_xfer *xfer; USBHIST_FUNC(); USBHIST_CALLARGS(usbdebug, "pipe = %#jx", (uintptr_t)pipe, 0, 0, 0); SDT_PROBE1(usb, device, pipe, abort__start, pipe); ASSERT_SLEEPABLE(); KASSERT(mutex_owned(pipe->up_dev->ud_bus->ub_lock)); /* * Allow only one thread at a time to abort the pipe, so we * don't get confused if upm_abort drops the lock in the middle * of the abort to wait for hardware completion softints to * stop using the xfer before returning. */ KASSERTMSG(pipe->up_abortlwp == NULL, "pipe->up_abortlwp=%p", pipe->up_abortlwp); pipe->up_abortlwp = curlwp; #ifdef USB_DEBUG if (usbdebug > 5) usbd_dump_queue(pipe); #endif pipe->up_repeat = 0; pipe->up_running = 0; pipe->up_aborting = 1; while ((xfer = SIMPLEQ_FIRST(&pipe->up_queue)) != NULL) { USBHIST_LOG(usbdebug, "pipe = %#jx xfer = %#jx " "(methods = %#jx)", (uintptr_t)pipe, (uintptr_t)xfer, (uintptr_t)pipe->up_methods, 0); if (xfer->ux_status == USBD_NOT_STARTED) { SDT_PROBE1(usb, device, xfer, preabort, xfer); #ifdef DIAGNOSTIC xfer->ux_state = XFER_BUSY; #endif SIMPLEQ_REMOVE_HEAD(&pipe->up_queue, ux_next); } else { /* Make the HC abort it (and invoke the callback). */ SDT_PROBE1(usb, device, xfer, abort, xfer); pipe->up_methods->upm_abort(xfer); while (pipe->up_callingxfer == xfer) { USBHIST_LOG(usbdebug, "wait for callback" "pipe = %#jx xfer = %#jx", (uintptr_t)pipe, (uintptr_t)xfer, 0, 0); cv_wait(&pipe->up_callingcv, pipe->up_dev->ud_bus->ub_lock); } /* XXX only for non-0 usbd_clear_endpoint_stall(pipe); */ } } /* * There may be an xfer callback already in progress which was * taken off the queue before we got to it. We must wait for * the callback to finish before returning control to the * caller. */ while (pipe->up_callingxfer) { USBHIST_LOG(usbdebug, "wait for callback" "pipe = %#jx xfer = %#jx", (uintptr_t)pipe, (uintptr_t)pipe->up_callingxfer, 0, 0); cv_wait(&pipe->up_callingcv, pipe->up_dev->ud_bus->ub_lock); } KASSERT(mutex_owned(pipe->up_dev->ud_bus->ub_lock)); KASSERTMSG(pipe->up_abortlwp == curlwp, "pipe->up_abortlwp=%p", pipe->up_abortlwp); pipe->up_abortlwp = NULL; SDT_PROBE1(usb, device, pipe, abort__done, pipe); } /* Called with USB lock held. */ void usb_transfer_complete(struct usbd_xfer *xfer) { struct usbd_pipe *pipe = xfer->ux_pipe; struct usbd_bus *bus = pipe->up_dev->ud_bus; int sync = xfer->ux_flags & USBD_SYNCHRONOUS; int erred; int polling = bus->ub_usepolling; int repeat = pipe->up_repeat; USBHIST_FUNC(); USBHIST_CALLARGS(usbdebug, "pipe = %#jx xfer = %#jx status = %jd " "actlen = %jd", (uintptr_t)pipe, (uintptr_t)xfer, xfer->ux_status, xfer->ux_actlen); KASSERT(polling || mutex_owned(pipe->up_dev->ud_bus->ub_lock)); KASSERTMSG(xfer->ux_state == XFER_ONQU, "xfer %p state is %x", xfer, xfer->ux_state); KASSERT(pipe != NULL); /* * If device is known to miss out ack, then pretend that * output timeout is a success. Userland should handle * the logic to verify that the operation succeeded. */ if (pipe->up_dev->ud_quirks && pipe->up_dev->ud_quirks->uq_flags & UQ_MISS_OUT_ACK && xfer->ux_status == USBD_TIMEOUT && !usbd_xfer_isread(xfer)) { USBHIST_LOG(usbdebug, "Possible output ack miss for xfer %#jx: " "hiding write timeout to %jd.%jd for %ju bytes written", (uintptr_t)xfer, curlwp->l_proc->p_pid, curlwp->l_lid, xfer->ux_length); xfer->ux_status = USBD_NORMAL_COMPLETION; xfer->ux_actlen = xfer->ux_length; } erred = xfer->ux_status == USBD_CANCELLED || xfer->ux_status == USBD_TIMEOUT; if (!repeat) { /* Remove request from queue. */ KASSERTMSG(!SIMPLEQ_EMPTY(&pipe->up_queue), "pipe %p is empty, but xfer %p wants to complete", pipe, xfer); KASSERTMSG(xfer == SIMPLEQ_FIRST(&pipe->up_queue), "xfer %p is not start of queue (%p is at start)", xfer, SIMPLEQ_FIRST(&pipe->up_queue)); #ifdef DIAGNOSTIC xfer->ux_state = XFER_BUSY; #endif SIMPLEQ_REMOVE_HEAD(&pipe->up_queue, ux_next); } USBHIST_LOG(usbdebug, "xfer %#jx: repeat %jd new head = %#jx", (uintptr_t)xfer, repeat, (uintptr_t)SIMPLEQ_FIRST(&pipe->up_queue), 0); /* Count completed transfers. */ ++pipe->up_dev->ud_bus->ub_stats.uds_requests [pipe->up_endpoint->ue_edesc->bmAttributes & UE_XFERTYPE]; xfer->ux_done = 1; if (!xfer->ux_status && xfer->ux_actlen < xfer->ux_length && !(xfer->ux_flags & USBD_SHORT_XFER_OK)) { USBHIST_LOG(usbdebug, "short transfer %jd < %jd", xfer->ux_actlen, xfer->ux_length, 0, 0); xfer->ux_status = USBD_SHORT_XFER; } USBHIST_LOG(usbdebug, "xfer %#jx doing done %#jx", (uintptr_t)xfer, (uintptr_t)pipe->up_methods->upm_done, 0, 0); SDT_PROBE2(usb, device, xfer, done, xfer, xfer->ux_status); pipe->up_methods->upm_done(xfer); if (xfer->ux_length != 0 && xfer->ux_buffer != xfer->ux_buf) { KDASSERTMSG(xfer->ux_actlen <= xfer->ux_length, "actlen %d length %d",xfer->ux_actlen, xfer->ux_length); /* Only if IN transfer */ if (usbd_xfer_isread(xfer)) { memcpy(xfer->ux_buffer, xfer->ux_buf, xfer->ux_actlen); } } USBHIST_LOG(usbdebug, "xfer %#jx doing callback %#jx status %jd", (uintptr_t)xfer, (uintptr_t)xfer->ux_callback, xfer->ux_status, 0); if (xfer->ux_callback) { if (!polling) { KASSERT(pipe->up_callingxfer == NULL); pipe->up_callingxfer = xfer; mutex_exit(pipe->up_dev->ud_bus->ub_lock); if (!(pipe->up_flags & USBD_MPSAFE)) KERNEL_LOCK(1, curlwp); } xfer->ux_callback(xfer, xfer->ux_priv, xfer->ux_status); if (!polling) { if (!(pipe->up_flags & USBD_MPSAFE)) KERNEL_UNLOCK_ONE(curlwp); mutex_enter(pipe->up_dev->ud_bus->ub_lock); KASSERT(pipe->up_callingxfer == xfer); pipe->up_callingxfer = NULL; cv_broadcast(&pipe->up_callingcv); } } if (sync && !polling) { USBHIST_LOG(usbdebug, "<- done xfer %#jx, wakeup", (uintptr_t)xfer, 0, 0, 0); cv_broadcast(&xfer->ux_cv); } if (repeat) { xfer->ux_actlen = 0; xfer->ux_status = USBD_NOT_STARTED; } else { /* XXX should we stop the queue on all errors? */ if (erred && pipe->up_iface != NULL) /* not control pipe */ pipe->up_running = 0; } if (pipe->up_running && pipe->up_serialise) usbd_start_next(pipe); } /* Called with USB lock held. */ void usbd_start_next(struct usbd_pipe *pipe) { struct usbd_xfer *xfer; usbd_status err; USBHIST_FUNC(); KASSERT(pipe != NULL); KASSERT(pipe->up_methods != NULL); KASSERT(pipe->up_methods->upm_start != NULL); KASSERT(pipe->up_serialise == true); int polling = pipe->up_dev->ud_bus->ub_usepolling; KASSERT(polling || mutex_owned(pipe->up_dev->ud_bus->ub_lock)); /* Get next request in queue. */ xfer = SIMPLEQ_FIRST(&pipe->up_queue); USBHIST_CALLARGS(usbdebug, "pipe = %#jx, xfer = %#jx", (uintptr_t)pipe, (uintptr_t)xfer, 0, 0); if (xfer == NULL) { pipe->up_running = 0; } else { SDT_PROBE2(usb, device, pipe, start, pipe, xfer); err = pipe->up_methods->upm_start(xfer); if (err != USBD_IN_PROGRESS) { USBHIST_LOG(usbdebug, "error = %jd", err, 0, 0, 0); pipe->up_running = 0; /* XXX do what? */ } } KASSERT(polling || mutex_owned(pipe->up_dev->ud_bus->ub_lock)); } usbd_status usbd_do_request(struct usbd_device *dev, usb_device_request_t *req, void *data) { return usbd_do_request_flags(dev, req, data, 0, 0, USBD_DEFAULT_TIMEOUT); } usbd_status usbd_do_request_flags(struct usbd_device *dev, usb_device_request_t *req, void *data, uint16_t flags, int *actlen, uint32_t timeout) { size_t len = UGETW(req->wLength); return usbd_do_request_len(dev, req, len, data, flags, actlen, timeout); } usbd_status usbd_do_request_len(struct usbd_device *dev, usb_device_request_t *req, size_t len, void *data, uint16_t flags, int *actlen, uint32_t timeout) { struct usbd_xfer *xfer; usbd_status err; KASSERT(len >= UGETW(req->wLength)); USBHIST_FUNC(); USBHIST_CALLARGS(usbdebug, "dev=%#jx req=%jx flags=%jx len=%jx", (uintptr_t)dev, (uintptr_t)req, flags, len); ASSERT_SLEEPABLE(); SDT_PROBE5(usb, device, request, start, dev, req, len, flags, timeout); int error = usbd_create_xfer(dev->ud_pipe0, len, 0, 0, &xfer); if (error) { SDT_PROBE7(usb, device, request, done, dev, req, /*actlen*/0, flags, timeout, data, USBD_NOMEM); return USBD_NOMEM; } usbd_setup_default_xfer(xfer, dev, 0, timeout, req, data, UGETW(req->wLength), flags, NULL); KASSERT(xfer->ux_pipe == dev->ud_pipe0); err = usbd_sync_transfer(xfer); #if defined(USB_DEBUG) || defined(DIAGNOSTIC) if (xfer->ux_actlen > xfer->ux_length) { USBHIST_LOG(usbdebug, "overrun addr = %jd type = 0x%02jx", dev->ud_addr, xfer->ux_request.bmRequestType, 0, 0); USBHIST_LOG(usbdebug, " req = 0x%02jx val = %jd " "index = %jd", xfer->ux_request.bRequest, UGETW(xfer->ux_request.wValue), UGETW(xfer->ux_request.wIndex), 0); USBHIST_LOG(usbdebug, " rlen = %jd length = %jd " "actlen = %jd", UGETW(xfer->ux_request.wLength), xfer->ux_length, xfer->ux_actlen, 0); } #endif if (actlen != NULL) *actlen = xfer->ux_actlen; usbd_destroy_xfer(xfer); SDT_PROBE7(usb, device, request, done, dev, req, xfer->ux_actlen, flags, timeout, data, err); if (err) { USBHIST_LOG(usbdebug, "returning err = %jd", err, 0, 0, 0); } return err; } const struct usbd_quirks * usbd_get_quirks(struct usbd_device *dev) { #ifdef DIAGNOSTIC if (dev == NULL) { printf("usbd_get_quirks: dev == NULL\n"); return 0; } #endif return dev->ud_quirks; } /* XXX do periodic free() of free list */ /* * Called from keyboard driver when in polling mode. */ void usbd_dopoll(struct usbd_interface *iface) { iface->ui_dev->ud_bus->ub_methods->ubm_dopoll(iface->ui_dev->ud_bus); } /* * This is for keyboard driver as well, which only operates in polling * mode from the ask root, etc., prompt and from DDB. */ void usbd_set_polling(struct usbd_device *dev, int on) { if (on) dev->ud_bus->ub_usepolling++; else dev->ud_bus->ub_usepolling--; /* Kick the host controller when switching modes */ mutex_enter(dev->ud_bus->ub_lock); dev->ud_bus->ub_methods->ubm_softint(dev->ud_bus); mutex_exit(dev->ud_bus->ub_lock); } usb_endpoint_descriptor_t * usbd_get_endpoint_descriptor(struct usbd_interface *iface, uint8_t address) { struct usbd_endpoint *ep; int i; for (i = 0; i < iface->ui_idesc->bNumEndpoints; i++) { ep = &iface->ui_endpoints[i]; if (ep->ue_edesc->bEndpointAddress == address) return iface->ui_endpoints[i].ue_edesc; } return NULL; } /* * usbd_ratecheck() can limit the number of error messages that occurs. * When a device is unplugged it may take up to 0.25s for the hub driver * to notice it. If the driver continuously tries to do I/O operations * this can generate a large number of messages. */ int usbd_ratecheck(struct timeval *last) { static struct timeval errinterval = { 0, 250000 }; /* 0.25 s*/ return ratecheck(last, &errinterval); } /* * Search for a vendor/product pair in an array. The item size is * given as an argument. */ const struct usb_devno * usb_match_device(const struct usb_devno *tbl, u_int nentries, u_int sz, uint16_t vendor, uint16_t product) { while (nentries-- > 0) { uint16_t tproduct = tbl->ud_product; if (tbl->ud_vendor == vendor && (tproduct == product || tproduct == USB_PRODUCT_ANY)) return tbl; tbl = (const struct usb_devno *)((const char *)tbl + sz); } return NULL; } usbd_status usbd_get_string(struct usbd_device *dev, int si, char *buf) { return usbd_get_string0(dev, si, buf, 1); } usbd_status usbd_get_string0(struct usbd_device *dev, int si, char *buf, int unicode) { int swap = dev->ud_quirks->uq_flags & UQ_SWAP_UNICODE; usb_string_descriptor_t us; char *s; int i, n; uint16_t c; usbd_status err; int size; USBHIST_FUNC(); USBHIST_CALLED(usbdebug); buf[0] = '\0'; if (si == 0) return USBD_INVAL; if (dev->ud_quirks->uq_flags & UQ_NO_STRINGS) return USBD_STALLED; if (dev->ud_langid == USBD_NOLANG) { /* Set up default language */ err = usbd_get_string_desc(dev, USB_LANGUAGE_TABLE, 0, &us, &size); if (err || size < 4) { USBHIST_LOG(usbdebug, "getting lang failed, using 0", 0, 0, 0, 0); dev->ud_langid = 0; /* Well, just pick something then */ } else { /* Pick the first language as the default. */ dev->ud_langid = UGETW(us.bString[0]); } } err = usbd_get_string_desc(dev, si, dev->ud_langid, &us, &size); if (err) return err; s = buf; n = size / 2 - 1; if (unicode) { for (i = 0; i < n; i++) { c = UGETW(us.bString[i]); if (swap) c = (c >> 8) | (c << 8); s += wput_utf8(s, 3, c); } *s++ = 0; } #ifdef COMPAT_30 else { for (i = 0; i < n; i++) { c = UGETW(us.bString[i]); if (swap) c = (c >> 8) | (c << 8); *s++ = (c < 0x80) ? c : '?'; } *s++ = 0; } #endif return USBD_NORMAL_COMPLETION; } /* * usbd_xfer_trycomplete(xfer) * * Try to claim xfer for completion. Return true if successful, * false if the xfer has been synchronously aborted or has timed * out. * * If this returns true, caller is responsible for setting * xfer->ux_status and calling usb_transfer_complete. To be used * in a host controller interrupt handler. * * Caller must either hold the bus lock or have the bus in polling * mode. If this succeeds, caller must proceed to call * usb_complete_transfer under the bus lock or with polling * enabled -- must not release and reacquire the bus lock in the * meantime. Failing to heed this rule may lead to catastrophe * with abort or timeout. */ bool usbd_xfer_trycomplete(struct usbd_xfer *xfer) { struct usbd_bus *bus __diagused = xfer->ux_bus; KASSERT(bus->ub_usepolling || mutex_owned(bus->ub_lock)); /* * If software has completed it, either by synchronous abort or * by timeout, too late. */ if (xfer->ux_status != USBD_IN_PROGRESS) return false; /* * We are completing the xfer. Cancel the timeout if we can, * but only asynchronously. See usbd_xfer_cancel_timeout_async * for why we need not wait for the callout or task here. */ usbd_xfer_cancel_timeout_async(xfer); /* Success! Note: Caller must set xfer->ux_status afterwar. */ return true; } /* * usbd_xfer_abort(xfer) * * Try to claim xfer to abort. If successful, mark it completed * with USBD_CANCELLED and call the bus-specific method to abort * at the hardware level. * * To be called in thread context from struct * usbd_pipe_methods::upm_abort. * * Caller must hold the bus lock. */ void usbd_xfer_abort(struct usbd_xfer *xfer) { struct usbd_bus *bus = xfer->ux_bus; KASSERT(mutex_owned(bus->ub_lock)); /* * If host controller interrupt or timer interrupt has * completed it, too late. But the xfer cannot be * cancelled already -- only one caller can synchronously * abort. */ KASSERT(xfer->ux_status != USBD_CANCELLED); if (xfer->ux_status != USBD_IN_PROGRESS) return; /* * Cancel the timeout if we can, but only asynchronously; see * usbd_xfer_cancel_timeout_async for why we need not wait for * the callout or task here. */ usbd_xfer_cancel_timeout_async(xfer); /* * We beat everyone else. Claim the status as cancelled, do * the bus-specific dance to abort the hardware, and complete * the xfer. */ xfer->ux_status = USBD_CANCELLED; bus->ub_methods->ubm_abortx(xfer); usb_transfer_complete(xfer); } /* * usbd_xfer_timeout(xfer) * * Called at IPL_SOFTCLOCK when too much time has elapsed waiting * for xfer to complete. Since we can't abort the xfer at * IPL_SOFTCLOCK, defer to a usb_task to run it in thread context, * unless the xfer has completed or aborted concurrently -- and if * the xfer has also been resubmitted, take care of rescheduling * the callout. */ static void usbd_xfer_timeout(void *cookie) { struct usbd_xfer *xfer = cookie; struct usbd_bus *bus = xfer->ux_bus; struct usbd_device *dev = xfer->ux_pipe->up_dev; /* Acquire the lock so we can transition the timeout state. */ mutex_enter(bus->ub_lock); /* * Use usbd_xfer_probe_timeout to check whether the timeout is * still valid, or to reschedule the callout if necessary. If * it is still valid, schedule the task. */ if (usbd_xfer_probe_timeout(xfer)) usb_add_task(dev, &xfer->ux_aborttask, USB_TASKQ_HC); /* * Notify usbd_xfer_cancel_timeout_async that we may have * scheduled the task. This causes callout_invoking to return * false in usbd_xfer_cancel_timeout_async so that it can tell * which stage in the callout->task->abort process we're at. */ callout_ack(&xfer->ux_callout); /* All done -- release the lock. */ mutex_exit(bus->ub_lock); } /* * usbd_xfer_timeout_task(xfer) * * Called in thread context when too much time has elapsed waiting * for xfer to complete. Abort the xfer with USBD_TIMEOUT, unless * it has completed or aborted concurrently -- and if the xfer has * also been resubmitted, take care of rescheduling the callout. */ static void usbd_xfer_timeout_task(void *cookie) { struct usbd_xfer *xfer = cookie; struct usbd_bus *bus = xfer->ux_bus; /* Acquire the lock so we can transition the timeout state. */ mutex_enter(bus->ub_lock); /* * Use usbd_xfer_probe_timeout to check whether the timeout is * still valid, or to reschedule the callout if necessary. If * it is not valid -- the timeout has been asynchronously * cancelled, or the xfer has already been resubmitted -- then * we're done here. */ if (!usbd_xfer_probe_timeout(xfer)) goto out; /* * May have completed or been aborted, but we're the only one * who can time it out. If it has completed or been aborted, * no need to timeout. */ KASSERT(xfer->ux_status != USBD_TIMEOUT); if (xfer->ux_status != USBD_IN_PROGRESS) goto out; /* * We beat everyone else. Claim the status as timed out, do * the bus-specific dance to abort the hardware, and complete * the xfer. */ xfer->ux_status = USBD_TIMEOUT; bus->ub_methods->ubm_abortx(xfer); usb_transfer_complete(xfer); out: /* All done -- release the lock. */ mutex_exit(bus->ub_lock); } /* * usbd_xfer_probe_timeout(xfer) * * Probe the status of xfer's timeout. Acknowledge and process a * request to reschedule. Return true if the timeout is still * valid and the caller should take further action (queueing a * task or aborting the xfer), false if it must stop here. */ static bool usbd_xfer_probe_timeout(struct usbd_xfer *xfer) { struct usbd_bus *bus = xfer->ux_bus; bool valid; KASSERT(bus->ub_usepolling || mutex_owned(bus->ub_lock)); /* The timeout must be set. */ KASSERT(xfer->ux_timeout_set); /* * Neither callout nor task may be pending; they execute * alternately in lock step. */ KASSERT(!callout_pending(&xfer->ux_callout)); KASSERT(!usb_task_pending(xfer->ux_pipe->up_dev, &xfer->ux_aborttask)); /* There are a few cases... */ if (bus->ub_methods->ubm_dying(bus)) { /* Host controller dying. Drop it all on the floor. */ xfer->ux_timeout_set = false; xfer->ux_timeout_reset = false; valid = false; } else if (xfer->ux_timeout_reset) { /* * The xfer completed _and_ got resubmitted while we * waited for the lock. Acknowledge the request to * reschedule, and reschedule it if there is a timeout * and the bus is not polling. */ xfer->ux_timeout_reset = false; if (xfer->ux_timeout && !bus->ub_usepolling) { KASSERT(xfer->ux_timeout_set); callout_schedule(&xfer->ux_callout, mstohz(xfer->ux_timeout)); } else { /* No more callout or task scheduled. */ xfer->ux_timeout_set = false; } valid = false; } else if (xfer->ux_status != USBD_IN_PROGRESS) { /* * The xfer has completed by hardware completion or by * software abort, and has not been resubmitted, so the * timeout must be unset, and is no longer valid for * the caller. */ xfer->ux_timeout_set = false; valid = false; } else { /* * The xfer has not yet completed, so the timeout is * valid. */ valid = true; } /* Any reset must have been processed. */ KASSERT(!xfer->ux_timeout_reset); /* * Either we claim the timeout is set, or the callout is idle. * If the timeout is still set, we may be handing off to the * task instead, so this is an if but not an iff. */ KASSERT(xfer->ux_timeout_set || !callout_pending(&xfer->ux_callout)); /* * The task must be idle now. * * - If the caller is the callout, _and_ the timeout is still * valid, the caller will schedule it, but it hasn't been * scheduled yet. (If the timeout is not valid, the task * should not be scheduled.) * * - If the caller is the task, it cannot be scheduled again * until the callout runs again, which won't happen until we * next release the lock. */ KASSERT(!usb_task_pending(xfer->ux_pipe->up_dev, &xfer->ux_aborttask)); KASSERT(bus->ub_usepolling || mutex_owned(bus->ub_lock)); return valid; } /* * usbd_xfer_schedule_timeout(xfer) * * Ensure that xfer has a timeout. If the callout is already * queued or the task is already running, request that they * reschedule the callout. If not, and if we're not polling, * schedule the callout anew. * * To be called in thread context from struct * usbd_pipe_methods::upm_start. */ void usbd_xfer_schedule_timeout(struct usbd_xfer *xfer) { struct usbd_bus *bus = xfer->ux_bus; KASSERT(bus->ub_usepolling || mutex_owned(bus->ub_lock)); if (xfer->ux_timeout_set) { /* * Callout or task has fired from a prior completed * xfer but has not yet noticed that the xfer is done. * Ask it to reschedule itself to ux_timeout. */ xfer->ux_timeout_reset = true; } else if (xfer->ux_timeout && !bus->ub_usepolling) { /* Callout is not scheduled. Schedule it. */ KASSERT(!callout_pending(&xfer->ux_callout)); callout_schedule(&xfer->ux_callout, mstohz(xfer->ux_timeout)); xfer->ux_timeout_set = true; } KASSERT(bus->ub_usepolling || mutex_owned(bus->ub_lock)); } /* * usbd_xfer_cancel_timeout_async(xfer) * * Cancel the callout and the task of xfer, which have not yet run * to completion, but don't wait for the callout or task to finish * running. * * If they have already fired, at worst they are waiting for the * bus lock. They will see that the xfer is no longer in progress * and give up, or they will see that the xfer has been * resubmitted with a new timeout and reschedule the callout. * * If a resubmitted request completed so fast that the callout * didn't have time to process a timer reset, just cancel the * timer reset. */ static void usbd_xfer_cancel_timeout_async(struct usbd_xfer *xfer) { struct usbd_bus *bus __diagused = xfer->ux_bus; KASSERT(bus->ub_usepolling || mutex_owned(bus->ub_lock)); /* * If the timer wasn't running anyway, forget about it. This * can happen if we are completing an isochronous transfer * which doesn't use the same timeout logic. */ if (!xfer->ux_timeout_set) return; xfer->ux_timeout_reset = false; if (!callout_stop(&xfer->ux_callout)) { /* * We stopped the callout before it ran. The timeout * is no longer set. */ xfer->ux_timeout_set = false; } else if (callout_invoking(&xfer->ux_callout)) { /* * The callout has begun to run but it has not yet * acquired the lock and called callout_ack. The task * cannot be queued yet, and the callout cannot have * been rescheduled yet. * * By the time the callout acquires the lock, we will * have transitioned from USBD_IN_PROGRESS to a * completed status, and possibly also resubmitted the * xfer and set xfer->ux_timeout_reset = true. In both * cases, the callout will DTRT, so no further action * is needed here. */ } else if (usb_rem_task(xfer->ux_pipe->up_dev, &xfer->ux_aborttask)) { /* * The callout had fired and scheduled the task, but we * stopped the task before it could run. The timeout * is therefore no longer set -- the next resubmission * of the xfer must schedule a new timeout. * * The callout should not be pending at this point: * it is scheduled only under the lock, and only when * xfer->ux_timeout_set is false, or by the callout or * task itself when xfer->ux_timeout_reset is true. */ xfer->ux_timeout_set = false; } /* * The callout cannot be scheduled and the task cannot be * queued at this point. Either we cancelled them, or they are * already running and waiting for the bus lock. */ KASSERT(!callout_pending(&xfer->ux_callout)); KASSERT(!usb_task_pending(xfer->ux_pipe->up_dev, &xfer->ux_aborttask)); KASSERT(bus->ub_usepolling || mutex_owned(bus->ub_lock)); } |
| 97 215 210 368 368 37 9 31 1345 877 659 1614 2 1614 91 1614 184 159 4 22 159 143 143 134 1 1 2 4 6 6 1 136 24 71 18 37 125 1 4 27 1 30 19 11 28 105 129 1 15 15 15 15 22 3 19 6 1 1 24 23 23 23 23 13 13 2 10 12 9 9 9 9 9 8 5 4 2 138 138 138 137 1 138 138 138 138 134 13 2 1 11 11 11 10 38 38 35 35 35 35 2 37 71 71 2 1 56 65 2 54 9 56 10 65 3 62 561 561 561 539 41 538 40 46 45 53 53 46 42 5 5 2 13 3 10 26 2 2 2 2 2 2 2 2 1 2 2 2 1 63 479 466 13 479 347 347 347 338 9 346 2 340 340 115 114 1 41 102 115 1277 19 1260 29 29 29 29 2 28 2 20 12 28 29 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 197 198 199 200 201 202 203 204 205 206 207 208 209 210 211 212 213 214 215 216 217 218 219 220 221 222 223 224 225 226 227 228 229 230 231 232 233 234 235 236 237 238 239 240 241 242 243 244 245 246 247 248 249 250 251 252 253 254 255 256 257 258 259 260 261 262 263 264 265 266 267 268 269 270 271 272 273 274 275 276 277 278 279 280 281 282 283 284 285 286 287 288 289 290 291 292 293 294 295 296 297 298 299 300 301 302 303 304 305 306 307 308 309 310 311 312 313 314 315 316 317 318 319 320 321 322 323 324 325 326 327 328 329 330 331 332 333 334 335 336 337 338 339 340 341 342 343 344 345 346 347 348 349 350 351 352 353 354 355 356 357 358 359 360 361 362 363 364 365 366 367 368 369 370 371 372 373 374 375 376 377 378 379 380 381 382 383 384 385 386 387 388 389 390 391 392 393 394 395 396 397 398 399 400 401 402 403 404 405 406 407 408 409 410 411 412 413 414 415 416 417 418 419 420 421 422 423 424 425 426 427 428 429 430 431 432 433 434 435 436 437 438 439 440 441 442 443 444 445 446 447 448 449 450 451 452 453 454 455 456 457 458 459 460 461 462 463 464 465 466 467 468 469 470 471 472 473 474 475 476 477 478 479 480 481 482 483 484 485 486 487 488 489 490 491 492 493 494 495 496 497 498 499 500 501 502 503 504 505 506 507 508 509 510 511 512 513 514 515 516 517 518 519 520 521 522 523 524 525 526 527 528 529 530 531 532 533 534 535 536 537 538 539 540 541 542 543 544 545 546 547 548 549 550 551 552 553 554 555 556 557 558 559 560 561 562 563 564 565 566 567 568 569 570 571 572 573 574 575 576 577 578 579 580 581 582 583 584 585 586 587 588 589 590 591 592 593 594 595 596 597 598 599 600 601 602 603 604 605 606 607 608 609 610 611 612 613 614 615 616 617 618 619 620 621 622 623 624 625 626 627 628 629 630 631 632 633 634 635 636 637 638 639 640 641 642 643 644 645 646 647 648 649 650 651 652 653 654 655 656 657 658 659 660 661 662 663 664 665 666 667 668 669 670 671 672 673 674 675 676 677 678 679 680 681 682 683 684 685 686 687 688 689 690 691 692 693 694 695 696 697 698 699 700 701 702 703 704 705 706 707 708 709 710 711 712 713 714 715 716 717 718 719 720 721 722 723 724 725 726 727 728 729 730 731 732 733 734 735 736 737 738 739 740 741 742 743 744 745 746 747 748 749 750 751 752 753 754 755 756 757 758 759 760 761 762 763 764 765 766 767 768 769 770 771 772 773 774 775 776 777 778 779 780 781 782 783 784 785 786 787 788 789 790 791 792 793 794 795 796 797 798 799 800 801 802 803 804 805 806 807 808 809 810 811 812 813 814 815 816 817 818 819 820 821 822 823 824 825 826 827 828 829 830 831 832 833 834 835 836 837 838 839 840 841 842 843 844 845 846 847 848 849 850 851 852 853 854 855 856 857 858 859 860 861 862 863 864 865 866 867 868 869 870 871 872 873 874 875 876 877 878 879 880 881 882 883 884 885 886 887 888 889 890 891 892 893 894 895 896 897 898 899 900 901 902 903 904 905 906 907 908 909 910 911 912 913 914 915 916 917 918 919 920 921 922 923 924 925 926 927 928 929 930 931 932 933 934 935 936 937 938 939 940 941 942 943 944 945 946 947 948 949 950 951 952 953 954 955 956 957 958 959 960 961 962 963 964 965 966 967 968 969 970 971 972 973 974 975 976 977 978 979 980 981 982 983 984 985 986 987 988 989 990 991 992 993 994 995 996 997 998 999 1000 1001 1002 1003 1004 1005 1006 1007 1008 1009 1010 1011 1012 1013 1014 1015 1016 1017 1018 1019 1020 1021 1022 1023 1024 1025 1026 1027 1028 1029 1030 1031 1032 1033 1034 1035 1036 1037 1038 1039 1040 1041 1042 1043 1044 1045 1046 1047 1048 1049 1050 1051 1052 1053 1054 1055 1056 1057 1058 1059 1060 1061 1062 1063 1064 1065 1066 1067 1068 1069 1070 1071 1072 1073 1074 1075 1076 1077 1078 1079 1080 1081 1082 1083 1084 1085 1086 1087 1088 1089 1090 1091 1092 1093 1094 1095 1096 1097 1098 1099 1100 1101 1102 1103 1104 1105 1106 1107 1108 1109 1110 1111 1112 1113 1114 1115 1116 1117 1118 1119 1120 1121 1122 1123 1124 1125 1126 1127 1128 1129 1130 1131 1132 1133 1134 1135 1136 1137 1138 1139 1140 1141 1142 1143 1144 1145 1146 1147 1148 1149 1150 1151 1152 1153 1154 1155 1156 1157 1158 1159 1160 1161 1162 1163 1164 1165 1166 1167 1168 1169 1170 1171 1172 1173 1174 1175 1176 1177 1178 1179 1180 1181 1182 1183 1184 1185 1186 1187 1188 1189 1190 1191 1192 1193 1194 1195 1196 1197 1198 1199 1200 1201 1202 1203 1204 1205 1206 1207 1208 1209 1210 1211 1212 1213 1214 1215 1216 1217 1218 1219 1220 1221 1222 1223 1224 1225 1226 1227 1228 1229 1230 1231 1232 1233 1234 1235 1236 1237 1238 1239 1240 1241 1242 1243 1244 1245 1246 1247 1248 1249 1250 1251 1252 1253 1254 1255 1256 1257 1258 1259 1260 1261 1262 1263 1264 1265 1266 1267 1268 1269 1270 1271 1272 1273 1274 1275 1276 1277 1278 1279 1280 1281 1282 1283 1284 1285 1286 1287 1288 1289 1290 1291 1292 1293 1294 1295 1296 1297 1298 1299 1300 1301 1302 1303 1304 1305 1306 1307 1308 1309 1310 1311 1312 1313 1314 1315 1316 1317 1318 1319 1320 1321 1322 1323 1324 1325 1326 1327 1328 1329 1330 1331 1332 1333 1334 1335 1336 1337 1338 1339 1340 1341 1342 1343 1344 1345 1346 1347 1348 1349 1350 1351 1352 1353 1354 1355 1356 1357 1358 1359 1360 1361 1362 1363 1364 1365 1366 1367 1368 1369 1370 1371 1372 1373 1374 1375 1376 1377 1378 1379 1380 1381 1382 1383 1384 1385 1386 1387 1388 1389 1390 1391 1392 1393 1394 1395 1396 1397 1398 1399 1400 1401 1402 1403 1404 1405 1406 1407 1408 1409 1410 1411 1412 1413 1414 1415 1416 1417 1418 1419 1420 1421 1422 1423 1424 1425 1426 1427 1428 1429 1430 1431 1432 1433 1434 1435 1436 1437 1438 1439 1440 1441 1442 1443 1444 1445 1446 1447 1448 1449 1450 1451 1452 1453 1454 1455 1456 1457 1458 1459 1460 1461 1462 1463 1464 1465 1466 1467 1468 1469 1470 1471 1472 1473 1474 1475 1476 1477 1478 1479 1480 1481 1482 1483 1484 1485 1486 1487 1488 1489 1490 1491 1492 1493 1494 1495 1496 1497 1498 1499 1500 1501 1502 1503 1504 1505 1506 1507 1508 1509 1510 1511 1512 1513 1514 1515 1516 1517 1518 1519 1520 1521 1522 1523 1524 1525 1526 1527 1528 1529 1530 1531 1532 1533 1534 1535 1536 1537 1538 1539 1540 1541 1542 1543 1544 1545 1546 1547 1548 1549 1550 1551 1552 1553 1554 1555 1556 1557 1558 1559 1560 1561 1562 1563 1564 1565 1566 1567 1568 1569 1570 1571 1572 1573 1574 1575 1576 1577 1578 1579 1580 1581 1582 1583 1584 1585 1586 1587 1588 1589 1590 1591 1592 1593 1594 1595 1596 1597 1598 1599 1600 1601 1602 1603 1604 1605 1606 1607 1608 1609 1610 1611 1612 1613 1614 1615 1616 1617 1618 1619 1620 1621 1622 1623 1624 1625 1626 1627 1628 1629 1630 1631 1632 1633 1634 1635 1636 1637 1638 1639 1640 1641 1642 1643 1644 1645 1646 1647 1648 1649 1650 1651 1652 1653 1654 1655 1656 1657 1658 1659 1660 1661 1662 1663 1664 1665 1666 1667 1668 1669 1670 1671 1672 1673 1674 1675 1676 1677 1678 1679 1680 1681 1682 1683 1684 1685 1686 1687 1688 1689 1690 1691 1692 1693 1694 1695 1696 1697 1698 1699 1700 1701 1702 1703 1704 1705 1706 1707 1708 1709 1710 1711 1712 1713 1714 1715 1716 1717 1718 1719 1720 1721 1722 1723 1724 1725 1726 1727 1728 1729 1730 1731 1732 1733 1734 1735 1736 1737 1738 1739 1740 1741 1742 1743 1744 1745 1746 1747 1748 1749 1750 1751 1752 1753 1754 1755 1756 1757 1758 1759 1760 1761 1762 1763 1764 1765 1766 1767 1768 1769 1770 1771 1772 1773 1774 1775 1776 1777 1778 1779 1780 1781 1782 1783 1784 1785 1786 1787 1788 1789 1790 1791 1792 1793 1794 1795 1796 1797 1798 1799 1800 1801 1802 1803 1804 1805 1806 1807 1808 1809 1810 1811 1812 1813 1814 1815 1816 1817 1818 1819 1820 1821 1822 1823 1824 1825 1826 1827 1828 1829 1830 1831 1832 1833 1834 1835 1836 1837 1838 1839 1840 1841 1842 1843 1844 1845 1846 1847 1848 1849 1850 1851 1852 1853 1854 1855 1856 1857 1858 1859 1860 1861 1862 1863 1864 1865 1866 1867 1868 1869 1870 1871 1872 1873 1874 1875 1876 1877 1878 1879 1880 1881 1882 1883 1884 1885 1886 1887 1888 1889 1890 1891 1892 1893 1894 1895 1896 1897 1898 1899 1900 1901 1902 1903 1904 1905 1906 1907 1908 1909 1910 1911 1912 1913 1914 1915 1916 1917 1918 1919 1920 1921 1922 1923 1924 1925 1926 1927 1928 1929 1930 1931 1932 1933 1934 1935 1936 1937 1938 1939 1940 1941 1942 1943 1944 1945 1946 1947 1948 1949 1950 1951 1952 1953 1954 1955 1956 1957 1958 1959 1960 1961 1962 1963 1964 1965 1966 1967 1968 1969 1970 1971 1972 1973 1974 1975 1976 1977 1978 1979 1980 1981 1982 1983 1984 1985 1986 1987 1988 1989 1990 1991 1992 1993 1994 1995 1996 1997 1998 1999 2000 2001 2002 2003 2004 2005 2006 2007 2008 2009 2010 2011 2012 2013 2014 2015 2016 2017 2018 2019 2020 2021 2022 2023 2024 2025 2026 2027 2028 2029 2030 2031 2032 2033 2034 2035 2036 2037 2038 2039 2040 2041 2042 2043 2044 2045 2046 2047 2048 2049 2050 2051 2052 2053 2054 2055 2056 2057 2058 2059 2060 2061 2062 2063 2064 2065 2066 2067 2068 2069 2070 2071 2072 2073 2074 2075 2076 2077 2078 2079 2080 2081 2082 2083 2084 2085 2086 2087 2088 2089 2090 2091 2092 2093 2094 2095 2096 2097 2098 2099 2100 2101 2102 2103 2104 2105 2106 2107 2108 2109 2110 2111 2112 2113 2114 2115 2116 2117 2118 2119 2120 2121 2122 2123 2124 2125 2126 2127 2128 2129 2130 2131 2132 2133 2134 2135 2136 2137 2138 2139 2140 2141 2142 2143 2144 2145 2146 2147 2148 2149 2150 2151 2152 2153 2154 2155 2156 2157 2158 2159 2160 2161 2162 2163 2164 2165 2166 2167 2168 2169 2170 2171 2172 2173 2174 2175 2176 2177 2178 2179 2180 2181 2182 2183 2184 2185 2186 2187 2188 2189 2190 2191 2192 2193 2194 2195 2196 2197 2198 2199 2200 2201 2202 2203 2204 2205 2206 2207 2208 2209 2210 2211 2212 2213 2214 2215 2216 2217 2218 2219 2220 2221 2222 2223 2224 2225 2226 2227 2228 2229 2230 2231 2232 2233 2234 2235 2236 2237 2238 2239 2240 2241 2242 2243 2244 2245 2246 2247 2248 2249 2250 2251 2252 2253 2254 2255 2256 2257 2258 2259 2260 2261 2262 2263 2264 2265 2266 2267 2268 2269 2270 2271 2272 2273 2274 2275 2276 2277 2278 2279 2280 2281 2282 2283 2284 2285 2286 2287 2288 2289 2290 2291 2292 2293 2294 2295 2296 2297 2298 2299 2300 2301 2302 2303 2304 2305 2306 2307 2308 2309 2310 2311 2312 2313 2314 2315 2316 2317 2318 2319 2320 2321 2322 2323 2324 2325 2326 2327 2328 2329 2330 2331 2332 2333 2334 2335 2336 2337 2338 2339 2340 2341 2342 2343 2344 2345 2346 2347 2348 2349 2350 2351 2352 2353 2354 2355 2356 2357 2358 2359 2360 2361 2362 2363 2364 2365 2366 2367 2368 2369 2370 2371 2372 2373 2374 2375 2376 2377 2378 2379 2380 2381 2382 2383 2384 2385 2386 2387 2388 2389 2390 2391 2392 2393 2394 2395 2396 2397 2398 2399 2400 2401 2402 2403 2404 2405 2406 2407 2408 2409 2410 2411 2412 2413 2414 2415 2416 2417 2418 2419 2420 2421 2422 2423 | /* $NetBSD: ufs_vnops.c,v 1.262 2022/03/27 16:24:59 christos Exp $ */ /*- * Copyright (c) 2008, 2020 The NetBSD Foundation, Inc. * All rights reserved. * * This code is derived from software contributed to The NetBSD Foundation * by Wasabi Systems, Inc. * * Redistribution and use in source and binary forms, with or without * modification, are permitted provided that the following conditions * are met: * 1. Redistributions of source code must retain the above copyright * notice, this list of conditions and the following disclaimer. * 2. Redistributions in binary form must reproduce the above copyright * notice, this list of conditions and the following disclaimer in the * documentation and/or other materials provided with the distribution. * * THIS SOFTWARE IS PROVIDED BY THE NETBSD FOUNDATION, INC. AND CONTRIBUTORS * ``AS IS'' AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED * TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR * PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE FOUNDATION OR CONTRIBUTORS * BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR * CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF * SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS * INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN * CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) * ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE * POSSIBILITY OF SUCH DAMAGE. */ /* * Copyright (c) 1982, 1986, 1989, 1993, 1995 * The Regents of the University of California. All rights reserved. * (c) UNIX System Laboratories, Inc. * All or some portions of this file are derived from material licensed * to the University of California by American Telephone and Telegraph * Co. or Unix System Laboratories, Inc. and are reproduced herein with * the permission of UNIX System Laboratories, Inc. * * Redistribution and use in source and binary forms, with or without * modification, are permitted provided that the following conditions * are met: * 1. Redistributions of source code must retain the above copyright * notice, this list of conditions and the following disclaimer. * 2. Redistributions in binary form must reproduce the above copyright * notice, this list of conditions and the following disclaimer in the * documentation and/or other materials provided with the distribution. * 3. Neither the name of the University nor the names of its contributors * may be used to endorse or promote products derived from this software * without specific prior written permission. * * THIS SOFTWARE IS PROVIDED BY THE REGENTS AND CONTRIBUTORS ``AS IS'' AND * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE * ARE DISCLAIMED. IN NO EVENT SHALL THE REGENTS OR CONTRIBUTORS BE LIABLE * FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL * DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS * OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) * HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT * LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY * OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF * SUCH DAMAGE. * * @(#)ufs_vnops.c 8.28 (Berkeley) 7/31/95 */ #include <sys/cdefs.h> __KERNEL_RCSID(0, "$NetBSD: ufs_vnops.c,v 1.262 2022/03/27 16:24:59 christos Exp $"); #if defined(_KERNEL_OPT) #include "opt_ffs.h" #include "opt_quota.h" #include "opt_uvmhist.h" #endif #include <sys/param.h> #include <sys/systm.h> #include <sys/namei.h> #include <sys/resourcevar.h> #include <sys/kernel.h> #include <sys/file.h> #include <sys/stat.h> #include <sys/buf.h> #include <sys/proc.h> #include <sys/mount.h> #include <sys/vnode.h> #include <sys/fstrans.h> #include <sys/kmem.h> #include <sys/malloc.h> #include <sys/dirent.h> #include <sys/lockf.h> #include <sys/kauth.h> #include <sys/wapbl.h> #include <miscfs/specfs/specdev.h> #include <miscfs/fifofs/fifo.h> #include <miscfs/genfs/genfs.h> #include <ufs/ufs/acl.h> #include <ufs/ufs/inode.h> #include <ufs/ufs/dir.h> #include <ufs/ufs/ufsmount.h> #include <ufs/ufs/ufs_bswap.h> #include <ufs/ufs/ufs_extern.h> #include <ufs/ufs/ufs_wapbl.h> #ifdef UFS_DIRHASH #include <ufs/ufs/dirhash.h> #endif #include <ufs/ext2fs/ext2fs_extern.h> #include <ufs/ext2fs/ext2fs_dir.h> #include <ufs/ffs/ffs_extern.h> #include <ufs/lfs/lfs_extern.h> #include <ufs/lfs/lfs.h> #ifdef UVMHIST #include <uvm/uvm.h> #endif #include <uvm/uvm_extern.h> #include <uvm/uvm_stat.h> __CTASSERT(EXT2FS_MAXNAMLEN == FFS_MAXNAMLEN); __CTASSERT(LFS_MAXNAMLEN == FFS_MAXNAMLEN); static int ufs_chmod(struct vnode *, int, kauth_cred_t, struct lwp *); static int ufs_chown(struct vnode *, uid_t, gid_t, kauth_cred_t, struct lwp *); static int ufs_makeinode(struct vattr *, struct vnode *, const struct ufs_lookup_results *, struct vnode **, struct componentname *); /* * A virgin directory (no blushing please). */ static const struct dirtemplate mastertemplate = { 0, 12, DT_DIR, 1, ".", 0, UFS_DIRBLKSIZ - 12, DT_DIR, 2, ".." }; /* * Create a regular file */ int ufs_create(void *v) { struct vop_create_v3_args /* { struct vnode *a_dvp; struct vnode **a_vpp; struct componentname *a_cnp; struct vattr *a_vap; } */ *ap = v; int error; struct vnode *dvp = ap->a_dvp; struct ufs_lookup_results *ulr; /* XXX should handle this material another way */ ulr = &VTOI(dvp)->i_crap; UFS_CHECK_CRAPCOUNTER(VTOI(dvp)); /* * UFS_WAPBL_BEGIN(dvp->v_mount) performed by successful * ufs_makeinode */ error = ufs_makeinode(ap->a_vap, dvp, ulr, ap->a_vpp, ap->a_cnp); if (error) { return (error); } UFS_WAPBL_END(dvp->v_mount); VOP_UNLOCK(*ap->a_vpp); return (0); } /* * Mknod vnode call */ /* ARGSUSED */ int ufs_mknod(void *v) { struct vop_mknod_v3_args /* { struct vnode *a_dvp; struct vnode **a_vpp; struct componentname *a_cnp; struct vattr *a_vap; } */ *ap = v; struct vattr *vap; struct vnode **vpp; struct inode *ip; int error; struct ufs_lookup_results *ulr; vap = ap->a_vap; vpp = ap->a_vpp; /* XXX should handle this material another way */ ulr = &VTOI(ap->a_dvp)->i_crap; UFS_CHECK_CRAPCOUNTER(VTOI(ap->a_dvp)); /* * UFS_WAPBL_BEGIN(dvp->v_mount) performed by successful * ufs_makeinode */ if ((error = ufs_makeinode(vap, ap->a_dvp, ulr, vpp, ap->a_cnp)) != 0) goto out; ip = VTOI(*vpp); ip->i_flag |= IN_ACCESS | IN_CHANGE | IN_UPDATE; UFS_WAPBL_UPDATE(*vpp, NULL, NULL, 0); UFS_WAPBL_END(ap->a_dvp->v_mount); VOP_UNLOCK(*vpp); out: if (error != 0) { *vpp = NULL; return (error); } return (0); } /* * Open called. * * Nothing to do. */ /* ARGSUSED */ int ufs_open(void *v) { struct vop_open_args /* { struct vnode *a_vp; int a_mode; kauth_cred_t a_cred; } */ *ap = v; /* * Files marked append-only must be opened for appending. */ if ((VTOI(ap->a_vp)->i_flags & APPEND) && (ap->a_mode & (FWRITE | O_APPEND)) == FWRITE) return (EPERM); return (0); } /* * Close called. * * Update the times on the inode. */ /* ARGSUSED */ int ufs_close(void *v) { struct vop_close_args /* { struct vnode *a_vp; int a_fflag; kauth_cred_t a_cred; } */ *ap = v; struct vnode *vp; vp = ap->a_vp; if (vrefcnt(vp) > 1) UFS_ITIMES(vp, NULL, NULL, NULL); return (0); } static int ufs_check_possible(struct vnode *vp, struct inode *ip, accmode_t accmode, kauth_cred_t cred) { #if defined(QUOTA) || defined(QUOTA2) int error; #endif /* * Disallow write attempts on read-only file systems; * unless the file is a socket, fifo, or a block or * character device resident on the file system. */ if (accmode & VMODIFY_PERMS) { switch (vp->v_type) { case VDIR: case VLNK: case VREG: if (vp->v_mount->mnt_flag & MNT_RDONLY) return EROFS; #if defined(QUOTA) || defined(QUOTA2) error = chkdq(ip, 0, cred, 0); if (error != 0) return error; #endif break; case VBAD: case VBLK: case VCHR: case VSOCK: case VFIFO: case VNON: default: break; } } /* If it is a snapshot, nobody gets access to it. */ if ((ip->i_flags & SF_SNAPSHOT)) return EPERM; /* * If immutable bit set, nobody gets to write it. "& ~VADMIN_PERMS" * permits the owner of the file to remove the IMMUTABLE flag. */ if ((accmode & (VMODIFY_PERMS & ~VADMIN_PERMS)) && (ip->i_flags & IMMUTABLE)) return EPERM; return 0; } static int ufs_check_permitted(struct vnode *vp, struct inode *ip, struct acl *acl, accmode_t accmode, kauth_cred_t cred, int (*func)(struct vnode *, kauth_cred_t, uid_t, gid_t, mode_t, struct acl *, accmode_t)) { return kauth_authorize_vnode(cred, KAUTH_ACCESS_ACTION(accmode, vp->v_type, ip->i_mode & ALLPERMS), vp, NULL, (*func)(vp, cred, ip->i_uid, ip->i_gid, ip->i_mode & ALLPERMS, acl, accmode)); } int ufs_accessx(void *v) { struct vop_accessx_args /* { struct vnode *a_vp; accmode_t a_accmode; kauth_cred_t a_cred; } */ *ap = v; struct vnode *vp = ap->a_vp; struct inode *ip = VTOI(vp); accmode_t accmode = ap->a_accmode; int error; #ifdef UFS_ACL struct acl *acl; acl_type_t type; #endif error = ufs_check_possible(vp, ip, accmode, ap->a_cred); if (error) return error; #ifdef UFS_ACL if ((vp->v_mount->mnt_flag & (MNT_POSIX1EACLS | MNT_NFS4ACLS)) != 0) { if (vp->v_mount->mnt_flag & MNT_NFS4ACLS) type = ACL_TYPE_NFS4; else type = ACL_TYPE_ACCESS; acl = acl_alloc(KM_SLEEP); if (type == ACL_TYPE_NFS4) error = ufs_getacl_nfs4_internal(vp, acl, curlwp); else error = VOP_GETACL(vp, type, acl, ap->a_cred); if (!error) { if (type == ACL_TYPE_NFS4) { error = ufs_check_permitted(vp, ip, acl, accmode, ap->a_cred, genfs_can_access_acl_nfs4); } else { error = vfs_unixify_accmode(&accmode); if (error == 0) error = ufs_check_permitted(vp, ip, acl, accmode, ap->a_cred, genfs_can_access_acl_posix1e); } acl_free(acl); return error; } if (error != EOPNOTSUPP) printf("%s: Error retrieving ACL: %d\n", __func__, error); /* * XXX: Fall back until debugged. Should * eventually possibly log an error, and return * EPERM for safety. */ acl_free(acl); } #endif /* !UFS_ACL */ error = vfs_unixify_accmode(&accmode); if (error) return error; return ufs_check_permitted(vp, ip, NULL, accmode, ap->a_cred, genfs_can_access); } /* ARGSUSED */ int ufs_getattr(void *v) { struct vop_getattr_args /* { struct vnode *a_vp; struct vattr *a_vap; kauth_cred_t a_cred; } */ *ap = v; struct vnode *vp; struct inode *ip; struct vattr *vap; vp = ap->a_vp; ip = VTOI(vp); vap = ap->a_vap; UFS_ITIMES(vp, NULL, NULL, NULL); /* * Copy from inode table */ vap->va_fsid = ip->i_dev; vap->va_fileid = ip->i_number; vap->va_mode = ip->i_mode & ALLPERMS; vap->va_nlink = ip->i_nlink; vap->va_uid = ip->i_uid; vap->va_gid = ip->i_gid; vap->va_size = vp->v_size; if (ip->i_ump->um_fstype == UFS1) { switch (vp->v_type) { case VBLK: case VCHR: vap->va_rdev = (dev_t)ufs_rw32(ip->i_ffs1_rdev, UFS_MPNEEDSWAP(ip->i_ump)); break; default: vap->va_rdev = NODEV; break; } vap->va_atime.tv_sec = ip->i_ffs1_atime; vap->va_atime.tv_nsec = ip->i_ffs1_atimensec; vap->va_mtime.tv_sec = ip->i_ffs1_mtime; vap->va_mtime.tv_nsec = ip->i_ffs1_mtimensec; vap->va_ctime.tv_sec = ip->i_ffs1_ctime; vap->va_ctime.tv_nsec = ip->i_ffs1_ctimensec; vap->va_birthtime.tv_sec = 0; vap->va_birthtime.tv_nsec = 0; vap->va_bytes = dbtob((u_quad_t)ip->i_ffs1_blocks); } else { switch (vp->v_type) { case VBLK: case VCHR: vap->va_rdev = (dev_t)ufs_rw64(ip->i_ffs2_rdev, UFS_MPNEEDSWAP(ip->i_ump)); break; default: vap->va_rdev = NODEV; break; } vap->va_atime.tv_sec = ip->i_ffs2_atime; vap->va_atime.tv_nsec = ip->i_ffs2_atimensec; vap->va_mtime.tv_sec = ip->i_ffs2_mtime; vap->va_mtime.tv_nsec = ip->i_ffs2_mtimensec; vap->va_ctime.tv_sec = ip->i_ffs2_ctime; vap->va_ctime.tv_nsec = ip->i_ffs2_ctimensec; vap->va_birthtime.tv_sec = ip->i_ffs2_birthtime; vap->va_birthtime.tv_nsec = ip->i_ffs2_birthnsec; vap->va_bytes = dbtob(ip->i_ffs2_blocks); } vap->va_gen = ip->i_gen; vap->va_flags = ip->i_flags; /* this doesn't belong here */ if (vp->v_type == VBLK) vap->va_blocksize = BLKDEV_IOSIZE; else if (vp->v_type == VCHR) vap->va_blocksize = MAXBSIZE; else vap->va_blocksize = vp->v_mount->mnt_stat.f_iosize; vap->va_type = vp->v_type; vap->va_filerev = ip->i_modrev; return (0); } /* * Set attribute vnode op. called from several syscalls */ int ufs_setattr(void *v) { struct vop_setattr_args /* { struct vnode *a_vp; struct vattr *a_vap; kauth_cred_t a_cred; } */ *ap = v; struct vattr *vap; struct vnode *vp; struct inode *ip; kauth_cred_t cred; struct lwp *l; int error; kauth_action_t action; bool changing_sysflags; vap = ap->a_vap; vp = ap->a_vp; ip = VTOI(vp); cred = ap->a_cred; l = curlwp; action = KAUTH_VNODE_WRITE_FLAGS; changing_sysflags = false; /* * Check for unsettable attributes. */ if ((vap->va_type != VNON) || (vap->va_nlink != VNOVAL) || (vap->va_fsid != VNOVAL) || (vap->va_fileid != VNOVAL) || (vap->va_blocksize != VNOVAL) || (vap->va_rdev != VNOVAL) || ((int)vap->va_bytes != VNOVAL) || (vap->va_gen != VNOVAL)) { return (EINVAL); } UFS_WAPBL_JUNLOCK_ASSERT(vp->v_mount); if (vap->va_flags != VNOVAL) { if (vp->v_mount->mnt_flag & MNT_RDONLY) { error = EROFS; goto out; } /* Snapshot flag cannot be set or cleared */ if ((vap->va_flags & (SF_SNAPSHOT | SF_SNAPINVAL)) != (ip->i_flags & (SF_SNAPSHOT | SF_SNAPINVAL))) { error = EPERM; goto out; } if (ip->i_flags & (SF_IMMUTABLE | SF_APPEND)) { action |= KAUTH_VNODE_HAS_SYSFLAGS; } if ((vap->va_flags & SF_SETTABLE) != (ip->i_flags & SF_SETTABLE)) { action |= KAUTH_VNODE_WRITE_SYSFLAGS; changing_sysflags = true; } error = kauth_authorize_vnode(cred, action, vp, NULL, genfs_can_chflags(vp, cred, ip->i_uid, changing_sysflags)); if (error) goto out; if (changing_sysflags) { error = UFS_WAPBL_BEGIN(vp->v_mount); if (error) goto out; ip->i_flags = vap->va_flags; DIP_ASSIGN(ip, flags, ip->i_flags); } else { error = UFS_WAPBL_BEGIN(vp->v_mount); if (error) goto out; ip->i_flags &= SF_SETTABLE; ip->i_flags |= (vap->va_flags & UF_SETTABLE); DIP_ASSIGN(ip, flags, ip->i_flags); } ip->i_flag |= IN_CHANGE; UFS_WAPBL_UPDATE(vp, NULL, NULL, 0); UFS_WAPBL_END(vp->v_mount); if (vap->va_flags & (IMMUTABLE | APPEND)) { error = 0; goto out; } } if (ip->i_flags & (IMMUTABLE | APPEND)) { error = EPERM; goto out; } /* * Go through the fields and update iff not VNOVAL. */ if (vap->va_uid != (uid_t)VNOVAL || vap->va_gid != (gid_t)VNOVAL) { if (vp->v_mount->mnt_flag & MNT_RDONLY) { error = EROFS; goto out; } error = UFS_WAPBL_BEGIN(vp->v_mount); if (error) goto out; error = ufs_chown(vp, vap->va_uid, vap->va_gid, cred, l); UFS_WAPBL_END(vp->v_mount); if (error) goto out; } if (vap->va_size != VNOVAL) { /* * Disallow write attempts on read-only file systems; * unless the file is a socket, fifo, or a block or * character device resident on the file system. */ switch (vp->v_type) { case VDIR: error = EISDIR; goto out; case VCHR: case VBLK: case VFIFO: break; case VREG: if (vp->v_mount->mnt_flag & MNT_RDONLY) { error = EROFS; goto out; } if ((ip->i_flags & SF_SNAPSHOT) != 0) { error = EPERM; goto out; } error = ufs_truncate_retry(vp, 0, vap->va_size, cred); if (error) goto out; break; default: error = EOPNOTSUPP; goto out; } } ip = VTOI(vp); if (vap->va_atime.tv_sec != VNOVAL || vap->va_mtime.tv_sec != VNOVAL || vap->va_birthtime.tv_sec != VNOVAL) { if (vp->v_mount->mnt_flag & MNT_RDONLY) { error = EROFS; goto out; } if ((ip->i_flags & SF_SNAPSHOT) != 0) { error = EPERM; goto out; } error = kauth_authorize_vnode(cred, KAUTH_VNODE_WRITE_TIMES, vp, NULL, genfs_can_chtimes(vp, cred, ip->i_uid, vap->va_vaflags)); if (error) goto out; error = UFS_WAPBL_BEGIN(vp->v_mount); if (error) goto out; if (vap->va_atime.tv_sec != VNOVAL) if (!(vp->v_mount->mnt_flag & MNT_NOATIME)) ip->i_flag |= IN_ACCESS; if (vap->va_mtime.tv_sec != VNOVAL) { ip->i_flag |= IN_CHANGE | IN_UPDATE; if (vp->v_mount->mnt_flag & MNT_RELATIME) ip->i_flag |= IN_ACCESS; } if (vap->va_birthtime.tv_sec != VNOVAL && ip->i_ump->um_fstype == UFS2) { ip->i_ffs2_birthtime = vap->va_birthtime.tv_sec; ip->i_ffs2_birthnsec = vap->va_birthtime.tv_nsec; } error = UFS_UPDATE(vp, &vap->va_atime, &vap->va_mtime, 0); UFS_WAPBL_END(vp->v_mount); if (error) goto out; } error = 0; if (vap->va_mode != (mode_t)VNOVAL) { if (vp->v_mount->mnt_flag & MNT_RDONLY) { error = EROFS; goto out; } if ((ip->i_flags & SF_SNAPSHOT) != 0 && (vap->va_mode & (S_IXUSR | S_IWUSR | S_IXGRP | S_IWGRP | S_IXOTH | S_IWOTH))) { error = EPERM; goto out; } error = UFS_WAPBL_BEGIN(vp->v_mount); if (error) goto out; error = ufs_chmod(vp, (int)vap->va_mode, cred, l); UFS_WAPBL_END(vp->v_mount); } out: cache_enter_id(vp, ip->i_mode, ip->i_uid, ip->i_gid, !HAS_ACLS(ip)); return (error); } #ifdef UFS_ACL static int ufs_update_nfs4_acl_after_mode_change(struct vnode *vp, int mode, int file_owner_id, kauth_cred_t cred, struct lwp *l) { int error; struct acl *aclp; aclp = acl_alloc(KM_SLEEP); error = ufs_getacl_nfs4_internal(vp, aclp, l); /* * We don't have to handle EOPNOTSUPP here, as the filesystem claims * it supports ACLs. */ if (error) goto out; acl_nfs4_sync_acl_from_mode(aclp, mode, file_owner_id); error = ufs_setacl_nfs4_internal(vp, aclp, l, false); out: acl_free(aclp); return (error); } #endif /* UFS_ACL */ /* * Change the mode on a file. * Inode must be locked before calling. */ static int ufs_chmod(struct vnode *vp, int mode, kauth_cred_t cred, struct lwp *l) { struct inode *ip; int error; UFS_WAPBL_JLOCK_ASSERT(vp->v_mount); ip = VTOI(vp); #ifdef UFS_ACL /* * To modify the permissions on a file, must possess VADMIN * for that file. */ if ((error = VOP_ACCESSX(vp, VWRITE_ACL, cred)) != 0) return error; #endif error = kauth_authorize_vnode(cred, KAUTH_VNODE_WRITE_SECURITY, vp, NULL, genfs_can_chmod(vp, cred, ip->i_uid, ip->i_gid, mode)); if (error) return (error); #ifdef UFS_ACL if ((vp->v_mount->mnt_flag & MNT_NFS4ACLS) != 0) { error = ufs_update_nfs4_acl_after_mode_change(vp, mode, ip->i_uid, cred, l); if (error) return error; } #endif ip->i_mode &= ~ALLPERMS; ip->i_mode |= (mode & ALLPERMS); ip->i_flag |= IN_CHANGE; DIP_ASSIGN(ip, mode, ip->i_mode); UFS_WAPBL_UPDATE(vp, NULL, NULL, 0); cache_enter_id(vp, ip->i_mode, ip->i_uid, ip->i_gid, !HAS_ACLS(ip)); return (0); } /* * Perform chown operation on inode ip; * inode must be locked prior to call. */ static int ufs_chown(struct vnode *vp, uid_t uid, gid_t gid, kauth_cred_t cred, struct lwp *l) { struct inode *ip; int error = 0; #if defined(QUOTA) || defined(QUOTA2) uid_t ouid; gid_t ogid; int64_t change; #endif ip = VTOI(vp); error = 0; if (uid == (uid_t)VNOVAL) uid = ip->i_uid; if (gid == (gid_t)VNOVAL) gid = ip->i_gid; #ifdef UFS_ACL /* * To modify the ownership of a file, must possess VADMIN for that * file. */ if ((error = VOP_ACCESSX(vp, VWRITE_OWNER, cred)) != 0) return error; #endif error = kauth_authorize_vnode(cred, KAUTH_VNODE_CHANGE_OWNERSHIP, vp, NULL, genfs_can_chown(vp, cred, ip->i_uid, ip->i_gid, uid, gid)); if (error) return (error); #if defined(QUOTA) || defined(QUOTA2) ogid = ip->i_gid; ouid = ip->i_uid; change = DIP(ip, blocks); (void) chkdq(ip, -change, cred, 0); (void) chkiq(ip, -1, cred, 0); #endif ip->i_gid = gid; DIP_ASSIGN(ip, gid, gid); ip->i_uid = uid; DIP_ASSIGN(ip, uid, uid); #if defined(QUOTA) || defined(QUOTA2) if ((error = chkdq(ip, change, cred, 0)) == 0) { if ((error = chkiq(ip, 1, cred, 0)) == 0) goto good; else (void) chkdq(ip, -change, cred, FORCE); } ip->i_gid = ogid; DIP_ASSIGN(ip, gid, ogid); ip->i_uid = ouid; DIP_ASSIGN(ip, uid, ouid); (void) chkdq(ip, change, cred, FORCE); (void) chkiq(ip, 1, cred, FORCE); return (error); good: #endif /* QUOTA || QUOTA2 */ ip->i_flag |= IN_CHANGE; UFS_WAPBL_UPDATE(vp, NULL, NULL, 0); cache_enter_id(vp, ip->i_mode, ip->i_uid, ip->i_gid, !HAS_ACLS(ip)); return (0); } int ufs_remove(void *v) { struct vop_remove_v3_args /* { struct vnode *a_dvp; struct vnode *a_vp; struct componentname *a_cnp; nlink_t ctx_vp_new_nlink; } */ *ap = v; struct vnode *vp, *dvp; struct inode *ip; struct mount *mp; int error; struct ufs_lookup_results *ulr; vp = ap->a_vp; dvp = ap->a_dvp; ip = VTOI(vp); mp = dvp->v_mount; KASSERT(mp == vp->v_mount); /* XXX Not stable without lock. */ #ifdef UFS_ACL #ifdef notyet /* We don't do this because if the filesystem is mounted without ACLs * this goes through vfs_unixify_accmode() and we get EPERM. */ error = VOP_ACCESSX(vp, VDELETE, ap->a_cnp->cn_cred); if (error) goto err; #endif #endif /* XXX should handle this material another way */ ulr = &VTOI(dvp)->i_crap; UFS_CHECK_CRAPCOUNTER(VTOI(dvp)); if (vp->v_type == VDIR || (ip->i_flags & (IMMUTABLE | APPEND)) || (VTOI(dvp)->i_flags & APPEND)) error = EPERM; else { error = UFS_WAPBL_BEGIN(mp); if (error == 0) { error = ufs_dirremove(dvp, ulr, ip, ap->a_cnp->cn_flags, 0); UFS_WAPBL_END(mp); if (error == 0) { ap->ctx_vp_new_nlink = ip->i_nlink; } } } #ifdef notyet err: #endif if (dvp == vp) vrele(vp); else vput(vp); return (error); } /* * ufs_link: create hard link. */ int ufs_link(void *v) { struct vop_link_v2_args /* { struct vnode *a_dvp; struct vnode *a_vp; struct componentname *a_cnp; } */ *ap = v; struct vnode *dvp = ap->a_dvp; struct vnode *vp = ap->a_vp; struct componentname *cnp = ap->a_cnp; struct mount *mp = dvp->v_mount; struct inode *ip; struct direct *newdir; int error, abrt = 1; struct ufs_lookup_results *ulr; KASSERT(dvp != vp); KASSERT(vp->v_type != VDIR); KASSERT(mp == vp->v_mount); /* XXX Not stable without lock. */ /* XXX should handle this material another way */ ulr = &VTOI(dvp)->i_crap; UFS_CHECK_CRAPCOUNTER(VTOI(dvp)); error = vn_lock(vp, LK_EXCLUSIVE); if (error) goto out2; ip = VTOI(vp); if ((nlink_t)ip->i_nlink >= LINK_MAX) { error = EMLINK; goto out1; } if (ip->i_flags & (IMMUTABLE | APPEND)) { error = EPERM; goto out1; } error = kauth_authorize_vnode(cnp->cn_cred, KAUTH_VNODE_ADD_LINK, vp, dvp, 0); if (error) goto out1; error = UFS_WAPBL_BEGIN(mp); if (error) goto out1; ip->i_nlink++; DIP_ASSIGN(ip, nlink, ip->i_nlink); ip->i_flag |= IN_CHANGE; abrt = 0; error = UFS_UPDATE(vp, NULL, NULL, UPDATE_DIROP); if (!error) { newdir = pool_cache_get(ufs_direct_cache, PR_WAITOK); ufs_makedirentry(ip, cnp, newdir); error = ufs_direnter(dvp, ulr, vp, newdir, cnp, NULL); pool_cache_put(ufs_direct_cache, newdir); } if (error) { ip->i_nlink--; DIP_ASSIGN(ip, nlink, ip->i_nlink); ip->i_flag |= IN_CHANGE; UFS_WAPBL_UPDATE(vp, NULL, NULL, UPDATE_DIROP); } UFS_WAPBL_END(mp); out1: VOP_UNLOCK(vp); out2: if (abrt) VOP_ABORTOP(dvp, cnp); return (error); } /* * whiteout vnode call */ int ufs_whiteout(void *v) { struct vop_whiteout_args /* { struct vnode *a_dvp; struct componentname *a_cnp; int a_flags; } */ *ap = v; struct vnode *dvp = ap->a_dvp; struct componentname *cnp = ap->a_cnp; struct direct *newdir; int error; struct ufsmount *ump = VFSTOUFS(dvp->v_mount); struct ufs_lookup_results *ulr; /* XXX should handle this material another way */ ulr = &VTOI(dvp)->i_crap; UFS_CHECK_CRAPCOUNTER(VTOI(dvp)); error = 0; switch (ap->a_flags) { case LOOKUP: /* 4.4 format directories support whiteout operations */ if (ump->um_maxsymlinklen > 0) return (0); return (EOPNOTSUPP); case CREATE: /* create a new directory whiteout */ error = UFS_WAPBL_BEGIN(dvp->v_mount); if (error) break; KASSERTMSG((ump->um_maxsymlinklen > 0), "ufs_whiteout: old format filesystem"); newdir = pool_cache_get(ufs_direct_cache, PR_WAITOK); newdir->d_ino = UFS_WINO; newdir->d_namlen = cnp->cn_namelen; memcpy(newdir->d_name, cnp->cn_nameptr, (size_t)cnp->cn_namelen); /* NUL terminate and zero out padding */ memset(&newdir->d_name[cnp->cn_namelen], 0, UFS_NAMEPAD(cnp->cn_namelen)); newdir->d_type = DT_WHT; error = ufs_direnter(dvp, ulr, NULL, newdir, cnp, NULL); pool_cache_put(ufs_direct_cache, newdir); break; case DELETE: /* remove an existing directory whiteout */ error = UFS_WAPBL_BEGIN(dvp->v_mount); if (error) break; KASSERTMSG((ump->um_maxsymlinklen > 0), "ufs_whiteout: old format filesystem"); cnp->cn_flags &= ~DOWHITEOUT; error = ufs_dirremove(dvp, ulr, NULL, cnp->cn_flags, 0); break; default: panic("ufs_whiteout: unknown op"); /* NOTREACHED */ } UFS_WAPBL_END(dvp->v_mount); return (error); } #ifdef UFS_ACL static int ufs_do_posix1e_acl_inheritance_dir(struct vnode *dvp, struct vnode *tvp, mode_t dmode, kauth_cred_t cred, struct lwp *l) { int error; struct inode *ip = VTOI(tvp); struct acl *dacl, *acl; acl = acl_alloc(KM_SLEEP); dacl = acl_alloc(KM_SLEEP); /* * Retrieve default ACL from parent, if any. */ error = VOP_GETACL(dvp, ACL_TYPE_DEFAULT, acl, cred); switch (error) { case 0: /* * Retrieved a default ACL, so merge mode and ACL if * necessary. If the ACL is empty, fall through to * the "not defined or available" case. */ if (acl->acl_cnt != 0) { dmode = acl_posix1e_newfilemode(dmode, acl); ip->i_mode = dmode; DIP_ASSIGN(ip, mode, dmode); *dacl = *acl; ufs_sync_acl_from_inode(ip, acl); break; } /* FALLTHROUGH */ case EOPNOTSUPP: /* * Just use the mode as-is. */ ip->i_mode = dmode; DIP_ASSIGN(ip, mode, dmode); error = 0; goto out; default: goto out; } /* * XXX: If we abort now, will Soft Updates notify the extattr * code that the EAs for the file need to be released? */ UFS_WAPBL_END(tvp->v_mount); error = ufs_setacl_posix1e(tvp, ACL_TYPE_ACCESS, acl, cred, l); if (error == 0) error = ufs_setacl_posix1e(tvp, ACL_TYPE_DEFAULT, dacl, cred, l); UFS_WAPBL_BEGIN(tvp->v_mount); switch (error) { case 0: break; case EOPNOTSUPP: /* * XXX: This should not happen, as EOPNOTSUPP above * was supposed to free acl. */ printf("ufs_mkdir: VOP_GETACL() but no VOP_SETACL()\n"); /* panic("ufs_mkdir: VOP_GETACL() but no VOP_SETACL()"); */ break; default: goto out; } out: acl_free(acl); acl_free(dacl); return (error); } static int ufs_do_posix1e_acl_inheritance_file(struct vnode *dvp, struct vnode *tvp, mode_t mode, kauth_cred_t cred, struct lwp *l) { int error; struct inode *ip = VTOI(tvp); struct acl *acl; acl = acl_alloc(KM_SLEEP); /* * Retrieve default ACL for parent, if any. */ error = VOP_GETACL(dvp, ACL_TYPE_DEFAULT, acl, cred); switch (error) { case 0: /* * Retrieved a default ACL, so merge mode and ACL if * necessary. */ if (acl->acl_cnt != 0) { /* * Two possible ways for default ACL to not * be present. First, the EA can be * undefined, or second, the default ACL can * be blank. If it's blank, fall through to * the it's not defined case. */ mode = acl_posix1e_newfilemode(mode, acl); ip->i_mode = mode; DIP_ASSIGN(ip, mode, mode); ufs_sync_acl_from_inode(ip, acl); break; } /* FALLTHROUGH */ case EOPNOTSUPP: /* * Just use the mode as-is. */ ip->i_mode = mode; DIP_ASSIGN(ip, mode, mode); error = 0; goto out; default: goto out; } UFS_WAPBL_END(tvp->v_mount); /* * XXX: If we abort now, will Soft Updates notify the extattr * code that the EAs for the file need to be released? */ error = VOP_SETACL(tvp, ACL_TYPE_ACCESS, acl, cred); UFS_WAPBL_BEGIN(tvp->v_mount); switch (error) { case 0: break; case EOPNOTSUPP: /* * XXX: This should not happen, as EOPNOTSUPP above was * supposed to free acl. */ printf("%s: VOP_GETACL() but no VOP_SETACL()\n", __func__); /* panic("%s: VOP_GETACL() but no VOP_SETACL()", __func__); */ break; default: goto out; } out: acl_free(acl); return (error); } static int ufs_do_nfs4_acl_inheritance(struct vnode *dvp, struct vnode *tvp, mode_t child_mode, kauth_cred_t cred, struct lwp *l) { int error; struct acl *parent_aclp, *child_aclp; parent_aclp = acl_alloc(KM_SLEEP); child_aclp = acl_alloc(KM_SLEEP); error = ufs_getacl_nfs4_internal(dvp, parent_aclp, l); if (error) goto out; acl_nfs4_compute_inherited_acl(parent_aclp, child_aclp, child_mode, VTOI(tvp)->i_uid, tvp->v_type == VDIR); error = ufs_setacl_nfs4_internal(tvp, child_aclp, l, false); if (error) goto out; out: acl_free(parent_aclp); acl_free(child_aclp); return (error); } #endif int ufs_mkdir(void *v) { struct vop_mkdir_v3_args /* { struct vnode *a_dvp; struct vnode **a_vpp; struct componentname *a_cnp; struct vattr *a_vap; } */ *ap = v; struct vnode *dvp = ap->a_dvp, *tvp; struct vattr *vap = ap->a_vap; struct componentname *cnp = ap->a_cnp; struct inode *ip, *dp = VTOI(dvp); struct buf *bp; struct dirtemplate dirtemplate; struct direct *newdir; int error; struct ufsmount *ump = dp->i_ump; int dirblksiz = ump->um_dirblksiz; struct ufs_lookup_results *ulr; /* XXX should handle this material another way */ ulr = &dp->i_crap; UFS_CHECK_CRAPCOUNTER(dp); KASSERT(vap->va_type == VDIR); if ((nlink_t)dp->i_nlink >= LINK_MAX) { error = EMLINK; goto out; } /* * Must simulate part of ufs_makeinode here to acquire the inode, * but not have it entered in the parent directory. The entry is * made later after writing "." and ".." entries. */ error = vcache_new(dvp->v_mount, dvp, vap, cnp->cn_cred, NULL, ap->a_vpp); if (error) goto out; error = vn_lock(*ap->a_vpp, LK_EXCLUSIVE); if (error) { vrele(*ap->a_vpp); *ap->a_vpp = NULL; goto out; } error = UFS_WAPBL_BEGIN(ap->a_dvp->v_mount); if (error) { vput(*ap->a_vpp); goto out; } tvp = *ap->a_vpp; ip = VTOI(tvp); ip->i_flag |= IN_ACCESS | IN_CHANGE | IN_UPDATE; ip->i_nlink = 2; DIP_ASSIGN(ip, nlink, 2); if (cnp->cn_flags & ISWHITEOUT) { ip->i_flags |= UF_OPAQUE; DIP_ASSIGN(ip, flags, ip->i_flags); } /* * Bump link count in parent directory to reflect work done below. * Should be done before reference is created so cleanup is * possible if we crash. */ dp->i_nlink++; DIP_ASSIGN(dp, nlink, dp->i_nlink); dp->i_flag |= IN_CHANGE; if ((error = UFS_UPDATE(dvp, NULL, NULL, UPDATE_DIROP)) != 0) goto bad; #ifdef UFS_ACL mode_t dmode = (vap->va_mode & 0777) | IFDIR; struct lwp *l = curlwp; if (dvp->v_mount->mnt_flag & MNT_POSIX1EACLS) { error = ufs_do_posix1e_acl_inheritance_dir(dvp, tvp, dmode, cnp->cn_cred, l); if (error) goto bad; } else if (dvp->v_mount->mnt_flag & MNT_NFS4ACLS) { error = ufs_do_nfs4_acl_inheritance(dvp, tvp, dmode, cnp->cn_cred, l); if (error) goto bad; } #endif /* !UFS_ACL */ /* * Initialize directory with "." and ".." from static template. */ dirtemplate = mastertemplate; dirtemplate.dotdot_reclen = dirblksiz - dirtemplate.dot_reclen; dirtemplate.dot_ino = ufs_rw32(ip->i_number, UFS_MPNEEDSWAP(ump)); dirtemplate.dotdot_ino = ufs_rw32(dp->i_number, UFS_MPNEEDSWAP(ump)); dirtemplate.dot_reclen = ufs_rw16(dirtemplate.dot_reclen, UFS_MPNEEDSWAP(ump)); dirtemplate.dotdot_reclen = ufs_rw16(dirtemplate.dotdot_reclen, UFS_MPNEEDSWAP(ump)); if (ump->um_maxsymlinklen <= 0) { #if BYTE_ORDER == LITTLE_ENDIAN if (UFS_MPNEEDSWAP(ump) == 0) #else if (UFS_MPNEEDSWAP(ump) != 0) #endif { dirtemplate.dot_type = dirtemplate.dot_namlen; dirtemplate.dotdot_type = dirtemplate.dotdot_namlen; dirtemplate.dot_namlen = dirtemplate.dotdot_namlen = 0; } else dirtemplate.dot_type = dirtemplate.dotdot_type = 0; } if ((error = UFS_BALLOC(tvp, (off_t)0, dirblksiz, cnp->cn_cred, B_CLRBUF, &bp)) != 0) goto bad; ip->i_size = dirblksiz; DIP_ASSIGN(ip, size, dirblksiz); ip->i_flag |= IN_ACCESS | IN_CHANGE | IN_UPDATE; uvm_vnp_setsize(tvp, ip->i_size); memcpy((void *)bp->b_data, (void *)&dirtemplate, sizeof dirtemplate); /* * Directory set up, now install its entry in the parent directory. * We must write out the buffer containing the new directory body * before entering the new name in the parent. */ if ((error = VOP_BWRITE(bp->b_vp, bp)) != 0) goto bad; if ((error = UFS_UPDATE(tvp, NULL, NULL, UPDATE_DIROP)) != 0) { goto bad; } newdir = pool_cache_get(ufs_direct_cache, PR_WAITOK); ufs_makedirentry(ip, cnp, newdir); error = ufs_direnter(dvp, ulr, tvp, newdir, cnp, bp); pool_cache_put(ufs_direct_cache, newdir); bad: if (error == 0) { VOP_UNLOCK(tvp); UFS_WAPBL_END(dvp->v_mount); } else { dp->i_nlink--; DIP_ASSIGN(dp, nlink, dp->i_nlink); dp->i_flag |= IN_CHANGE; UFS_WAPBL_UPDATE(dvp, NULL, NULL, UPDATE_DIROP); /* * No need to do an explicit UFS_TRUNCATE here, vrele will * do this for us because we set the link count to 0. */ ip->i_nlink = 0; DIP_ASSIGN(ip, nlink, 0); ip->i_flag |= IN_CHANGE; UFS_WAPBL_UPDATE(tvp, NULL, NULL, UPDATE_DIROP); UFS_WAPBL_END(dvp->v_mount); vput(tvp); } out: return (error); } int ufs_rmdir(void *v) { struct vop_rmdir_v2_args /* { struct vnode *a_dvp; struct vnode *a_vp; struct componentname *a_cnp; } */ *ap = v; struct vnode *vp, *dvp; struct componentname *cnp; struct inode *ip, *dp; int error; struct ufs_lookup_results *ulr; vp = ap->a_vp; dvp = ap->a_dvp; cnp = ap->a_cnp; ip = VTOI(vp); dp = VTOI(dvp); #ifdef UFS_ACL #ifdef notyet /* We don't do this because if the filesystem is mounted without ACLs * this goes through vfs_unixify_accmode() and we get EPERM. */ error = VOP_ACCESSX(vp, VDELETE, cnp->cn_cred); if (error) goto err; #endif #endif /* XXX should handle this material another way */ ulr = &dp->i_crap; UFS_CHECK_CRAPCOUNTER(dp); /* * No rmdir "." or of mounted directories please. */ if (dp == ip || vp->v_mountedhere != NULL) { error = EINVAL; goto err; } /* * Do not remove a directory that is in the process of being renamed. * Verify that the directory is empty (and valid). (Rmdir ".." won't * be valid since ".." will contain a reference to the current * directory and thus be non-empty.) */ error = 0; if (ip->i_nlink != 2 || !ufs_dirempty(ip, dp->i_number, cnp->cn_cred)) { error = ENOTEMPTY; goto out; } if ((dp->i_flags & APPEND) || (ip->i_flags & (IMMUTABLE | APPEND))) { error = EPERM; goto out; } error = UFS_WAPBL_BEGIN(dvp->v_mount); if (error) goto out; /* * Delete reference to directory before purging * inode. If we crash in between, the directory * will be reattached to lost+found, */ error = ufs_dirremove(dvp, ulr, ip, cnp->cn_flags, 1); if (error) { UFS_WAPBL_END(dvp->v_mount); goto out; } cache_purge(dvp); /* * Truncate inode. The only stuff left in the directory is "." and * "..". The "." reference is inconsequential since we're quashing * it. */ dp->i_nlink--; DIP_ASSIGN(dp, nlink, dp->i_nlink); dp->i_flag |= IN_CHANGE; UFS_WAPBL_UPDATE(dvp, NULL, NULL, UPDATE_DIROP); ip->i_nlink--; DIP_ASSIGN(ip, nlink, ip->i_nlink); ip->i_flag |= IN_CHANGE; (void) UFS_TRUNCATE(vp, (off_t)0, IO_SYNC, cnp->cn_cred); cache_purge(vp); /* * Unlock the log while we still have reference to unlinked * directory vp so that it will not get locked for recycling */ UFS_WAPBL_END(dvp->v_mount); #ifdef UFS_DIRHASH if (ip->i_dirhash != NULL) ufsdirhash_free(ip); #endif out: vput(vp); return error; err: if (dp == ip) vrele(vp); else vput(vp); return error; } /* * symlink -- make a symbolic link */ int ufs_symlink(void *v) { struct vop_symlink_v3_args /* { struct vnode *a_dvp; struct vnode **a_vpp; struct componentname *a_cnp; struct vattr *a_vap; char *a_target; } */ *ap = v; struct vnode *vp, **vpp; struct inode *ip; int len, error; struct ufs_lookup_results *ulr; vpp = ap->a_vpp; /* XXX should handle this material another way */ ulr = &VTOI(ap->a_dvp)->i_crap; UFS_CHECK_CRAPCOUNTER(VTOI(ap->a_dvp)); /* * UFS_WAPBL_BEGIN(dvp->v_mount) performed by successful * ufs_makeinode */ KASSERT(ap->a_vap->va_type == VLNK); error = ufs_makeinode(ap->a_vap, ap->a_dvp, ulr, vpp, ap->a_cnp); if (error) goto out; vp = *vpp; len = strlen(ap->a_target); ip = VTOI(vp); /* * This test is off by one. um_maxsymlinklen contains the * number of bytes available, and we aren't storing a \0, so * the test should properly be <=. However, it cannot be * changed as this would break compatibility with existing fs * images -- see the way ufs_readlink() works. */ if (len < ip->i_ump->um_maxsymlinklen) { memcpy((char *)SHORTLINK(ip), ap->a_target, len); ip->i_size = len; DIP_ASSIGN(ip, size, len); uvm_vnp_setsize(vp, ip->i_size); ip->i_flag |= IN_CHANGE | IN_UPDATE; if (vp->v_mount->mnt_flag & MNT_RELATIME) ip->i_flag |= IN_ACCESS; UFS_WAPBL_UPDATE(vp, NULL, NULL, 0); } else error = ufs_bufio(UIO_WRITE, vp, ap->a_target, len, (off_t)0, IO_NODELOCKED | IO_JOURNALLOCKED, ap->a_cnp->cn_cred, NULL, NULL); UFS_WAPBL_END(ap->a_dvp->v_mount); VOP_UNLOCK(vp); if (error) vrele(vp); out: return (error); } /* * Vnode op for reading directories. * * This routine handles converting from the on-disk directory format * "struct direct" to the in-memory format "struct dirent" as well as * byte swapping the entries if necessary. */ int ufs_readdir(void *v) { struct vop_readdir_args /* { struct vnode *a_vp; struct uio *a_uio; kauth_cred_t a_cred; int *a_eofflag; off_t **a_cookies; int *a_ncookies; } */ *ap = v; /* vnode and fs */ struct vnode *vp = ap->a_vp; struct ufsmount *ump = VFSTOUFS(vp->v_mount); int nswap = UFS_MPNEEDSWAP(ump); #if BYTE_ORDER == LITTLE_ENDIAN int needswap = ump->um_maxsymlinklen <= 0 && nswap == 0; #else int needswap = ump->um_maxsymlinklen <= 0 && nswap != 0; #endif /* caller's buffer */ struct uio *calleruio = ap->a_uio; off_t startoffset, endoffset; size_t callerbytes; off_t curoffset; /* dirent production buffer */ char *direntbuf; size_t direntbufmax; struct dirent *dirent, *stopdirent; /* output cookies array */ off_t *cookies; size_t numcookies, maxcookies; /* disk buffer */ off_t physstart, physend; size_t skipstart, dropend; char *rawbuf; size_t rawbufmax, rawbytes; struct uio rawuio; struct iovec rawiov; struct direct *rawdp, *stoprawdp; /* general */ int error; KASSERT(VOP_ISLOCKED(vp)); /* * Figure out where the user wants us to read and how much. * * XXX: there should probably be an upper bound on callerbytes * to avoid silliness trying to do large kernel allocations. */ callerbytes = calleruio->uio_resid; startoffset = calleruio->uio_offset; endoffset = startoffset + callerbytes; if (callerbytes < _DIRENT_MINSIZE(dirent)) { /* no room for even one struct dirent */ return EINVAL; } /* * Now figure out where to actually start reading. Round the * start down to a block boundary: we need to start at the * beginning of a block in order to read the directory * correctly. * * We also want to always read a whole number of blocks so * that the copying code below doesn't have to worry about * partial entries. (It used to try at one point, and was a * horrible mess.) * * Furthermore, since blocks have to be scanned from the * beginning, if we go partially into another block now we'll * just have to rescan it on the next readdir call, which * doesn't really serve any useful purpose. * * So, round down the end as well. It's ok to underpopulate * the transfer buffer, as long as we send back at least one * dirent so as to avoid giving a bogus EOF indication. * * Note that because dirents are larger than ffs struct * directs, despite the rounding down we may not be able to * send all the entries in the blocks we read and may have to * rescan some of them on the next call anyway. Alternatively * if there's empty space on disk we might have actually been * able to fit the next block in, and so forth. None of this * actually matters that much in practice. * * XXX: what does ffs do if a directory block becomes * completely empty, and what happens if all the blocks we * read are completely empty even though we aren't at EOF? As * of this writing I (dholland) can't remember the details. */ physstart = rounddown2(startoffset, ump->um_dirblksiz); physend = rounddown2(endoffset, ump->um_dirblksiz); if (physstart >= physend) { /* Need at least one block */ return EINVAL; } /* * skipstart is the number of bytes we need to read in * (because we need to start at the beginning of a block) but * not transfer to the user. * * dropend is the number of bytes to ignore at the end of the * user's buffer. */ skipstart = startoffset - physstart; dropend = endoffset - physend; /* * Make a transfer buffer. * * Note: rawbufmax = physend - physstart. Proof: * * physend - physstart = physend - physstart * = physend - physstart + startoffset - startoffset * = physend + (startoffset - physstart) - startoffset * = physend + skipstart - startoffset * = physend + skipstart - startoffset + endoffset - endoffset * = skipstart - startoffset + endoffset - (endoffset - physend) * = skipstart - startoffset + endoffset - dropend * = skipstart - startoffset + (startoffset + callerbytes) - dropend * = skipstart + callerbytes - dropend * = rawbufmax * Qed. * * XXX: this should just use physend - physstart. * * XXX: this should be rewritten to read the directs straight * out of bufferio buffers instead of copying twice. This would * also let us adapt better to the user's buffer size. */ /* Base buffer space for CALLERBYTES of new data */ rawbufmax = callerbytes + skipstart; if (rawbufmax < callerbytes) return EINVAL; rawbufmax -= dropend; if (rawbufmax < _DIRENT_MINSIZE(rawdp)) { /* no room for even one struct direct */ return EINVAL; } /* read it */ rawbuf = kmem_alloc(rawbufmax, KM_SLEEP); rawiov.iov_base = rawbuf; rawiov.iov_len = rawbufmax; rawuio.uio_iov = &rawiov; rawuio.uio_iovcnt = 1; rawuio.uio_offset = physstart; rawuio.uio_resid = rawbufmax; UIO_SETUP_SYSSPACE(&rawuio); rawuio.uio_rw = UIO_READ; error = UFS_BUFRD(vp, &rawuio, 0, ap->a_cred); if (error != 0) { kmem_free(rawbuf, rawbufmax); return error; } rawbytes = rawbufmax - rawuio.uio_resid; /* the raw entries to iterate over */ rawdp = (struct direct *)(void *)rawbuf; stoprawdp = (struct direct *)(void *)&rawbuf[rawbytes]; /* allocate space to produce dirents into */ direntbufmax = callerbytes; direntbuf = kmem_alloc(direntbufmax, KM_SLEEP); /* the dirents to iterate over */ dirent = (struct dirent *)(void *)direntbuf; stopdirent = (struct dirent *)(void *)&direntbuf[direntbufmax]; /* the output "cookies" (seek positions of directory entries) */ if (ap->a_cookies) { numcookies = 0; maxcookies = rawbytes / _DIRENT_RECLEN(rawdp, 1); cookies = malloc(maxcookies * sizeof(*cookies), M_TEMP, M_WAITOK); } else { /* XXX: GCC */ maxcookies = 0; cookies = NULL; } /* now produce the dirents */ curoffset = calleruio->uio_offset; while (rawdp < stoprawdp) { rawdp->d_reclen = ufs_rw16(rawdp->d_reclen, nswap); if (skipstart > 0) { /* drain skipstart */ if (rawdp->d_reclen <= skipstart) { skipstart -= rawdp->d_reclen; rawdp = _DIRENT_NEXT(rawdp); continue; } /* caller's start position wasn't on an entry */ error = EINVAL; goto out; } if (rawdp->d_reclen == 0) { struct dirent *save = dirent; dirent->d_reclen = _DIRENT_MINSIZE(dirent); dirent = _DIRENT_NEXT(dirent); save->d_reclen = 0; rawdp = stoprawdp; break; } /* copy the header */ if (needswap) { dirent->d_type = rawdp->d_namlen; dirent->d_namlen = rawdp->d_type; } else { dirent->d_type = rawdp->d_type; dirent->d_namlen = rawdp->d_namlen; } dirent->d_reclen = _DIRENT_RECLEN(dirent, dirent->d_namlen); /* stop if there isn't room for the name AND another header */ if ((char *)(void *)dirent + dirent->d_reclen + _DIRENT_MINSIZE(dirent) > (char *)(void *)stopdirent) break; /* copy the name (and inode (XXX: why after the test?)) */ dirent->d_fileno = ufs_rw32(rawdp->d_ino, nswap); (void)memcpy(dirent->d_name, rawdp->d_name, dirent->d_namlen); memset(&dirent->d_name[dirent->d_namlen], 0, dirent->d_reclen - _DIRENT_NAMEOFF(dirent) - dirent->d_namlen); /* onward */ curoffset += rawdp->d_reclen; if (ap->a_cookies) { KASSERT(numcookies < maxcookies); cookies[numcookies++] = curoffset; } dirent = _DIRENT_NEXT(dirent); rawdp = _DIRENT_NEXT(rawdp); } /* transfer the dirents to the caller's buffer */ callerbytes = ((char *)(void *)dirent - direntbuf); error = uiomove(direntbuf, callerbytes, calleruio); out: calleruio->uio_offset = curoffset; if (ap->a_cookies) { if (error) { free(cookies, M_TEMP); *ap->a_cookies = NULL; *ap->a_ncookies = 0; } else { *ap->a_cookies = cookies; *ap->a_ncookies = numcookies; } } kmem_free(direntbuf, direntbufmax); kmem_free(rawbuf, rawbufmax); *ap->a_eofflag = VTOI(vp)->i_size <= calleruio->uio_offset; return error; } /* * Return target name of a symbolic link */ int ufs_readlink(void *v) { struct vop_readlink_args /* { struct vnode *a_vp; struct uio *a_uio; kauth_cred_t a_cred; } */ *ap = v; struct vnode *vp = ap->a_vp; struct inode *ip = VTOI(vp); struct ufsmount *ump = VFSTOUFS(vp->v_mount); int isize; /* * The test against um_maxsymlinklen is off by one; it should * theoretically be <=, not <. However, it cannot be changed * as that would break compatibility with existing fs images. */ isize = ip->i_size; if (isize < ump->um_maxsymlinklen || (ump->um_maxsymlinklen == 0 && DIP(ip, blocks) == 0)) { uiomove((char *)SHORTLINK(ip), isize, ap->a_uio); return (0); } return (UFS_BUFRD(vp, ap->a_uio, 0, ap->a_cred)); } /* * Calculate the logical to physical mapping if not done already, * then call the device strategy routine. */ int ufs_strategy(void *v) { struct vop_strategy_args /* { struct vnode *a_vp; struct buf *a_bp; } */ *ap = v; struct buf *bp; struct vnode *vp; struct inode *ip; struct mount *mp; int error; bp = ap->a_bp; vp = ap->a_vp; ip = VTOI(vp); if (vp->v_type == VBLK || vp->v_type == VCHR) panic("ufs_strategy: spec"); KASSERT(fstrans_held(vp->v_mount)); KASSERT(bp->b_bcount != 0); if (bp->b_blkno == bp->b_lblkno) { error = VOP_BMAP(vp, bp->b_lblkno, NULL, &bp->b_blkno, NULL); if (error) { bp->b_error = error; biodone(bp); return (error); } if (bp->b_blkno == -1) /* no valid data */ clrbuf(bp); } if (bp->b_blkno < 0) { /* block is not on disk */ biodone(bp); return (0); } vp = ip->i_devvp; error = VOP_STRATEGY(vp, bp); if (error) return error; if (!BUF_ISREAD(bp)) return 0; mp = wapbl_vptomp(vp); if (mp == NULL || mp->mnt_wapbl_replay == NULL || !WAPBL_REPLAY_ISOPEN(mp) || !WAPBL_REPLAY_CAN_READ(mp, bp->b_blkno, bp->b_bcount)) return 0; error = biowait(bp); if (error) return error; error = WAPBL_REPLAY_READ(mp, bp->b_data, bp->b_blkno, bp->b_bcount); if (error) { mutex_enter(&bufcache_lock); SET(bp->b_cflags, BC_INVAL); mutex_exit(&bufcache_lock); } return error; } /* * Print out the contents of an inode. */ int ufs_print(void *v) { struct vop_print_args /* { struct vnode *a_vp; } */ *ap = v; struct vnode *vp; struct inode *ip; vp = ap->a_vp; ip = VTOI(vp); printf("tag VT_UFS, ino %llu, on dev %llu, %llu", (unsigned long long)ip->i_number, (unsigned long long)major(ip->i_dev), (unsigned long long)minor(ip->i_dev)); printf(" flags 0x%x, nlink %d\n", ip->i_flag, ip->i_nlink); printf("\tmode 0%o, owner %d, group %d, size %qd", ip->i_mode, ip->i_uid, ip->i_gid, (long long)ip->i_size); if (vp->v_type == VFIFO) VOCALL(fifo_vnodeop_p, VOFFSET(vop_print), v); printf("\n"); return (0); } /* * Read wrapper for special devices. */ int ufsspec_read(void *v) { struct vop_read_args /* { struct vnode *a_vp; struct uio *a_uio; int a_ioflag; kauth_cred_t a_cred; } */ *ap = v; /* * Set access flag. */ if ((ap->a_vp->v_mount->mnt_flag & MNT_NODEVMTIME) == 0) VTOI(ap->a_vp)->i_flag |= IN_ACCESS; return (VOCALL (spec_vnodeop_p, VOFFSET(vop_read), ap)); } /* * Write wrapper for special devices. */ int ufsspec_write(void *v) { struct vop_write_args /* { struct vnode *a_vp; struct uio *a_uio; int a_ioflag; kauth_cred_t a_cred; } */ *ap = v; /* * Set update and change flags. */ if ((ap->a_vp->v_mount->mnt_flag & MNT_NODEVMTIME) == 0) VTOI(ap->a_vp)->i_flag |= IN_MODIFY; return (VOCALL (spec_vnodeop_p, VOFFSET(vop_write), ap)); } /* * Close wrapper for special devices. * * Update the times on the inode then do device close. */ int ufsspec_close(void *v) { struct vop_close_args /* { struct vnode *a_vp; int a_fflag; kauth_cred_t a_cred; } */ *ap = v; struct vnode *vp; vp = ap->a_vp; if (vrefcnt(vp) > 1) UFS_ITIMES(vp, NULL, NULL, NULL); return (VOCALL (spec_vnodeop_p, VOFFSET(vop_close), ap)); } /* * Read wrapper for fifo's */ int ufsfifo_read(void *v) { struct vop_read_args /* { struct vnode *a_vp; struct uio *a_uio; int a_ioflag; kauth_cred_t a_cred; } */ *ap = v; /* * Set access flag. */ VTOI(ap->a_vp)->i_flag |= IN_ACCESS; return (VOCALL (fifo_vnodeop_p, VOFFSET(vop_read), ap)); } /* * Write wrapper for fifo's. */ int ufsfifo_write(void *v) { struct vop_write_args /* { struct vnode *a_vp; struct uio *a_uio; int a_ioflag; kauth_cred_t a_cred; } */ *ap = v; /* * Set update and change flags. */ VTOI(ap->a_vp)->i_flag |= IN_MODIFY; return (VOCALL (fifo_vnodeop_p, VOFFSET(vop_write), ap)); } /* * Close wrapper for fifo's. * * Update the times on the inode then do device close. */ int ufsfifo_close(void *v) { struct vop_close_args /* { struct vnode *a_vp; int a_fflag; kauth_cred_t a_cred; } */ *ap = v; struct vnode *vp; vp = ap->a_vp; if (vrefcnt(ap->a_vp) > 1) UFS_ITIMES(vp, NULL, NULL, NULL); return (VOCALL (fifo_vnodeop_p, VOFFSET(vop_close), ap)); } /* * Return POSIX pathconf information applicable to ufs filesystems. */ int ufs_pathconf(void *v) { struct vop_pathconf_args /* { struct vnode *a_vp; int a_name; register_t *a_retval; } */ *ap = v; switch (ap->a_name) { case _PC_LINK_MAX: *ap->a_retval = LINK_MAX; return (0); case _PC_NAME_MAX: *ap->a_retval = FFS_MAXNAMLEN; return (0); case _PC_PATH_MAX: *ap->a_retval = PATH_MAX; return (0); case _PC_PIPE_BUF: *ap->a_retval = PIPE_BUF; return (0); case _PC_CHOWN_RESTRICTED: *ap->a_retval = 1; return (0); case _PC_NO_TRUNC: *ap->a_retval = 1; return (0); #ifdef UFS_ACL case _PC_ACL_EXTENDED: if (ap->a_vp->v_mount->mnt_flag & MNT_POSIX1EACLS) *ap->a_retval = 1; else *ap->a_retval = 0; return 0; case _PC_ACL_NFS4: if (ap->a_vp->v_mount->mnt_flag & MNT_NFS4ACLS) *ap->a_retval = 1; else *ap->a_retval = 0; return 0; #endif case _PC_ACL_PATH_MAX: #ifdef UFS_ACL if (ap->a_vp->v_mount->mnt_flag & (MNT_POSIX1EACLS | MNT_NFS4ACLS)) *ap->a_retval = ACL_MAX_ENTRIES; else *ap->a_retval = 3; #else *ap->a_retval = 3; #endif return 0; case _PC_SYNC_IO: *ap->a_retval = 1; return (0); case _PC_FILESIZEBITS: *ap->a_retval = 42; return (0); case _PC_SYMLINK_MAX: *ap->a_retval = MAXPATHLEN; return (0); case _PC_2_SYMLINKS: *ap->a_retval = 1; return (0); default: return (EINVAL); } /* NOTREACHED */ } /* * Advisory record locking support */ int ufs_advlock(void *v) { struct vop_advlock_args /* { struct vnode *a_vp; void * a_id; int a_op; struct flock *a_fl; int a_flags; } */ *ap = v; struct inode *ip; ip = VTOI(ap->a_vp); return lf_advlock(ap, &ip->i_lockf, ip->i_size); } /* * Initialize the vnode associated with a new inode, handle aliased * vnodes. */ void ufs_vinit(struct mount *mntp, int (**specops)(void *), int (**fifoops)(void *), struct vnode **vpp) { struct timeval tv; struct inode *ip; struct vnode *vp; dev_t rdev; struct ufsmount *ump; vp = *vpp; ip = VTOI(vp); switch(vp->v_type = IFTOVT(ip->i_mode)) { case VCHR: case VBLK: vp->v_op = specops; ump = ip->i_ump; if (ump->um_fstype == UFS1) rdev = (dev_t)ufs_rw32(ip->i_ffs1_rdev, UFS_MPNEEDSWAP(ump)); else rdev = (dev_t)ufs_rw64(ip->i_ffs2_rdev, UFS_MPNEEDSWAP(ump)); spec_node_init(vp, rdev); break; case VFIFO: vp->v_op = fifoops; break; case VNON: case VBAD: case VSOCK: case VLNK: case VDIR: case VREG: break; } if (ip->i_number == UFS_ROOTINO) vp->v_vflag |= VV_ROOT; /* * Initialize modrev times */ getmicrouptime(&tv); ip->i_modrev = (uint64_t)(uint)tv.tv_sec << 32 | tv.tv_usec * 4294u; *vpp = vp; } /* * Allocate a new inode. */ static int ufs_makeinode(struct vattr *vap, struct vnode *dvp, const struct ufs_lookup_results *ulr, struct vnode **vpp, struct componentname *cnp) { struct inode *ip; struct direct *newdir; struct vnode *tvp; int error; UFS_WAPBL_JUNLOCK_ASSERT(dvp->v_mount); error = vcache_new(dvp->v_mount, dvp, vap, cnp->cn_cred, NULL, &tvp); if (error) return error; error = vn_lock(tvp, LK_EXCLUSIVE); if (error) { vrele(tvp); return error; } *vpp = tvp; ip = VTOI(tvp); error = UFS_WAPBL_BEGIN(dvp->v_mount); if (error) { vput(tvp); return (error); } ip->i_flag |= IN_ACCESS | IN_CHANGE | IN_UPDATE; ip->i_nlink = 1; DIP_ASSIGN(ip, nlink, 1); /* Authorize setting SGID if needed. */ if (ip->i_mode & ISGID) { error = kauth_authorize_vnode(cnp->cn_cred, KAUTH_VNODE_WRITE_SECURITY, tvp, NULL, genfs_can_chmod(tvp, cnp->cn_cred, ip->i_uid, ip->i_gid, MAKEIMODE(vap->va_type, vap->va_mode))); if (error) { ip->i_mode &= ~ISGID; DIP_ASSIGN(ip, mode, ip->i_mode); } } if (cnp->cn_flags & ISWHITEOUT) { ip->i_flags |= UF_OPAQUE; DIP_ASSIGN(ip, flags, ip->i_flags); } /* * Make sure inode goes to disk before directory entry. */ if ((error = UFS_UPDATE(tvp, NULL, NULL, UPDATE_DIROP)) != 0) goto bad; #ifdef UFS_ACL struct lwp *l = curlwp; if (dvp->v_mount->mnt_flag & MNT_POSIX1EACLS) { error = ufs_do_posix1e_acl_inheritance_file(dvp, tvp, ip->i_mode, cnp->cn_cred, l); if (error) goto bad; } else if (dvp->v_mount->mnt_flag & MNT_NFS4ACLS) { error = ufs_do_nfs4_acl_inheritance(dvp, tvp, ip->i_mode, cnp->cn_cred, l); if (error) goto bad; } #endif /* !UFS_ACL */ newdir = pool_cache_get(ufs_direct_cache, PR_WAITOK); ufs_makedirentry(ip, cnp, newdir); error = ufs_direnter(dvp, ulr, tvp, newdir, cnp, NULL); pool_cache_put(ufs_direct_cache, newdir); if (error) goto bad; *vpp = tvp; cache_enter(dvp, *vpp, cnp->cn_nameptr, cnp->cn_namelen, cnp->cn_flags); return (0); bad: /* * Write error occurred trying to update the inode * or the directory so must deallocate the inode. */ ip->i_nlink = 0; DIP_ASSIGN(ip, nlink, 0); ip->i_flag |= IN_CHANGE; UFS_WAPBL_UPDATE(tvp, NULL, NULL, 0); UFS_WAPBL_END(dvp->v_mount); vput(tvp); return (error); } /* * Allocate len bytes at offset off. */ int ufs_gop_alloc(struct vnode *vp, off_t off, off_t len, int flags, kauth_cred_t cred) { struct inode *ip = VTOI(vp); int error, delta, bshift, bsize; UVMHIST_FUNC("ufs_gop_alloc"); UVMHIST_CALLED(ubchist); error = 0; bshift = vp->v_mount->mnt_fs_bshift; bsize = 1 << bshift; delta = off & (bsize - 1); off -= delta; len += delta; while (len > 0) { bsize = MIN(bsize, len); error = UFS_BALLOC(vp, off, bsize, cred, flags, NULL); if (error) { goto out; } /* * increase file size now, UFS_BALLOC() requires that * EOF be up-to-date before each call. */ if (ip->i_size < off + bsize) { UVMHIST_LOG(ubchist, "vp %#jx old 0x%jx new 0x%x", (uintptr_t)vp, ip->i_size, off + bsize, 0); ip->i_size = off + bsize; DIP_ASSIGN(ip, size, ip->i_size); } off += bsize; len -= bsize; } out: UFS_WAPBL_UPDATE(vp, NULL, NULL, 0); return error; } void ufs_gop_markupdate(struct vnode *vp, int flags) { u_int32_t mask = 0; if ((flags & GOP_UPDATE_ACCESSED) != 0) { mask = IN_ACCESS; } if ((flags & GOP_UPDATE_MODIFIED) != 0) { if (vp->v_type == VREG) { mask |= IN_CHANGE | IN_UPDATE; } else { mask |= IN_MODIFY; } } if (mask) { struct inode *ip = VTOI(vp); ip->i_flag |= mask; } } int ufs_bufio(enum uio_rw rw, struct vnode *vp, void *buf, size_t len, off_t off, int ioflg, kauth_cred_t cred, size_t *aresid, struct lwp *l) { struct iovec iov; struct uio uio; int error; KASSERT(ISSET(ioflg, IO_NODELOCKED)); KASSERT(VOP_ISLOCKED(vp)); KASSERT(rw != UIO_WRITE || VOP_ISLOCKED(vp) == LK_EXCLUSIVE); KASSERT(rw != UIO_WRITE || vp->v_mount->mnt_wapbl == NULL || ISSET(ioflg, IO_JOURNALLOCKED)); iov.iov_base = buf; iov.iov_len = len; uio.uio_iov = &iov; uio.uio_iovcnt = 1; uio.uio_resid = len; uio.uio_offset = off; uio.uio_rw = rw; UIO_SETUP_SYSSPACE(&uio); switch (rw) { case UIO_READ: error = UFS_BUFRD(vp, &uio, ioflg, cred); break; case UIO_WRITE: error = UFS_BUFWR(vp, &uio, ioflg, cred); break; default: panic("invalid uio rw: %d", (int)rw); } if (aresid) *aresid = uio.uio_resid; else if (uio.uio_resid && error == 0) error = EIO; KASSERT(VOP_ISLOCKED(vp)); KASSERT(rw != UIO_WRITE || VOP_ISLOCKED(vp) == LK_EXCLUSIVE); return error; } |
| 410 962 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 197 198 199 200 201 202 203 204 205 206 207 208 209 210 211 212 213 214 215 216 217 218 219 220 221 222 223 224 225 226 227 228 229 230 231 232 233 234 235 236 237 238 239 240 241 242 243 244 245 246 247 248 249 250 251 252 253 254 255 256 257 258 259 260 261 262 263 264 265 266 267 268 269 270 271 272 273 274 275 276 277 278 279 280 281 282 283 284 285 286 287 288 289 290 291 292 293 294 295 296 297 298 299 300 301 302 303 304 305 306 307 308 309 310 311 312 313 314 315 316 317 318 319 320 321 322 323 324 325 326 327 328 329 330 331 332 333 334 335 336 337 338 339 340 341 342 343 344 345 346 347 348 349 350 351 352 353 354 355 356 357 358 359 360 361 362 363 364 365 366 367 368 369 370 371 372 373 374 375 376 377 378 379 380 | /* $NetBSD: pmap_private.h,v 1.2 2022/08/20 23:49:31 riastradh Exp $ */ /* * Copyright (c) 1997 Charles D. Cranor and Washington University. * All rights reserved. * * Redistribution and use in source and binary forms, with or without * modification, are permitted provided that the following conditions * are met: * 1. Redistributions of source code must retain the above copyright * notice, this list of conditions and the following disclaimer. * 2. Redistributions in binary form must reproduce the above copyright * notice, this list of conditions and the following disclaimer in the * documentation and/or other materials provided with the distribution. * * THIS SOFTWARE IS PROVIDED BY THE AUTHOR ``AS IS'' AND ANY EXPRESS OR * IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES * OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. * IN NO EVENT SHALL THE AUTHOR BE LIABLE FOR ANY DIRECT, INDIRECT, * INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT * NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, * DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY * THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT * (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF * THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. */ /* * Copyright (c) 2001 Wasabi Systems, Inc. * All rights reserved. * * Written by Frank van der Linden for Wasabi Systems, Inc. * * Redistribution and use in source and binary forms, with or without * modification, are permitted provided that the following conditions * are met: * 1. Redistributions of source code must retain the above copyright * notice, this list of conditions and the following disclaimer. * 2. Redistributions in binary form must reproduce the above copyright * notice, this list of conditions and the following disclaimer in the * documentation and/or other materials provided with the distribution. * 3. All advertising materials mentioning features or use of this software * must display the following acknowledgement: * This product includes software developed for the NetBSD Project by * Wasabi Systems, Inc. * 4. The name of Wasabi Systems, Inc. may not be used to endorse * or promote products derived from this software without specific prior * written permission. * * THIS SOFTWARE IS PROVIDED BY WASABI SYSTEMS, INC. ``AS IS'' AND * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED * TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR * PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL WASABI SYSTEMS, INC * BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR * CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF * SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS * INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN * CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) * ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE * POSSIBILITY OF SUCH DAMAGE. */ #ifndef _X86_PMAP_PRIVATE_H_ #define _X86_PMAP_PRIVATE_H_ #ifndef _MACHINE_PMAP_PRIVATE_H_X86 #error Include machine/pmap_private.h, not x86/pmap_private.h. #endif #ifdef _KERNEL_OPT #include "opt_svs.h" #endif #include <sys/param.h> #include <sys/types.h> #include <sys/kcpuset.h> #include <sys/mutex.h> #include <sys/pool.h> #include <sys/queue.h> #include <sys/rwlock.h> #include <machine/pte.h> #include <machine/vmparam.h> #include <uvm/uvm_object.h> #include <uvm/uvm_pmap.h> struct pmap; #define SLAREA_USER 0 #define SLAREA_PTE 1 #define SLAREA_MAIN 2 #define SLAREA_PCPU 3 #define SLAREA_DMAP 4 #define SLAREA_HYPV 5 #define SLAREA_ASAN 6 #define SLAREA_MSAN 7 #define SLAREA_KERN 8 #define SLSPACE_NAREAS 9 struct slotspace { struct { size_t sslot; /* start slot */ size_t nslot; /* # of slots */ bool active; /* area is active */ } area[SLSPACE_NAREAS]; }; extern struct slotspace slotspace; #include <x86/gdt.h> struct pcpu_entry { uint8_t gdt[MAXGDTSIZ]; uint8_t ldt[MAX_USERLDT_SIZE]; uint8_t idt[PAGE_SIZE]; uint8_t tss[PAGE_SIZE]; uint8_t ist0[PAGE_SIZE]; uint8_t ist1[PAGE_SIZE]; uint8_t ist2[PAGE_SIZE]; uint8_t ist3[PAGE_SIZE]; uint8_t rsp0[2 * PAGE_SIZE]; } __packed; struct pcpu_area { #ifdef SVS uint8_t utls[PAGE_SIZE]; #endif uint8_t ldt[PAGE_SIZE]; struct pcpu_entry ent[MAXCPUS]; } __packed; extern struct pcpu_area *pcpuarea; #define PMAP_PCID_KERN 0 #define PMAP_PCID_USER 1 /* * pmap data structures: see pmap.c for details of locking. */ /* * we maintain a list of all non-kernel pmaps */ LIST_HEAD(pmap_head, pmap); /* struct pmap_head: head of a pmap list */ /* * linked list of all non-kernel pmaps */ extern struct pmap_head pmaps; extern kmutex_t pmaps_lock; /* protects pmaps */ /* * pool_cache(9) that pmaps are allocated from */ extern struct pool_cache pmap_cache; /* * the pmap structure * * note that the pm_obj contains the lock pointer, the reference count, * page list, and number of PTPs within the pmap. * * pm_lock is the same as the lock for vm object 0. Changes to * the other objects may only be made if that lock has been taken * (the other object locks are only used when uvm_pagealloc is called) */ struct pv_page; struct pmap { struct uvm_object pm_obj[PTP_LEVELS-1];/* objects for lvl >= 1) */ LIST_ENTRY(pmap) pm_list; /* list of all pmaps */ pd_entry_t *pm_pdir; /* VA of PD */ paddr_t pm_pdirpa[PDP_SIZE]; /* PA of PDs (read-only after create) */ struct vm_page *pm_ptphint[PTP_LEVELS-1]; /* pointer to a PTP in our pmap */ struct pmap_statistics pm_stats; /* pmap stats */ struct pv_entry *pm_pve; /* spare pv_entry */ LIST_HEAD(, pv_page) pm_pvp_part; LIST_HEAD(, pv_page) pm_pvp_empty; LIST_HEAD(, pv_page) pm_pvp_full; #if !defined(__x86_64__) vaddr_t pm_hiexec; /* highest executable mapping */ #endif /* !defined(__x86_64__) */ union descriptor *pm_ldt; /* user-set LDT */ size_t pm_ldt_len; /* XXX unused, remove */ int pm_ldt_sel; /* LDT selector */ kcpuset_t *pm_cpus; /* mask of CPUs using pmap */ kcpuset_t *pm_kernel_cpus; /* mask of CPUs using kernel part of pmap */ kcpuset_t *pm_xen_ptp_cpus; /* mask of CPUs which have this pmap's ptp mapped */ uint64_t pm_ncsw; /* for assertions */ LIST_HEAD(,vm_page) pm_gc_ptp; /* PTPs queued for free */ /* Used by NVMM and Xen */ int (*pm_enter)(struct pmap *, vaddr_t, paddr_t, vm_prot_t, u_int); bool (*pm_extract)(struct pmap *, vaddr_t, paddr_t *); void (*pm_remove)(struct pmap *, vaddr_t, vaddr_t); int (*pm_sync_pv)(struct vm_page *, vaddr_t, paddr_t, int, uint8_t *, pt_entry_t *); void (*pm_pp_remove_ent)(struct pmap *, struct vm_page *, pt_entry_t, vaddr_t); void (*pm_write_protect)(struct pmap *, vaddr_t, vaddr_t, vm_prot_t); void (*pm_unwire)(struct pmap *, vaddr_t); void (*pm_tlb_flush)(struct pmap *); void *pm_data; kmutex_t pm_lock /* locks for pm_objs */ __aligned(64); /* give lock own cache line */ krwlock_t pm_dummy_lock; /* ugly hack for abusing uvm_object */ }; /* macro to access pm_pdirpa slots */ #ifdef PAE #define pmap_pdirpa(pmap, index) \ ((pmap)->pm_pdirpa[l2tol3(index)] + l2tol2(index) * sizeof(pd_entry_t)) #else #define pmap_pdirpa(pmap, index) \ ((pmap)->pm_pdirpa[0] + (index) * sizeof(pd_entry_t)) #endif /* * global kernel variables */ /* * PDPpaddr is the physical address of the kernel's PDP. * - i386 non-PAE and amd64: PDPpaddr corresponds directly to the %cr3 * value associated to the kernel process, proc0. * - i386 PAE: it still represents the PA of the kernel's PDP (L2). Due to * the L3 PD, it cannot be considered as the equivalent of a %cr3 any more. * - Xen: it corresponds to the PFN of the kernel's PDP. */ extern u_long PDPpaddr; extern pd_entry_t pmap_pg_g; /* do we support PTE_G? */ extern pd_entry_t pmap_pg_nx; /* do we support PTE_NX? */ extern int pmap_largepages; extern long nkptp[PTP_LEVELS]; #define pmap_valid_entry(E) ((E) & PTE_P) /* is PDE or PTE valid? */ void pmap_map_ptes(struct pmap *, struct pmap **, pd_entry_t **, pd_entry_t * const **); void pmap_unmap_ptes(struct pmap *, struct pmap *); bool pmap_pdes_valid(vaddr_t, pd_entry_t * const *, pd_entry_t *, int *lastlvl); bool pmap_is_curpmap(struct pmap *); void pmap_ept_transform(struct pmap *); #ifndef __HAVE_DIRECT_MAP void pmap_vpage_cpu_init(struct cpu_info *); #endif vaddr_t slotspace_rand(int, size_t, size_t, size_t, vaddr_t); vaddr_t reserve_dumppages(vaddr_t); /* XXX: not a pmap fn */ typedef enum tlbwhy { TLBSHOOT_REMOVE_ALL, TLBSHOOT_KENTER, TLBSHOOT_KREMOVE, TLBSHOOT_FREE_PTP, TLBSHOOT_REMOVE_PTE, TLBSHOOT_SYNC_PV, TLBSHOOT_WRITE_PROTECT, TLBSHOOT_ENTER, TLBSHOOT_NVMM, TLBSHOOT_BUS_DMA, TLBSHOOT_BUS_SPACE, TLBSHOOT__MAX, } tlbwhy_t; void pmap_tlb_init(void); void pmap_tlb_cpu_init(struct cpu_info *); void pmap_tlb_shootdown(pmap_t, vaddr_t, pt_entry_t, tlbwhy_t); void pmap_tlb_shootnow(void); void pmap_tlb_intr(void); /* * inline functions */ /* * pmap_update_pg: flush one page from the TLB (or flush the whole thing * if hardware doesn't support one-page flushing) */ __inline static void __unused pmap_update_pg(vaddr_t va) { invlpg(va); } /* * various address inlines * * vtopte: return a pointer to the PTE mapping a VA, works only for * user and PT addresses * * kvtopte: return a pointer to the PTE mapping a kernel VA */ #include <lib/libkern/libkern.h> static __inline pt_entry_t * __unused vtopte(vaddr_t va) { KASSERT(va < VM_MIN_KERNEL_ADDRESS); return (PTE_BASE + pl1_i(va)); } static __inline pt_entry_t * __unused kvtopte(vaddr_t va) { pd_entry_t *pde; KASSERT(va >= VM_MIN_KERNEL_ADDRESS); pde = L2_BASE + pl2_i(va); if (*pde & PTE_PS) return ((pt_entry_t *)pde); return (PTE_BASE + pl1_i(va)); } #ifdef XENPV #include <sys/bitops.h> #define XPTE_MASK L1_FRAME /* Selects the index of a PTE in (A)PTE_BASE */ #define XPTE_SHIFT (L1_SHIFT - ilog2(sizeof(pt_entry_t))) /* PTE access inline functions */ /* * Get the machine address of the pointed pte * We use hardware MMU to get value so works only for levels 1-3 */ static __inline paddr_t xpmap_ptetomach(pt_entry_t *pte) { pt_entry_t *up_pte; vaddr_t va = (vaddr_t) pte; va = ((va & XPTE_MASK) >> XPTE_SHIFT) | (vaddr_t) PTE_BASE; up_pte = (pt_entry_t *) va; return (paddr_t) (((*up_pte) & PTE_FRAME) + (((vaddr_t) pte) & (~PTE_FRAME & ~VA_SIGN_MASK))); } /* Xen helpers to change bits of a pte */ #define XPMAP_UPDATE_DIRECT 1 /* Update direct map entry flags too */ paddr_t vtomach(vaddr_t); #define vtomfn(va) (vtomach(va) >> PAGE_SHIFT) #endif /* XENPV */ #ifdef __HAVE_PCPU_AREA extern struct pcpu_area *pcpuarea; #define PDIR_SLOT_PCPU 510 #define PMAP_PCPU_BASE (VA_SIGN_NEG((PDIR_SLOT_PCPU * NBPD_L4))) #endif void svs_quad_copy(void *, void *, long); #endif /* _X86_PMAP_PRIVATE_H_ */ |
| 8 226 256 252 5 8 222 223 218 6 6 218 8 8 8 8 8 8 6 6 62 62 25 24 25 25 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 197 198 199 200 201 202 203 204 205 206 207 208 209 210 211 212 213 214 215 216 217 218 219 220 221 222 223 224 225 226 227 228 229 230 231 232 233 234 235 236 237 238 239 240 241 242 243 244 245 246 247 248 249 250 251 252 253 254 255 256 257 258 259 260 261 262 263 264 265 266 267 268 269 270 271 272 273 274 275 276 277 278 279 280 281 282 283 284 285 286 287 288 289 290 291 292 293 294 295 296 297 298 299 300 301 302 303 304 305 306 307 308 309 310 311 312 313 314 315 316 317 318 319 320 321 322 323 324 325 326 327 328 329 330 331 332 333 334 335 336 337 338 339 340 341 342 343 344 345 346 347 348 349 350 351 352 353 354 355 356 357 358 359 360 361 362 363 364 365 366 367 368 369 370 371 372 373 374 375 376 377 378 379 380 381 382 383 384 385 386 387 388 389 390 391 392 393 394 395 396 397 398 399 400 401 402 403 404 405 406 407 408 409 410 411 412 413 414 415 416 417 418 419 420 421 422 423 424 425 426 427 428 429 430 431 432 433 434 435 436 437 438 439 440 441 442 443 444 445 446 447 448 449 450 451 452 453 454 455 456 457 458 459 460 461 462 463 464 465 466 467 468 469 470 471 472 473 474 475 476 477 478 479 480 481 482 483 484 485 486 487 488 489 490 491 492 493 494 495 496 497 498 499 500 501 502 503 504 505 506 507 508 509 510 511 512 513 514 515 516 517 518 519 520 521 522 523 524 525 526 527 528 529 530 531 532 533 534 535 536 537 538 539 540 541 542 543 544 545 546 547 548 549 550 551 552 553 554 555 556 557 558 559 560 561 562 563 564 565 566 567 568 569 570 571 572 573 574 575 576 577 578 579 580 581 582 583 584 585 586 587 588 589 590 591 592 593 594 595 596 597 598 599 600 601 602 603 604 605 606 607 608 609 610 611 612 613 614 615 616 617 618 619 620 621 622 623 624 625 626 627 628 629 630 631 632 633 634 635 636 637 638 639 640 641 642 643 644 645 646 647 648 649 650 651 652 653 654 655 656 657 658 659 660 661 662 663 664 665 666 667 668 669 670 671 672 673 674 675 676 677 678 679 680 681 682 683 684 685 686 687 688 689 690 691 692 693 694 695 696 697 698 699 700 701 702 703 704 705 706 707 708 709 710 711 712 713 714 715 716 717 718 719 720 721 722 723 724 725 726 727 728 729 730 731 732 733 734 735 736 737 738 739 740 741 742 743 744 745 746 747 748 749 750 751 752 753 754 755 756 757 758 759 760 761 762 763 764 765 766 767 768 769 770 771 772 773 774 775 776 777 778 779 780 781 782 783 784 785 786 787 788 789 790 791 792 793 794 795 796 797 798 799 800 801 802 803 804 805 806 807 808 809 810 811 812 813 814 815 816 817 818 819 820 821 822 823 824 825 826 827 828 829 830 | /* $NetBSD: kern_pax.c,v 1.62 2021/08/30 01:25:10 rin Exp $ */ /* * Copyright (c) 2015, 2020 The NetBSD Foundation, Inc. * All rights reserved. * * This code is derived from software contributed to The NetBSD Foundation * by Maxime Villard. * * Redistribution and use in source and binary forms, with or without * modification, are permitted provided that the following conditions * are met: * 1. Redistributions of source code must retain the above copyright * notice, this list of conditions and the following disclaimer. * 2. Redistributions in binary form must reproduce the above copyright * notice, this list of conditions and the following disclaimer in the * documentation and/or other materials provided with the distribution. * * THIS SOFTWARE IS PROVIDED BY THE NETBSD FOUNDATION, INC. AND CONTRIBUTORS * ``AS IS'' AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED * TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR * PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE FOUNDATION OR CONTRIBUTORS * BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR * CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF * SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS * INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN * CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) * ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE * POSSIBILITY OF SUCH DAMAGE. */ /* * Copyright (c) 2006 Elad Efrat <elad@NetBSD.org> * All rights reserved. * * Redistribution and use in source and binary forms, with or without * modification, are permitted provided that the following conditions * are met: * 1. Redistributions of source code must retain the above copyright * notice, this list of conditions and the following disclaimer. * 2. Redistributions in binary form must reproduce the above copyright * notice, this list of conditions and the following disclaimer in the * documentation and/or other materials provided with the distribution. * 3. The name of the author may not be used to endorse or promote products * derived from this software without specific prior written permission. * * THIS SOFTWARE IS PROVIDED BY THE AUTHOR ``AS IS'' AND ANY EXPRESS OR * IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES * OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. * IN NO EVENT SHALL THE AUTHOR BE LIABLE FOR ANY DIRECT, INDIRECT, * INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT * NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, * DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY * THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT * (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF * THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. */ #include <sys/cdefs.h> __KERNEL_RCSID(0, "$NetBSD: kern_pax.c,v 1.62 2021/08/30 01:25:10 rin Exp $"); #include "opt_pax.h" #include <sys/param.h> #include <sys/proc.h> #include <sys/exec.h> #include <sys/exec_elf.h> #include <sys/pax.h> #include <sys/sysctl.h> #include <sys/kmem.h> #include <sys/mman.h> #include <sys/syslog.h> #include <sys/vnode.h> #include <sys/queue.h> #include <sys/bitops.h> #include <sys/kauth.h> #include <sys/cprng.h> #ifdef PAX_ASLR_DEBUG #define PAX_DPRINTF(_fmt, args...) \ do if (pax_aslr_debug) uprintf("%s: " _fmt "\n", __func__, ##args); \ while (/*CONSTCOND*/0) #else #define PAX_DPRINTF(_fmt, args...) do {} while (/*CONSTCOND*/0) #endif #ifdef PAX_ASLR #include <sys/mman.h> int pax_aslr_enabled = 1; int pax_aslr_global = PAX_ASLR; #ifndef PAX_ASLR_DELTA_MMAP_LSB #define PAX_ASLR_DELTA_MMAP_LSB PGSHIFT #endif #ifndef PAX_ASLR_DELTA_MMAP_LEN #define PAX_ASLR_DELTA_MMAP_LEN ((sizeof(void *) * NBBY) / 2) #endif #ifndef PAX_ASLR_DELTA_MMAP_LEN32 #define PAX_ASLR_DELTA_MMAP_LEN32 ((sizeof(uint32_t) * NBBY) / 2) #endif #ifndef PAX_ASLR_DELTA_STACK_LSB #define PAX_ASLR_DELTA_STACK_LSB PGSHIFT #endif #ifndef PAX_ASLR_DELTA_STACK_LEN #define PAX_ASLR_DELTA_STACK_LEN ((sizeof(void *) * NBBY) / 4) #endif #ifndef PAX_ASLR_DELTA_STACK_LEN32 #define PAX_ASLR_DELTA_STACK_LEN32 ((sizeof(uint32_t) * NBBY) / 4) #endif #define PAX_ASLR_MAX_STACK_WASTE 8 #ifdef PAX_ASLR_DEBUG int pax_aslr_debug; /* flag set means disable */ int pax_aslr_flags; uint32_t pax_aslr_rand; #define PAX_ASLR_STACK 0x01 #define PAX_ASLR_STACK_GAP 0x02 #define PAX_ASLR_MMAP 0x04 #define PAX_ASLR_EXEC_OFFSET 0x08 #define PAX_ASLR_RTLD_OFFSET 0x10 #define PAX_ASLR_FIXED 0x20 #endif static bool pax_aslr_elf_flags_active(uint32_t); #endif /* PAX_ASLR */ #ifdef PAX_MPROTECT static int pax_mprotect_enabled = 1; static int pax_mprotect_global = PAX_MPROTECT; static int pax_mprotect_ptrace = 1; static bool pax_mprotect_elf_flags_active(uint32_t); #endif /* PAX_MPROTECT */ #ifdef PAX_MPROTECT_DEBUG int pax_mprotect_debug; #endif #ifdef PAX_SEGVGUARD #ifndef PAX_SEGVGUARD_EXPIRY #define PAX_SEGVGUARD_EXPIRY (2 * 60) #endif #ifndef PAX_SEGVGUARD_SUSPENSION #define PAX_SEGVGUARD_SUSPENSION (10 * 60) #endif #ifndef PAX_SEGVGUARD_MAXCRASHES #define PAX_SEGVGUARD_MAXCRASHES 5 #endif static int pax_segvguard_enabled = 1; static int pax_segvguard_global = PAX_SEGVGUARD; static int pax_segvguard_expiry = PAX_SEGVGUARD_EXPIRY; static int pax_segvguard_suspension = PAX_SEGVGUARD_SUSPENSION; static int pax_segvguard_maxcrashes = PAX_SEGVGUARD_MAXCRASHES; struct pax_segvguard_uid_entry { uid_t sue_uid; size_t sue_ncrashes; time_t sue_expiry; time_t sue_suspended; LIST_ENTRY(pax_segvguard_uid_entry) sue_list; }; struct pax_segvguard_entry { LIST_HEAD(, pax_segvguard_uid_entry) segv_uids; }; static bool pax_segvguard_elf_flags_active(uint32_t); #endif /* PAX_SEGVGUARD */ SYSCTL_SETUP(sysctl_security_pax_setup, "sysctl security.pax setup") { const struct sysctlnode *rnode = NULL, *cnode; sysctl_createv(clog, 0, NULL, &rnode, CTLFLAG_PERMANENT, CTLTYPE_NODE, "pax", SYSCTL_DESCR("PaX (exploit mitigation) features."), NULL, 0, NULL, 0, CTL_SECURITY, CTL_CREATE, CTL_EOL); cnode = rnode; #ifdef PAX_MPROTECT rnode = cnode; sysctl_createv(clog, 0, &rnode, &rnode, CTLFLAG_PERMANENT, CTLTYPE_NODE, "mprotect", SYSCTL_DESCR("mprotect(2) W^X restrictions."), NULL, 0, NULL, 0, CTL_CREATE, CTL_EOL); sysctl_createv(clog, 0, &rnode, NULL, CTLFLAG_PERMANENT|CTLFLAG_READWRITE, CTLTYPE_INT, "enabled", SYSCTL_DESCR("Restrictions enabled."), NULL, 0, &pax_mprotect_enabled, 0, CTL_CREATE, CTL_EOL); sysctl_createv(clog, 0, &rnode, NULL, CTLFLAG_PERMANENT|CTLFLAG_READWRITE, CTLTYPE_INT, "global", SYSCTL_DESCR("When enabled, unless explicitly " "specified, apply restrictions to " "all processes."), NULL, 0, &pax_mprotect_global, 0, CTL_CREATE, CTL_EOL); sysctl_createv(clog, 0, &rnode, NULL, CTLFLAG_PERMANENT|CTLFLAG_READWRITE, CTLTYPE_INT, "ptrace", SYSCTL_DESCR("When enabled, allow ptrace(2) to " "override mprotect permissions on traced " "processes"), NULL, 0, &pax_mprotect_ptrace, 0, CTL_CREATE, CTL_EOL); #ifdef PAX_MPROTECT_DEBUG sysctl_createv(clog, 0, &rnode, NULL, CTLFLAG_PERMANENT|CTLFLAG_READWRITE, CTLTYPE_INT, "debug", SYSCTL_DESCR("print mprotect changes."), NULL, 0, &pax_mprotect_debug, 0, CTL_CREATE, CTL_EOL); #endif #endif /* PAX_MPROTECT */ #ifdef PAX_SEGVGUARD rnode = cnode; sysctl_createv(clog, 0, &rnode, &rnode, CTLFLAG_PERMANENT, CTLTYPE_NODE, "segvguard", SYSCTL_DESCR("PaX segvguard."), NULL, 0, NULL, 0, CTL_CREATE, CTL_EOL); sysctl_createv(clog, 0, &rnode, NULL, CTLFLAG_PERMANENT|CTLFLAG_READWRITE, CTLTYPE_INT, "enabled", SYSCTL_DESCR("segvguard enabled."), NULL, 0, &pax_segvguard_enabled, 0, CTL_CREATE, CTL_EOL); sysctl_createv(clog, 0, &rnode, NULL, CTLFLAG_PERMANENT|CTLFLAG_READWRITE, CTLTYPE_INT, "global", SYSCTL_DESCR("segvguard all programs."), NULL, 0, &pax_segvguard_global, 0, CTL_CREATE, CTL_EOL); sysctl_createv(clog, 0, &rnode, NULL, CTLFLAG_PERMANENT|CTLFLAG_READWRITE, CTLTYPE_INT, "expiry_timeout", SYSCTL_DESCR("Entry expiry timeout (in seconds)."), NULL, 0, &pax_segvguard_expiry, 0, CTL_CREATE, CTL_EOL); sysctl_createv(clog, 0, &rnode, NULL, CTLFLAG_PERMANENT|CTLFLAG_READWRITE, CTLTYPE_INT, "suspend_timeout", SYSCTL_DESCR("Entry suspension timeout (in seconds)."), NULL, 0, &pax_segvguard_suspension, 0, CTL_CREATE, CTL_EOL); sysctl_createv(clog, 0, &rnode, NULL, CTLFLAG_PERMANENT|CTLFLAG_READWRITE, CTLTYPE_INT, "max_crashes", SYSCTL_DESCR("Max number of crashes before expiry."), NULL, 0, &pax_segvguard_maxcrashes, 0, CTL_CREATE, CTL_EOL); #endif /* PAX_SEGVGUARD */ #ifdef PAX_ASLR rnode = cnode; sysctl_createv(clog, 0, &rnode, &rnode, CTLFLAG_PERMANENT, CTLTYPE_NODE, "aslr", SYSCTL_DESCR("Address Space Layout Randomization."), NULL, 0, NULL, 0, CTL_CREATE, CTL_EOL); sysctl_createv(clog, 0, &rnode, NULL, CTLFLAG_PERMANENT|CTLFLAG_READWRITE, CTLTYPE_INT, "enabled", SYSCTL_DESCR("Restrictions enabled."), NULL, 0, &pax_aslr_enabled, 0, CTL_CREATE, CTL_EOL); sysctl_createv(clog, 0, &rnode, NULL, CTLFLAG_PERMANENT|CTLFLAG_READWRITE, CTLTYPE_INT, "global", SYSCTL_DESCR("When enabled, unless explicitly " "specified, apply to all processes."), NULL, 0, &pax_aslr_global, 0, CTL_CREATE, CTL_EOL); #ifdef PAX_ASLR_DEBUG sysctl_createv(clog, 0, &rnode, NULL, CTLFLAG_PERMANENT|CTLFLAG_READWRITE, CTLTYPE_INT, "debug", SYSCTL_DESCR("Print ASLR selected addresses."), NULL, 0, &pax_aslr_debug, 0, CTL_CREATE, CTL_EOL); sysctl_createv(clog, 0, &rnode, NULL, CTLFLAG_PERMANENT|CTLFLAG_READWRITE, CTLTYPE_INT, "flags", SYSCTL_DESCR("Disable/Enable select ASLR features."), NULL, 0, &pax_aslr_flags, 0, CTL_CREATE, CTL_EOL); sysctl_createv(clog, 0, &rnode, NULL, CTLFLAG_PERMANENT|CTLFLAG_READWRITE, CTLTYPE_INT, "rand", SYSCTL_DESCR("Use the given fixed random value"), NULL, 0, &pax_aslr_rand, 0, CTL_CREATE, CTL_EOL); #endif sysctl_createv(clog, 0, &rnode, NULL, CTLFLAG_PERMANENT|CTLFLAG_IMMEDIATE, CTLTYPE_INT, "mmap_len", SYSCTL_DESCR("Number of bits randomized for " "mmap(2) calls."), NULL, PAX_ASLR_DELTA_MMAP_LEN, NULL, 0, CTL_CREATE, CTL_EOL); sysctl_createv(clog, 0, &rnode, NULL, CTLFLAG_PERMANENT|CTLFLAG_IMMEDIATE, CTLTYPE_INT, "stack_len", SYSCTL_DESCR("Number of bits randomized for " "the stack."), NULL, PAX_ASLR_DELTA_STACK_LEN, NULL, 0, CTL_CREATE, CTL_EOL); sysctl_createv(clog, 0, &rnode, NULL, CTLFLAG_PERMANENT|CTLFLAG_IMMEDIATE, CTLTYPE_INT, "exec_len", SYSCTL_DESCR("Number of bits randomized for " "the PIE exec base."), NULL, PAX_ASLR_DELTA_EXEC_LEN, NULL, 0, CTL_CREATE, CTL_EOL); #endif /* PAX_ASLR */ } /* * Initialize PaX. */ void pax_init(void) { #ifdef PAX_ASLR /* Adjust maximum stack by the size we can consume for ASLR */ extern rlim_t maxsmap; maxsmap = MAXSSIZ - (MAXSSIZ / PAX_ASLR_MAX_STACK_WASTE); // XXX: compat32 is not handled. #endif } void pax_set_flags(struct exec_package *epp, struct proc *p) { p->p_pax = epp->ep_pax_flags; #ifdef PAX_MPROTECT if (pax_mprotect_ptrace == 0) return; /* * If we are running under the debugger, turn off MPROTECT so * the debugger can insert/delete breakpoints */ if (p->p_slflag & PSL_TRACED) p->p_pax &= ~P_PAX_MPROTECT; #endif } void pax_setup_elf_flags(struct exec_package *epp, uint32_t elf_flags) { uint32_t flags = 0; #ifdef PAX_ASLR if (pax_aslr_elf_flags_active(elf_flags)) { flags |= P_PAX_ASLR; } #endif #ifdef PAX_MPROTECT if (pax_mprotect_elf_flags_active(elf_flags)) { flags |= P_PAX_MPROTECT; } #endif #ifdef PAX_SEGVGUARD if (pax_segvguard_elf_flags_active(elf_flags)) { flags |= P_PAX_GUARD; } #endif epp->ep_pax_flags = flags; } #if defined(PAX_MPROTECT) || defined(PAX_SEGVGUARD) || defined(PAX_ASLR) static inline bool pax_flags_active(uint32_t flags, uint32_t opt) { if (!(flags & opt)) return false; return true; } #endif /* PAX_MPROTECT || PAX_SEGVGUARD || PAX_ASLR */ #ifdef PAX_MPROTECT static bool pax_mprotect_elf_flags_active(uint32_t flags) { if (!pax_mprotect_enabled) return false; if (pax_mprotect_global && (flags & ELF_NOTE_PAX_NOMPROTECT) != 0) { /* Mprotect explicitly disabled */ return false; } if (!pax_mprotect_global && (flags & ELF_NOTE_PAX_MPROTECT) == 0) { /* Mprotect not requested */ return false; } return true; } vm_prot_t pax_mprotect_maxprotect( #ifdef PAX_MPROTECT_DEBUG const char *file, size_t line, #endif struct lwp *l, vm_prot_t active, vm_prot_t extra, vm_prot_t maxprot) { uint32_t flags; flags = l->l_proc->p_pax; if (!pax_flags_active(flags, P_PAX_MPROTECT)) return maxprot; return (active|extra) & maxprot; } int pax_mprotect_validate( #ifdef PAX_MPROTECT_DEBUG const char *file, size_t line, #endif struct lwp *l, vm_prot_t prot) { uint32_t flags; flags = l->l_proc->p_pax; if (!pax_flags_active(flags, P_PAX_MPROTECT)) return 0; if ((prot & (VM_PROT_WRITE|VM_PROT_EXECUTE)) == (VM_PROT_WRITE|VM_PROT_EXECUTE)) { #ifdef PAX_MPROTECT_DEBUG struct proc *p = l->l_proc; if (pax_mprotect_debug) printf("%s: %s,%zu: %d.%d (%s): WX rejected\n", __func__, file, line, p->p_pid, l->l_lid, p->p_comm); #endif return EACCES; } return 0; } /* * Bypass MPROTECT for traced processes */ int pax_mprotect_prot(struct lwp *l) { uint32_t flags; flags = l->l_proc->p_pax; if (!pax_flags_active(flags, P_PAX_MPROTECT)) return 0; if (pax_mprotect_ptrace < 2) return 0; return UVM_EXTRACT_PROT_ALL; } #endif /* PAX_MPROTECT */ #ifdef PAX_ASLR static bool pax_aslr_elf_flags_active(uint32_t flags) { if (!pax_aslr_enabled) return false; if (pax_aslr_global && (flags & ELF_NOTE_PAX_NOASLR) != 0) { /* ASLR explicitly disabled */ return false; } if (!pax_aslr_global && (flags & ELF_NOTE_PAX_ASLR) == 0) { /* ASLR not requested */ return false; } return true; } static bool pax_aslr_epp_active(struct exec_package *epp) { if (__predict_false((epp->ep_flags & (EXEC_32|EXEC_TOPDOWN_VM)) == 0)) return false; return pax_flags_active(epp->ep_pax_flags, P_PAX_ASLR); } static bool pax_aslr_active(struct lwp *l) { return pax_flags_active(l->l_proc->p_pax, P_PAX_ASLR); } void pax_aslr_init_vm(struct lwp *l, struct vmspace *vm, struct exec_package *ep) { if (!pax_aslr_active(l)) return; if (__predict_false((ep->ep_flags & (EXEC_32|EXEC_TOPDOWN_VM)) == 0)) return; #ifdef PAX_ASLR_DEBUG if (pax_aslr_flags & PAX_ASLR_MMAP) return; #endif uint32_t len = (ep->ep_flags & EXEC_32) ? PAX_ASLR_DELTA_MMAP_LEN32 : PAX_ASLR_DELTA_MMAP_LEN; uint32_t rand = cprng_fast32(); #ifdef PAX_ASLR_DEBUG if (pax_aslr_flags & PAX_ASLR_FIXED) rand = pax_aslr_rand; #endif vm->vm_aslr_delta_mmap = PAX_ASLR_DELTA(rand, PAX_ASLR_DELTA_MMAP_LSB, len); PAX_DPRINTF("delta_mmap=%#jx/%u", (uintmax_t)vm->vm_aslr_delta_mmap, len); } void pax_aslr_mmap(struct lwp *l, vaddr_t *addr, vaddr_t orig_addr, int f) { if (!pax_aslr_active(l)) return; #ifdef PAX_ASLR_DEBUG char buf[256]; if (pax_aslr_flags & PAX_ASLR_MMAP) return; if (pax_aslr_debug) snprintb(buf, sizeof(buf), MAP_FMT, f); else buf[0] = '\0'; #endif if (!(f & MAP_FIXED) && ((orig_addr == 0) || !(f & MAP_ANON))) { PAX_DPRINTF("applying to %#jx orig_addr=%#jx f=%s", (uintmax_t)*addr, (uintmax_t)orig_addr, buf); if (!(l->l_proc->p_vmspace->vm_map.flags & VM_MAP_TOPDOWN)) *addr += l->l_proc->p_vmspace->vm_aslr_delta_mmap; else *addr -= l->l_proc->p_vmspace->vm_aslr_delta_mmap; PAX_DPRINTF("result %#jx", (uintmax_t)*addr); } else { PAX_DPRINTF("not applying to %#jx orig_addr=%#jx f=%s", (uintmax_t)*addr, (uintmax_t)orig_addr, buf); } } static vaddr_t pax_aslr_offset(vaddr_t align) { size_t pax_align, l2, delta; uint32_t rand; vaddr_t offset; pax_align = align == 0 ? PAGE_SIZE : align; l2 = ilog2(pax_align); rand = cprng_fast32(); #ifdef PAX_ASLR_DEBUG if (pax_aslr_flags & PAX_ASLR_FIXED) rand = pax_aslr_rand; #endif #define PAX_TRUNC(a, b) ((a) & ~((b) - 1)) delta = PAX_ASLR_DELTA(rand, l2, PAX_ASLR_DELTA_EXEC_LEN); offset = PAX_TRUNC(delta, pax_align); offset = MAX(offset, pax_align); PAX_DPRINTF("rand=%#x l2=%#zx pax_align=%#zx delta=%#zx offset=%#jx", rand, l2, pax_align, delta, (uintmax_t)offset); return offset; } vaddr_t pax_aslr_exec_offset(struct exec_package *epp, vaddr_t align) { if (!pax_aslr_epp_active(epp)) goto out; #ifdef PAX_ASLR_DEBUG if (pax_aslr_flags & PAX_ASLR_EXEC_OFFSET) goto out; #endif return pax_aslr_offset(align); out: return MAX(align, PAGE_SIZE); } voff_t pax_aslr_rtld_offset(struct exec_package *epp, vaddr_t align, int use_topdown) { voff_t offset; if (!pax_aslr_epp_active(epp)) return 0; #ifdef PAX_ASLR_DEBUG if (pax_aslr_flags & PAX_ASLR_RTLD_OFFSET) return 0; #endif offset = pax_aslr_offset(align); if (use_topdown) offset = -offset; return offset; } void pax_aslr_stack(struct exec_package *epp, vsize_t *max_stack_size) { if (!pax_aslr_epp_active(epp)) return; #ifdef PAX_ASLR_DEBUG if (pax_aslr_flags & PAX_ASLR_STACK) return; #endif uint32_t len = (epp->ep_flags & EXEC_32) ? PAX_ASLR_DELTA_STACK_LEN32 : PAX_ASLR_DELTA_STACK_LEN; uint32_t rand = cprng_fast32(); #ifdef PAX_ASLR_DEBUG if (pax_aslr_flags & PAX_ASLR_FIXED) rand = pax_aslr_rand; #endif u_long d = PAX_ASLR_DELTA(rand, PAX_ASLR_DELTA_STACK_LSB, len); d &= (*max_stack_size / PAX_ASLR_MAX_STACK_WASTE) - 1; u_long newminsaddr = (u_long)STACK_GROW(epp->ep_minsaddr, d); PAX_DPRINTF("old minsaddr=%#jx delta=%#lx new minsaddr=%#lx", (uintmax_t)epp->ep_minsaddr, d, newminsaddr); epp->ep_minsaddr = (vaddr_t)newminsaddr; *max_stack_size -= d; } uint32_t pax_aslr_stack_gap(struct exec_package *epp) { if (!pax_aslr_epp_active(epp)) return 0; #ifdef PAX_ASLR_DEBUG if (pax_aslr_flags & PAX_ASLR_STACK_GAP) return 0; #endif uint32_t rand = cprng_fast32(); #ifdef PAX_ASLR_DEBUG if (pax_aslr_flags & PAX_ASLR_FIXED) rand = pax_aslr_rand; #endif rand %= PAGE_SIZE; PAX_DPRINTF("stack gap=%#x\n", rand); return rand; } #endif /* PAX_ASLR */ #ifdef PAX_SEGVGUARD static bool pax_segvguard_elf_flags_active(uint32_t flags) { if (!pax_segvguard_enabled) return false; if (pax_segvguard_global && (flags & ELF_NOTE_PAX_NOGUARD) != 0) { /* Segvguard explicitly disabled */ return false; } if (!pax_segvguard_global && (flags & ELF_NOTE_PAX_GUARD) == 0) { /* Segvguard not requested */ return false; } return true; } void pax_segvguard_cleanup(struct vnode *vp) { struct pax_segvguard_entry *p = vp->v_segvguard; struct pax_segvguard_uid_entry *up; if (__predict_true(p == NULL)) { return; } while ((up = LIST_FIRST(&p->segv_uids)) != NULL) { LIST_REMOVE(up, sue_list); kmem_free(up, sizeof(*up)); } kmem_free(p, sizeof(*p)); vp->v_segvguard = NULL; } /* * Called when a process of image vp generated a segfault. * * => exec_lock must be held by the caller * => if "crashed" is true, exec_lock must be held for write */ int pax_segvguard(struct lwp *l, struct vnode *vp, const char *name, bool crashed) { struct pax_segvguard_entry *p; struct pax_segvguard_uid_entry *up; struct timeval tv; uid_t uid; uint32_t flags; bool have_uid; KASSERT(rw_lock_held(&exec_lock)); KASSERT(!crashed || rw_write_held(&exec_lock)); flags = l->l_proc->p_pax; if (!pax_flags_active(flags, P_PAX_GUARD)) return 0; if (vp == NULL) return EFAULT; /* Fast-path if starting a program we don't know. */ if ((p = vp->v_segvguard) == NULL && !crashed) return 0; microtime(&tv); /* * If a program we don't know crashed, we need to create a new entry * for it. */ if (p == NULL) { p = kmem_alloc(sizeof(*p), KM_SLEEP); vp->v_segvguard = p; LIST_INIT(&p->segv_uids); /* * Initialize a new entry with "crashes so far" of 1. * The expiry time is when we purge the entry if it didn't * reach the limit. */ up = kmem_alloc(sizeof(*up), KM_SLEEP); up->sue_uid = kauth_cred_getuid(l->l_cred); up->sue_ncrashes = 1; up->sue_expiry = tv.tv_sec + pax_segvguard_expiry; up->sue_suspended = 0; LIST_INSERT_HEAD(&p->segv_uids, up, sue_list); return 0; } /* * A program we "know" either executed or crashed again. * See if it's a culprit we're familiar with. */ uid = kauth_cred_getuid(l->l_cred); have_uid = false; LIST_FOREACH(up, &p->segv_uids, sue_list) { if (up->sue_uid == uid) { have_uid = true; break; } } /* * It's someone else. Add an entry for him if we crashed. */ if (!have_uid) { if (crashed) { up = kmem_alloc(sizeof(*up), KM_SLEEP); up->sue_uid = uid; up->sue_ncrashes = 1; up->sue_expiry = tv.tv_sec + pax_segvguard_expiry; up->sue_suspended = 0; LIST_INSERT_HEAD(&p->segv_uids, up, sue_list); } return 0; } if (crashed) { /* Check if timer on previous crashes expired first. */ if (up->sue_expiry < tv.tv_sec) { log(LOG_INFO, "PaX Segvguard: [%s] Suspension" " expired.\n", name ? name : "unknown"); up->sue_ncrashes = 1; up->sue_expiry = tv.tv_sec + pax_segvguard_expiry; up->sue_suspended = 0; return 0; } up->sue_ncrashes++; if (up->sue_ncrashes >= pax_segvguard_maxcrashes) { log(LOG_ALERT, "PaX Segvguard: [%s] Suspending " "execution for %d seconds after %zu crashes.\n", name ? name : "unknown", pax_segvguard_suspension, up->sue_ncrashes); /* Suspend this program for a while. */ up->sue_suspended = tv.tv_sec + pax_segvguard_suspension; up->sue_ncrashes = 0; up->sue_expiry = 0; } } else { /* Are we supposed to be suspended? */ if (up->sue_suspended > tv.tv_sec) { log(LOG_ALERT, "PaX Segvguard: [%s] Preventing " "execution due to repeated segfaults.\n", name ? name : "unknown"); return EPERM; } } return 0; } #endif /* PAX_SEGVGUARD */ |
| 2 83 2 43 18 23 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 197 198 199 200 201 202 203 204 205 206 207 208 209 210 211 212 213 214 215 216 217 218 219 220 221 222 223 224 225 226 227 228 229 230 231 232 233 234 235 236 237 238 239 240 241 242 243 244 245 246 247 248 249 250 251 252 253 254 255 256 257 258 259 260 261 262 263 264 265 266 267 268 269 270 271 272 273 274 275 276 277 278 279 280 281 282 283 284 285 286 287 288 289 290 291 292 293 294 295 296 297 298 299 300 301 302 303 304 305 306 307 308 309 310 311 312 313 314 315 316 317 318 319 320 321 322 323 324 325 326 327 328 329 330 331 332 333 334 335 336 337 338 339 340 341 342 343 344 345 346 347 348 349 350 351 352 353 354 355 356 357 358 359 360 361 362 363 364 365 366 367 368 369 370 371 372 373 374 375 376 377 378 379 380 381 382 383 384 385 386 387 388 389 390 391 392 393 394 395 396 397 398 399 400 401 402 403 404 405 406 407 408 409 410 411 412 413 414 415 416 417 418 419 420 421 422 423 424 425 426 427 428 429 430 431 432 433 434 435 436 437 438 439 440 441 442 443 444 445 446 447 448 449 450 451 452 453 454 455 456 457 458 459 460 461 462 463 464 465 466 467 468 469 470 471 472 473 474 475 476 477 478 479 480 481 482 483 484 485 486 487 488 489 490 491 492 493 494 495 496 497 498 499 500 501 502 503 504 | /* $NetBSD: in_var.h,v 1.102 2021/03/08 22:01:18 christos Exp $ */ /*- * Copyright (c) 1998 The NetBSD Foundation, Inc. * All rights reserved. * * This code is derived from software contributed to The NetBSD Foundation * by Public Access Networks Corporation ("Panix"). It was developed under * contract to Panix by Eric Haszlakiewicz and Thor Lancelot Simon. * * Redistribution and use in source and binary forms, with or without * modification, are permitted provided that the following conditions * are met: * 1. Redistributions of source code must retain the above copyright * notice, this list of conditions and the following disclaimer. * 2. Redistributions in binary form must reproduce the above copyright * notice, this list of conditions and the following disclaimer in the * documentation and/or other materials provided with the distribution. * * THIS SOFTWARE IS PROVIDED BY THE NETBSD FOUNDATION, INC. AND CONTRIBUTORS * ``AS IS'' AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED * TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR * PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE FOUNDATION OR CONTRIBUTORS * BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR * CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF * SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS * INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN * CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) * ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE * POSSIBILITY OF SUCH DAMAGE. */ /* * Copyright (c) 1985, 1986, 1993 * The Regents of the University of California. All rights reserved. * * Redistribution and use in source and binary forms, with or without * modification, are permitted provided that the following conditions * are met: * 1. Redistributions of source code must retain the above copyright * notice, this list of conditions and the following disclaimer. * 2. Redistributions in binary form must reproduce the above copyright * notice, this list of conditions and the following disclaimer in the * documentation and/or other materials provided with the distribution. * 3. Neither the name of the University nor the names of its contributors * may be used to endorse or promote products derived from this software * without specific prior written permission. * * THIS SOFTWARE IS PROVIDED BY THE REGENTS AND CONTRIBUTORS ``AS IS'' AND * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE * ARE DISCLAIMED. IN NO EVENT SHALL THE REGENTS OR CONTRIBUTORS BE LIABLE * FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL * DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS * OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) * HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT * LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY * OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF * SUCH DAMAGE. * * @(#)in_var.h 8.2 (Berkeley) 1/9/95 */ #ifndef _NETINET_IN_VAR_H_ #define _NETINET_IN_VAR_H_ #include <sys/queue.h> #define IN_IFF_TENTATIVE 0x01 /* tentative address */ #define IN_IFF_DUPLICATED 0x02 /* DAD detected duplicate */ #define IN_IFF_DETACHED 0x04 /* may be detached from the link */ #define IN_IFF_TRYTENTATIVE 0x08 /* intent to try DAD */ #define IN_IFFBITS \ "\020\1TENTATIVE\2DUPLICATED\3DETACHED\4TRYTENTATIVE" /* do not input/output */ #define IN_IFF_NOTREADY \ (IN_IFF_TRYTENTATIVE | IN_IFF_TENTATIVE | IN_IFF_DUPLICATED) /* * Interface address, Internet version. One of these structures * is allocated for each interface with an Internet address. * The ifaddr structure contains the protocol-independent part * of the structure and is assumed to be first. */ struct in_ifaddr { struct ifaddr ia_ifa; /* protocol-independent info */ #define ia_ifp ia_ifa.ifa_ifp #define ia_flags ia_ifa.ifa_flags /* ia_{,sub}net{,mask} in host order */ u_int32_t ia_net; /* network number of interface */ u_int32_t ia_netmask; /* mask of net part */ u_int32_t ia_subnet; /* subnet number, including net */ u_int32_t ia_subnetmask; /* mask of subnet part */ struct in_addr ia_netbroadcast; /* to recognize net broadcasts */ LIST_ENTRY(in_ifaddr) ia_hash; /* entry in bucket of inet addresses */ TAILQ_ENTRY(in_ifaddr) ia_list; /* list of internet addresses */ struct sockaddr_in ia_addr; /* reserve space for interface name */ struct sockaddr_in ia_dstaddr; /* reserve space for broadcast addr */ #define ia_broadaddr ia_dstaddr struct sockaddr_in ia_sockmask; /* reserve space for general netmask */ LIST_HEAD(, in_multi) ia_multiaddrs; /* list of multicast addresses */ struct in_multi *ia_allhosts; /* multicast address record for the allhosts multicast group */ uint16_t ia_idsalt; /* ip_id salt for this ia */ int ia4_flags; /* address flags */ void (*ia_dad_start) (struct ifaddr *); /* DAD start function */ void (*ia_dad_stop) (struct ifaddr *); /* DAD stop function */ time_t ia_dad_defended; /* last time of DAD defence */ #ifdef _KERNEL struct pslist_entry ia_hash_pslist_entry; struct pslist_entry ia_pslist_entry; #endif }; struct in_nbrinfo { char ifname[IFNAMSIZ]; /* if name, e.g. "en0" */ struct in_addr addr; /* IPv4 address of the neighbor */ long asked; /* number of queries already sent for this addr */ int state; /* reachability state */ int expire; /* lifetime for NDP state transition */ }; #ifdef _KERNEL static __inline void ia4_acquire(struct in_ifaddr *ia, struct psref *psref) { KASSERT(ia != NULL); ifa_acquire(&ia->ia_ifa, psref); } static __inline void ia4_release(struct in_ifaddr *ia, struct psref *psref) { if (ia == NULL) return; ifa_release(&ia->ia_ifa, psref); } #endif struct in_aliasreq { char ifra_name[IFNAMSIZ]; /* if name, e.g. "en0" */ struct sockaddr_in ifra_addr; struct sockaddr_in ifra_dstaddr; #define ifra_broadaddr ifra_dstaddr struct sockaddr_in ifra_mask; }; /* * Given a pointer to an in_ifaddr (ifaddr), * return a pointer to the addr as a sockaddr_in. */ #define IA_SIN(ia) (&(((struct in_ifaddr *)(ia))->ia_addr)) #ifdef _KERNEL /* Note: 61, 127, 251, 509, 1021, 2039 are good. */ #ifndef IN_IFADDR_HASH_SIZE #define IN_IFADDR_HASH_SIZE 509 #endif /* * This is a bit unconventional, and wastes a little bit of space, but * because we want a very even hash function we don't use & in_ifaddrhash * here, but rather % the hash size, which should obviously be prime. */ #define IN_IFADDR_HASH(x) in_ifaddrhashtbl[(u_long)(x) % IN_IFADDR_HASH_SIZE] LIST_HEAD(in_ifaddrhashhead, in_ifaddr); /* Type of the hash head */ TAILQ_HEAD(in_ifaddrhead, in_ifaddr); /* Type of the list head */ extern u_long in_ifaddrhash; /* size of hash table - 1 */ extern struct in_ifaddrhashhead *in_ifaddrhashtbl; /* Hash table head */ extern struct in_ifaddrhead in_ifaddrhead; /* List head (in ip_input) */ extern pserialize_t in_ifaddrhash_psz; extern struct pslist_head *in_ifaddrhashtbl_pslist; extern u_long in_ifaddrhash_pslist; extern struct pslist_head in_ifaddrhead_pslist; #define IN_IFADDR_HASH_PSLIST(x) \ in_ifaddrhashtbl_pslist[(u_long)(x) % IN_IFADDR_HASH_SIZE] #define IN_ADDRHASH_READER_FOREACH(__ia, __addr) \ PSLIST_READER_FOREACH((__ia), &IN_IFADDR_HASH_PSLIST(__addr), \ struct in_ifaddr, ia_hash_pslist_entry) #define IN_ADDRHASH_WRITER_INSERT_HEAD(__ia) \ PSLIST_WRITER_INSERT_HEAD( \ &IN_IFADDR_HASH_PSLIST((__ia)->ia_addr.sin_addr.s_addr), \ (__ia), ia_hash_pslist_entry) #define IN_ADDRHASH_WRITER_REMOVE(__ia) \ PSLIST_WRITER_REMOVE((__ia), ia_hash_pslist_entry) #define IN_ADDRHASH_ENTRY_INIT(__ia) \ PSLIST_ENTRY_INIT((__ia), ia_hash_pslist_entry); #define IN_ADDRHASH_ENTRY_DESTROY(__ia) \ PSLIST_ENTRY_DESTROY((__ia), ia_hash_pslist_entry); #define IN_ADDRHASH_READER_NEXT(__ia) \ PSLIST_READER_NEXT((__ia), struct in_ifaddr, ia_hash_pslist_entry) #define IN_ADDRLIST_ENTRY_INIT(__ia) \ PSLIST_ENTRY_INIT((__ia), ia_pslist_entry) #define IN_ADDRLIST_ENTRY_DESTROY(__ia) \ PSLIST_ENTRY_DESTROY((__ia), ia_pslist_entry); #define IN_ADDRLIST_READER_EMPTY() \ (PSLIST_READER_FIRST(&in_ifaddrhead_pslist, struct in_ifaddr, \ ia_pslist_entry) == NULL) #define IN_ADDRLIST_READER_FIRST() \ PSLIST_READER_FIRST(&in_ifaddrhead_pslist, struct in_ifaddr, \ ia_pslist_entry) #define IN_ADDRLIST_READER_NEXT(__ia) \ PSLIST_READER_NEXT((__ia), struct in_ifaddr, ia_pslist_entry) #define IN_ADDRLIST_READER_FOREACH(__ia) \ PSLIST_READER_FOREACH((__ia), &in_ifaddrhead_pslist, \ struct in_ifaddr, ia_pslist_entry) #define IN_ADDRLIST_WRITER_INSERT_HEAD(__ia) \ PSLIST_WRITER_INSERT_HEAD(&in_ifaddrhead_pslist, (__ia), \ ia_pslist_entry) #define IN_ADDRLIST_WRITER_REMOVE(__ia) \ PSLIST_WRITER_REMOVE((__ia), ia_pslist_entry) #define IN_ADDRLIST_WRITER_FOREACH(__ia) \ PSLIST_WRITER_FOREACH((__ia), &in_ifaddrhead_pslist, \ struct in_ifaddr, ia_pslist_entry) #define IN_ADDRLIST_WRITER_FIRST() \ PSLIST_WRITER_FIRST(&in_ifaddrhead_pslist, struct in_ifaddr, \ ia_pslist_entry) #define IN_ADDRLIST_WRITER_NEXT(__ia) \ PSLIST_WRITER_NEXT((__ia), struct in_ifaddr, ia_pslist_entry) #define IN_ADDRLIST_WRITER_INSERT_AFTER(__ia, __new) \ PSLIST_WRITER_INSERT_AFTER((__ia), (__new), ia_pslist_entry) #define IN_ADDRLIST_WRITER_EMPTY() \ (PSLIST_WRITER_FIRST(&in_ifaddrhead_pslist, struct in_ifaddr, \ ia_pslist_entry) == NULL) #define IN_ADDRLIST_WRITER_INSERT_TAIL(__new) \ do { \ if (IN_ADDRLIST_WRITER_EMPTY()) { \ IN_ADDRLIST_WRITER_INSERT_HEAD((__new)); \ } else { \ struct in_ifaddr *__ia; \ IN_ADDRLIST_WRITER_FOREACH(__ia) { \ if (IN_ADDRLIST_WRITER_NEXT(__ia) == NULL) { \ IN_ADDRLIST_WRITER_INSERT_AFTER(__ia,\ (__new)); \ break; \ } \ } \ } \ } while (0) extern const int inetctlerrmap[]; /* * Find whether an internet address (in_addr) belongs to one * of our interfaces (in_ifaddr). NULL if the address isn't ours. */ static __inline struct in_ifaddr * in_get_ia(struct in_addr addr) { struct in_ifaddr *ia; IN_ADDRHASH_READER_FOREACH(ia, addr.s_addr) { if (in_hosteq(ia->ia_addr.sin_addr, addr)) break; } return ia; } static __inline struct in_ifaddr * in_get_ia_psref(struct in_addr addr, struct psref *psref) { struct in_ifaddr *ia; int s; s = pserialize_read_enter(); ia = in_get_ia(addr); if (ia != NULL) ia4_acquire(ia, psref); pserialize_read_exit(s); return ia; } /* * Find whether an internet address (in_addr) belongs to a specified * interface. NULL if the address isn't ours. */ static __inline struct in_ifaddr * in_get_ia_on_iface(struct in_addr addr, struct ifnet *ifp) { struct in_ifaddr *ia; IN_ADDRHASH_READER_FOREACH(ia, addr.s_addr) { if (in_hosteq(ia->ia_addr.sin_addr, addr) && ia->ia_ifp == ifp) break; } return ia; } static __inline struct in_ifaddr * in_get_ia_on_iface_psref(struct in_addr addr, struct ifnet *ifp, struct psref *psref) { struct in_ifaddr *ia; int s; s = pserialize_read_enter(); ia = in_get_ia_on_iface(addr, ifp); if (ia != NULL) ia4_acquire(ia, psref); pserialize_read_exit(s); return ia; } /* * Find an internet address structure (in_ifaddr) corresponding * to a given interface (ifnet structure). */ static __inline struct in_ifaddr * in_get_ia_from_ifp(struct ifnet *ifp) { struct ifaddr *ifa; IFADDR_READER_FOREACH(ifa, ifp) { if (ifa->ifa_addr->sa_family == AF_INET) break; } return ifatoia(ifa); } static __inline struct in_ifaddr * in_get_ia_from_ifp_psref(struct ifnet *ifp, struct psref *psref) { struct in_ifaddr *ia; int s; s = pserialize_read_enter(); ia = in_get_ia_from_ifp(ifp); if (ia != NULL) ia4_acquire(ia, psref); pserialize_read_exit(s); return ia; } #include <netinet/in_selsrc.h> /* * IPv4 per-interface state. */ struct in_ifinfo { struct lltable *ii_llt; /* ARP state */ struct in_ifsysctl *ii_selsrc; }; #endif /* _KERNEL */ /* * Internet multicast address structure. There is one of these for each IP * multicast group to which this host belongs on a given network interface. * They are kept in a linked list, rooted in the interface's in_ifaddr * structure. */ struct router_info; struct in_multi { LIST_ENTRY(in_multi) inm_list; /* list of multicast addresses */ struct router_info *inm_rti; /* router version info */ struct ifnet *inm_ifp; /* back pointer to ifnet */ struct in_addr inm_addr; /* IP multicast address */ u_int inm_refcount; /* no. membership claims by sockets */ u_int inm_timer; /* IGMP membership report timer */ u_int inm_state; /* state of membership */ }; #ifdef _KERNEL #include <net/pktqueue.h> #include <sys/cprng.h> extern pktqueue_t *ip_pktq; extern int ip_dad_count; /* Duplicate Address Detection probes */ static inline bool ip_dad_enabled(void) { #if NARP > 0 return ip_dad_count > 0; #else return false; #endif } #if defined(INET) && NARP > 0 extern int arp_debug; #define ARPLOGADDR(a) IN_PRINT(_ipbuf, a) #define ARPLOG(level, fmt, args...) \ do { \ char _ipbuf[INET_ADDRSTRLEN]; \ (void)_ipbuf; \ if (arp_debug) \ log(level, "%s: " fmt, __func__, ##args); \ } while (/*CONSTCOND*/0) #else #define ARPLOG(level, fmt, args...) #endif /* * Structure used by functions below to remember position when stepping * through all of the in_multi records. */ struct in_multistep { int i_n; struct in_multi *i_inm; }; bool in_multi_group(struct in_addr, struct ifnet *, int); struct in_multi *in_first_multi(struct in_multistep *); struct in_multi *in_next_multi(struct in_multistep *); struct in_multi *in_lookup_multi(struct in_addr, struct ifnet *); struct in_multi *in_addmulti(struct in_addr *, struct ifnet *); void in_delmulti(struct in_multi *); void in_multi_lock(int); void in_multi_unlock(void); int in_multi_lock_held(void); struct ifaddr; int in_ifinit(struct ifnet *, struct in_ifaddr *, const struct sockaddr_in *, const struct sockaddr_in *, int); void in_savemkludge(struct in_ifaddr *); void in_restoremkludge(struct in_ifaddr *, struct ifnet *); void in_purgemkludge(struct ifnet *); void in_setmaxmtu(void); int in_control(struct socket *, u_long, void *, struct ifnet *); void in_purgeaddr(struct ifaddr *); void in_purgeif(struct ifnet *); void in_addrhash_insert(struct in_ifaddr *); void in_addrhash_remove(struct in_ifaddr *); int ipflow_fastforward(struct mbuf *); extern uint16_t ip_id; extern int ip_do_randomid; static __inline uint16_t ip_randomid(void) { uint16_t id = (uint16_t)cprng_fast32(); return id ? id : 1; } /* * ip_newid_range: "allocate" num contiguous IP IDs. * * => Return the first ID. */ static __inline uint16_t ip_newid_range(const struct in_ifaddr *ia, u_int num) { uint16_t id; if (ip_do_randomid) { /* XXX ignore num */ return ip_randomid(); } /* Never allow an IP ID of 0 (detect wrap). */ if ((uint16_t)(ip_id + num) < ip_id) { ip_id = 1; } id = htons(ip_id); ip_id += num; return id; } static __inline uint16_t ip_newid(const struct in_ifaddr *ia) { return ip_newid_range(ia, 1); } #ifdef SYSCTLFN_PROTO int sysctl_inpcblist(SYSCTLFN_PROTO); #endif #define LLTABLE(ifp) \ ((struct in_ifinfo *)(ifp)->if_afdata[AF_INET])->ii_llt #endif /* !_KERNEL */ /* INET6 stuff */ #include <netinet6/in6_var.h> #endif /* !_NETINET_IN_VAR_H_ */ |
| 1817 1816 598 595 597 914 914 916 913 913 8 8 8 8 737 735 100 100 144 143 100 101 1336 1335 1152 382 382 17 382 382 381 367 2979 2982 2958 258 258 256 256 256 256 1187 14 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 197 198 199 200 201 202 203 204 205 206 207 208 209 210 211 212 213 214 215 216 217 218 219 220 221 222 223 224 225 226 227 228 229 230 231 232 233 234 235 236 237 238 239 240 241 242 243 244 245 246 247 248 249 250 251 252 253 254 255 256 257 258 259 260 261 262 263 264 265 266 267 268 269 270 271 272 273 274 275 276 277 278 279 280 281 282 283 284 285 286 287 288 289 290 291 292 293 294 295 296 297 298 299 300 301 302 303 304 305 306 307 308 309 310 311 312 313 314 315 316 317 318 319 320 321 322 323 324 325 326 327 328 329 330 331 332 333 334 335 336 337 338 339 340 341 342 343 344 345 346 347 348 349 350 351 352 353 354 355 356 357 358 359 360 361 362 363 364 365 366 367 368 369 370 371 372 373 374 375 376 377 378 379 380 381 382 383 384 385 386 387 388 389 390 391 392 393 394 395 396 397 398 399 400 401 402 403 404 405 406 407 408 409 410 411 412 413 414 415 416 417 418 419 420 421 422 423 424 425 426 427 428 429 430 431 432 433 434 435 436 437 438 439 440 441 442 443 444 445 446 447 448 449 450 451 452 453 454 455 456 457 458 459 460 461 462 463 464 465 466 467 468 469 470 471 472 473 474 475 476 477 478 479 480 481 482 483 484 485 486 487 488 489 490 491 492 493 494 495 496 497 498 499 500 501 502 503 504 505 506 507 508 509 510 511 512 513 514 515 516 517 518 519 520 521 522 523 524 525 526 527 528 529 530 531 532 533 534 535 536 537 538 539 540 541 542 543 544 545 546 547 548 549 550 551 552 553 554 555 556 557 558 559 560 561 562 563 564 565 566 | /* $NetBSD: kern_condvar.c,v 1.54 2022/06/29 22:27:01 riastradh Exp $ */ /*- * Copyright (c) 2006, 2007, 2008, 2019, 2020 The NetBSD Foundation, Inc. * All rights reserved. * * This code is derived from software contributed to The NetBSD Foundation * by Andrew Doran. * * Redistribution and use in source and binary forms, with or without * modification, are permitted provided that the following conditions * are met: * 1. Redistributions of source code must retain the above copyright * notice, this list of conditions and the following disclaimer. * 2. Redistributions in binary form must reproduce the above copyright * notice, this list of conditions and the following disclaimer in the * documentation and/or other materials provided with the distribution. * * THIS SOFTWARE IS PROVIDED BY THE NETBSD FOUNDATION, INC. AND CONTRIBUTORS * ``AS IS'' AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED * TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR * PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE FOUNDATION OR CONTRIBUTORS * BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR * CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF * SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS * INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN * CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) * ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE * POSSIBILITY OF SUCH DAMAGE. */ /* * Kernel condition variable implementation. */ #include <sys/cdefs.h> __KERNEL_RCSID(0, "$NetBSD: kern_condvar.c,v 1.54 2022/06/29 22:27:01 riastradh Exp $"); #include <sys/param.h> #include <sys/systm.h> #include <sys/lwp.h> #include <sys/condvar.h> #include <sys/sleepq.h> #include <sys/lockdebug.h> #include <sys/cpu.h> #include <sys/kernel.h> /* * Accessors for the private contents of the kcondvar_t data type. * * cv_opaque[0] sleepq_t * cv_opaque[1] description for ps(1) * * cv_opaque[0] is protected by the interlock passed to cv_wait() (enqueue * only), and the sleep queue lock acquired with sleepq_hashlock() (enqueue * and dequeue). * * cv_opaque[1] (the wmesg) is static and does not change throughout the life * of the CV. */ #define CV_SLEEPQ(cv) ((sleepq_t *)(cv)->cv_opaque) #define CV_WMESG(cv) ((const char *)(cv)->cv_opaque[1]) #define CV_SET_WMESG(cv, v) (cv)->cv_opaque[1] = __UNCONST(v) #define CV_DEBUG_P(cv) (CV_WMESG(cv) != nodebug) #define CV_RA ((uintptr_t)__builtin_return_address(0)) static void cv_unsleep(lwp_t *, bool); static inline void cv_wakeup_one(kcondvar_t *); static inline void cv_wakeup_all(kcondvar_t *); syncobj_t cv_syncobj = { .sobj_flag = SOBJ_SLEEPQ_SORTED, .sobj_unsleep = cv_unsleep, .sobj_changepri = sleepq_changepri, .sobj_lendpri = sleepq_lendpri, .sobj_owner = syncobj_noowner, }; static const char deadcv[] = "deadcv"; /* * cv_init: * * Initialize a condition variable for use. */ void cv_init(kcondvar_t *cv, const char *wmesg) { KASSERT(wmesg != NULL); CV_SET_WMESG(cv, wmesg); sleepq_init(CV_SLEEPQ(cv)); } /* * cv_destroy: * * Tear down a condition variable. */ void cv_destroy(kcondvar_t *cv) { sleepq_destroy(CV_SLEEPQ(cv)); #ifdef DIAGNOSTIC KASSERT(cv_is_valid(cv)); KASSERT(!cv_has_waiters(cv)); CV_SET_WMESG(cv, deadcv); #endif } /* * cv_enter: * * Look up and lock the sleep queue corresponding to the given * condition variable, and increment the number of waiters. */ static inline void cv_enter(kcondvar_t *cv, kmutex_t *mtx, lwp_t *l, bool catch_p) { sleepq_t *sq; kmutex_t *mp; KASSERT(cv_is_valid(cv)); KASSERT(!cpu_intr_p()); KASSERT((l->l_pflag & LP_INTR) == 0 || panicstr != NULL); l->l_kpriority = true; mp = sleepq_hashlock(cv); sq = CV_SLEEPQ(cv); sleepq_enter(sq, l, mp); sleepq_enqueue(sq, cv, CV_WMESG(cv), &cv_syncobj, catch_p); mutex_exit(mtx); KASSERT(cv_has_waiters(cv)); } /* * cv_unsleep: * * Remove an LWP from the condition variable and sleep queue. This * is called when the LWP has not been awoken normally but instead * interrupted: for example, when a signal is received. Must be * called with the LWP locked. Will unlock if "unlock" is true. */ static void cv_unsleep(lwp_t *l, bool unlock) { kcondvar_t *cv __diagused; cv = (kcondvar_t *)(uintptr_t)l->l_wchan; KASSERT(l->l_wchan == (wchan_t)cv); KASSERT(l->l_sleepq == CV_SLEEPQ(cv)); KASSERT(cv_is_valid(cv)); KASSERT(cv_has_waiters(cv)); sleepq_unsleep(l, unlock); } /* * cv_wait: * * Wait non-interruptably on a condition variable until awoken. */ void cv_wait(kcondvar_t *cv, kmutex_t *mtx) { lwp_t *l = curlwp; KASSERT(mutex_owned(mtx)); cv_enter(cv, mtx, l, false); (void)sleepq_block(0, false, &cv_syncobj); mutex_enter(mtx); } /* * cv_wait_sig: * * Wait on a condition variable until a awoken or a signal is received. * Will also return early if the process is exiting. Returns zero if * awoken normally, ERESTART if a signal was received and the system * call is restartable, or EINTR otherwise. */ int cv_wait_sig(kcondvar_t *cv, kmutex_t *mtx) { lwp_t *l = curlwp; int error; KASSERT(mutex_owned(mtx)); cv_enter(cv, mtx, l, true); error = sleepq_block(0, true, &cv_syncobj); mutex_enter(mtx); return error; } /* * cv_timedwait: * * Wait on a condition variable until awoken or the specified timeout * expires. Returns zero if awoken normally or EWOULDBLOCK if the * timeout expired. * * timo is a timeout in ticks. timo = 0 specifies an infinite timeout. */ int cv_timedwait(kcondvar_t *cv, kmutex_t *mtx, int timo) { lwp_t *l = curlwp; int error; KASSERT(mutex_owned(mtx)); cv_enter(cv, mtx, l, false); error = sleepq_block(timo, false, &cv_syncobj); mutex_enter(mtx); return error; } /* * cv_timedwait_sig: * * Wait on a condition variable until a timeout expires, awoken or a * signal is received. Will also return early if the process is * exiting. Returns zero if awoken normally, EWOULDBLOCK if the * timeout expires, ERESTART if a signal was received and the system * call is restartable, or EINTR otherwise. * * timo is a timeout in ticks. timo = 0 specifies an infinite timeout. */ int cv_timedwait_sig(kcondvar_t *cv, kmutex_t *mtx, int timo) { lwp_t *l = curlwp; int error; KASSERT(mutex_owned(mtx)); cv_enter(cv, mtx, l, true); error = sleepq_block(timo, true, &cv_syncobj); mutex_enter(mtx); return error; } /* * Given a number of seconds, sec, and 2^64ths of a second, frac, we * want a number of ticks for a timeout: * * timo = hz*(sec + frac/2^64) * = hz*sec + hz*frac/2^64 * = hz*sec + hz*(frachi*2^32 + fraclo)/2^64 * = hz*sec + hz*frachi/2^32 + hz*fraclo/2^64, * * where frachi is the high 32 bits of frac and fraclo is the * low 32 bits. * * We assume hz < INT_MAX/2 < UINT32_MAX, so * * hz*fraclo/2^64 < fraclo*2^32/2^64 <= 1, * * since fraclo < 2^32. * * We clamp the result at INT_MAX/2 for a timeout in ticks, since we * can't represent timeouts higher than INT_MAX in cv_timedwait, and * spurious wakeup is OK. Moreover, we don't want to wrap around, * because we compute end - start in ticks in order to compute the * remaining timeout, and that difference cannot wrap around, so we use * a timeout less than INT_MAX. Using INT_MAX/2 provides plenty of * margin for paranoia and will exceed most waits in practice by far. */ static unsigned bintime2timo(const struct bintime *bt) { KASSERT(hz < INT_MAX/2); CTASSERT(INT_MAX/2 < UINT32_MAX); if (bt->sec > ((INT_MAX/2)/hz)) return INT_MAX/2; if ((hz*(bt->frac >> 32) >> 32) > (INT_MAX/2 - hz*bt->sec)) return INT_MAX/2; return hz*bt->sec + (hz*(bt->frac >> 32) >> 32); } /* * timo is in units of ticks. We want units of seconds and 2^64ths of * a second. We know hz = 1 sec/tick, and 2^64 = 1 sec/(2^64th of a * second), from which we can conclude 2^64 / hz = 1 (2^64th of a * second)/tick. So for the fractional part, we compute * * frac = rem * 2^64 / hz * = ((rem * 2^32) / hz) * 2^32 * * Using truncating integer division instead of real division will * leave us with only about 32 bits of precision, which means about * 1/4-nanosecond resolution, which is good enough for our purposes. */ static struct bintime timo2bintime(unsigned timo) { return (struct bintime) { .sec = timo / hz, .frac = (((uint64_t)(timo % hz) << 32)/hz << 32), }; } /* * cv_timedwaitbt: * * Wait on a condition variable until awoken or the specified * timeout expires. Returns zero if awoken normally or * EWOULDBLOCK if the timeout expires. * * On entry, bt is a timeout in bintime. cv_timedwaitbt subtracts * the time slept, so on exit, bt is the time remaining after * sleeping, possibly negative if the complete time has elapsed. * No infinite timeout; use cv_wait_sig instead. * * epsilon is a requested maximum error in timeout (excluding * spurious wakeups). Currently not used, will be used in the * future to choose between low- and high-resolution timers. * Actual wakeup time will be somewhere in [t, t + max(e, r) + s) * where r is the finest resolution of clock available and s is * scheduling delays for scheduler overhead and competing threads. * Time is measured by the interrupt source implementing the * timeout, not by another timecounter. */ int cv_timedwaitbt(kcondvar_t *cv, kmutex_t *mtx, struct bintime *bt, const struct bintime *epsilon __diagused) { struct bintime slept; unsigned start, end; int timo; int error; KASSERTMSG(bt->sec >= 0, "negative timeout"); KASSERTMSG(epsilon != NULL, "specify maximum requested delay"); /* If there's nothing left to wait, time out. */ if (bt->sec == 0 && bt->frac == 0) return EWOULDBLOCK; /* Convert to ticks, but clamp to be >=1. */ timo = bintime2timo(bt); KASSERTMSG(timo >= 0, "negative ticks: %d", timo); if (timo == 0) timo = 1; /* * getticks() is technically int, but nothing special * happens instead of overflow, so we assume two's-complement * wraparound and just treat it as unsigned. */ start = getticks(); error = cv_timedwait(cv, mtx, timo); end = getticks(); /* * Set it to the time left, or zero, whichever is larger. We * do not fail with EWOULDBLOCK here because this may have been * an explicit wakeup, so the caller needs to check before they * give up or else cv_signal would be lost. */ slept = timo2bintime(end - start); if (bintimecmp(bt, &slept, <=)) { bt->sec = 0; bt->frac = 0; } else { /* bt := bt - slept */ bintime_sub(bt, &slept); } return error; } /* * cv_timedwaitbt_sig: * * Wait on a condition variable until awoken, the specified * timeout expires, or interrupted by a signal. Returns zero if * awoken normally, EWOULDBLOCK if the timeout expires, or * EINTR/ERESTART if interrupted by a signal. * * On entry, bt is a timeout in bintime. cv_timedwaitbt_sig * subtracts the time slept, so on exit, bt is the time remaining * after sleeping. No infinite timeout; use cv_wait instead. * * epsilon is a requested maximum error in timeout (excluding * spurious wakeups). Currently not used, will be used in the * future to choose between low- and high-resolution timers. */ int cv_timedwaitbt_sig(kcondvar_t *cv, kmutex_t *mtx, struct bintime *bt, const struct bintime *epsilon __diagused) { struct bintime slept; unsigned start, end; int timo; int error; KASSERTMSG(bt->sec >= 0, "negative timeout"); KASSERTMSG(epsilon != NULL, "specify maximum requested delay"); /* If there's nothing left to wait, time out. */ if (bt->sec == 0 && bt->frac == 0) return EWOULDBLOCK; /* Convert to ticks, but clamp to be >=1. */ timo = bintime2timo(bt); KASSERTMSG(timo >= 0, "negative ticks: %d", timo); if (timo == 0) timo = 1; /* * getticks() is technically int, but nothing special * happens instead of overflow, so we assume two's-complement * wraparound and just treat it as unsigned. */ start = getticks(); error = cv_timedwait_sig(cv, mtx, timo); end = getticks(); /* * Set it to the time left, or zero, whichever is larger. We * do not fail with EWOULDBLOCK here because this may have been * an explicit wakeup, so the caller needs to check before they * give up or else cv_signal would be lost. */ slept = timo2bintime(end - start); if (bintimecmp(bt, &slept, <=)) { bt->sec = 0; bt->frac = 0; } else { /* bt := bt - slept */ bintime_sub(bt, &slept); } return error; } /* * cv_signal: * * Wake the highest priority LWP waiting on a condition variable. * Must be called with the interlocking mutex held. */ void cv_signal(kcondvar_t *cv) { KASSERT(cv_is_valid(cv)); if (__predict_false(!LIST_EMPTY(CV_SLEEPQ(cv)))) cv_wakeup_one(cv); } /* * cv_wakeup_one: * * Slow path for cv_signal(). Deliberately marked __noinline to * prevent the compiler pulling it in to cv_signal(), which adds * extra prologue and epilogue code. */ static __noinline void cv_wakeup_one(kcondvar_t *cv) { sleepq_t *sq; kmutex_t *mp; lwp_t *l; /* * Keep waking LWPs until a non-interruptable waiter is found. An * interruptable waiter could fail to do something useful with the * wakeup due to an error return from cv_[timed]wait_sig(), and the * caller of cv_signal() may not expect such a scenario. * * This isn't a problem for non-interruptable waits (untimed and * timed), because if such a waiter is woken here it will not return * an error. */ mp = sleepq_hashlock(cv); sq = CV_SLEEPQ(cv); while ((l = LIST_FIRST(sq)) != NULL) { KASSERT(l->l_sleepq == sq); KASSERT(l->l_mutex == mp); KASSERT(l->l_wchan == cv); if ((l->l_flag & LW_SINTR) == 0) { sleepq_remove(sq, l); break; } else sleepq_remove(sq, l); } mutex_spin_exit(mp); } /* * cv_broadcast: * * Wake all LWPs waiting on a condition variable. Must be called * with the interlocking mutex held. */ void cv_broadcast(kcondvar_t *cv) { KASSERT(cv_is_valid(cv)); if (__predict_false(!LIST_EMPTY(CV_SLEEPQ(cv)))) cv_wakeup_all(cv); } /* * cv_wakeup_all: * * Slow path for cv_broadcast(). Deliberately marked __noinline to * prevent the compiler pulling it in to cv_broadcast(), which adds * extra prologue and epilogue code. */ static __noinline void cv_wakeup_all(kcondvar_t *cv) { sleepq_t *sq; kmutex_t *mp; lwp_t *l; mp = sleepq_hashlock(cv); sq = CV_SLEEPQ(cv); while ((l = LIST_FIRST(sq)) != NULL) { KASSERT(l->l_sleepq == sq); KASSERT(l->l_mutex == mp); KASSERT(l->l_wchan == cv); sleepq_remove(sq, l); } mutex_spin_exit(mp); } /* * cv_has_waiters: * * For diagnostic assertions: return non-zero if a condition * variable has waiters. */ bool cv_has_waiters(kcondvar_t *cv) { return !LIST_EMPTY(CV_SLEEPQ(cv)); } /* * cv_is_valid: * * For diagnostic assertions: return non-zero if a condition * variable appears to be valid. No locks need be held. */ bool cv_is_valid(kcondvar_t *cv) { return CV_WMESG(cv) != deadcv && CV_WMESG(cv) != NULL; } |
| 11 9 1 1 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 | /* $NetBSD: if_media_80.c,v 1.5 2022/08/03 01:38:51 riastradh Exp $ */ /*- * Copyright (c) 1998 The NetBSD Foundation, Inc. * All rights reserved. * * This code is derived from software contributed to The NetBSD Foundation * by Jason R. Thorpe of the Numerical Aerospace Simulation Facility, * NASA Ames Research Center. * * Redistribution and use in source and binary forms, with or without * modification, are permitted provided that the following conditions * are met: * 1. Redistributions of source code must retain the above copyright * notice, this list of conditions and the following disclaimer. * 2. Redistributions in binary form must reproduce the above copyright * notice, this list of conditions and the following disclaimer in the * documentation and/or other materials provided with the distribution. * * THIS SOFTWARE IS PROVIDED BY THE NETBSD FOUNDATION, INC. AND CONTRIBUTORS * ``AS IS'' AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED * TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR * PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE FOUNDATION OR CONTRIBUTORS * BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR * CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF * SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS * INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN * CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) * ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE * POSSIBILITY OF SUCH DAMAGE. */ /* * Copyright (c) 1997 * Jonathan Stone and Jason R. Thorpe. All rights reserved. * * This software is derived from information provided by Matt Thomas. * * Redistribution and use in source and binary forms, with or without * modification, are permitted provided that the following conditions * are met: * 1. Redistributions of source code must retain the above copyright * notice, this list of conditions and the following disclaimer. * 2. Redistributions in binary form must reproduce the above copyright * notice, this list of conditions and the following disclaimer in the * documentation and/or other materials provided with the distribution. * 3. All advertising materials mentioning features or use of this software * must display the following acknowledgement: * This product includes software developed by Jonathan Stone * and Jason R. Thorpe for the NetBSD Project. * 4. The names of the authors may not be used to endorse or promote products * derived from this software without specific prior written permission. * * THIS SOFTWARE IS PROVIDED BY THE AUTHORS ``AS IS'' AND ANY EXPRESS OR * IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES * OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. * IN NO EVENT SHALL THE AUTHOR BE LIABLE FOR ANY DIRECT, INDIRECT, * INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, * BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; * LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED * AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, * OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY * OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF * SUCH DAMAGE. */ #include <sys/cdefs.h> __KERNEL_RCSID(0, "$NetBSD: if_media_80.c,v 1.5 2022/08/03 01:38:51 riastradh Exp $"); #include <sys/param.h> #include <sys/kernel.h> #include <sys/syscallargs.h> #include <sys/errno.h> #include <sys/malloc.h> #include <sys/proc.h> #include <sys/compat_stub.h> #include <net/if.h> #include <net/if_media.h> #include <compat/sys/sockio.h> #include <compat/common/compat_mod.h> static void ifmword_n2o(int *oldwd, int *newwd) { if (IFM_SUBTYPE(*newwd) > IFM_OTHER) *oldwd = (*newwd & ~(_IFM_ETH_XTMASK | IFM_TMASK)) | IFM_OTHER; else *oldwd = *newwd; } /*ARGSUSED*/ static int compat_ifmediareq_pre(struct ifreq *ifr, u_long *cmd, bool *do_post) { struct ifmediareq *ifmr = (struct ifmediareq *)ifr; switch (*cmd) { case SIOCSIFMEDIA_80: *cmd = SIOCSIFMEDIA; /* Convert to new one */ if ((IFM_TYPE(ifr->ifr_media) == IFM_ETHER) && IFM_SUBTYPE(ifr->ifr_media) > IFM_OTHER) { /* Clear unused bits to not to change to wrong media */ ifr->ifr_media &= ~_IFM_ETH_XTMASK; } return 0; case SIOCGIFMEDIA_80: *cmd = SIOCGIFMEDIA; /* Convert to new one */ if (ifmr->ifm_count != 0) { /* * Tell the upper layer to try to convert each ifmedia * entry in the post process. */ *do_post = true; } return 0; default: return 0; } } /*ARGSUSED*/ static int compat_ifmediareq_post(struct ifreq *ifr, u_long cmd) { struct ifmediareq *ifmr = (struct ifmediareq *)ifr; size_t minwords; size_t count; int error, *kptr; switch (cmd) { case SIOCSIFMEDIA: return 0; case SIOCGIFMEDIA: if (ifmr->ifm_count < 0) return EINVAL; /* * ifmr->ifm_count was already ajusted in ifmedia_ioctl(), so * there is no problem to trust ifm_count. */ minwords = ifmr->ifm_count; kptr = malloc(minwords * sizeof(*kptr), M_TEMP, M_WAITOK|M_ZERO); if (kptr == NULL) return ENOMEM; /* * Convert ifm_current and ifm_active. * It's not required to convert ifm_mask. */ ifmword_n2o(&ifmr->ifm_current, &ifmr->ifm_current); ifmword_n2o(&ifmr->ifm_active, &ifmr->ifm_active); /* Convert ifm_ulist array */ for (count = 0; count < minwords; count++) { int oldmwd; error = ufetch_int(&ifmr->ifm_ulist[count], &oldmwd); if (error != 0) goto out; ifmword_n2o(&kptr[count], &oldmwd); } /* Copy to userland in old format */ error = copyout(kptr, ifmr->ifm_ulist, minwords * sizeof(*kptr)); out: free(kptr, M_TEMP); return error; default: return 0; } } void ifmedia_80_init(void) { MODULE_HOOK_SET(ifmedia_80_pre_hook, compat_ifmediareq_pre); MODULE_HOOK_SET(ifmedia_80_post_hook, compat_ifmediareq_post); } void ifmedia_80_fini(void) { MODULE_HOOK_UNSET(ifmedia_80_post_hook); MODULE_HOOK_UNSET(ifmedia_80_pre_hook); } |
| 1 1 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 197 198 199 200 201 202 203 204 205 206 207 208 209 210 211 212 213 214 215 216 217 218 219 220 221 222 223 224 225 226 227 228 229 230 231 232 233 234 235 236 237 238 239 240 241 242 243 244 245 246 247 248 249 250 251 252 253 254 255 256 257 258 259 260 261 262 263 264 265 266 267 268 269 270 271 272 273 274 275 276 277 278 279 280 281 282 283 284 285 286 287 288 289 290 291 292 293 294 295 296 297 298 299 300 301 302 303 304 305 306 307 308 309 310 311 312 313 314 315 316 317 318 319 320 321 322 323 324 325 326 327 328 329 330 331 332 333 334 335 336 337 338 339 340 341 342 343 344 345 346 347 348 349 350 351 352 353 354 355 356 357 358 359 360 361 362 363 364 365 366 367 368 369 370 371 372 373 374 375 376 377 378 379 | /* $NetBSD: rf_compat80.c,v 1.17 2022/06/28 03:13:27 oster Exp $ */ /* * Copyright (c) 2017 Matthew R. Green * All rights reserved. * * Redistribution and use in source and binary forms, with or without * modification, are permitted provided that the following conditions * are met: * 1. Redistributions of source code must retain the above copyright * notice, this list of conditions and the following disclaimer. * 2. Redistributions in binary form must reproduce the above copyright * notice, this list of conditions and the following disclaimer in the * documentation and/or other materials provided with the distribution. * * THIS SOFTWARE IS PROVIDED BY THE AUTHOR ``AS IS'' AND ANY EXPRESS OR * IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES * OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. * IN NO EVENT SHALL THE AUTHOR BE LIABLE FOR ANY DIRECT, INDIRECT, * INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, * BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; * LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED * AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, * OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY * OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF * SUCH DAMAGE. */ #include <sys/types.h> #include <sys/param.h> #include <sys/systm.h> #include <sys/module.h> #include <sys/compat_stub.h> #include <dev/raidframe/raidframeio.h> #include <dev/raidframe/raidframevar.h> #include "rf_raid.h" #include "rf_compat80.h" #include "rf_kintf.h" /* NetBSD 8.99.x removed the row, raidPtr and next members */ struct rf_recon_req80 { RF_RowCol_t row, col; RF_ReconReqFlags_t flags; void *raidPtr; /* used internally; need not be set at ioctl * time */ struct rf_recon_req *next; /* used internally; need not be set at * ioctl time */ }; /* NetBSD 8.99.x made this structure alignment neutral */ typedef struct RF_RaidDisk_s80 { char devname[56]; /* name of device file */ RF_DiskStatus_t status; /* whether it is up or down */ RF_RowCol_t spareRow; /* if in status "spared", this identifies the * spare disk */ RF_RowCol_t spareCol; /* if in status "spared", this identifies the * spare disk */ RF_SectorCount_t numBlocks; /* number of blocks, obtained via READ * CAPACITY */ int blockSize; RF_SectorCount_t partitionSize; /* The *actual* and *full* size of the partition, from the disklabel */ int auto_configured;/* 1 if this component was autoconfigured. 0 otherwise. */ dev_t dev; } RF_RaidDisk_t80; typedef struct RF_DeviceConfig_s80 { u_int rows; u_int cols; u_int maxqdepth; int ndevs; RF_RaidDisk_t80 devs[RF_MAX_DISKS]; int nspares; RF_RaidDisk_t80 spares[RF_MAX_DISKS]; } RF_DeviceConfig_t80; typedef struct RF_Config_s80 { RF_RowCol_t numRow, numCol, numSpare; /* number of rows, columns, * and spare disks */ dev_t devs[RF_MAXROW][RF_MAXCOL]; /* device numbers for disks * comprising array */ char devnames[RF_MAXROW][RF_MAXCOL][50]; /* device names */ dev_t spare_devs[RF_MAXSPARE]; /* device numbers for spare * disks */ char spare_names[RF_MAXSPARE][50]; /* device names */ RF_SectorNum_t sectPerSU; /* sectors per stripe unit */ RF_StripeNum_t SUsPerPU;/* stripe units per parity unit */ RF_StripeNum_t SUsPerRU;/* stripe units per reconstruction unit */ RF_ParityConfig_t parityConfig; /* identifies the RAID architecture to * be used */ RF_DiskQueueType_t diskQueueType; /* 'f' = fifo, 'c' = cvscan, * not used in kernel */ char maxOutstandingDiskReqs; /* # concurrent reqs to be sent to a * disk. not used in kernel. */ char debugVars[RF_MAXDBGV][50]; /* space for specifying debug * variables & their values */ unsigned int layoutSpecificSize; /* size in bytes of * layout-specific info */ void *layoutSpecific; /* a pointer to a layout-specific structure to * be copied in */ int force; /* if !0, ignore many fatal configuration conditions */ /* "force" is used to override cases where the component labels would indicate that configuration should not proceed without user intervention */ } RF_Config_t80; static int rf_check_recon_status_ext80(RF_Raid_t *raidPtr, void *data) { RF_ProgressInfo_t info, **infoPtr = data; rf_check_recon_status_ext(raidPtr, &info); return copyout(&info, *infoPtr, sizeof(info)); } static int rf_check_parityrewrite_status_ext80(RF_Raid_t *raidPtr, void *data) { RF_ProgressInfo_t info, **infoPtr = data; rf_check_parityrewrite_status_ext(raidPtr, &info); return copyout(&info, *infoPtr, sizeof(info)); } static int rf_check_copyback_status_ext80(RF_Raid_t *raidPtr, void *data) { RF_ProgressInfo_t info, **infoPtr = data; rf_check_copyback_status_ext(raidPtr, &info); return copyout(&info, *infoPtr, sizeof(info)); } static void rf_copy_raiddisk80(RF_RaidDisk_t *disk, RF_RaidDisk_t80 *disk80) { /* Be sure the padding areas don't have kernel memory. */ memset(disk80, 0, sizeof(*disk80)); memcpy(disk80->devname, disk->devname, sizeof(disk80->devname)); disk80->status = disk->status; disk80->spareRow = 0; disk80->spareCol = disk->spareCol; disk80->numBlocks = disk->numBlocks; disk80->blockSize = disk->blockSize; disk80->partitionSize = disk->partitionSize; disk80->auto_configured = disk->auto_configured; disk80->dev = disk->dev; } static int rf_get_info80(RF_Raid_t *raidPtr, void *data) { RF_DeviceConfig_t *config; RF_DeviceConfig_t80 *config80, **configPtr80 = data; int rv; config = RF_Malloc(sizeof(*config)); if (config == NULL) return ENOMEM; config80 = RF_Malloc(sizeof(*config80)); if (config80 == NULL) { RF_Free(config, sizeof(*config)); return ENOMEM; } rv = rf_get_info(raidPtr, config); if (rv == 0) { /* convert new to old */ config80->rows = 1; config80->cols = config->cols; config80->maxqdepth = config->maxqdepth; config80->ndevs = config->ndevs; config80->nspares = config->nspares; for (size_t i = 0; i < RF_MAX_DISKS; i++) { rf_copy_raiddisk80(&config->devs[i], &config80->devs[i]); rf_copy_raiddisk80(&config->spares[i], &config80->spares[i]); } rv = copyout(config80, *configPtr80, sizeof(*config80)); } RF_Free(config, sizeof(*config)); RF_Free(config80, sizeof(*config80)); return rv; } static int rf_get_component_label80(RF_Raid_t *raidPtr, void *data) { RF_ComponentLabel_t **clabel_ptr = (RF_ComponentLabel_t **)data; RF_ComponentLabel_t *clabel; int retcode; /* * Perhaps there should be an option to skip the in-core * copy and hit the disk, as with disklabel(8). */ clabel = RF_Malloc(sizeof(*clabel)); if (clabel == NULL) return ENOMEM; retcode = copyin(*clabel_ptr, clabel, sizeof(*clabel)); if (retcode) { RF_Free(clabel, sizeof(*clabel)); return retcode; } rf_get_component_label(raidPtr, clabel); /* Fix-up for userland. */ if (clabel->version == bswap32(RF_COMPONENT_LABEL_VERSION)) clabel->version = RF_COMPONENT_LABEL_VERSION; retcode = copyout(clabel, *clabel_ptr, sizeof(**clabel_ptr)); RF_Free(clabel, sizeof(*clabel)); return retcode; } static int rf_config80(struct raid_softc *rs, void *data) { RF_Config_t80 *u80_cfg, *k80_cfg; RF_Config_t *k_cfg; RF_Raid_t *raidPtr = rf_get_raid(rs); size_t i, j; int error; if (raidPtr->valid) { /* There is a valid RAID set running on this unit! */ printf("raid%d: Device already configured!\n", rf_get_unit(rs)); return EINVAL; } /* copy-in the configuration information */ /* data points to a pointer to the configuration structure */ u80_cfg = *((RF_Config_t80 **) data); k80_cfg = RF_Malloc(sizeof(*k80_cfg)); if (k80_cfg == NULL) return ENOMEM; error = copyin(u80_cfg, k80_cfg, sizeof(*k80_cfg)); if (error) { RF_Free(k80_cfg, sizeof(*k80_cfg)); return error; } k_cfg = RF_Malloc(sizeof(*k_cfg)); if (k_cfg == NULL) { RF_Free(k80_cfg, sizeof(*k80_cfg)); return ENOMEM; } k_cfg->numCol = k80_cfg->numCol; k_cfg->numSpare = k80_cfg->numSpare; for (i = 0; i < RF_MAXROW; i++) for (j = 0; j < RF_MAXCOL; j++) k_cfg->devs[i][j] = k80_cfg->devs[i][j]; memcpy(k_cfg->devnames, k80_cfg->devnames, sizeof(k_cfg->devnames)); for (i = 0; i < RF_MAXSPARE; i++) k_cfg->spare_devs[i] = k80_cfg->spare_devs[i]; memcpy(k_cfg->spare_names, k80_cfg->spare_names, sizeof(k_cfg->spare_names)); k_cfg->sectPerSU = k80_cfg->sectPerSU; k_cfg->SUsPerPU = k80_cfg->SUsPerPU; k_cfg->SUsPerRU = k80_cfg->SUsPerRU; k_cfg->parityConfig = k80_cfg->parityConfig; memcpy(k_cfg->diskQueueType, k80_cfg->diskQueueType, sizeof(k_cfg->diskQueueType)); k_cfg->maxOutstandingDiskReqs = k80_cfg->maxOutstandingDiskReqs; memcpy(k_cfg->debugVars, k80_cfg->debugVars, sizeof(k_cfg->debugVars)); k_cfg->layoutSpecificSize = k80_cfg->layoutSpecificSize; k_cfg->layoutSpecific = k80_cfg->layoutSpecific; k_cfg->force = k80_cfg->force; RF_Free(k80_cfg, sizeof(*k80_cfg)); return rf_construct(rs, k_cfg); } static int rf_fail_disk80(RF_Raid_t *raidPtr, struct rf_recon_req80 *req80) { struct rf_recon_req req = { .col = req80->col, .flags = req80->flags, }; return rf_fail_disk(raidPtr, &req); } static int raidframe_ioctl_80(struct raid_softc *rs, u_long cmd, void *data) { RF_Raid_t *raidPtr = rf_get_raid(rs); switch (cmd) { case RAIDFRAME_CHECK_RECON_STATUS_EXT80: case RAIDFRAME_CHECK_PARITYREWRITE_STATUS_EXT80: case RAIDFRAME_CHECK_COPYBACK_STATUS_EXT80: case RAIDFRAME_GET_INFO80: case RAIDFRAME_GET_COMPONENT_LABEL80: case RAIDFRAME_FAIL_DISK80: if (!rf_inited(rs)) return ENXIO; break; case RAIDFRAME_CONFIGURE80: break; default: return EPASSTHROUGH; } switch (cmd) { case RAIDFRAME_CHECK_RECON_STATUS_EXT80: return rf_check_recon_status_ext80(raidPtr, data); case RAIDFRAME_CHECK_PARITYREWRITE_STATUS_EXT80: return rf_check_parityrewrite_status_ext80(raidPtr, data); case RAIDFRAME_CHECK_COPYBACK_STATUS_EXT80: return rf_check_copyback_status_ext80(raidPtr, data); case RAIDFRAME_GET_INFO80: return rf_get_info80(raidPtr, data); case RAIDFRAME_GET_COMPONENT_LABEL80: return rf_get_component_label80(raidPtr, data); case RAIDFRAME_CONFIGURE80: return rf_config80(rs, data); case RAIDFRAME_FAIL_DISK80: return rf_fail_disk80(raidPtr, data); default: /* abort really */ return EPASSTHROUGH; } } static void raidframe_80_init(void) { MODULE_HOOK_SET(raidframe_ioctl_80_hook, raidframe_ioctl_80); } static void raidframe_80_fini(void) { MODULE_HOOK_UNSET(raidframe_ioctl_80_hook); } MODULE(MODULE_CLASS_EXEC, compat_raid_80, "raid,compat_80"); static int compat_raid_80_modcmd(modcmd_t cmd, void *arg) { switch (cmd) { case MODULE_CMD_INIT: raidframe_80_init(); return 0; case MODULE_CMD_FINI: raidframe_80_fini(); return 0; default: return ENOTTY; } } |
| 1 1 1 1 1 1 3 5 1 5 4 1 2 2 1 1 1 2 1 1 1 1 1 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 197 198 199 200 201 202 203 204 205 206 207 208 209 210 211 212 213 214 215 216 217 218 219 220 221 222 223 224 225 226 227 228 229 230 231 232 233 234 235 236 237 238 239 240 241 242 243 244 245 246 247 248 249 250 251 252 253 254 255 256 257 258 259 260 261 262 263 264 265 266 267 268 269 270 271 272 273 274 275 276 277 278 279 280 281 282 283 284 285 286 287 288 289 290 291 292 293 294 295 296 297 298 299 300 301 302 303 304 305 306 307 308 309 310 311 312 313 314 315 316 317 318 319 320 321 322 323 324 325 326 327 328 329 330 331 332 333 334 335 336 337 338 339 340 341 342 343 344 345 346 347 348 349 350 351 352 353 354 355 356 357 358 359 360 361 362 363 364 365 366 367 368 369 370 371 372 373 374 375 376 377 378 379 380 381 382 383 384 385 386 387 388 389 390 391 392 393 394 395 396 397 398 399 400 401 402 403 404 405 406 407 408 409 410 411 412 413 414 415 416 417 418 419 420 421 422 423 424 425 426 427 428 429 430 431 432 433 434 435 436 437 438 439 440 441 442 443 444 445 446 447 448 449 450 451 452 453 454 455 456 457 458 459 460 461 462 463 464 465 466 467 468 469 470 471 472 473 474 475 476 477 478 479 480 481 482 483 484 485 486 487 488 489 490 491 492 493 494 495 496 497 498 499 500 501 502 503 504 505 506 507 508 509 510 511 512 513 514 515 516 517 518 519 520 521 522 523 524 525 526 527 528 529 530 531 532 533 534 535 536 537 538 539 540 541 542 543 544 545 546 547 548 549 550 551 552 553 554 555 556 | /* $NetBSD: wskbdutil.c,v 1.19 2017/11/03 19:20:27 maya Exp $ */ /*- * Copyright (c) 1997 The NetBSD Foundation, Inc. * All rights reserved. * * This code is derived from software contributed to The NetBSD Foundation * by Juergen Hannken-Illjes. * * Redistribution and use in source and binary forms, with or without * modification, are permitted provided that the following conditions * are met: * 1. Redistributions of source code must retain the above copyright * notice, this list of conditions and the following disclaimer. * 2. Redistributions in binary form must reproduce the above copyright * notice, this list of conditions and the following disclaimer in the * documentation and/or other materials provided with the distribution. * * THIS SOFTWARE IS PROVIDED BY THE NETBSD FOUNDATION, INC. AND CONTRIBUTORS * ``AS IS'' AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED * TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR * PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE FOUNDATION OR CONTRIBUTORS * BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR * CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF * SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS * INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN * CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) * ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE * POSSIBILITY OF SUCH DAMAGE. */ #include <sys/cdefs.h> __KERNEL_RCSID(0, "$NetBSD: wskbdutil.c,v 1.19 2017/11/03 19:20:27 maya Exp $"); #include <sys/param.h> #include <sys/errno.h> #include <sys/systm.h> #include <sys/malloc.h> #include <dev/wscons/wsksymdef.h> #include <dev/wscons/wsksymvar.h> static struct compose_tab_s { keysym_t elem[2]; keysym_t result; } compose_tab[] = { { { KS_plus, KS_plus }, KS_numbersign }, { { KS_a, KS_a }, KS_at }, { { KS_parenleft, KS_parenleft }, KS_bracketleft }, { { KS_slash, KS_slash }, KS_backslash }, { { KS_parenright, KS_parenright }, KS_bracketright }, { { KS_parenleft, KS_minus }, KS_braceleft }, { { KS_slash, KS_minus }, KS_bar }, { { KS_parenright, KS_minus }, KS_braceright }, { { KS_exclam, KS_exclam }, KS_exclamdown }, { { KS_c, KS_slash }, KS_cent }, { { KS_l, KS_minus }, KS_sterling }, { { KS_y, KS_minus }, KS_yen }, { { KS_s, KS_o }, KS_section }, { { KS_x, KS_o }, KS_currency }, { { KS_c, KS_o }, KS_copyright }, { { KS_less, KS_less }, KS_guillemotleft }, { { KS_greater, KS_greater }, KS_guillemotright }, { { KS_question, KS_question }, KS_questiondown }, { { KS_dead_acute, KS_space }, KS_acute }, { { KS_dead_grave, KS_space }, KS_grave }, { { KS_dead_tilde, KS_space }, KS_asciitilde }, { { KS_dead_circumflex, KS_space }, KS_asciicircum }, { { KS_dead_circumflex, KS_A }, KS_Acircumflex }, { { KS_dead_diaeresis, KS_A }, KS_Adiaeresis }, { { KS_dead_grave, KS_A }, KS_Agrave }, { { KS_dead_abovering, KS_A }, KS_Aring }, { { KS_dead_tilde, KS_A }, KS_Atilde }, { { KS_dead_cedilla, KS_C }, KS_Ccedilla }, { { KS_dead_acute, KS_E }, KS_Eacute }, { { KS_dead_circumflex, KS_E }, KS_Ecircumflex }, { { KS_dead_diaeresis, KS_E }, KS_Ediaeresis }, { { KS_dead_grave, KS_E }, KS_Egrave }, { { KS_dead_acute, KS_I }, KS_Iacute }, { { KS_dead_circumflex, KS_I }, KS_Icircumflex }, { { KS_dead_diaeresis, KS_I }, KS_Idiaeresis }, { { KS_dead_grave, KS_I }, KS_Igrave }, { { KS_dead_tilde, KS_N }, KS_Ntilde }, { { KS_dead_acute, KS_O }, KS_Oacute }, { { KS_dead_circumflex, KS_O }, KS_Ocircumflex }, { { KS_dead_diaeresis, KS_O }, KS_Odiaeresis }, { { KS_dead_grave, KS_O }, KS_Ograve }, { { KS_dead_tilde, KS_O }, KS_Otilde }, { { KS_dead_acute, KS_U }, KS_Uacute }, { { KS_dead_circumflex, KS_U }, KS_Ucircumflex }, { { KS_dead_diaeresis, KS_U }, KS_Udiaeresis }, { { KS_dead_grave, KS_U }, KS_Ugrave }, { { KS_dead_acute, KS_Y }, KS_Yacute }, { { KS_dead_acute, KS_a }, KS_aacute }, { { KS_dead_circumflex, KS_a }, KS_acircumflex }, { { KS_dead_diaeresis, KS_a }, KS_adiaeresis }, { { KS_dead_grave, KS_a }, KS_agrave }, { { KS_dead_abovering, KS_a }, KS_aring }, { { KS_dead_tilde, KS_a }, KS_atilde }, { { KS_dead_cedilla, KS_c }, KS_ccedilla }, { { KS_dead_acute, KS_e }, KS_eacute }, { { KS_dead_circumflex, KS_e }, KS_ecircumflex }, { { KS_dead_diaeresis, KS_e }, KS_ediaeresis }, { { KS_dead_grave, KS_e }, KS_egrave }, { { KS_dead_acute, KS_i }, KS_iacute }, { { KS_dead_circumflex, KS_i }, KS_icircumflex }, { { KS_dead_diaeresis, KS_i }, KS_idiaeresis }, { { KS_dead_grave, KS_i }, KS_igrave }, { { KS_dead_tilde, KS_n }, KS_ntilde }, { { KS_dead_acute, KS_o }, KS_oacute }, { { KS_dead_circumflex, KS_o }, KS_ocircumflex }, { { KS_dead_diaeresis, KS_o }, KS_odiaeresis }, { { KS_dead_grave, KS_o }, KS_ograve }, { { KS_dead_tilde, KS_o }, KS_otilde }, { { KS_dead_acute, KS_u }, KS_uacute }, { { KS_dead_circumflex, KS_u }, KS_ucircumflex }, { { KS_dead_diaeresis, KS_u }, KS_udiaeresis }, { { KS_dead_grave, KS_u }, KS_ugrave }, { { KS_dead_acute, KS_y }, KS_yacute }, { { KS_dead_diaeresis, KS_y }, KS_ydiaeresis }, { { KS_quotedbl, KS_A }, KS_Adiaeresis }, { { KS_quotedbl, KS_E }, KS_Ediaeresis }, { { KS_quotedbl, KS_I }, KS_Idiaeresis }, { { KS_quotedbl, KS_O }, KS_Odiaeresis }, { { KS_quotedbl, KS_U }, KS_Udiaeresis }, { { KS_quotedbl, KS_a }, KS_adiaeresis }, { { KS_quotedbl, KS_e }, KS_ediaeresis }, { { KS_quotedbl, KS_i }, KS_idiaeresis }, { { KS_quotedbl, KS_o }, KS_odiaeresis }, { { KS_quotedbl, KS_u }, KS_udiaeresis }, { { KS_quotedbl, KS_y }, KS_ydiaeresis }, { { KS_acute, KS_A }, KS_Aacute }, { { KS_asciicircum, KS_A }, KS_Acircumflex }, { { KS_grave, KS_A }, KS_Agrave }, { { KS_asterisk, KS_A }, KS_Aring }, { { KS_asciitilde, KS_A }, KS_Atilde }, { { KS_cedilla, KS_C }, KS_Ccedilla }, { { KS_acute, KS_E }, KS_Eacute }, { { KS_asciicircum, KS_E }, KS_Ecircumflex }, { { KS_grave, KS_E }, KS_Egrave }, { { KS_acute, KS_I }, KS_Iacute }, { { KS_asciicircum, KS_I }, KS_Icircumflex }, { { KS_grave, KS_I }, KS_Igrave }, { { KS_asciitilde, KS_N }, KS_Ntilde }, { { KS_acute, KS_O }, KS_Oacute }, { { KS_asciicircum, KS_O }, KS_Ocircumflex }, { { KS_grave, KS_O }, KS_Ograve }, { { KS_asciitilde, KS_O }, KS_Otilde }, { { KS_acute, KS_U }, KS_Uacute }, { { KS_asciicircum, KS_U }, KS_Ucircumflex }, { { KS_grave, KS_U }, KS_Ugrave }, { { KS_acute, KS_Y }, KS_Yacute }, { { KS_acute, KS_a }, KS_aacute }, { { KS_asciicircum, KS_a }, KS_acircumflex }, { { KS_grave, KS_a }, KS_agrave }, { { KS_asterisk, KS_a }, KS_aring }, { { KS_asciitilde, KS_a }, KS_atilde }, { { KS_cedilla, KS_c }, KS_ccedilla }, { { KS_acute, KS_e }, KS_eacute }, { { KS_asciicircum, KS_e }, KS_ecircumflex }, { { KS_grave, KS_e }, KS_egrave }, { { KS_acute, KS_i }, KS_iacute }, { { KS_asciicircum, KS_i }, KS_icircumflex }, { { KS_grave, KS_i }, KS_igrave }, { { KS_asciitilde, KS_n }, KS_ntilde }, { { KS_acute, KS_o }, KS_oacute }, { { KS_asciicircum, KS_o }, KS_ocircumflex }, { { KS_grave, KS_o }, KS_ograve }, { { KS_asciitilde, KS_o }, KS_otilde }, { { KS_acute, KS_u }, KS_uacute }, { { KS_asciicircum, KS_u }, KS_ucircumflex }, { { KS_grave, KS_u }, KS_ugrave }, { { KS_acute, KS_y }, KS_yacute }, { { KS_dead_semi, KS_gr_A }, KS_gr_At }, { { KS_dead_semi, KS_gr_E }, KS_gr_Et }, { { KS_dead_semi, KS_gr_H }, KS_gr_Ht }, { { KS_dead_semi, KS_gr_I }, KS_gr_It }, { { KS_dead_semi, KS_gr_O }, KS_gr_Ot }, { { KS_dead_semi, KS_gr_Y }, KS_gr_Yt }, { { KS_dead_semi, KS_gr_V }, KS_gr_Vt }, { { KS_dead_colon, KS_gr_I }, KS_gr_Id }, { { KS_dead_colon, KS_gr_Y }, KS_gr_Yd }, { { KS_dead_semi, KS_gr_a }, KS_gr_at }, { { KS_dead_semi, KS_gr_e }, KS_gr_et }, { { KS_dead_semi, KS_gr_h }, KS_gr_ht }, { { KS_dead_semi, KS_gr_i }, KS_gr_it }, { { KS_dead_semi, KS_gr_o }, KS_gr_ot }, { { KS_dead_semi, KS_gr_y }, KS_gr_yt }, { { KS_dead_semi, KS_gr_v }, KS_gr_vt }, { { KS_dead_colon, KS_gr_i }, KS_gr_id }, { { KS_dead_colon, KS_gr_y }, KS_gr_yd }, /* Latin 2*/ { { KS_dead_acute, KS_S }, KS_Sacute }, { { KS_dead_acute, KS_Z }, KS_Zacute }, { { KS_dead_acute, KS_s }, KS_sacute }, { { KS_dead_acute, KS_z }, KS_zacute }, { { KS_dead_acute, KS_R }, KS_Racute }, { { KS_dead_acute, KS_A }, KS_Aacute }, { { KS_dead_acute, KS_L }, KS_Lacute }, { { KS_dead_acute, KS_C }, KS_Cacute }, { { KS_dead_acute, KS_E }, KS_Eacute }, { { KS_dead_acute, KS_I }, KS_Iacute }, { { KS_dead_acute, KS_N }, KS_Nacute }, { { KS_dead_acute, KS_O }, KS_Oacute }, { { KS_dead_acute, KS_U }, KS_Uacute }, { { KS_dead_acute, KS_Y }, KS_Yacute }, { { KS_dead_acute, KS_r }, KS_racute }, { { KS_dead_acute, KS_a }, KS_aacute }, { { KS_dead_acute, KS_l }, KS_lacute }, { { KS_dead_acute, KS_c }, KS_cacute }, { { KS_dead_acute, KS_e }, KS_eacute }, { { KS_dead_acute, KS_i }, KS_iacute }, { { KS_dead_acute, KS_n }, KS_nacute }, { { KS_dead_acute, KS_o }, KS_oacute }, { { KS_dead_acute, KS_u }, KS_uacute }, { { KS_dead_acute, KS_y }, KS_yacute }, { { KS_dead_breve, KS_A }, KS_Abreve }, { { KS_dead_breve, KS_a }, KS_abreve }, { { KS_dead_caron, KS_L }, KS_Lcaron }, { { KS_dead_caron, KS_S }, KS_Scaron }, { { KS_dead_caron, KS_T }, KS_Tcaron }, { { KS_dead_caron, KS_Z }, KS_Zcaron }, { { KS_dead_caron, KS_l }, KS_lcaron }, { { KS_dead_caron, KS_s }, KS_scaron }, { { KS_dead_caron, KS_t }, KS_tcaron }, { { KS_dead_caron, KS_z }, KS_zcaron }, { { KS_dead_caron, KS_C }, KS_Ccaron }, { { KS_dead_caron, KS_E }, KS_Ecaron }, { { KS_dead_caron, KS_D }, KS_Dcaron }, { { KS_dead_caron, KS_N }, KS_Ncaron }, { { KS_dead_caron, KS_R }, KS_Rcaron }, { { KS_dead_caron, KS_c }, KS_ccaron }, { { KS_dead_caron, KS_e }, KS_ecaron }, { { KS_dead_caron, KS_d }, KS_dcaron }, { { KS_dead_caron, KS_n }, KS_ncaron }, { { KS_dead_caron, KS_r }, KS_rcaron }, { { KS_dead_cedilla, KS_S }, KS_Scedilla }, { { KS_dead_cedilla, KS_s }, KS_scedilla }, { { KS_dead_cedilla, KS_C }, KS_Ccedilla }, { { KS_dead_cedilla, KS_T }, KS_Tcedilla }, { { KS_dead_cedilla, KS_c }, KS_ccedilla }, { { KS_dead_cedilla, KS_t }, KS_tcedilla }, { { KS_dead_circumflex, KS_A }, KS_Acircumflex }, { { KS_dead_circumflex, KS_I }, KS_Icircumflex }, { { KS_dead_circumflex, KS_O }, KS_Ocircumflex }, { { KS_dead_circumflex, KS_a }, KS_acircumflex }, { { KS_dead_circumflex, KS_i }, KS_icircumflex }, { { KS_dead_circumflex, KS_o }, KS_ocircumflex }, { { KS_dead_diaeresis, KS_A }, KS_Adiaeresis }, { { KS_dead_diaeresis, KS_E }, KS_Ediaeresis }, { { KS_dead_diaeresis, KS_O }, KS_Odiaeresis }, { { KS_dead_diaeresis, KS_U }, KS_Udiaeresis }, { { KS_dead_diaeresis, KS_a }, KS_adiaeresis }, { { KS_dead_diaeresis, KS_e }, KS_ediaeresis }, { { KS_dead_diaeresis, KS_o }, KS_odiaeresis }, { { KS_dead_diaeresis, KS_u }, KS_udiaeresis }, { { KS_dead_dotaccent, KS_Z }, KS_Zabovedot }, { { KS_dead_dotaccent, KS_z }, KS_zabovedot }, { { KS_dead_hungarumlaut, KS_O }, KS_Odoubleacute }, { { KS_dead_hungarumlaut, KS_U }, KS_Udoubleacute }, { { KS_dead_hungarumlaut, KS_o }, KS_odoubleacute }, { { KS_dead_hungarumlaut, KS_u }, KS_udoubleacute }, { { KS_dead_ogonek, KS_A }, KS_Aogonek }, { { KS_dead_ogonek, KS_a }, KS_aogonek }, { { KS_dead_ogonek, KS_E }, KS_Eogonek }, { { KS_dead_ogonek, KS_e }, KS_eogonek }, { { KS_dead_abovering, KS_U }, KS_Uabovering }, { { KS_dead_abovering, KS_u }, KS_uabovering }, { { KS_dead_slash, KS_L }, KS_Lstroke }, { { KS_dead_slash, KS_l }, KS_lstroke } }; #define COMPOSE_SIZE __arraycount(compose_tab) static int compose_tab_inorder = 0; static inline int compose_tab_cmp(struct compose_tab_s *, struct compose_tab_s *); static keysym_t ksym_upcase(keysym_t); static void fillmapentry(const keysym_t *, int, struct wscons_keymap *); static inline int compose_tab_cmp(struct compose_tab_s *i, struct compose_tab_s *j) { if (i->elem[0] == j->elem[0]) return(i->elem[1] - j->elem[1]); else return(i->elem[0] - j->elem[0]); } keysym_t wskbd_compose_value(keysym_t *compose_buf) { int i, j, r; struct compose_tab_s v; if (! compose_tab_inorder) { /* Insertion sort. */ for (i = 1; i < COMPOSE_SIZE; i++) { v = compose_tab[i]; /* find correct slot, moving others up */ for (j = i; --j >= 0 && compose_tab_cmp(& v, & compose_tab[j]) < 0; ) compose_tab[j + 1] = compose_tab[j]; compose_tab[j + 1] = v; } compose_tab_inorder = 1; } for (j = 0, i = COMPOSE_SIZE; i != 0; i /= 2) { if (compose_tab[j + i/2].elem[0] == compose_buf[0]) { if (compose_tab[j + i/2].elem[1] == compose_buf[1]) return(compose_tab[j + i/2].result); r = compose_tab[j + i/2].elem[1] < compose_buf[1]; } else r = compose_tab[j + i/2].elem[0] < compose_buf[0]; if (r) { j += i/2 + 1; i--; } } return(KS_voidSymbol); } static const u_char latin1_to_upper[256] = { /* 0 8 1 9 2 a 3 b 4 c 5 d 6 e 7 f */ 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, /* 0 */ 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, /* 0 */ 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, /* 1 */ 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, /* 1 */ 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, /* 2 */ 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, /* 2 */ 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, /* 3 */ 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, /* 3 */ 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, /* 4 */ 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, /* 4 */ 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, /* 5 */ 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, /* 5 */ 0x00, 'A', 'B', 'C', 'D', 'E', 'F', 'G', /* 6 */ 'H', 'I', 'J', 'K', 'L', 'M', 'N', 'O', /* 6 */ 'P', 'Q', 'R', 'S', 'T', 'U', 'V', 'W', /* 7 */ 'X', 'Y', 'Z', 0x00, 0x00, 0x00, 0x00, 0x00, /* 7 */ 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, /* 8 */ 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, /* 8 */ 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, /* 9 */ 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, /* 9 */ 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, /* a */ 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, /* a */ 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, /* b */ 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, /* b */ 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, /* c */ 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, /* c */ 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, /* d */ 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, /* d */ 0xc0, 0xc1, 0xc2, 0xc3, 0xc4, 0xc5, 0xc6, 0xc7, /* e */ 0xc8, 0xc9, 0xca, 0xcb, 0xcc, 0xcd, 0xce, 0xcf, /* e */ 0xd0, 0xd1, 0xd2, 0xd3, 0xd4, 0xd5, 0xd6, 0x00, /* f */ 0xd8, 0xd9, 0xda, 0xdb, 0xdc, 0xdd, 0xde, 0x00, /* f */ }; static keysym_t ksym_upcase(keysym_t ksym) { if (ksym >= KS_f1 && ksym <= KS_f20) return(KS_F1 - KS_f1 + ksym); if (KS_GROUP(ksym) == KS_GROUP_Plain && ksym <= 0xff && latin1_to_upper[ksym] != 0x00) return(latin1_to_upper[ksym]); return(ksym); } static void fillmapentry(const keysym_t *kp, int len, struct wscons_keymap *mapentry) { switch (len) { case 0: mapentry->group1[0] = KS_voidSymbol; mapentry->group1[1] = KS_voidSymbol; mapentry->group2[0] = KS_voidSymbol; mapentry->group2[1] = KS_voidSymbol; break; case 1: mapentry->group1[0] = kp[0]; mapentry->group1[1] = ksym_upcase(kp[0]); mapentry->group2[0] = mapentry->group1[0]; mapentry->group2[1] = mapentry->group1[1]; break; case 2: mapentry->group1[0] = kp[0]; mapentry->group1[1] = kp[1]; mapentry->group2[0] = mapentry->group1[0]; mapentry->group2[1] = mapentry->group1[1]; break; case 3: mapentry->group1[0] = kp[0]; mapentry->group1[1] = kp[1]; mapentry->group2[0] = kp[2]; mapentry->group2[1] = ksym_upcase(kp[2]); break; case 4: mapentry->group1[0] = kp[0]; mapentry->group1[1] = kp[1]; mapentry->group2[0] = kp[2]; mapentry->group2[1] = kp[3]; break; } } void wskbd_get_mapentry(const struct wskbd_mapdata *mapdata, int kc, struct wscons_keymap *mapentry) { kbd_t cur; const keysym_t *kp; const struct wscons_keydesc *mp; int l; mapentry->command = KS_voidSymbol; mapentry->group1[0] = KS_voidSymbol; mapentry->group1[1] = KS_voidSymbol; mapentry->group2[0] = KS_voidSymbol; mapentry->group2[1] = KS_voidSymbol; for (cur = mapdata->layout & ~KB_HANDLEDBYWSKBD; cur != 0; ) { mp = mapdata->keydesc; while (mp->map_size > 0) { if (mp->name == cur) break; mp++; } /* If map not found, return */ if (mp->map_size <= 0) return; for (kp = mp->map; kp < mp->map + mp->map_size; kp++) if (KS_GROUP(*kp) == KS_GROUP_Keycode && KS_VALUE(*kp) == kc) { /* First skip keycode and possible command */ kp++; if (KS_GROUP(*kp) == KS_GROUP_Command || *kp == KS_Cmd || *kp == KS_Cmd1 || *kp == KS_Cmd2) mapentry->command = *kp++; for (l = 0; kp + l < mp->map + mp->map_size; l++) if (KS_GROUP(kp[l]) == KS_GROUP_Keycode) break; if (l > 4) panic("wskbd_get_mapentry: %d(%d): bad entry", mp->name, *kp); fillmapentry(kp, l, mapentry); return; } cur = mp->base; } } void wskbd_init_keymap(int newlen, struct wscons_keymap **map, int *maplen) { int i; if (newlen != *maplen) { if (*maplen > 0) free(*map, M_TEMP); *maplen = newlen; *map = malloc(newlen*sizeof(struct wscons_keymap), M_TEMP, M_WAITOK); } for (i = 0; i < *maplen; i++) { (*map)[i].command = KS_voidSymbol; (*map)[i].group1[0] = KS_voidSymbol; (*map)[i].group1[1] = KS_voidSymbol; (*map)[i].group2[0] = KS_voidSymbol; (*map)[i].group2[1] = KS_voidSymbol; } } int wskbd_load_keymap(const struct wskbd_mapdata *mapdata, struct wscons_keymap **map, int *maplen) { int i, s, kc, stack_ptr; const keysym_t *kp; const struct wscons_keydesc *mp, *stack[10]; kbd_t cur; for (cur = mapdata->layout & ~KB_HANDLEDBYWSKBD, stack_ptr = 0; cur != 0; stack_ptr++) { mp = mapdata->keydesc; while (mp->map_size > 0) { if (cur == 0 || mp->name == cur) { break; } mp++; } if (stack_ptr == __arraycount(stack)) panic("wskbd_load_keymap: %d: recursion too deep", mapdata->layout); if (mp->map_size <= 0) return(EINVAL); stack[stack_ptr] = mp; cur = mp->base; } for (i = 0, s = stack_ptr - 1; s >= 0; s--) { mp = stack[s]; for (kp = mp->map; kp < mp->map + mp->map_size; kp++) if (KS_GROUP(*kp) == KS_GROUP_Keycode && KS_VALUE(*kp) > i) i = KS_VALUE(*kp); } wskbd_init_keymap(i + 1, map, maplen); for (s = stack_ptr - 1; s >= 0; s--) { mp = stack[s]; for (kp = mp->map; kp < mp->map + mp->map_size; ) { if (KS_GROUP(*kp) != KS_GROUP_Keycode) panic("wskbd_load_keymap: %d(%d): bad entry", mp->name, *kp); kc = KS_VALUE(*kp); kp++; if (KS_GROUP(*kp) == KS_GROUP_Command || *kp == KS_Cmd || *kp == KS_Cmd1 || *kp == KS_Cmd2) { (*map)[kc].command = *kp; kp++; } for (i = 0; kp + i < mp->map + mp->map_size; i++) if (KS_GROUP(kp[i]) == KS_GROUP_Keycode) break; if (i > 4) panic("wskbd_load_keymap: %d(%d): bad entry", mp->name, *kp); fillmapentry(kp, i, &(*map)[kc]); kp += i; } } return(0); } |
| 8 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 197 198 199 200 201 202 203 204 205 206 207 208 209 210 211 212 213 214 215 216 217 218 219 220 221 222 223 224 225 226 227 228 229 230 231 232 233 234 235 236 237 238 239 240 241 242 243 244 245 246 247 248 249 250 251 252 253 254 255 256 257 258 259 260 261 262 263 264 265 266 267 268 269 270 271 272 273 274 275 276 277 278 279 280 281 282 283 284 285 286 287 288 289 290 291 292 293 294 295 296 297 298 299 300 301 302 303 304 305 306 307 308 309 310 311 312 313 314 315 316 317 318 319 320 321 322 323 324 325 326 327 328 329 330 331 332 333 334 335 336 337 338 339 340 341 342 343 344 345 346 347 348 349 350 351 352 353 354 355 356 357 358 359 360 361 362 363 364 365 366 367 368 369 370 371 372 373 374 375 376 377 378 379 380 381 382 383 384 385 386 387 388 389 390 391 392 393 394 395 396 397 398 399 400 401 402 403 404 405 406 407 408 409 410 411 412 413 414 415 416 417 418 419 420 421 422 423 424 425 426 427 428 429 430 431 432 433 434 435 436 437 438 439 440 441 442 443 444 445 446 447 448 449 450 451 452 453 454 455 456 457 458 459 460 461 462 463 464 465 466 467 468 469 470 471 472 473 474 475 476 477 478 479 480 481 482 483 484 485 486 487 488 489 490 491 492 493 494 495 496 497 498 499 500 501 502 503 504 505 506 507 508 509 510 511 512 513 514 515 516 517 518 519 520 521 522 523 524 525 526 527 528 529 530 531 532 533 534 535 536 537 538 539 540 541 542 543 544 545 546 547 548 549 550 551 552 553 554 555 556 557 558 559 560 561 562 563 564 565 566 567 568 569 570 571 572 573 574 575 576 577 578 579 580 581 582 583 584 585 586 587 588 589 590 591 592 593 594 595 596 597 598 599 600 601 602 603 604 605 606 607 608 609 610 611 612 613 614 615 616 617 618 619 620 621 622 623 624 625 626 627 628 629 630 631 632 633 634 635 636 637 638 639 640 641 642 643 644 645 646 647 648 649 650 651 652 653 654 655 656 657 658 659 660 661 662 663 664 665 666 667 668 669 670 671 672 673 674 675 676 677 678 679 680 681 682 683 684 685 686 687 688 689 690 691 692 693 694 695 696 697 698 699 700 701 702 703 704 705 706 707 708 709 710 711 712 713 714 715 716 717 718 719 720 721 722 723 724 725 726 727 728 729 730 731 732 733 734 735 736 737 738 739 740 741 742 743 744 745 746 747 748 749 750 751 752 753 754 755 756 757 758 759 760 761 762 763 764 765 766 767 768 769 770 771 772 773 774 775 776 777 778 779 780 781 782 783 784 785 786 787 788 789 790 791 792 793 794 795 796 797 798 799 800 801 802 803 804 805 806 807 808 809 810 811 812 813 814 815 816 817 818 819 820 821 822 823 824 825 826 827 828 829 830 831 832 833 834 835 836 837 838 839 840 841 842 843 844 845 846 847 848 849 850 851 852 853 854 855 856 857 858 859 860 861 862 863 864 865 866 867 868 869 870 871 872 873 874 875 876 877 878 879 880 881 882 883 884 885 886 887 888 889 890 891 892 893 894 895 896 897 898 899 900 901 902 903 904 905 906 907 908 909 910 911 912 913 914 915 916 917 918 919 920 921 922 923 924 925 926 927 928 929 930 931 932 933 934 935 936 937 938 939 940 941 942 943 944 945 946 947 948 949 950 951 952 953 954 955 956 957 958 959 960 961 962 963 964 965 966 967 968 969 970 971 972 973 974 975 976 977 978 979 980 981 982 983 984 985 986 987 988 989 990 991 992 993 994 995 996 997 998 999 1000 1001 1002 1003 1004 1005 1006 1007 1008 1009 1010 1011 1012 1013 1014 1015 1016 1017 1018 1019 1020 1021 1022 1023 1024 1025 1026 1027 1028 1029 1030 1031 1032 1033 1034 1035 1036 1037 1038 1039 1040 1041 1042 1043 1044 1045 1046 1047 1048 1049 1050 1051 1052 1053 1054 1055 1056 1057 1058 1059 1060 1061 1062 1063 1064 1065 1066 1067 1068 1069 1070 1071 1072 1073 1074 1075 1076 1077 1078 1079 1080 1081 1082 1083 1084 1085 1086 1087 1088 1089 1090 1091 1092 1093 1094 1095 1096 1097 1098 1099 1100 1101 1102 1103 1104 1105 1106 1107 1108 1109 1110 1111 1112 1113 1114 1115 1116 1117 1118 1119 1120 1121 1122 1123 1124 1125 1126 1127 1128 1129 1130 1131 1132 1133 1134 1135 1136 1137 1138 1139 1140 1141 1142 1143 1144 1145 1146 1147 1148 1149 1150 1151 1152 1153 1154 1155 1156 1157 1158 1159 1160 1161 1162 1163 1164 1165 1166 1167 1168 1169 1170 1171 1172 1173 1174 1175 1176 1177 1178 1179 1180 1181 1182 1183 1184 1185 1186 1187 1188 1189 1190 1191 1192 1193 1194 1195 1196 1197 1198 1199 1200 1201 1202 1203 1204 1205 1206 1207 1208 1209 1210 1211 1212 1213 1214 1215 1216 1217 1218 1219 1220 1221 1222 1223 1224 1225 1226 1227 1228 1229 1230 1231 1232 1233 1234 1235 1236 1237 1238 1239 1240 1241 1242 1243 1244 1245 1246 1247 1248 1249 1250 1251 1252 1253 1254 1255 1256 1257 1258 1259 1260 1261 1262 1263 1264 1265 1266 1267 1268 1269 1270 1271 1272 1273 1274 1275 1276 1277 1278 1279 1280 1281 1282 1283 1284 1285 1286 1287 1288 1289 1290 1291 1292 1293 1294 1295 1296 1297 1298 1299 1300 1301 1302 1303 1304 1305 1306 1307 1308 1309 1310 1311 1312 1313 1314 1315 1316 1317 1318 1319 1320 1321 1322 1323 1324 1325 1326 1327 1328 1329 1330 1331 1332 1333 1334 1335 1336 1337 1338 1339 1340 1341 1342 1343 1344 1345 1346 1347 1348 1349 1350 1351 1352 1353 1354 1355 1356 1357 1358 1359 1360 1361 1362 1363 1364 1365 1366 1367 1368 1369 1370 1371 1372 1373 1374 1375 1376 1377 1378 1379 1380 1381 1382 1383 1384 1385 1386 1387 1388 1389 1390 1391 1392 1393 1394 1395 1396 1397 1398 1399 1400 1401 1402 1403 1404 1405 1406 1407 1408 1409 1410 1411 1412 1413 1414 1415 1416 1417 1418 1419 1420 1421 1422 1423 1424 1425 1426 1427 1428 1429 1430 1431 1432 1433 1434 1435 1436 1437 1438 1439 1440 1441 1442 1443 1444 1445 1446 1447 1448 1449 1450 1451 1452 1453 1454 1455 1456 1457 1458 1459 1460 1461 1462 1463 1464 1465 1466 1467 1468 1469 1470 1471 1472 1473 1474 1475 1476 1477 1478 1479 1480 1481 1482 1483 1484 1485 1486 1487 1488 1489 1490 1491 1492 1493 1494 1495 1496 1497 1498 1499 1500 1501 1502 1503 1504 1505 1506 1507 1508 1509 1510 1511 1512 1513 1514 1515 1516 1517 1518 1519 1520 1521 1522 1523 1524 1525 1526 1527 1528 1529 1530 1531 1532 1533 1534 1535 1536 1537 1538 1539 1540 1541 1542 1543 1544 1545 1546 1547 1548 1549 1550 1551 1552 1553 1554 1555 1556 1557 1558 1559 1560 1561 1562 1563 1564 1565 1566 1567 1568 1569 1570 1571 1572 1573 1574 1575 1576 1577 1578 1579 1580 1581 1582 1583 1584 1585 1586 1587 1588 1589 1590 1591 1592 1593 1594 1595 1596 1597 1598 1599 1600 1601 1602 1603 1604 1605 1606 1607 1608 1609 1610 1611 1612 1613 1614 1615 1616 1617 1618 1619 1620 1621 1622 1623 1624 1625 1626 1627 1628 1629 1630 1631 1632 1633 1634 1635 1636 1637 1638 1639 1640 1641 1642 1643 1644 1645 1646 1647 1648 1649 1650 1651 1652 1653 1654 1655 1656 1657 1658 1659 1660 1661 1662 1663 1664 1665 1666 1667 1668 1669 1670 1671 1672 1673 1674 1675 1676 1677 1678 1679 1680 1681 1682 1683 1684 1685 1686 1687 1688 1689 1690 1691 1692 1693 1694 1695 1696 1697 1698 1699 1700 1701 1702 1703 1704 1705 1706 1707 1708 1709 1710 1711 1712 1713 1714 1715 1716 1717 1718 1719 1720 1721 1722 1723 1724 1725 1726 1727 1728 1729 1730 1731 1732 1733 1734 1735 1736 1737 1738 1739 1740 1741 1742 1743 1744 1745 1746 1747 1748 1749 1750 1751 1752 1753 1754 1755 1756 1757 1758 1759 1760 1761 1762 1763 1764 1765 1766 1767 1768 1769 1770 1771 1772 1773 1774 1775 1776 1777 1778 1779 1780 1781 1782 1783 1784 1785 1786 1787 1788 1789 1790 1791 1792 1793 1794 1795 1796 1797 1798 1799 1800 1801 1802 1803 1804 1805 1806 1807 1808 1809 1810 1811 1812 1813 1814 1815 1816 1817 1818 1819 1820 1821 1822 1823 1824 1825 1826 1827 1828 1829 1830 1831 1832 1833 1834 1835 1836 1837 1838 1839 1840 1841 1842 1843 1844 1845 1846 1847 1848 1849 1850 1851 1852 1853 1854 1855 1856 1857 1858 1859 1860 1861 1862 1863 1864 1865 1866 1867 1868 1869 1870 1871 1872 1873 1874 1875 1876 1877 1878 1879 1880 1881 1882 1883 1884 1885 1886 1887 1888 1889 1890 1891 1892 1893 1894 1895 1896 1897 1898 1899 1900 1901 1902 1903 1904 1905 1906 1907 1908 1909 1910 1911 1912 1913 1914 1915 1916 1917 1918 1919 1920 1921 1922 1923 1924 1925 1926 1927 1928 1929 1930 1931 1932 1933 1934 1935 1936 1937 1938 1939 1940 1941 1942 1943 1944 1945 1946 1947 1948 1949 1950 1951 1952 1953 1954 1955 1956 1957 1958 1959 1960 1961 1962 1963 1964 1965 1966 1967 1968 1969 1970 1971 1972 1973 1974 1975 1976 1977 1978 1979 1980 1981 1982 1983 1984 1985 1986 1987 1988 1989 1990 1991 1992 1993 1994 1995 1996 1997 1998 1999 2000 2001 2002 2003 2004 2005 2006 2007 2008 2009 2010 2011 2012 2013 2014 2015 2016 2017 2018 2019 2020 2021 2022 2023 2024 2025 2026 2027 2028 2029 2030 2031 2032 2033 2034 2035 2036 2037 2038 2039 2040 2041 2042 2043 2044 2045 2046 2047 2048 2049 2050 2051 2052 2053 2054 2055 2056 2057 2058 2059 2060 2061 2062 2063 2064 2065 2066 2067 2068 2069 2070 2071 2072 2073 2074 2075 2076 2077 2078 2079 2080 2081 2082 2083 2084 2085 2086 2087 2088 2089 2090 2091 2092 2093 2094 2095 2096 2097 2098 2099 2100 2101 2102 2103 2104 2105 2106 2107 2108 2109 2110 2111 2112 2113 2114 2115 2116 2117 2118 2119 2120 2121 2122 2123 2124 2125 2126 2127 2128 2129 2130 2131 2132 2133 2134 2135 2136 2137 2138 2139 2140 2141 2142 2143 2144 2145 2146 2147 2148 2149 2150 2151 2152 2153 2154 2155 2156 2157 2158 2159 2160 2161 2162 2163 2164 2165 2166 2167 2168 2169 2170 2171 2172 2173 2174 2175 2176 2177 2178 2179 2180 2181 2182 2183 2184 2185 2186 2187 2188 2189 2190 2191 2192 2193 2194 2195 2196 2197 2198 2199 2200 2201 2202 2203 2204 2205 2206 2207 2208 2209 2210 2211 2212 2213 2214 2215 2216 2217 2218 2219 2220 2221 2222 2223 2224 2225 2226 2227 2228 2229 2230 2231 2232 2233 2234 2235 2236 2237 2238 2239 2240 2241 2242 2243 2244 2245 2246 2247 2248 2249 2250 2251 2252 2253 2254 2255 2256 2257 2258 2259 2260 2261 2262 2263 2264 2265 2266 2267 2268 2269 2270 2271 2272 2273 2274 2275 2276 2277 2278 2279 2280 2281 2282 2283 2284 2285 2286 2287 2288 2289 2290 2291 2292 2293 2294 2295 2296 2297 2298 2299 2300 2301 2302 2303 2304 2305 2306 2307 2308 2309 2310 2311 2312 2313 2314 2315 2316 2317 2318 2319 2320 2321 2322 2323 2324 2325 2326 2327 2328 2329 2330 2331 2332 2333 2334 2335 2336 2337 2338 2339 2340 2341 2342 2343 2344 2345 2346 2347 2348 2349 2350 2351 2352 2353 2354 2355 2356 2357 2358 2359 2360 2361 2362 2363 2364 2365 2366 2367 2368 2369 2370 2371 2372 2373 2374 2375 2376 2377 2378 2379 2380 2381 2382 2383 2384 2385 2386 2387 2388 2389 2390 2391 2392 2393 2394 2395 2396 2397 2398 2399 2400 2401 2402 2403 2404 2405 2406 2407 2408 2409 2410 2411 2412 2413 2414 2415 2416 2417 2418 2419 2420 2421 2422 2423 2424 2425 2426 2427 2428 2429 2430 2431 2432 2433 2434 2435 2436 2437 2438 2439 2440 2441 2442 2443 2444 2445 2446 2447 2448 2449 2450 2451 2452 2453 2454 2455 2456 2457 2458 2459 2460 2461 2462 2463 2464 2465 2466 2467 2468 2469 2470 2471 2472 2473 2474 2475 2476 2477 2478 2479 2480 2481 2482 2483 2484 2485 2486 2487 2488 2489 2490 2491 2492 2493 2494 2495 2496 2497 2498 2499 2500 2501 2502 2503 2504 2505 2506 2507 2508 2509 2510 2511 2512 2513 2514 2515 2516 2517 2518 2519 2520 2521 2522 2523 2524 2525 2526 2527 2528 2529 2530 2531 2532 2533 2534 2535 2536 2537 2538 2539 2540 2541 2542 2543 2544 2545 2546 2547 2548 2549 2550 2551 2552 2553 2554 2555 2556 2557 2558 2559 2560 2561 2562 2563 2564 2565 2566 2567 2568 2569 2570 2571 2572 2573 2574 2575 2576 2577 2578 2579 2580 2581 2582 2583 2584 2585 2586 2587 2588 2589 2590 2591 2592 2593 2594 2595 2596 2597 2598 2599 2600 2601 2602 2603 2604 2605 2606 2607 2608 2609 2610 2611 2612 2613 2614 2615 2616 2617 2618 2619 2620 2621 2622 2623 2624 2625 2626 2627 2628 2629 2630 2631 2632 2633 2634 2635 2636 2637 2638 2639 2640 2641 2642 2643 2644 2645 2646 2647 2648 2649 2650 2651 2652 2653 2654 2655 2656 2657 2658 2659 2660 2661 2662 2663 2664 2665 2666 2667 2668 2669 2670 2671 2672 2673 2674 2675 2676 2677 2678 2679 2680 2681 2682 2683 2684 2685 2686 2687 2688 2689 2690 2691 2692 2693 2694 2695 2696 2697 2698 2699 2700 2701 2702 2703 2704 2705 2706 2707 2708 2709 2710 2711 2712 2713 2714 2715 2716 2717 2718 2719 2720 2721 2722 2723 2724 2725 2726 2727 2728 2729 2730 2731 2732 2733 2734 2735 2736 2737 2738 2739 2740 2741 2742 2743 2744 2745 2746 2747 2748 2749 2750 2751 2752 2753 2754 2755 2756 2757 2758 2759 2760 2761 2762 2763 2764 2765 2766 2767 2768 2769 2770 2771 2772 2773 2774 2775 2776 2777 2778 2779 2780 2781 2782 2783 2784 2785 2786 2787 2788 2789 2790 2791 2792 2793 2794 2795 2796 2797 2798 2799 2800 2801 2802 2803 2804 2805 2806 2807 2808 2809 2810 2811 2812 2813 2814 2815 2816 2817 2818 2819 2820 2821 2822 2823 2824 2825 2826 2827 2828 2829 2830 2831 2832 2833 2834 2835 2836 2837 2838 2839 2840 2841 2842 2843 2844 2845 2846 2847 2848 2849 2850 2851 2852 2853 2854 2855 2856 2857 2858 2859 2860 2861 2862 2863 2864 2865 2866 2867 2868 2869 2870 2871 2872 2873 2874 2875 2876 2877 2878 2879 2880 2881 2882 2883 2884 2885 2886 2887 2888 2889 2890 2891 2892 2893 2894 2895 2896 2897 2898 2899 2900 2901 2902 2903 2904 2905 2906 2907 2908 2909 2910 2911 2912 2913 2914 2915 2916 2917 2918 2919 2920 2921 2922 2923 2924 2925 2926 2927 2928 2929 2930 2931 2932 2933 2934 2935 2936 2937 2938 2939 2940 2941 2942 2943 2944 2945 2946 2947 2948 2949 2950 2951 2952 2953 2954 2955 2956 2957 2958 2959 2960 2961 2962 2963 2964 2965 2966 2967 2968 2969 2970 2971 2972 2973 2974 2975 2976 2977 2978 2979 2980 2981 2982 2983 2984 2985 2986 2987 2988 2989 2990 2991 2992 2993 2994 2995 2996 2997 2998 2999 3000 3001 3002 3003 3004 3005 3006 3007 3008 3009 3010 3011 3012 3013 3014 3015 3016 3017 3018 3019 3020 3021 3022 3023 3024 3025 3026 3027 3028 3029 3030 3031 3032 3033 3034 3035 3036 3037 3038 3039 3040 3041 3042 3043 3044 3045 3046 3047 3048 3049 3050 3051 3052 3053 3054 3055 3056 3057 3058 3059 3060 3061 3062 3063 3064 3065 3066 3067 3068 3069 3070 3071 3072 3073 3074 3075 3076 3077 3078 3079 3080 3081 3082 3083 3084 3085 3086 3087 3088 3089 3090 3091 3092 3093 3094 3095 3096 3097 3098 3099 3100 3101 3102 3103 3104 3105 3106 3107 3108 3109 3110 3111 3112 3113 3114 3115 3116 3117 3118 3119 3120 3121 3122 3123 3124 3125 3126 3127 3128 3129 3130 3131 3132 3133 3134 3135 3136 3137 3138 3139 3140 3141 3142 3143 3144 3145 3146 3147 3148 3149 3150 3151 3152 3153 3154 3155 3156 3157 3158 3159 3160 3161 3162 3163 3164 3165 3166 3167 3168 3169 3170 3171 3172 3173 3174 3175 3176 3177 3178 3179 3180 3181 3182 3183 3184 3185 3186 3187 3188 3189 3190 3191 3192 3193 3194 3195 3196 3197 3198 3199 3200 3201 3202 3203 3204 3205 3206 3207 3208 3209 3210 3211 3212 3213 3214 3215 3216 3217 3218 3219 3220 3221 3222 3223 3224 3225 3226 3227 3228 3229 3230 3231 3232 3233 3234 3235 3236 3237 3238 3239 3240 3241 3242 3243 3244 3245 3246 3247 3248 3249 3250 3251 3252 3253 3254 3255 3256 3257 3258 3259 3260 3261 3262 3263 3264 3265 3266 3267 3268 3269 3270 3271 3272 3273 3274 3275 3276 3277 3278 3279 3280 3281 3282 3283 3284 3285 3286 3287 3288 3289 3290 3291 3292 3293 3294 3295 3296 3297 3298 3299 3300 3301 3302 3303 3304 3305 3306 3307 3308 3309 3310 3311 3312 3313 3314 3315 3316 3317 3318 3319 3320 3321 3322 3323 3324 3325 3326 3327 3328 3329 3330 3331 3332 3333 3334 3335 3336 3337 3338 3339 3340 3341 3342 3343 3344 3345 3346 3347 3348 3349 3350 3351 3352 3353 3354 3355 3356 3357 3358 3359 3360 3361 3362 3363 3364 3365 3366 3367 3368 3369 3370 3371 3372 3373 3374 3375 3376 3377 3378 3379 3380 3381 3382 3383 3384 3385 3386 3387 3388 3389 3390 3391 3392 3393 3394 3395 3396 3397 3398 3399 3400 3401 3402 3403 3404 3405 3406 3407 3408 3409 3410 3411 3412 3413 3414 3415 3416 3417 3418 3419 3420 3421 3422 3423 3424 3425 3426 3427 3428 3429 3430 3431 3432 3433 3434 3435 3436 3437 3438 3439 3440 3441 3442 3443 3444 3445 3446 3447 3448 3449 3450 3451 3452 3453 3454 3455 3456 3457 3458 3459 3460 3461 3462 3463 3464 3465 3466 3467 3468 3469 3470 3471 3472 3473 3474 3475 3476 3477 3478 3479 3480 3481 3482 3483 3484 3485 3486 3487 3488 3489 3490 3491 3492 3493 3494 3495 3496 3497 3498 3499 3500 3501 3502 3503 3504 3505 3506 3507 3508 3509 3510 3511 3512 3513 3514 3515 3516 3517 3518 3519 3520 3521 3522 3523 3524 3525 3526 3527 3528 3529 3530 3531 3532 3533 3534 3535 3536 3537 3538 3539 3540 3541 3542 3543 3544 3545 3546 3547 3548 3549 3550 3551 3552 3553 3554 3555 3556 3557 3558 3559 3560 3561 3562 3563 3564 3565 3566 3567 3568 3569 3570 3571 3572 3573 3574 3575 3576 3577 3578 3579 3580 3581 3582 3583 3584 3585 3586 3587 3588 3589 3590 3591 3592 3593 3594 3595 3596 3597 3598 3599 3600 3601 3602 3603 3604 3605 3606 3607 3608 3609 3610 3611 3612 3613 3614 3615 3616 3617 3618 3619 3620 3621 3622 3623 3624 3625 3626 3627 3628 3629 3630 3631 3632 3633 3634 3635 3636 3637 3638 3639 3640 3641 3642 3643 3644 3645 3646 3647 3648 3649 3650 3651 3652 3653 3654 3655 3656 3657 3658 3659 3660 3661 3662 3663 3664 3665 3666 3667 3668 3669 3670 3671 3672 3673 3674 3675 3676 3677 3678 3679 3680 3681 3682 3683 3684 3685 3686 3687 3688 3689 3690 3691 3692 3693 3694 3695 3696 3697 3698 3699 3700 3701 3702 3703 3704 3705 3706 3707 3708 3709 3710 3711 3712 3713 3714 3715 3716 3717 3718 3719 3720 3721 3722 3723 3724 3725 3726 3727 3728 3729 3730 3731 3732 3733 3734 3735 3736 3737 3738 3739 3740 3741 3742 3743 3744 3745 3746 3747 3748 3749 3750 3751 3752 3753 3754 3755 3756 3757 3758 3759 3760 3761 3762 3763 3764 3765 3766 3767 3768 3769 3770 3771 3772 3773 3774 3775 3776 3777 3778 3779 3780 3781 3782 3783 3784 3785 3786 3787 3788 3789 3790 3791 3792 3793 3794 3795 3796 3797 3798 3799 3800 3801 3802 3803 3804 3805 3806 3807 3808 3809 3810 3811 3812 3813 3814 3815 3816 3817 3818 3819 3820 3821 3822 3823 3824 3825 3826 3827 3828 3829 3830 3831 3832 3833 3834 3835 3836 3837 3838 3839 3840 3841 3842 3843 3844 3845 3846 3847 3848 3849 3850 3851 3852 3853 3854 3855 3856 3857 3858 3859 3860 3861 3862 3863 3864 3865 3866 3867 3868 3869 3870 3871 3872 3873 3874 3875 3876 3877 3878 3879 3880 3881 3882 3883 3884 3885 3886 3887 3888 3889 3890 3891 3892 3893 3894 3895 3896 3897 3898 3899 3900 3901 3902 3903 3904 3905 3906 3907 3908 3909 3910 3911 3912 3913 3914 3915 3916 3917 3918 3919 3920 3921 3922 3923 3924 3925 3926 3927 3928 3929 3930 3931 3932 3933 3934 3935 3936 3937 3938 3939 3940 3941 3942 3943 3944 3945 3946 3947 3948 3949 3950 3951 3952 3953 3954 3955 3956 3957 3958 3959 3960 3961 3962 3963 3964 3965 3966 3967 3968 3969 3970 3971 3972 3973 3974 3975 3976 3977 3978 3979 3980 3981 3982 3983 3984 3985 3986 3987 3988 3989 3990 3991 3992 3993 3994 3995 3996 3997 3998 3999 4000 4001 4002 4003 4004 4005 4006 4007 4008 4009 4010 4011 4012 4013 4014 4015 4016 4017 4018 4019 4020 4021 4022 4023 4024 4025 4026 4027 4028 4029 4030 4031 4032 4033 4034 4035 4036 4037 4038 4039 4040 4041 4042 4043 4044 4045 4046 4047 4048 4049 4050 4051 4052 4053 4054 4055 4056 4057 4058 4059 4060 4061 4062 4063 4064 4065 4066 4067 4068 4069 4070 4071 4072 4073 4074 4075 4076 4077 4078 4079 4080 4081 4082 4083 4084 4085 4086 4087 4088 4089 4090 4091 4092 4093 4094 4095 4096 4097 4098 4099 4100 4101 4102 4103 4104 4105 4106 4107 4108 4109 4110 4111 4112 4113 4114 4115 4116 4117 4118 4119 4120 4121 4122 4123 4124 4125 4126 4127 4128 4129 4130 4131 4132 4133 4134 4135 4136 4137 4138 4139 4140 4141 4142 4143 4144 4145 4146 4147 4148 4149 4150 4151 4152 4153 4154 4155 4156 4157 4158 4159 4160 4161 4162 4163 4164 4165 4166 4167 4168 4169 4170 4171 4172 4173 4174 4175 4176 4177 4178 4179 4180 4181 4182 4183 4184 4185 4186 4187 4188 4189 4190 4191 4192 4193 4194 4195 4196 4197 4198 4199 4200 4201 4202 4203 4204 4205 4206 4207 4208 4209 4210 4211 4212 4213 4214 4215 4216 4217 4218 4219 4220 4221 4222 4223 4224 4225 4226 4227 4228 4229 4230 4231 4232 4233 4234 4235 4236 4237 4238 4239 4240 4241 4242 4243 4244 4245 4246 4247 4248 4249 4250 4251 4252 4253 4254 4255 4256 4257 4258 4259 4260 4261 4262 4263 4264 4265 4266 4267 4268 4269 4270 4271 4272 4273 4274 4275 4276 4277 4278 4279 4280 4281 4282 4283 4284 4285 4286 4287 4288 4289 4290 4291 4292 4293 4294 4295 4296 4297 4298 4299 4300 4301 4302 4303 4304 4305 4306 4307 4308 4309 4310 4311 4312 4313 4314 4315 4316 4317 4318 4319 4320 4321 4322 4323 4324 4325 4326 4327 4328 4329 4330 4331 4332 4333 4334 4335 4336 4337 4338 4339 4340 4341 4342 4343 4344 4345 4346 4347 4348 4349 4350 4351 4352 4353 4354 4355 4356 4357 4358 4359 4360 4361 4362 4363 4364 4365 4366 4367 4368 4369 4370 4371 4372 4373 4374 4375 4376 4377 4378 4379 4380 4381 4382 4383 4384 4385 4386 4387 4388 4389 4390 4391 4392 4393 4394 4395 4396 4397 4398 4399 4400 4401 4402 4403 4404 4405 4406 4407 4408 4409 4410 4411 4412 4413 4414 4415 4416 4417 4418 4419 4420 4421 4422 4423 4424 4425 4426 4427 4428 4429 4430 4431 4432 4433 4434 4435 4436 4437 4438 4439 4440 4441 4442 4443 4444 4445 4446 4447 4448 4449 4450 4451 4452 4453 4454 4455 4456 4457 4458 4459 4460 4461 4462 4463 4464 4465 4466 4467 4468 4469 4470 4471 4472 4473 4474 4475 4476 4477 4478 4479 4480 4481 4482 4483 4484 4485 4486 4487 4488 4489 4490 4491 4492 4493 4494 4495 4496 4497 4498 4499 4500 4501 4502 4503 4504 4505 4506 4507 4508 4509 4510 4511 4512 4513 4514 4515 4516 4517 4518 4519 4520 4521 4522 4523 4524 4525 4526 4527 4528 4529 4530 4531 4532 4533 4534 4535 4536 4537 4538 4539 4540 4541 4542 4543 4544 4545 4546 4547 4548 4549 4550 4551 4552 4553 4554 4555 4556 4557 4558 4559 4560 4561 4562 4563 4564 4565 4566 4567 4568 4569 4570 4571 4572 4573 4574 4575 4576 4577 4578 4579 4580 4581 4582 4583 4584 4585 4586 4587 4588 4589 4590 4591 4592 4593 4594 4595 4596 4597 4598 4599 4600 4601 4602 4603 4604 4605 4606 4607 4608 4609 4610 4611 4612 4613 4614 4615 4616 4617 4618 4619 4620 4621 4622 4623 4624 4625 4626 4627 4628 4629 4630 4631 4632 4633 4634 4635 4636 4637 4638 4639 4640 4641 4642 4643 4644 4645 4646 4647 4648 4649 4650 4651 4652 4653 4654 4655 4656 4657 4658 4659 4660 4661 4662 4663 4664 4665 4666 4667 4668 4669 4670 4671 4672 4673 4674 4675 4676 4677 4678 4679 4680 4681 4682 4683 4684 4685 4686 4687 4688 4689 4690 4691 4692 4693 4694 4695 4696 4697 4698 4699 4700 4701 4702 4703 4704 4705 4706 4707 4708 4709 4710 4711 4712 4713 4714 4715 4716 4717 4718 4719 4720 4721 4722 4723 4724 4725 4726 4727 4728 4729 4730 4731 4732 4733 4734 4735 4736 4737 4738 4739 4740 4741 4742 4743 4744 4745 4746 4747 4748 4749 4750 4751 4752 4753 4754 4755 4756 4757 4758 4759 4760 4761 4762 4763 4764 4765 4766 4767 4768 4769 4770 4771 4772 4773 4774 4775 4776 4777 4778 4779 4780 4781 4782 4783 4784 4785 4786 4787 4788 4789 4790 4791 4792 4793 4794 4795 4796 4797 4798 4799 4800 4801 4802 4803 4804 4805 4806 4807 4808 4809 4810 4811 4812 4813 4814 4815 4816 4817 4818 4819 4820 4821 4822 4823 4824 4825 4826 4827 4828 4829 4830 4831 4832 4833 4834 4835 4836 4837 4838 4839 4840 4841 4842 4843 4844 4845 4846 4847 4848 4849 4850 4851 4852 4853 4854 4855 4856 4857 4858 4859 4860 4861 4862 4863 4864 4865 4866 4867 4868 4869 4870 4871 4872 4873 4874 4875 4876 4877 4878 4879 4880 4881 4882 4883 4884 4885 4886 4887 4888 4889 4890 4891 4892 4893 4894 4895 4896 4897 4898 4899 4900 4901 4902 4903 4904 4905 4906 4907 4908 4909 4910 4911 4912 4913 4914 4915 4916 4917 4918 4919 4920 4921 4922 4923 4924 4925 4926 4927 4928 4929 4930 4931 4932 4933 4934 4935 4936 4937 4938 4939 4940 4941 4942 4943 4944 4945 4946 4947 4948 4949 4950 4951 4952 4953 4954 4955 4956 4957 4958 4959 4960 4961 4962 4963 4964 4965 4966 4967 4968 4969 4970 4971 4972 4973 4974 4975 4976 4977 4978 4979 4980 4981 4982 4983 4984 4985 4986 4987 4988 4989 4990 4991 4992 4993 4994 4995 4996 4997 4998 4999 5000 5001 5002 5003 5004 5005 5006 5007 5008 5009 5010 5011 5012 5013 5014 5015 5016 5017 5018 5019 5020 5021 5022 5023 5024 5025 5026 5027 5028 5029 5030 5031 5032 5033 5034 5035 5036 5037 5038 5039 5040 5041 5042 5043 5044 5045 5046 5047 5048 5049 5050 5051 5052 5053 5054 5055 5056 5057 5058 5059 5060 5061 5062 5063 5064 5065 5066 5067 5068 5069 5070 5071 5072 5073 5074 5075 5076 5077 5078 5079 5080 5081 5082 5083 5084 5085 5086 5087 5088 5089 5090 5091 5092 5093 5094 5095 5096 5097 5098 5099 5100 5101 5102 5103 5104 5105 5106 5107 5108 5109 5110 5111 5112 5113 5114 5115 5116 5117 5118 5119 5120 5121 5122 5123 5124 5125 5126 5127 5128 5129 5130 5131 5132 5133 5134 5135 5136 5137 5138 5139 5140 5141 5142 5143 5144 5145 5146 5147 5148 5149 5150 5151 5152 5153 5154 5155 5156 5157 5158 5159 5160 5161 5162 5163 5164 5165 5166 5167 5168 5169 5170 5171 5172 5173 5174 5175 5176 5177 5178 5179 5180 5181 5182 5183 5184 5185 5186 5187 5188 5189 5190 5191 5192 5193 5194 5195 5196 5197 5198 5199 5200 5201 5202 5203 5204 5205 5206 5207 5208 5209 5210 5211 5212 5213 5214 5215 5216 5217 5218 5219 5220 5221 5222 5223 5224 5225 5226 5227 5228 5229 5230 | /* $NetBSD: if_urtwn.c,v 1.105 2022/07/31 12:59:26 mlelstv Exp $ */ /* $OpenBSD: if_urtwn.c,v 1.42 2015/02/10 23:25:46 mpi Exp $ */ /*- * Copyright (c) 2010 Damien Bergamini <damien.bergamini@free.fr> * Copyright (c) 2014 Kevin Lo <kevlo@FreeBSD.org> * Copyright (c) 2016 Nathanial Sloss <nathanialsloss@yahoo.com.au> * * Permission to use, copy, modify, and distribute this software for any * purpose with or without fee is hereby granted, provided that the above * copyright notice and this permission notice appear in all copies. * * THE SOFTWARE IS PROVIDED "AS IS" AND THE AUTHOR DISCLAIMS ALL WARRANTIES * WITH REGARD TO THIS SOFTWARE INCLUDING ALL IMPLIED WARRANTIES OF * MERCHANTABILITY AND FITNESS. IN NO EVENT SHALL THE AUTHOR BE LIABLE FOR * ANY SPECIAL, DIRECT, INDIRECT, OR CONSEQUENTIAL DAMAGES OR ANY DAMAGES * WHATSOEVER RESULTING FROM LOSS OF USE, DATA OR PROFITS, WHETHER IN AN * ACTION OF CONTRACT, NEGLIGENCE OR OTHER TORTIOUS ACTION, ARISING OUT OF * OR IN CONNECTION WITH THE USE OR PERFORMANCE OF THIS SOFTWARE. */ /*- * Driver for Realtek RTL8188CE-VAU/RTL8188CUS/RTL8188EU/RTL8188RU/RTL8192CU * RTL8192EU. */ #include <sys/cdefs.h> __KERNEL_RCSID(0, "$NetBSD: if_urtwn.c,v 1.105 2022/07/31 12:59:26 mlelstv Exp $"); #ifdef _KERNEL_OPT #include "opt_inet.h" #include "opt_usb.h" #endif #include <sys/param.h> #include <sys/sockio.h> #include <sys/sysctl.h> #include <sys/mbuf.h> #include <sys/kernel.h> #include <sys/socket.h> #include <sys/systm.h> #include <sys/module.h> #include <sys/conf.h> #include <sys/device.h> #include <sys/rndsource.h> #include <sys/bus.h> #include <machine/endian.h> #include <sys/intr.h> #include <net/bpf.h> #include <net/if.h> #include <net/if_arp.h> #include <net/if_dl.h> #include <net/if_ether.h> #include <net/if_media.h> #include <net/if_types.h> #include <netinet/in.h> #include <netinet/in_systm.h> #include <netinet/in_var.h> #include <netinet/ip.h> #include <netinet/if_inarp.h> #include <net80211/ieee80211_netbsd.h> #include <net80211/ieee80211_var.h> #include <net80211/ieee80211_radiotap.h> #include <dev/firmload.h> #include <dev/usb/usb.h> #include <dev/usb/usbdi.h> #include <dev/usb/usbdivar.h> #include <dev/usb/usbdi_util.h> #include <dev/usb/usbdevs.h> #include <dev/usb/usbhist.h> #include <dev/ic/rtwnreg.h> #include <dev/ic/rtwn_data.h> #include <dev/usb/if_urtwnreg.h> #include <dev/usb/if_urtwnvar.h> /* * The sc_write_mtx locking is to prevent sequences of writes from * being intermingled with each other. I don't know if this is really * needed. I have added it just to be on the safe side. */ #ifdef URTWN_DEBUG #define DBG_INIT __BIT(0) #define DBG_FN __BIT(1) #define DBG_TX __BIT(2) #define DBG_RX __BIT(3) #define DBG_STM __BIT(4) #define DBG_RF __BIT(5) #define DBG_REG __BIT(6) #define DBG_ALL 0xffffffffU u_int urtwn_debug = 0; #define DPRINTFN(n, fmt, a, b, c, d) do { \ if (urtwn_debug & (n)) { \ KERNHIST_LOG(usbhist, fmt, a, b, c, d); \ } \ } while (/*CONSTCOND*/0) #define URTWNHIST_FUNC() USBHIST_FUNC() #define URTWNHIST_CALLED() do { \ if (urtwn_debug & DBG_FN) { \ KERNHIST_CALLED(usbhist); \ } \ } while(/*CONSTCOND*/0) #define URTWNHIST_CALLARGS(fmt, a, b, c, d) do { \ if (urtwn_debug & DBG_FN) { \ KERNHIST_CALLARGS(usbhist, fmt, a, b, c, d); \ } \ } while(/*CONSTCOND*/0) #else #define DPRINTFN(n, fmt, a, b, c, d) #define URTWNHIST_FUNC() #define URTWNHIST_CALLED() #define URTWNHIST_CALLARGS(fmt, a, b, c, d) #endif #define URTWN_DEV(v,p) { { USB_VENDOR_##v, USB_PRODUCT_##v##_##p }, 0 } #define URTWN_RTL8188E_DEV(v,p) \ { { USB_VENDOR_##v, USB_PRODUCT_##v##_##p }, FLAG_RTL8188E } #define URTWN_RTL8192EU_DEV(v,p) \ { { USB_VENDOR_##v, USB_PRODUCT_##v##_##p }, FLAG_RTL8192E } static const struct urtwn_dev { struct usb_devno dev; uint32_t flags; #define FLAG_RTL8188E __BIT(0) #define FLAG_RTL8192E __BIT(1) } urtwn_devs[] = { URTWN_DEV(ABOCOM, RTL8188CU_1), URTWN_DEV(ABOCOM, RTL8188CU_2), URTWN_DEV(ABOCOM, RTL8192CU), URTWN_DEV(ASUSTEK, RTL8192CU), URTWN_DEV(ASUSTEK, RTL8192CU_3), URTWN_DEV(ASUSTEK, USBN10NANO), URTWN_DEV(ASUSTEK, RTL8192CU_3), URTWN_DEV(AZUREWAVE, RTL8188CE_1), URTWN_DEV(AZUREWAVE, RTL8188CE_2), URTWN_DEV(AZUREWAVE, RTL8188CU), URTWN_DEV(BELKIN, F7D2102), URTWN_DEV(BELKIN, RTL8188CU), URTWN_DEV(BELKIN, RTL8188CUS), URTWN_DEV(BELKIN, RTL8192CU), URTWN_DEV(BELKIN, RTL8192CU_1), URTWN_DEV(BELKIN, RTL8192CU_2), URTWN_DEV(CHICONY, RTL8188CUS_1), URTWN_DEV(CHICONY, RTL8188CUS_2), URTWN_DEV(CHICONY, RTL8188CUS_3), URTWN_DEV(CHICONY, RTL8188CUS_4), URTWN_DEV(CHICONY, RTL8188CUS_5), URTWN_DEV(CHICONY, RTL8188CUS_6), URTWN_DEV(COMPARE, RTL8192CU), URTWN_DEV(COREGA, RTL8192CU), URTWN_DEV(DLINK, DWA131B), URTWN_DEV(DLINK, RTL8188CU), URTWN_DEV(DLINK, RTL8192CU_1), URTWN_DEV(DLINK, RTL8192CU_2), URTWN_DEV(DLINK, RTL8192CU_3), URTWN_DEV(DLINK, RTL8192CU_4), URTWN_DEV(EDIMAX, RTL8188CU), URTWN_DEV(EDIMAX, RTL8192CU), URTWN_DEV(FEIXUN, RTL8188CU), URTWN_DEV(FEIXUN, RTL8192CU), URTWN_DEV(GUILLEMOT, HWNUP150), URTWN_DEV(GUILLEMOT, RTL8192CU), URTWN_DEV(HAWKING, RTL8192CU), URTWN_DEV(HAWKING, RTL8192CU_2), URTWN_DEV(HP3, RTL8188CU), URTWN_DEV(IODATA, WNG150UM), URTWN_DEV(IODATA, RTL8192CU), URTWN_DEV(NETGEAR, WNA1000M), URTWN_DEV(NETGEAR, RTL8192CU), URTWN_DEV(NETGEAR4, RTL8188CU), URTWN_DEV(NOVATECH, RTL8188CU), URTWN_DEV(PLANEX2, RTL8188CU_1), URTWN_DEV(PLANEX2, RTL8188CU_2), URTWN_DEV(PLANEX2, RTL8192CU), URTWN_DEV(PLANEX2, RTL8188CU_3), URTWN_DEV(PLANEX2, RTL8188CU_4), URTWN_DEV(PLANEX2, RTL8188CUS), URTWN_DEV(REALTEK, RTL8188CE_0), URTWN_DEV(REALTEK, RTL8188CE_1), URTWN_DEV(REALTEK, RTL8188CTV), URTWN_DEV(REALTEK, RTL8188CU_0), URTWN_DEV(REALTEK, RTL8188CU_1), URTWN_DEV(REALTEK, RTL8188CU_2), URTWN_DEV(REALTEK, RTL8188CU_3), URTWN_DEV(REALTEK, RTL8188CU_COMBO), URTWN_DEV(REALTEK, RTL8188CUS), URTWN_DEV(REALTEK, RTL8188RU), URTWN_DEV(REALTEK, RTL8188RU_2), URTWN_DEV(REALTEK, RTL8188RU_3), URTWN_DEV(REALTEK, RTL8191CU), URTWN_DEV(REALTEK, RTL8192CE), URTWN_DEV(REALTEK, RTL8192CU), URTWN_DEV(SITECOMEU, RTL8188CU), URTWN_DEV(SITECOMEU, RTL8188CU_2), URTWN_DEV(SITECOMEU, RTL8192CU), URTWN_DEV(SITECOMEU, RTL8192CUR2), URTWN_DEV(TPLINK, RTL8192CU), URTWN_DEV(TRENDNET, RTL8188CU), URTWN_DEV(TRENDNET, RTL8192CU), URTWN_DEV(TRENDNET, TEW648UBM), URTWN_DEV(ZYXEL, RTL8192CU), /* URTWN_RTL8188E */ URTWN_RTL8188E_DEV(DLINK, DWA125D1), URTWN_RTL8188E_DEV(ELECOM, WDC150SU2M), URTWN_RTL8188E_DEV(REALTEK, RTL8188ETV), URTWN_RTL8188E_DEV(REALTEK, RTL8188EU), URTWN_RTL8188E_DEV(ABOCOM, RTL8188EU), URTWN_RTL8188E_DEV(TPLINK, RTL8188EU), URTWN_RTL8188E_DEV(DLINK, DWA121B1), URTWN_RTL8188E_DEV(EDIMAX, EW7811UNV2), /* URTWN_RTL8192EU */ URTWN_RTL8192EU_DEV(DLINK, DWA131E), URTWN_RTL8192EU_DEV(REALTEK, RTL8192EU), URTWN_RTL8192EU_DEV(TPLINK, WN821NV5), URTWN_RTL8192EU_DEV(TPLINK, WN822NV4), URTWN_RTL8192EU_DEV(TPLINK, WN823NV2), }; #undef URTWN_DEV #undef URTWN_RTL8188E_DEV #undef URTWN_RTL8192EU_DEV static int urtwn_match(device_t, cfdata_t, void *); static void urtwn_attach(device_t, device_t, void *); static int urtwn_detach(device_t, int); static int urtwn_activate(device_t, enum devact); CFATTACH_DECL_NEW(urtwn, sizeof(struct urtwn_softc), urtwn_match, urtwn_attach, urtwn_detach, urtwn_activate); static int urtwn_open_pipes(struct urtwn_softc *); static void urtwn_close_pipes(struct urtwn_softc *); static int urtwn_alloc_rx_list(struct urtwn_softc *); static void urtwn_free_rx_list(struct urtwn_softc *); static int urtwn_alloc_tx_list(struct urtwn_softc *); static void urtwn_free_tx_list(struct urtwn_softc *); static void urtwn_task(void *); static void urtwn_do_async(struct urtwn_softc *, void (*)(struct urtwn_softc *, void *), void *, int); static void urtwn_wait_async(struct urtwn_softc *); static int urtwn_write_region_1(struct urtwn_softc *, uint16_t, uint8_t *, int); static void urtwn_write_1(struct urtwn_softc *, uint16_t, uint8_t); static void urtwn_write_2(struct urtwn_softc *, uint16_t, uint16_t); static void urtwn_write_4(struct urtwn_softc *, uint16_t, uint32_t); static int urtwn_write_region(struct urtwn_softc *, uint16_t, uint8_t *, int); static int urtwn_read_region_1(struct urtwn_softc *, uint16_t, uint8_t *, int); static uint8_t urtwn_read_1(struct urtwn_softc *, uint16_t); static uint16_t urtwn_read_2(struct urtwn_softc *, uint16_t); static uint32_t urtwn_read_4(struct urtwn_softc *, uint16_t); static int urtwn_fw_cmd(struct urtwn_softc *, uint8_t, const void *, int); static void urtwn_r92c_rf_write(struct urtwn_softc *, int, uint8_t, uint32_t); static void urtwn_r88e_rf_write(struct urtwn_softc *, int, uint8_t, uint32_t); static void urtwn_r92e_rf_write(struct urtwn_softc *, int, uint8_t, uint32_t); static uint32_t urtwn_rf_read(struct urtwn_softc *, int, uint8_t); static int urtwn_llt_write(struct urtwn_softc *, uint32_t, uint32_t); static uint8_t urtwn_efuse_read_1(struct urtwn_softc *, uint16_t); static void urtwn_efuse_read(struct urtwn_softc *); static void urtwn_efuse_switch_power(struct urtwn_softc *); static int urtwn_read_chipid(struct urtwn_softc *); #ifdef URTWN_DEBUG static void urtwn_dump_rom(struct urtwn_softc *, struct r92c_rom *); #endif static void urtwn_read_rom(struct urtwn_softc *); static void urtwn_r88e_read_rom(struct urtwn_softc *); static int urtwn_media_change(struct ifnet *); static int urtwn_ra_init(struct urtwn_softc *); static int urtwn_get_nettype(struct urtwn_softc *); static void urtwn_set_nettype0_msr(struct urtwn_softc *, uint8_t); static void urtwn_tsf_sync_enable(struct urtwn_softc *); static void urtwn_set_led(struct urtwn_softc *, int, int); static void urtwn_calib_to(void *); static void urtwn_calib_to_cb(struct urtwn_softc *, void *); static void urtwn_next_scan(void *); static int urtwn_newstate(struct ieee80211com *, enum ieee80211_state, int); static void urtwn_newstate_cb(struct urtwn_softc *, void *); static int urtwn_wme_update(struct ieee80211com *); static void urtwn_wme_update_cb(struct urtwn_softc *, void *); static void urtwn_update_avgrssi(struct urtwn_softc *, int, int8_t); static int8_t urtwn_get_rssi(struct urtwn_softc *, int, void *); static int8_t urtwn_r88e_get_rssi(struct urtwn_softc *, int, void *); static void urtwn_rx_frame(struct urtwn_softc *, uint8_t *, int); static void urtwn_rxeof(struct usbd_xfer *, void *, usbd_status); static void urtwn_txeof(struct usbd_xfer *, void *, usbd_status); static int urtwn_tx(struct urtwn_softc *, struct mbuf *, struct ieee80211_node *, struct urtwn_tx_data *); static struct urtwn_tx_data * urtwn_get_tx_data(struct urtwn_softc *, size_t); static void urtwn_start(struct ifnet *); static void urtwn_watchdog(struct ifnet *); static int urtwn_ioctl(struct ifnet *, u_long, void *); static int urtwn_r92c_power_on(struct urtwn_softc *); static int urtwn_r92e_power_on(struct urtwn_softc *); static int urtwn_r88e_power_on(struct urtwn_softc *); static int urtwn_llt_init(struct urtwn_softc *); static void urtwn_fw_reset(struct urtwn_softc *); static void urtwn_r88e_fw_reset(struct urtwn_softc *); static int urtwn_fw_loadpage(struct urtwn_softc *, int, uint8_t *, int); static int urtwn_load_firmware(struct urtwn_softc *); static int urtwn_r92c_dma_init(struct urtwn_softc *); static int urtwn_r88e_dma_init(struct urtwn_softc *); static void urtwn_mac_init(struct urtwn_softc *); static void urtwn_bb_init(struct urtwn_softc *); static void urtwn_rf_init(struct urtwn_softc *); static void urtwn_cam_init(struct urtwn_softc *); static void urtwn_pa_bias_init(struct urtwn_softc *); static void urtwn_rxfilter_init(struct urtwn_softc *); static void urtwn_edca_init(struct urtwn_softc *); static void urtwn_write_txpower(struct urtwn_softc *, int, uint16_t[]); static void urtwn_get_txpower(struct urtwn_softc *, size_t, u_int, u_int, uint16_t[]); static void urtwn_r88e_get_txpower(struct urtwn_softc *, size_t, u_int, u_int, uint16_t[]); static void urtwn_set_txpower(struct urtwn_softc *, u_int, u_int); static void urtwn_set_chan(struct urtwn_softc *, struct ieee80211_channel *, u_int); static void urtwn_iq_calib(struct urtwn_softc *, bool); static void urtwn_lc_calib(struct urtwn_softc *); static void urtwn_temp_calib(struct urtwn_softc *); static int urtwn_init(struct ifnet *); static void urtwn_stop(struct ifnet *, int); static int urtwn_reset(struct ifnet *); static void urtwn_chip_stop(struct urtwn_softc *); static void urtwn_newassoc(struct ieee80211_node *, int); static void urtwn_delay_ms(struct urtwn_softc *, int ms); /* Aliases. */ #define urtwn_bb_write urtwn_write_4 #define urtwn_bb_read urtwn_read_4 #define urtwn_lookup(d,v,p) ((const struct urtwn_dev *)usb_lookup(d,v,p)) static const uint16_t addaReg[] = { R92C_FPGA0_XCD_SWITCHCTL, R92C_BLUETOOTH, R92C_RX_WAIT_CCA, R92C_TX_CCK_RFON, R92C_TX_CCK_BBON, R92C_TX_OFDM_RFON, R92C_TX_OFDM_BBON, R92C_TX_TO_RX, R92C_TX_TO_TX, R92C_RX_CCK, R92C_RX_OFDM, R92C_RX_WAIT_RIFS, R92C_RX_TO_RX, R92C_STANDBY, R92C_SLEEP, R92C_PMPD_ANAEN }; static int urtwn_match(device_t parent, cfdata_t match, void *aux) { struct usb_attach_arg *uaa = aux; return urtwn_lookup(urtwn_devs, uaa->uaa_vendor, uaa->uaa_product) != NULL ? UMATCH_VENDOR_PRODUCT : UMATCH_NONE; } static void urtwn_attach(device_t parent, device_t self, void *aux) { struct urtwn_softc *sc = device_private(self); struct ieee80211com *ic = &sc->sc_ic; struct ifnet *ifp = &sc->sc_if; struct usb_attach_arg *uaa = aux; char *devinfop; const struct urtwn_dev *dev; usb_device_request_t req; size_t i; int error; URTWNHIST_FUNC(); URTWNHIST_CALLED(); sc->sc_dev = self; sc->sc_udev = uaa->uaa_device; sc->chip = 0; dev = urtwn_lookup(urtwn_devs, uaa->uaa_vendor, uaa->uaa_product); if (dev != NULL && ISSET(dev->flags, FLAG_RTL8188E)) SET(sc->chip, URTWN_CHIP_88E); if (dev != NULL && ISSET(dev->flags, FLAG_RTL8192E)) SET(sc->chip, URTWN_CHIP_92EU); aprint_naive("\n"); aprint_normal("\n"); devinfop = usbd_devinfo_alloc(sc->sc_udev, 0); aprint_normal_dev(self, "%s\n", devinfop); usbd_devinfo_free(devinfop); req.bmRequestType = UT_WRITE_DEVICE; req.bRequest = UR_SET_FEATURE; USETW(req.wValue, UF_DEVICE_REMOTE_WAKEUP); USETW(req.wIndex, UHF_PORT_SUSPEND); USETW(req.wLength, 0); (void) usbd_do_request(sc->sc_udev, &req, 0); cv_init(&sc->sc_task_cv, "urtwntsk"); mutex_init(&sc->sc_task_mtx, MUTEX_DEFAULT, IPL_NET); mutex_init(&sc->sc_tx_mtx, MUTEX_DEFAULT, IPL_NONE); mutex_init(&sc->sc_rx_mtx, MUTEX_DEFAULT, IPL_NONE); mutex_init(&sc->sc_fwcmd_mtx, MUTEX_DEFAULT, IPL_NONE); mutex_init(&sc->sc_write_mtx, MUTEX_DEFAULT, IPL_NONE); usb_init_task(&sc->sc_task, urtwn_task, sc, 0); callout_init(&sc->sc_scan_to, 0); callout_setfunc(&sc->sc_scan_to, urtwn_next_scan, sc); callout_init(&sc->sc_calib_to, 0); callout_setfunc(&sc->sc_calib_to, urtwn_calib_to, sc); rnd_attach_source(&sc->rnd_source, device_xname(sc->sc_dev), RND_TYPE_NET, RND_FLAG_DEFAULT); error = usbd_set_config_no(sc->sc_udev, 1, 0); if (error != 0) { aprint_error_dev(self, "failed to set configuration" ", err=%s\n", usbd_errstr(error)); goto fail; } /* Get the first interface handle. */ error = usbd_device2interface_handle(sc->sc_udev, 0, &sc->sc_iface); if (error != 0) { aprint_error_dev(self, "could not get interface handle\n"); goto fail; } error = urtwn_read_chipid(sc); if (error != 0) { aprint_error_dev(self, "unsupported test chip\n"); goto fail; } /* Determine number of Tx/Rx chains. */ if (sc->chip & URTWN_CHIP_92C) { sc->ntxchains = (sc->chip & URTWN_CHIP_92C_1T2R) ? 1 : 2; sc->nrxchains = 2; } else if (sc->chip & URTWN_CHIP_92EU) { sc->ntxchains = 2; sc->nrxchains = 2; } else { sc->ntxchains = 1; sc->nrxchains = 1; } if (ISSET(sc->chip, URTWN_CHIP_88E) || ISSET(sc->chip, URTWN_CHIP_92EU)) urtwn_r88e_read_rom(sc); else urtwn_read_rom(sc); aprint_normal_dev(self, "MAC/BB RTL%s, RF 6052 %zdT%zdR, address %s\n", (sc->chip & URTWN_CHIP_92EU) ? "8192EU" : (sc->chip & URTWN_CHIP_92C) ? "8192CU" : (sc->chip & URTWN_CHIP_88E) ? "8188EU" : (sc->board_type == R92C_BOARD_TYPE_HIGHPA) ? "8188RU" : (sc->board_type == R92C_BOARD_TYPE_MINICARD) ? "8188CE-VAU" : "8188CUS", sc->ntxchains, sc->nrxchains, ether_sprintf(ic->ic_myaddr)); error = urtwn_open_pipes(sc); if (error != 0) { aprint_error_dev(sc->sc_dev, "could not open pipes\n"); goto fail; } aprint_normal_dev(self, "%d rx pipe%s, %d tx pipe%s\n", sc->rx_npipe, sc->rx_npipe > 1 ? "s" : "", sc->tx_npipe, sc->tx_npipe > 1 ? "s" : ""); /* * Setup the 802.11 device. */ ic->ic_ifp = ifp; ic->ic_phytype = IEEE80211_T_OFDM; /* Not only, but not used. */ ic->ic_opmode = IEEE80211_M_STA; /* Default to BSS mode. */ ic->ic_state = IEEE80211_S_INIT; /* Set device capabilities. */ ic->ic_caps = IEEE80211_C_MONITOR | /* Monitor mode supported. */ IEEE80211_C_IBSS | /* IBSS mode supported */ IEEE80211_C_HOSTAP | /* HostAp mode supported */ IEEE80211_C_SHPREAMBLE | /* Short preamble supported. */ IEEE80211_C_SHSLOT | /* Short slot time supported. */ IEEE80211_C_WME | /* 802.11e */ IEEE80211_C_WPA; /* 802.11i */ /* Set supported .11b and .11g rates. */ ic->ic_sup_rates[IEEE80211_MODE_11B] = ieee80211_std_rateset_11b; ic->ic_sup_rates[IEEE80211_MODE_11G] = ieee80211_std_rateset_11g; /* Set supported .11b and .11g channels (1 through 14). */ for (i = 1; i <= 14; i++) { ic->ic_channels[i].ic_freq = ieee80211_ieee2mhz(i, IEEE80211_CHAN_2GHZ); ic->ic_channels[i].ic_flags = IEEE80211_CHAN_CCK | IEEE80211_CHAN_OFDM | IEEE80211_CHAN_DYN | IEEE80211_CHAN_2GHZ; } ifp->if_softc = sc; ifp->if_flags = IFF_BROADCAST | IFF_SIMPLEX | IFF_MULTICAST; ifp->if_init = urtwn_init; ifp->if_ioctl = urtwn_ioctl; ifp->if_start = urtwn_start; ifp->if_watchdog = urtwn_watchdog; IFQ_SET_READY(&ifp->if_snd); memcpy(ifp->if_xname, device_xname(sc->sc_dev), IFNAMSIZ); if_initialize(ifp); ieee80211_ifattach(ic); /* override default methods */ ic->ic_newassoc = urtwn_newassoc; ic->ic_reset = urtwn_reset; ic->ic_wme.wme_update = urtwn_wme_update; /* Override state transition machine. */ sc->sc_newstate = ic->ic_newstate; ic->ic_newstate = urtwn_newstate; /* XXX media locking needs revisiting */ mutex_init(&sc->sc_media_mtx, MUTEX_DEFAULT, IPL_SOFTUSB); ieee80211_media_init_with_lock(ic, urtwn_media_change, ieee80211_media_status, &sc->sc_media_mtx); bpf_attach2(ifp, DLT_IEEE802_11_RADIO, sizeof(struct ieee80211_frame) + IEEE80211_RADIOTAP_HDRLEN, &sc->sc_drvbpf); sc->sc_rxtap_len = sizeof(sc->sc_rxtapu); sc->sc_rxtap.wr_ihdr.it_len = htole16(sc->sc_rxtap_len); sc->sc_rxtap.wr_ihdr.it_present = htole32(URTWN_RX_RADIOTAP_PRESENT); sc->sc_txtap_len = sizeof(sc->sc_txtapu); sc->sc_txtap.wt_ihdr.it_len = htole16(sc->sc_txtap_len); sc->sc_txtap.wt_ihdr.it_present = htole32(URTWN_TX_RADIOTAP_PRESENT); ifp->if_percpuq = if_percpuq_create(ifp); if_register(ifp); ieee80211_announce(ic); usbd_add_drv_event(USB_EVENT_DRIVER_ATTACH, sc->sc_udev, sc->sc_dev); if (!pmf_device_register(self, NULL, NULL)) aprint_error_dev(self, "couldn't establish power handler\n"); SET(sc->sc_flags, URTWN_FLAG_ATTACHED); return; fail: sc->sc_dying = 1; aprint_error_dev(self, "attach failed\n"); } static int urtwn_detach(device_t self, int flags) { struct urtwn_softc *sc = device_private(self); struct ifnet *ifp = &sc->sc_if; int s; URTWNHIST_FUNC(); URTWNHIST_CALLED(); pmf_device_deregister(self); s = splusb(); sc->sc_dying = 1; callout_halt(&sc->sc_scan_to, NULL); callout_halt(&sc->sc_calib_to, NULL); if (ISSET(sc->sc_flags, URTWN_FLAG_ATTACHED)) { urtwn_stop(ifp, 0); usb_rem_task_wait(sc->sc_udev, &sc->sc_task, USB_TASKQ_DRIVER, NULL); ifp->if_flags &= ~(IFF_RUNNING | IFF_OACTIVE); bpf_detach(ifp); ieee80211_ifdetach(&sc->sc_ic); if_detach(ifp); mutex_destroy(&sc->sc_media_mtx); /* Close Tx/Rx pipes. Abort done by urtwn_stop. */ urtwn_close_pipes(sc); } splx(s); usbd_add_drv_event(USB_EVENT_DRIVER_DETACH, sc->sc_udev, sc->sc_dev); rnd_detach_source(&sc->rnd_source); callout_destroy(&sc->sc_scan_to); callout_destroy(&sc->sc_calib_to); cv_destroy(&sc->sc_task_cv); mutex_destroy(&sc->sc_write_mtx); mutex_destroy(&sc->sc_fwcmd_mtx); mutex_destroy(&sc->sc_tx_mtx); mutex_destroy(&sc->sc_rx_mtx); mutex_destroy(&sc->sc_task_mtx); return 0; } static int urtwn_activate(device_t self, enum devact act) { struct urtwn_softc *sc = device_private(self); URTWNHIST_FUNC(); URTWNHIST_CALLED(); switch (act) { case DVACT_DEACTIVATE: if_deactivate(sc->sc_ic.ic_ifp); return 0; default: return EOPNOTSUPP; } } static int urtwn_open_pipes(struct urtwn_softc *sc) { /* Bulk-out endpoints addresses (from highest to lowest prio). */ static uint8_t epaddr[R92C_MAX_EPOUT]; static uint8_t rxepaddr[R92C_MAX_EPIN]; usb_interface_descriptor_t *id; usb_endpoint_descriptor_t *ed; size_t i, ntx = 0, nrx = 0; int error; URTWNHIST_FUNC(); URTWNHIST_CALLED(); /* Determine the number of bulk-out pipes. */ id = usbd_get_interface_descriptor(sc->sc_iface); for (i = 0; i < id->bNumEndpoints; i++) { ed = usbd_interface2endpoint_descriptor(sc->sc_iface, i); if (ed == NULL || UE_GET_XFERTYPE(ed->bmAttributes) != UE_BULK) { continue; } if (UE_GET_DIR(ed->bEndpointAddress) == UE_DIR_OUT) { if (ntx < sizeof(epaddr)) epaddr[ntx] = ed->bEndpointAddress; ntx++; } if (UE_GET_DIR(ed->bEndpointAddress) == UE_DIR_IN) { if (nrx < sizeof(rxepaddr)) rxepaddr[nrx] = ed->bEndpointAddress; nrx++; } } if (nrx == 0 || nrx > R92C_MAX_EPIN) { aprint_error_dev(sc->sc_dev, "%zd: invalid number of Rx bulk pipes\n", nrx); return EIO; } if (ntx == 0 || ntx > R92C_MAX_EPOUT) { aprint_error_dev(sc->sc_dev, "%zd: invalid number of Tx bulk pipes\n", ntx); return EIO; } DPRINTFN(DBG_INIT, "found %jd/%jd bulk-in/out pipes", nrx, ntx, 0, 0); sc->rx_npipe = nrx; sc->tx_npipe = ntx; /* Open bulk-in pipe at address 0x81. */ for (i = 0; i < nrx; i++) { error = usbd_open_pipe(sc->sc_iface, rxepaddr[i], USBD_EXCLUSIVE_USE, &sc->rx_pipe[i]); if (error != 0) { aprint_error_dev(sc->sc_dev, "could not open Rx bulk pipe 0x%02x: %d\n", rxepaddr[i], error); goto fail; } } /* Open bulk-out pipes (up to 3). */ for (i = 0; i < ntx; i++) { error = usbd_open_pipe(sc->sc_iface, epaddr[i], USBD_EXCLUSIVE_USE, &sc->tx_pipe[i]); if (error != 0) { aprint_error_dev(sc->sc_dev, "could not open Tx bulk pipe 0x%02x: %d\n", epaddr[i], error); goto fail; } } /* Map 802.11 access categories to USB pipes. */ sc->ac2idx[WME_AC_BK] = sc->ac2idx[WME_AC_BE] = (ntx == 3) ? 2 : ((ntx == 2) ? 1 : 0); sc->ac2idx[WME_AC_VI] = (ntx == 3) ? 1 : 0; sc->ac2idx[WME_AC_VO] = 0; /* Always use highest prio. */ fail: if (error != 0) urtwn_close_pipes(sc); return error; } static void urtwn_close_pipes(struct urtwn_softc *sc) { struct usbd_pipe *pipe; size_t i; URTWNHIST_FUNC(); URTWNHIST_CALLED(); /* Close Rx pipes. */ CTASSERT(sizeof(pipe) == sizeof(void *)); for (i = 0; i < sc->rx_npipe; i++) { pipe = atomic_swap_ptr(&sc->rx_pipe[i], NULL); if (pipe != NULL) { usbd_close_pipe(pipe); } } /* Close Tx pipes. */ for (i = 0; i < sc->tx_npipe; i++) { pipe = atomic_swap_ptr(&sc->tx_pipe[i], NULL); if (pipe != NULL) { usbd_close_pipe(pipe); } } } static int __noinline urtwn_alloc_rx_list(struct urtwn_softc *sc) { struct urtwn_rx_data *data; size_t i; int error = 0; URTWNHIST_FUNC(); URTWNHIST_CALLED(); for (size_t j = 0; j < sc->rx_npipe; j++) { TAILQ_INIT(&sc->rx_free_list[j]); for (i = 0; i < URTWN_RX_LIST_COUNT; i++) { data = &sc->rx_data[j][i]; data->sc = sc; /* Backpointer for callbacks. */ error = usbd_create_xfer(sc->rx_pipe[j], URTWN_RXBUFSZ, 0, 0, &data->xfer); if (error) { aprint_error_dev(sc->sc_dev, "could not allocate xfer\n"); break; } data->buf = usbd_get_buffer(data->xfer); TAILQ_INSERT_TAIL(&sc->rx_free_list[j], data, next); } } if (error != 0) urtwn_free_rx_list(sc); return error; } static void urtwn_free_rx_list(struct urtwn_softc *sc) { struct usbd_xfer *xfer; size_t i; URTWNHIST_FUNC(); URTWNHIST_CALLED(); /* NB: Caller must abort pipe first. */ for (size_t j = 0; j < sc->rx_npipe; j++) { for (i = 0; i < URTWN_RX_LIST_COUNT; i++) { CTASSERT(sizeof(xfer) == sizeof(void *)); xfer = atomic_swap_ptr(&sc->rx_data[j][i].xfer, NULL); if (xfer != NULL) usbd_destroy_xfer(xfer); } } } static int __noinline urtwn_alloc_tx_list(struct urtwn_softc *sc) { struct urtwn_tx_data *data; size_t i; int error = 0; URTWNHIST_FUNC(); URTWNHIST_CALLED(); mutex_enter(&sc->sc_tx_mtx); for (size_t j = 0; j < sc->tx_npipe; j++) { TAILQ_INIT(&sc->tx_free_list[j]); for (i = 0; i < URTWN_TX_LIST_COUNT; i++) { data = &sc->tx_data[j][i]; data->sc = sc; /* Backpointer for callbacks. */ data->pidx = j; error = usbd_create_xfer(sc->tx_pipe[j], URTWN_TXBUFSZ, USBD_FORCE_SHORT_XFER, 0, &data->xfer); if (error) { aprint_error_dev(sc->sc_dev, "could not allocate xfer\n"); goto fail; } data->buf = usbd_get_buffer(data->xfer); /* Append this Tx buffer to our free list. */ TAILQ_INSERT_TAIL(&sc->tx_free_list[j], data, next); } } mutex_exit(&sc->sc_tx_mtx); return 0; fail: urtwn_free_tx_list(sc); mutex_exit(&sc->sc_tx_mtx); return error; } static void urtwn_free_tx_list(struct urtwn_softc *sc) { struct usbd_xfer *xfer; size_t i; URTWNHIST_FUNC(); URTWNHIST_CALLED(); /* NB: Caller must abort pipe first. */ for (size_t j = 0; j < sc->tx_npipe; j++) { for (i = 0; i < URTWN_TX_LIST_COUNT; i++) { CTASSERT(sizeof(xfer) == sizeof(void *)); xfer = atomic_swap_ptr(&sc->tx_data[j][i].xfer, NULL); if (xfer != NULL) usbd_destroy_xfer(xfer); } } } static int urtwn_tx_beacon(struct urtwn_softc *sc, struct mbuf *m, struct ieee80211_node *ni) { struct urtwn_tx_data *data = urtwn_get_tx_data(sc, sc->ac2idx[WME_AC_VO]); if (data == NULL) return ENOBUFS; return urtwn_tx(sc, m, ni, data); } static void urtwn_task(void *arg) { struct urtwn_softc *sc = arg; struct ieee80211com *ic = &sc->sc_ic; struct urtwn_host_cmd_ring *ring = &sc->cmdq; struct urtwn_host_cmd *cmd; int s; URTWNHIST_FUNC(); URTWNHIST_CALLED(); if (ic->ic_state == IEEE80211_S_RUN && (ic->ic_opmode == IEEE80211_M_HOSTAP || ic->ic_opmode == IEEE80211_M_IBSS)) { struct mbuf *m = ieee80211_beacon_alloc(ic, ic->ic_bss, &sc->sc_bo); if (m == NULL) { aprint_error_dev(sc->sc_dev, "could not allocate beacon"); } if (urtwn_tx_beacon(sc, m, ic->ic_bss) != 0) { aprint_error_dev(sc->sc_dev, "could not send beacon\n"); } /* beacon is no longer needed */ m_freem(m); } /* Process host commands. */ s = splusb(); mutex_spin_enter(&sc->sc_task_mtx); while (ring->next != ring->cur) { cmd = &ring->cmd[ring->next]; mutex_spin_exit(&sc->sc_task_mtx); splx(s); /* Invoke callback with kernel lock held. */ cmd->cb(sc, cmd->data); s = splusb(); mutex_spin_enter(&sc->sc_task_mtx); ring->queued--; ring->next = (ring->next + 1) % URTWN_HOST_CMD_RING_COUNT; } cv_broadcast(&sc->sc_task_cv); mutex_spin_exit(&sc->sc_task_mtx); splx(s); } static void urtwn_do_async(struct urtwn_softc *sc, void (*cb)(struct urtwn_softc *, void *), void *arg, int len) { struct urtwn_host_cmd_ring *ring = &sc->cmdq; struct urtwn_host_cmd *cmd; int s; URTWNHIST_FUNC(); URTWNHIST_CALLARGS("cb=%#jx, arg=%#jx, len=%jd", (uintptr_t)cb, (uintptr_t)arg, len, 0); s = splusb(); mutex_spin_enter(&sc->sc_task_mtx); cmd = &ring->cmd[ring->cur]; cmd->cb = cb; KASSERT(len <= sizeof(cmd->data)); memcpy(cmd->data, arg, len); ring->cur = (ring->cur + 1) % URTWN_HOST_CMD_RING_COUNT; /* If there is no pending command already, schedule a task. */ if (!sc->sc_dying && ++ring->queued == 1) { mutex_spin_exit(&sc->sc_task_mtx); usb_add_task(sc->sc_udev, &sc->sc_task, USB_TASKQ_DRIVER); } else mutex_spin_exit(&sc->sc_task_mtx); splx(s); } static void urtwn_wait_async(struct urtwn_softc *sc) { URTWNHIST_FUNC(); URTWNHIST_CALLED(); /* Wait for all queued asynchronous commands to complete. */ mutex_spin_enter(&sc->sc_task_mtx); while (sc->cmdq.queued > 0) cv_wait(&sc->sc_task_cv, &sc->sc_task_mtx); mutex_spin_exit(&sc->sc_task_mtx); } static int urtwn_write_region_1(struct urtwn_softc *sc, uint16_t addr, uint8_t *buf, int len) { usb_device_request_t req; usbd_status error; URTWNHIST_FUNC(); URTWNHIST_CALLED(); KASSERT(mutex_owned(&sc->sc_write_mtx)); req.bmRequestType = UT_WRITE_VENDOR_DEVICE; req.bRequest = R92C_REQ_REGS; USETW(req.wValue, addr); USETW(req.wIndex, 0); USETW(req.wLength, len); error = usbd_do_request(sc->sc_udev, &req, buf); if (error != USBD_NORMAL_COMPLETION) { DPRINTFN(DBG_REG, "error=%jd: addr=%#jx, len=%jd", error, addr, len, 0); } return error; } static void urtwn_write_1(struct urtwn_softc *sc, uint16_t addr, uint8_t val) { URTWNHIST_FUNC(); URTWNHIST_CALLED(); DPRINTFN(DBG_REG, "addr=%#jx, val=%#jx", addr, val, 0, 0); urtwn_write_region_1(sc, addr, &val, 1); } static void urtwn_write_2(struct urtwn_softc *sc, uint16_t addr, uint16_t val) { uint8_t buf[2]; URTWNHIST_FUNC(); URTWNHIST_CALLED(); DPRINTFN(DBG_REG, "addr=%#jx, val=%#jx", addr, val, 0, 0); buf[0] = (uint8_t)val; buf[1] = (uint8_t)(val >> 8); urtwn_write_region_1(sc, addr, buf, 2); } static void urtwn_write_4(struct urtwn_softc *sc, uint16_t addr, uint32_t val) { uint8_t buf[4]; URTWNHIST_FUNC(); URTWNHIST_CALLED(); DPRINTFN(DBG_REG, "addr=%#jx, val=%#jx", addr, val, 0, 0); buf[0] = (uint8_t)val; buf[1] = (uint8_t)(val >> 8); buf[2] = (uint8_t)(val >> 16); buf[3] = (uint8_t)(val >> 24); urtwn_write_region_1(sc, addr, buf, 4); } static int urtwn_write_region(struct urtwn_softc *sc, uint16_t addr, uint8_t *buf, int len) { URTWNHIST_FUNC(); URTWNHIST_CALLARGS("addr=%#jx, len=%#jx", addr, len, 0, 0); return urtwn_write_region_1(sc, addr, buf, len); } static int urtwn_read_region_1(struct urtwn_softc *sc, uint16_t addr, uint8_t *buf, int len) { usb_device_request_t req; usbd_status error; URTWNHIST_FUNC(); URTWNHIST_CALLED(); req.bmRequestType = UT_READ_VENDOR_DEVICE; req.bRequest = R92C_REQ_REGS; USETW(req.wValue, addr); USETW(req.wIndex, 0); USETW(req.wLength, len); error = usbd_do_request(sc->sc_udev, &req, buf); if (error != USBD_NORMAL_COMPLETION) { DPRINTFN(DBG_REG, "error=%jd: addr=%#jx, len=%jd", error, addr, len, 0); } return error; } static uint8_t urtwn_read_1(struct urtwn_softc *sc, uint16_t addr) { uint8_t val; URTWNHIST_FUNC(); URTWNHIST_CALLED(); if (urtwn_read_region_1(sc, addr, &val, 1) != USBD_NORMAL_COMPLETION) return 0xff; DPRINTFN(DBG_REG, "addr=%#jx, val=%#jx", addr, val, 0, 0); return val; } static uint16_t urtwn_read_2(struct urtwn_softc *sc, uint16_t addr) { uint8_t buf[2]; uint16_t val; URTWNHIST_FUNC(); URTWNHIST_CALLED(); if (urtwn_read_region_1(sc, addr, buf, 2) != USBD_NORMAL_COMPLETION) return 0xffff; val = LE_READ_2(&buf[0]); DPRINTFN(DBG_REG, "addr=%#jx, val=%#jx", addr, val, 0, 0); return val; } static uint32_t urtwn_read_4(struct urtwn_softc *sc, uint16_t addr) { uint8_t buf[4]; uint32_t val; URTWNHIST_FUNC(); URTWNHIST_CALLED(); if (urtwn_read_region_1(sc, addr, buf, 4) != USBD_NORMAL_COMPLETION) return 0xffffffff; val = LE_READ_4(&buf[0]); DPRINTFN(DBG_REG, "addr=%#jx, val=%#jx", addr, val, 0, 0); return val; } static int urtwn_fw_cmd(struct urtwn_softc *sc, uint8_t id, const void *buf, int len) { struct r92c_fw_cmd cmd; uint8_t *cp; int fwcur; int ntries; URTWNHIST_FUNC(); URTWNHIST_CALLED(); DPRINTFN(DBG_REG, "id=%jd, buf=%#jx, len=%jd", id, (uintptr_t)buf, len, 0); KASSERT(mutex_owned(&sc->sc_write_mtx)); mutex_enter(&sc->sc_fwcmd_mtx); fwcur = sc->fwcur; sc->fwcur = (sc->fwcur + 1) % R92C_H2C_NBOX; /* Wait for current FW box to be empty. */ for (ntries = 0; ntries < 100; ntries++) { if (!(urtwn_read_1(sc, R92C_HMETFR) & (1 << fwcur))) break; urtwn_delay_ms(sc, 2); } if (ntries == 100) { aprint_error_dev(sc->sc_dev, "could not send firmware command %d\n", id); mutex_exit(&sc->sc_fwcmd_mtx); return ETIMEDOUT; } memset(&cmd, 0, sizeof(cmd)); KASSERT(len <= sizeof(cmd.msg)); memcpy(cmd.msg, buf, len); /* Write the first word last since that will trigger the FW. */ cp = (uint8_t *)&cmd; cmd.id = id; if (len >= 4) { if (!ISSET(sc->chip, URTWN_CHIP_92EU)) { cmd.id |= R92C_CMD_FLAG_EXT; urtwn_write_region(sc, R92C_HMEBOX_EXT(fwcur), &cp[1], 2); urtwn_write_4(sc, R92C_HMEBOX(fwcur), cp[0] + (cp[3] << 8) + (cp[4] << 16) + ((uint32_t)cp[5] << 24)); } else { urtwn_write_region(sc, R92E_HMEBOX_EXT(fwcur), &cp[4], 2); urtwn_write_4(sc, R92C_HMEBOX(fwcur), cp[0] + (cp[1] << 8) + (cp[2] << 16) + ((uint32_t)cp[3] << 24)); } } else { urtwn_write_region(sc, R92C_HMEBOX(fwcur), cp, len); } mutex_exit(&sc->sc_fwcmd_mtx); return 0; } static __inline void urtwn_rf_write(struct urtwn_softc *sc, int chain, uint8_t addr, uint32_t val) { sc->sc_rf_write(sc, chain, addr, val); } static void urtwn_r92c_rf_write(struct urtwn_softc *sc, int chain, uint8_t addr, uint32_t val) { urtwn_bb_write(sc, R92C_LSSI_PARAM(chain), SM(R92C_LSSI_PARAM_ADDR, addr) | SM(R92C_LSSI_PARAM_DATA, val)); } static void urtwn_r88e_rf_write(struct urtwn_softc *sc, int chain, uint8_t addr, uint32_t val) { urtwn_bb_write(sc, R92C_LSSI_PARAM(chain), SM(R88E_LSSI_PARAM_ADDR, addr) | SM(R92C_LSSI_PARAM_DATA, val)); } static void urtwn_r92e_rf_write(struct urtwn_softc *sc, int chain, uint8_t addr, uint32_t val) { urtwn_bb_write(sc, R92C_LSSI_PARAM(chain), SM(R88E_LSSI_PARAM_ADDR, addr) | SM(R92C_LSSI_PARAM_DATA, val)); } static uint32_t urtwn_rf_read(struct urtwn_softc *sc, int chain, uint8_t addr) { uint32_t reg[R92C_MAX_CHAINS], val; reg[0] = urtwn_bb_read(sc, R92C_HSSI_PARAM2(0)); if (chain != 0) { reg[chain] = urtwn_bb_read(sc, R92C_HSSI_PARAM2(chain)); } urtwn_bb_write(sc, R92C_HSSI_PARAM2(0), reg[0] & ~R92C_HSSI_PARAM2_READ_EDGE); urtwn_delay_ms(sc, 1); urtwn_bb_write(sc, R92C_HSSI_PARAM2(chain), RW(reg[chain], R92C_HSSI_PARAM2_READ_ADDR, addr) | R92C_HSSI_PARAM2_READ_EDGE); urtwn_delay_ms(sc, 1); urtwn_bb_write(sc, R92C_HSSI_PARAM2(0), reg[0] | R92C_HSSI_PARAM2_READ_EDGE); urtwn_delay_ms(sc, 1); if (urtwn_bb_read(sc, R92C_HSSI_PARAM1(chain)) & R92C_HSSI_PARAM1_PI) { val = urtwn_bb_read(sc, R92C_HSPI_READBACK(chain)); } else { val = urtwn_bb_read(sc, R92C_LSSI_READBACK(chain)); } return MS(val, R92C_LSSI_READBACK_DATA); } static int urtwn_llt_write(struct urtwn_softc *sc, uint32_t addr, uint32_t data) { int ntries; KASSERT(mutex_owned(&sc->sc_write_mtx)); urtwn_write_4(sc, R92C_LLT_INIT, SM(R92C_LLT_INIT_OP, R92C_LLT_INIT_OP_WRITE) | SM(R92C_LLT_INIT_ADDR, addr) | SM(R92C_LLT_INIT_DATA, data)); /* Wait for write operation to complete. */ for (ntries = 0; ntries < 20; ntries++) { if (MS(urtwn_read_4(sc, R92C_LLT_INIT), R92C_LLT_INIT_OP) == R92C_LLT_INIT_OP_NO_ACTIVE) { /* Done */ return 0; } DELAY(5); } return ETIMEDOUT; } static uint8_t urtwn_efuse_read_1(struct urtwn_softc *sc, uint16_t addr) { uint32_t reg; int ntries; KASSERT(mutex_owned(&sc->sc_write_mtx)); reg = urtwn_read_4(sc, R92C_EFUSE_CTRL); reg = RW(reg, R92C_EFUSE_CTRL_ADDR, addr); reg &= ~R92C_EFUSE_CTRL_VALID; urtwn_write_4(sc, R92C_EFUSE_CTRL, reg); /* Wait for read operation to complete. */ for (ntries = 0; ntries < 100; ntries++) { reg = urtwn_read_4(sc, R92C_EFUSE_CTRL); if (reg & R92C_EFUSE_CTRL_VALID) { /* Done */ return MS(reg, R92C_EFUSE_CTRL_DATA); } DELAY(5); } aprint_error_dev(sc->sc_dev, "could not read efuse byte at address 0x%04x\n", addr); return 0xff; } static void urtwn_efuse_read(struct urtwn_softc *sc) { uint8_t *rom = (uint8_t *)&sc->rom; uint32_t reg; uint16_t addr = 0; uint8_t off, msk; size_t i; URTWNHIST_FUNC(); URTWNHIST_CALLED(); KASSERT(mutex_owned(&sc->sc_write_mtx)); urtwn_efuse_switch_power(sc); memset(&sc->rom, 0xff, sizeof(sc->rom)); while (addr < 512) { reg = urtwn_efuse_read_1(sc, addr); if (reg == 0xff) break; addr++; off = reg >> 4; msk = reg & 0xf; for (i = 0; i < 4; i++) { if (msk & (1U << i)) continue; rom[off * 8 + i * 2 + 0] = urtwn_efuse_read_1(sc, addr); addr++; rom[off * 8 + i * 2 + 1] = urtwn_efuse_read_1(sc, addr); addr++; } } #ifdef URTWN_DEBUG /* Dump ROM content. */ for (i = 0; i < (int)sizeof(sc->rom); i++) DPRINTFN(DBG_INIT, "%04jx: %02jx", i, rom[i], 0, 0); #endif } static void urtwn_efuse_switch_power(struct urtwn_softc *sc) { uint32_t reg; reg = urtwn_read_2(sc, R92C_SYS_ISO_CTRL); if (!(reg & R92C_SYS_ISO_CTRL_PWC_EV12V)) { urtwn_write_2(sc, R92C_SYS_ISO_CTRL, reg | R92C_SYS_ISO_CTRL_PWC_EV12V); } reg = urtwn_read_2(sc, R92C_SYS_FUNC_EN); if (!(reg & R92C_SYS_FUNC_EN_ELDR)) { urtwn_write_2(sc, R92C_SYS_FUNC_EN, reg | R92C_SYS_FUNC_EN_ELDR); } reg = urtwn_read_2(sc, R92C_SYS_CLKR); if ((reg & (R92C_SYS_CLKR_LOADER_EN | R92C_SYS_CLKR_ANA8M)) != (R92C_SYS_CLKR_LOADER_EN | R92C_SYS_CLKR_ANA8M)) { urtwn_write_2(sc, R92C_SYS_CLKR, reg | R92C_SYS_CLKR_LOADER_EN | R92C_SYS_CLKR_ANA8M); } } static int urtwn_read_chipid(struct urtwn_softc *sc) { uint32_t reg; URTWNHIST_FUNC(); URTWNHIST_CALLED(); if (ISSET(sc->chip, URTWN_CHIP_88E) || ISSET(sc->chip, URTWN_CHIP_92EU)) return 0; reg = urtwn_read_4(sc, R92C_SYS_CFG); if (reg & R92C_SYS_CFG_TRP_VAUX_EN) { /* test chip, not supported */ return EIO; } if (reg & R92C_SYS_CFG_TYPE_92C) { sc->chip |= URTWN_CHIP_92C; /* Check if it is a castrated 8192C. */ if (MS(urtwn_read_4(sc, R92C_HPON_FSM), R92C_HPON_FSM_CHIP_BONDING_ID) == R92C_HPON_FSM_CHIP_BONDING_ID_92C_1T2R) { sc->chip |= URTWN_CHIP_92C_1T2R; } } if (reg & R92C_SYS_CFG_VENDOR_UMC) { sc->chip |= URTWN_CHIP_UMC; if (MS(reg, R92C_SYS_CFG_CHIP_VER_RTL) == 0) { sc->chip |= URTWN_CHIP_UMC_A_CUT; } } return 0; } #ifdef URTWN_DEBUG static void urtwn_dump_rom(struct urtwn_softc *sc, struct r92c_rom *rp) { aprint_normal_dev(sc->sc_dev, "id 0x%04x, dbg_sel %#x, vid %#x, pid %#x\n", rp->id, rp->dbg_sel, rp->vid, rp->pid); aprint_normal_dev(sc->sc_dev, "usb_opt %#x, ep_setting %#x, usb_phy %#x\n", rp->usb_opt, rp->ep_setting, rp->usb_phy); aprint_normal_dev(sc->sc_dev, "macaddr %s\n", ether_sprintf(rp->macaddr)); aprint_normal_dev(sc->sc_dev, "string %s, subcustomer_id %#x\n", rp->string, rp->subcustomer_id); aprint_normal_dev(sc->sc_dev, "cck_tx_pwr c0: %d %d %d, c1: %d %d %d\n", rp->cck_tx_pwr[0][0], rp->cck_tx_pwr[0][1], rp->cck_tx_pwr[0][2], rp->cck_tx_pwr[1][0], rp->cck_tx_pwr[1][1], rp->cck_tx_pwr[1][2]); aprint_normal_dev(sc->sc_dev, "ht40_1s_tx_pwr c0 %d %d %d, c1 %d %d %d\n", rp->ht40_1s_tx_pwr[0][0], rp->ht40_1s_tx_pwr[0][1], rp->ht40_1s_tx_pwr[0][2], rp->ht40_1s_tx_pwr[1][0], rp->ht40_1s_tx_pwr[1][1], rp->ht40_1s_tx_pwr[1][2]); aprint_normal_dev(sc->sc_dev, "ht40_2s_tx_pwr_diff c0: %d %d %d, c1: %d %d %d\n", rp->ht40_2s_tx_pwr_diff[0] & 0xf, rp->ht40_2s_tx_pwr_diff[1] & 0xf, rp->ht40_2s_tx_pwr_diff[2] & 0xf, rp->ht40_2s_tx_pwr_diff[0] >> 4, rp->ht40_2s_tx_pwr_diff[1] & 0xf, rp->ht40_2s_tx_pwr_diff[2] >> 4); aprint_normal_dev(sc->sc_dev, "ht20_tx_pwr_diff c0: %d %d %d, c1: %d %d %d\n", rp->ht20_tx_pwr_diff[0] & 0xf, rp->ht20_tx_pwr_diff[1] & 0xf, rp->ht20_tx_pwr_diff[2] & 0xf, rp->ht20_tx_pwr_diff[0] >> 4, rp->ht20_tx_pwr_diff[1] >> 4, rp->ht20_tx_pwr_diff[2] >> 4); aprint_normal_dev(sc->sc_dev, "ofdm_tx_pwr_diff c0: %d %d %d, c1: %d %d %d\n", rp->ofdm_tx_pwr_diff[0] & 0xf, rp->ofdm_tx_pwr_diff[1] & 0xf, rp->ofdm_tx_pwr_diff[2] & 0xf, rp->ofdm_tx_pwr_diff[0] >> 4, rp->ofdm_tx_pwr_diff[1] >> 4, rp->ofdm_tx_pwr_diff[2] >> 4); aprint_normal_dev(sc->sc_dev, "ht40_max_pwr_offset c0: %d %d %d, c1: %d %d %d\n", rp->ht40_max_pwr[0] & 0xf, rp->ht40_max_pwr[1] & 0xf, rp->ht40_max_pwr[2] & 0xf, rp->ht40_max_pwr[0] >> 4, rp->ht40_max_pwr[1] >> 4, rp->ht40_max_pwr[2] >> 4); aprint_normal_dev(sc->sc_dev, "ht20_max_pwr_offset c0: %d %d %d, c1: %d %d %d\n", rp->ht20_max_pwr[0] & 0xf, rp->ht20_max_pwr[1] & 0xf, rp->ht20_max_pwr[2] & 0xf, rp->ht20_max_pwr[0] >> 4, rp->ht20_max_pwr[1] >> 4, rp->ht20_max_pwr[2] >> 4); aprint_normal_dev(sc->sc_dev, "xtal_calib %d, tssi %d %d, thermal %d\n", rp->xtal_calib, rp->tssi[0], rp->tssi[1], rp->thermal_meter); aprint_normal_dev(sc->sc_dev, "rf_opt1 %#x, rf_opt2 %#x, rf_opt3 %#x, rf_opt4 %#x\n", rp->rf_opt1, rp->rf_opt2, rp->rf_opt3, rp->rf_opt4); aprint_normal_dev(sc->sc_dev, "channnel_plan %d, version %d customer_id %#x\n", rp->channel_plan, rp->version, rp->curstomer_id); } #endif static void urtwn_read_rom(struct urtwn_softc *sc) { struct ieee80211com *ic = &sc->sc_ic; struct r92c_rom *rom = &sc->rom; URTWNHIST_FUNC(); URTWNHIST_CALLED(); mutex_enter(&sc->sc_write_mtx); /* Read full ROM image. */ urtwn_efuse_read(sc); #ifdef URTWN_DEBUG if (urtwn_debug & DBG_REG) urtwn_dump_rom(sc, rom); #endif /* XXX Weird but this is what the vendor driver does. */ sc->pa_setting = urtwn_efuse_read_1(sc, 0x1fa); sc->board_type = MS(rom->rf_opt1, R92C_ROM_RF1_BOARD_TYPE); sc->regulatory = MS(rom->rf_opt1, R92C_ROM_RF1_REGULATORY); DPRINTFN(DBG_INIT, "PA setting=%#jx, board=%#jx, regulatory=%jd", sc->pa_setting, sc->board_type, sc->regulatory, 0); IEEE80211_ADDR_COPY(ic->ic_myaddr, rom->macaddr); sc->sc_rf_write = urtwn_r92c_rf_write; sc->sc_power_on = urtwn_r92c_power_on; sc->sc_dma_init = urtwn_r92c_dma_init; mutex_exit(&sc->sc_write_mtx); } static void urtwn_r88e_read_rom(struct urtwn_softc *sc) { struct ieee80211com *ic = &sc->sc_ic; uint8_t *rom = sc->r88e_rom; uint32_t reg; uint16_t addr = 0; uint8_t off, msk, tmp; int i; URTWNHIST_FUNC(); URTWNHIST_CALLED(); mutex_enter(&sc->sc_write_mtx); off = 0; urtwn_efuse_switch_power(sc); /* Read full ROM image. */ memset(&sc->r88e_rom, 0xff, sizeof(sc->r88e_rom)); while (addr < 4096) { reg = urtwn_efuse_read_1(sc, addr); if (reg == 0xff) break; addr++; if ((reg & 0x1f) == 0x0f) { tmp = (reg & 0xe0) >> 5; reg = urtwn_efuse_read_1(sc, addr); if ((reg & 0x0f) != 0x0f) off = ((reg & 0xf0) >> 1) | tmp; addr++; } else off = reg >> 4; msk = reg & 0xf; for (i = 0; i < 4; i++) { if (msk & (1 << i)) continue; rom[off * 8 + i * 2 + 0] = urtwn_efuse_read_1(sc, addr); addr++; rom[off * 8 + i * 2 + 1] = urtwn_efuse_read_1(sc, addr); addr++; } } #ifdef URTWN_DEBUG if (urtwn_debug & DBG_REG) { } #endif addr = 0x10; for (i = 0; i < 6; i++) sc->cck_tx_pwr[i] = sc->r88e_rom[addr++]; for (i = 0; i < 5; i++) sc->ht40_tx_pwr[i] = sc->r88e_rom[addr++]; sc->bw20_tx_pwr_diff = (sc->r88e_rom[addr] & 0xf0) >> 4; if (sc->bw20_tx_pwr_diff & 0x08) sc->bw20_tx_pwr_diff |= 0xf0; sc->ofdm_tx_pwr_diff = (sc->r88e_rom[addr] & 0xf); if (sc->ofdm_tx_pwr_diff & 0x08) sc->ofdm_tx_pwr_diff |= 0xf0; sc->regulatory = MS(sc->r88e_rom[0xc1], R92C_ROM_RF1_REGULATORY); IEEE80211_ADDR_COPY(ic->ic_myaddr, &sc->r88e_rom[0xd7]); if (ISSET(sc->chip, URTWN_CHIP_92EU)) { sc->sc_power_on = urtwn_r92e_power_on; sc->sc_rf_write = urtwn_r92e_rf_write; } else { sc->sc_power_on = urtwn_r88e_power_on; sc->sc_rf_write = urtwn_r88e_rf_write; } sc->sc_dma_init = urtwn_r88e_dma_init; mutex_exit(&sc->sc_write_mtx); } static int urtwn_media_change(struct ifnet *ifp) { int error; URTWNHIST_FUNC(); URTWNHIST_CALLED(); if ((error = ieee80211_media_change(ifp)) != ENETRESET) return error; if ((ifp->if_flags & (IFF_UP | IFF_RUNNING)) == (IFF_UP | IFF_RUNNING)) { urtwn_init(ifp); } return 0; } /* * Initialize rate adaptation in firmware. */ static int __noinline urtwn_ra_init(struct urtwn_softc *sc) { static const uint8_t map[] = { 2, 4, 11, 22, 12, 18, 24, 36, 48, 72, 96, 108 }; struct ieee80211com *ic = &sc->sc_ic; struct ieee80211_node *ni = ic->ic_bss; struct ieee80211_rateset *rs = &ni->ni_rates; struct r92c_fw_cmd_macid_cfg cmd; uint32_t rates, basicrates; uint32_t rrsr_mask, rrsr_rate; uint8_t mode; size_t maxrate, maxbasicrate, i, j; int error; URTWNHIST_FUNC(); URTWNHIST_CALLED(); KASSERT(mutex_owned(&sc->sc_write_mtx)); /* Get normal and basic rates mask. */ rates = basicrates = 1; maxrate = maxbasicrate = 0; for (i = 0; i < rs->rs_nrates; i++) { /* Convert 802.11 rate to HW rate index. */ for (j = 0; j < __arraycount(map); j++) { if ((rs->rs_rates[i] & IEEE80211_RATE_VAL) == map[j]) { break; } } if (j == __arraycount(map)) { /* Unknown rate, skip. */ continue; } rates |= 1U << j; if (j > maxrate) { maxrate = j; } if (rs->rs_rates[i] & IEEE80211_RATE_BASIC) { basicrates |= 1U << j; if (j > maxbasicrate) { maxbasicrate = j; } } } if (ic->ic_curmode == IEEE80211_MODE_11B) { mode = R92C_RAID_11B; } else { mode = R92C_RAID_11BG; } DPRINTFN(DBG_INIT, "mode=%#jx", mode, 0, 0, 0); DPRINTFN(DBG_INIT, "rates=%#jx, basicrates=%#jx, " "maxrate=%jx, maxbasicrate=%jx", rates, basicrates, maxrate, maxbasicrate); if (ni->ni_capinfo & IEEE80211_CAPINFO_SHORT_PREAMBLE) { maxbasicrate |= R92C_RATE_SHORTGI; maxrate |= R92C_RATE_SHORTGI; } /* Set rates mask for group addressed frames. */ cmd.macid = RTWN_MACID_BC | RTWN_MACID_VALID; if (ni->ni_capinfo & IEEE80211_CAPINFO_SHORT_PREAMBLE) cmd.macid |= RTWN_MACID_SHORTGI; cmd.mask = htole32((mode << 28) | basicrates); error = urtwn_fw_cmd(sc, R92C_CMD_MACID_CONFIG, &cmd, sizeof(cmd)); if (error != 0) { aprint_error_dev(sc->sc_dev, "could not add broadcast station\n"); return error; } /* Set initial MRR rate. */ DPRINTFN(DBG_INIT, "maxbasicrate=%jd", maxbasicrate, 0, 0, 0); urtwn_write_1(sc, R92C_INIDATA_RATE_SEL(RTWN_MACID_BC), maxbasicrate); /* Set rates mask for unicast frames. */ cmd.macid = RTWN_MACID_BSS | RTWN_MACID_VALID; if (ni->ni_capinfo & IEEE80211_CAPINFO_SHORT_PREAMBLE) cmd.macid |= RTWN_MACID_SHORTGI; cmd.mask = htole32((mode << 28) | rates); error = urtwn_fw_cmd(sc, R92C_CMD_MACID_CONFIG, &cmd, sizeof(cmd)); if (error != 0) { aprint_error_dev(sc->sc_dev, "could not add BSS station\n"); return error; } /* Set initial MRR rate. */ DPRINTFN(DBG_INIT, "maxrate=%jd", maxrate, 0, 0, 0); urtwn_write_1(sc, R92C_INIDATA_RATE_SEL(RTWN_MACID_BSS), maxrate); rrsr_rate = ic->ic_fixed_rate; if (rrsr_rate == -1) rrsr_rate = 11; rrsr_mask = 0xffff >> (15 - rrsr_rate); urtwn_write_2(sc, R92C_RRSR, rrsr_mask); /* Indicate highest supported rate. */ ni->ni_txrate = rs->rs_nrates - 1; return 0; } static int urtwn_get_nettype(struct urtwn_softc *sc) { struct ieee80211com *ic = &sc->sc_ic; int type; URTWNHIST_FUNC(); URTWNHIST_CALLED(); switch (ic->ic_opmode) { case IEEE80211_M_STA: type = R92C_CR_NETTYPE_INFRA; break; case IEEE80211_M_IBSS: type = R92C_CR_NETTYPE_ADHOC; break; default: type = R92C_CR_NETTYPE_NOLINK; break; } return type; } static void urtwn_set_nettype0_msr(struct urtwn_softc *sc, uint8_t type) { uint8_t reg; URTWNHIST_FUNC(); URTWNHIST_CALLARGS("type=%jd", type, 0, 0, 0); KASSERT(mutex_owned(&sc->sc_write_mtx)); reg = urtwn_read_1(sc, R92C_CR + 2) & 0x0c; urtwn_write_1(sc, R92C_CR + 2, reg | type); } static void urtwn_tsf_sync_enable(struct urtwn_softc *sc) { struct ieee80211_node *ni = sc->sc_ic.ic_bss; uint64_t tsf; URTWNHIST_FUNC(); URTWNHIST_CALLED(); KASSERT(mutex_owned(&sc->sc_write_mtx)); /* Enable TSF synchronization. */ urtwn_write_1(sc, R92C_BCN_CTRL, urtwn_read_1(sc, R92C_BCN_CTRL) & ~R92C_BCN_CTRL_DIS_TSF_UDT0); /* Correct TSF */ urtwn_write_1(sc, R92C_BCN_CTRL, urtwn_read_1(sc, R92C_BCN_CTRL) & ~R92C_BCN_CTRL_EN_BCN); /* Set initial TSF. */ tsf = ni->ni_tstamp.tsf; tsf = le64toh(tsf); tsf = tsf - (tsf % (ni->ni_intval * IEEE80211_DUR_TU)); tsf -= IEEE80211_DUR_TU; urtwn_write_4(sc, R92C_TSFTR + 0, (uint32_t)tsf); urtwn_write_4(sc, R92C_TSFTR + 4, (uint32_t)(tsf >> 32)); urtwn_write_1(sc, R92C_BCN_CTRL, urtwn_read_1(sc, R92C_BCN_CTRL) | R92C_BCN_CTRL_EN_BCN); } static void urtwn_set_led(struct urtwn_softc *sc, int led, int on) { uint8_t reg; URTWNHIST_FUNC(); URTWNHIST_CALLARGS("led=%jd, on=%jd", led, on, 0, 0); KASSERT(mutex_owned(&sc->sc_write_mtx)); if (led == URTWN_LED_LINK) { if (ISSET(sc->chip, URTWN_CHIP_92EU)) { urtwn_write_1(sc, 0x64, urtwn_read_1(sc, 0x64) & 0xfe); reg = urtwn_read_1(sc, R92C_LEDCFG1) & R92E_LEDSON; urtwn_write_1(sc, R92C_LEDCFG1, reg | (R92C_LEDCFG0_DIS << 1)); if (on) { reg = urtwn_read_1(sc, R92C_LEDCFG1) & R92E_LEDSON; urtwn_write_1(sc, R92C_LEDCFG1, reg); } } else if (ISSET(sc->chip, URTWN_CHIP_88E)) { reg = urtwn_read_1(sc, R92C_LEDCFG2) & 0xf0; urtwn_write_1(sc, R92C_LEDCFG2, reg | 0x60); if (!on) { reg = urtwn_read_1(sc, R92C_LEDCFG2) & 0x90; urtwn_write_1(sc, R92C_LEDCFG2, reg | R92C_LEDCFG0_DIS); reg = urtwn_read_1(sc, R92C_MAC_PINMUX_CFG); urtwn_write_1(sc, R92C_MAC_PINMUX_CFG, reg & 0xfe); } } else { reg = urtwn_read_1(sc, R92C_LEDCFG0) & 0x70; if (!on) { reg |= R92C_LEDCFG0_DIS; } urtwn_write_1(sc, R92C_LEDCFG0, reg); } sc->ledlink = on; /* Save LED state. */ } } static void urtwn_calib_to(void *arg) { struct urtwn_softc *sc = arg; URTWNHIST_FUNC(); URTWNHIST_CALLED(); if (sc->sc_dying) return; /* Do it in a process context. */ urtwn_do_async(sc, urtwn_calib_to_cb, NULL, 0); } /* ARGSUSED */ static void urtwn_calib_to_cb(struct urtwn_softc *sc, void *arg) { struct r92c_fw_cmd_rssi cmd; struct r92e_fw_cmd_rssi cmde; URTWNHIST_FUNC(); URTWNHIST_CALLED(); if (sc->sc_ic.ic_state != IEEE80211_S_RUN) goto restart_timer; mutex_enter(&sc->sc_write_mtx); if (sc->avg_pwdb != -1) { /* Indicate Rx signal strength to FW for rate adaptation. */ memset(&cmd, 0, sizeof(cmd)); memset(&cmde, 0, sizeof(cmde)); cmd.macid = 0; /* BSS. */ cmde.macid = 0; /* BSS. */ cmd.pwdb = sc->avg_pwdb; cmde.pwdb = sc->avg_pwdb; DPRINTFN(DBG_RF, "sending RSSI command avg=%jd", sc->avg_pwdb, 0, 0, 0); if (!ISSET(sc->chip, URTWN_CHIP_92EU)) { urtwn_fw_cmd(sc, R92C_CMD_RSSI_SETTING, &cmd, sizeof(cmd)); } else { urtwn_fw_cmd(sc, R92E_CMD_RSSI_REPORT, &cmde, sizeof(cmde)); } } /* Do temperature compensation. */ urtwn_temp_calib(sc); mutex_exit(&sc->sc_write_mtx); restart_timer: if (!sc->sc_dying) { /* Restart calibration timer. */ callout_schedule(&sc->sc_calib_to, hz); } } static void urtwn_next_scan(void *arg) { struct urtwn_softc *sc = arg; int s; URTWNHIST_FUNC(); URTWNHIST_CALLED(); if (sc->sc_dying) return; s = splnet(); if (sc->sc_ic.ic_state == IEEE80211_S_SCAN) ieee80211_next_scan(&sc->sc_ic); splx(s); } static void urtwn_newassoc(struct ieee80211_node *ni, int isnew) { URTWNHIST_FUNC(); URTWNHIST_CALLARGS("new node %06jx%06jx", ni->ni_macaddr[0] << 2 | ni->ni_macaddr[1] << 1 | ni->ni_macaddr[2], ni->ni_macaddr[3] << 2 | ni->ni_macaddr[4] << 1 | ni->ni_macaddr[5], 0, 0); /* start with lowest Tx rate */ ni->ni_txrate = 0; } static int urtwn_newstate(struct ieee80211com *ic, enum ieee80211_state nstate, int arg) { struct urtwn_softc *sc = ic->ic_ifp->if_softc; struct urtwn_cmd_newstate cmd; URTWNHIST_FUNC(); URTWNHIST_CALLARGS("nstate=%jd, arg=%jd", nstate, arg, 0, 0); callout_stop(&sc->sc_scan_to); callout_stop(&sc->sc_calib_to); /* Do it in a process context. */ cmd.state = nstate; cmd.arg = arg; urtwn_do_async(sc, urtwn_newstate_cb, &cmd, sizeof(cmd)); return 0; } static void urtwn_newstate_cb(struct urtwn_softc *sc, void *arg) { struct urtwn_cmd_newstate *cmd = arg; struct ieee80211com *ic = &sc->sc_ic; struct ieee80211_node *ni; enum ieee80211_state ostate = ic->ic_state; enum ieee80211_state nstate = cmd->state; uint32_t reg; uint8_t sifs_time, msr; int s; URTWNHIST_FUNC(); URTWNHIST_CALLED(); DPRINTFN(DBG_STM, "%jd->%jd", ostate, nstate, 0, 0); s = splnet(); mutex_enter(&sc->sc_write_mtx); callout_stop(&sc->sc_scan_to); callout_stop(&sc->sc_calib_to); switch (ostate) { case IEEE80211_S_INIT: break; case IEEE80211_S_SCAN: if (nstate != IEEE80211_S_SCAN) { /* * End of scanning */ /* flush 4-AC Queue after site_survey */ urtwn_write_1(sc, R92C_TXPAUSE, 0x0); /* Allow Rx from our BSSID only. */ urtwn_write_4(sc, R92C_RCR, urtwn_read_4(sc, R92C_RCR) | R92C_RCR_CBSSID_DATA | R92C_RCR_CBSSID_BCN); } break; case IEEE80211_S_AUTH: case IEEE80211_S_ASSOC: break; case IEEE80211_S_RUN: /* Turn link LED off. */ urtwn_set_led(sc, URTWN_LED_LINK, 0); /* Set media status to 'No Link'. */ urtwn_set_nettype0_msr(sc, R92C_CR_NETTYPE_NOLINK); /* Stop Rx of data frames. */ urtwn_write_2(sc, R92C_RXFLTMAP2, 0); /* Reset TSF. */ urtwn_write_1(sc, R92C_DUAL_TSF_RST, 0x03); /* Disable TSF synchronization. */ urtwn_write_1(sc, R92C_BCN_CTRL, urtwn_read_1(sc, R92C_BCN_CTRL) | R92C_BCN_CTRL_DIS_TSF_UDT0); /* Back to 20MHz mode */ urtwn_set_chan(sc, ic->ic_curchan, IEEE80211_HTINFO_2NDCHAN_NONE); if (ic->ic_opmode == IEEE80211_M_IBSS || ic->ic_opmode == IEEE80211_M_HOSTAP) { /* Stop BCN */ urtwn_write_1(sc, R92C_BCN_CTRL, urtwn_read_1(sc, R92C_BCN_CTRL) & ~(R92C_BCN_CTRL_EN_BCN | R92C_BCN_CTRL_TXBCN_RPT)); } /* Reset EDCA parameters. */ urtwn_write_4(sc, R92C_EDCA_VO_PARAM, 0x002f3217); urtwn_write_4(sc, R92C_EDCA_VI_PARAM, 0x005e4317); urtwn_write_4(sc, R92C_EDCA_BE_PARAM, 0x00105320); urtwn_write_4(sc, R92C_EDCA_BK_PARAM, 0x0000a444); /* flush all cam entries */ urtwn_cam_init(sc); break; } switch (nstate) { case IEEE80211_S_INIT: /* Turn link LED off. */ urtwn_set_led(sc, URTWN_LED_LINK, 0); break; case IEEE80211_S_SCAN: if (ostate != IEEE80211_S_SCAN) { /* * Begin of scanning */ /* Set gain for scanning. */ reg = urtwn_bb_read(sc, R92C_OFDM0_AGCCORE1(0)); reg = RW(reg, R92C_OFDM0_AGCCORE1_GAIN, 0x20); urtwn_bb_write(sc, R92C_OFDM0_AGCCORE1(0), reg); if (!ISSET(sc->chip, URTWN_CHIP_88E)) { reg = urtwn_bb_read(sc, R92C_OFDM0_AGCCORE1(1)); reg = RW(reg, R92C_OFDM0_AGCCORE1_GAIN, 0x20); urtwn_bb_write(sc, R92C_OFDM0_AGCCORE1(1), reg); } /* Set media status to 'No Link'. */ urtwn_set_nettype0_msr(sc, R92C_CR_NETTYPE_NOLINK); /* Allow Rx from any BSSID. */ urtwn_write_4(sc, R92C_RCR, urtwn_read_4(sc, R92C_RCR) & ~(R92C_RCR_CBSSID_DATA | R92C_RCR_CBSSID_BCN)); /* Stop Rx of data frames. */ urtwn_write_2(sc, R92C_RXFLTMAP2, 0); /* Disable update TSF */ urtwn_write_1(sc, R92C_BCN_CTRL, urtwn_read_1(sc, R92C_BCN_CTRL) | R92C_BCN_CTRL_DIS_TSF_UDT0); } /* Make link LED blink during scan. */ urtwn_set_led(sc, URTWN_LED_LINK, !sc->ledlink); /* Pause AC Tx queues. */ urtwn_write_1(sc, R92C_TXPAUSE, urtwn_read_1(sc, R92C_TXPAUSE) | 0x0f); urtwn_set_chan(sc, ic->ic_curchan, IEEE80211_HTINFO_2NDCHAN_NONE); /* Start periodic scan. */ if (!sc->sc_dying) callout_schedule(&sc->sc_scan_to, hz / 5); break; case IEEE80211_S_AUTH: /* Set initial gain under link. */ reg = urtwn_bb_read(sc, R92C_OFDM0_AGCCORE1(0)); reg = RW(reg, R92C_OFDM0_AGCCORE1_GAIN, 0x32); urtwn_bb_write(sc, R92C_OFDM0_AGCCORE1(0), reg); if (!ISSET(sc->chip, URTWN_CHIP_88E)) { reg = urtwn_bb_read(sc, R92C_OFDM0_AGCCORE1(1)); reg = RW(reg, R92C_OFDM0_AGCCORE1_GAIN, 0x32); urtwn_bb_write(sc, R92C_OFDM0_AGCCORE1(1), reg); } /* Set media status to 'No Link'. */ urtwn_set_nettype0_msr(sc, R92C_CR_NETTYPE_NOLINK); /* Allow Rx from any BSSID. */ urtwn_write_4(sc, R92C_RCR, urtwn_read_4(sc, R92C_RCR) & ~(R92C_RCR_CBSSID_DATA | R92C_RCR_CBSSID_BCN)); urtwn_set_chan(sc, ic->ic_curchan, IEEE80211_HTINFO_2NDCHAN_NONE); break; case IEEE80211_S_ASSOC: break; case IEEE80211_S_RUN: ni = ic->ic_bss; /* XXX: Set 20MHz mode */ urtwn_set_chan(sc, ic->ic_curchan, IEEE80211_HTINFO_2NDCHAN_NONE); if (ic->ic_opmode == IEEE80211_M_MONITOR) { /* Back to 20MHz mode */ urtwn_set_chan(sc, ic->ic_curchan, IEEE80211_HTINFO_2NDCHAN_NONE); /* Set media status to 'No Link'. */ urtwn_set_nettype0_msr(sc, R92C_CR_NETTYPE_NOLINK); /* Enable Rx of data frames. */ urtwn_write_2(sc, R92C_RXFLTMAP2, 0xffff); /* Allow Rx from any BSSID. */ urtwn_write_4(sc, R92C_RCR, urtwn_read_4(sc, R92C_RCR) & ~(R92C_RCR_CBSSID_DATA | R92C_RCR_CBSSID_BCN)); /* Accept Rx data/control/management frames */ urtwn_write_4(sc, R92C_RCR, urtwn_read_4(sc, R92C_RCR) | R92C_RCR_ADF | R92C_RCR_ACF | R92C_RCR_AMF); /* Turn link LED on. */ urtwn_set_led(sc, URTWN_LED_LINK, 1); break; } /* Set media status to 'Associated'. */ urtwn_set_nettype0_msr(sc, urtwn_get_nettype(sc)); /* Set BSSID. */ urtwn_write_4(sc, R92C_BSSID + 0, LE_READ_4(&ni->ni_bssid[0])); urtwn_write_4(sc, R92C_BSSID + 4, LE_READ_2(&ni->ni_bssid[4])); if (ic->ic_curmode == IEEE80211_MODE_11B) { urtwn_write_1(sc, R92C_INIRTS_RATE_SEL, 0); } else { /* 802.11b/g */ urtwn_write_1(sc, R92C_INIRTS_RATE_SEL, 3); } /* Enable Rx of data frames. */ urtwn_write_2(sc, R92C_RXFLTMAP2, 0xffff); /* Set beacon interval. */ urtwn_write_2(sc, R92C_BCN_INTERVAL, ni->ni_intval); msr = urtwn_read_1(sc, R92C_MSR); msr &= R92C_MSR_MASK; switch (ic->ic_opmode) { case IEEE80211_M_STA: /* Allow Rx from our BSSID only. */ urtwn_write_4(sc, R92C_RCR, urtwn_read_4(sc, R92C_RCR) | R92C_RCR_CBSSID_DATA | R92C_RCR_CBSSID_BCN); /* Enable TSF synchronization. */ urtwn_tsf_sync_enable(sc); msr |= R92C_MSR_INFRA; break; case IEEE80211_M_HOSTAP: urtwn_write_2(sc, R92C_BCNTCFG, 0x000f); /* Allow Rx from any BSSID. */ urtwn_write_4(sc, R92C_RCR, urtwn_read_4(sc, R92C_RCR) & ~(R92C_RCR_CBSSID_DATA | R92C_RCR_CBSSID_BCN)); /* Reset TSF timer to zero. */ reg = urtwn_read_4(sc, R92C_TCR); reg &= ~0x01; urtwn_write_4(sc, R92C_TCR, reg); reg |= 0x01; urtwn_write_4(sc, R92C_TCR, reg); msr |= R92C_MSR_AP; break; default: msr |= R92C_MSR_ADHOC; break; } urtwn_write_1(sc, R92C_MSR, msr); sifs_time = 10; urtwn_write_1(sc, R92C_SIFS_CCK + 1, sifs_time); urtwn_write_1(sc, R92C_SIFS_OFDM + 1, sifs_time); urtwn_write_1(sc, R92C_SPEC_SIFS + 1, sifs_time); urtwn_write_1(sc, R92C_MAC_SPEC_SIFS + 1, sifs_time); urtwn_write_1(sc, R92C_R2T_SIFS + 1, sifs_time); urtwn_write_1(sc, R92C_T2T_SIFS + 1, sifs_time); /* Initialize rate adaptation. */ if (ISSET(sc->chip, URTWN_CHIP_88E) || ISSET(sc->chip, URTWN_CHIP_92EU)) ni->ni_txrate = ni->ni_rates.rs_nrates - 1; else urtwn_ra_init(sc); /* Turn link LED on. */ urtwn_set_led(sc, URTWN_LED_LINK, 1); /* Reset average RSSI. */ sc->avg_pwdb = -1; /* Reset temperature calibration state machine. */ sc->thcal_state = 0; sc->thcal_lctemp = 0; /* Start periodic calibration. */ if (!sc->sc_dying) callout_schedule(&sc->sc_calib_to, hz); break; } (*sc->sc_newstate)(ic, nstate, cmd->arg); mutex_exit(&sc->sc_write_mtx); splx(s); } static int urtwn_wme_update(struct ieee80211com *ic) { struct urtwn_softc *sc = ic->ic_ifp->if_softc; URTWNHIST_FUNC(); URTWNHIST_CALLED(); /* don't override default WME values if WME is not actually enabled */ if (!(ic->ic_flags & IEEE80211_F_WME)) return 0; /* Do it in a process context. */ urtwn_do_async(sc, urtwn_wme_update_cb, NULL, 0); return 0; } static void urtwn_wme_update_cb(struct urtwn_softc *sc, void *arg) { static const uint16_t ac2reg[WME_NUM_AC] = { R92C_EDCA_BE_PARAM, R92C_EDCA_BK_PARAM, R92C_EDCA_VI_PARAM, R92C_EDCA_VO_PARAM }; struct ieee80211com *ic = &sc->sc_ic; const struct wmeParams *wmep; int ac, aifs, slottime; int s; URTWNHIST_FUNC(); URTWNHIST_CALLED(); DPRINTFN(DBG_STM, "called", 0, 0, 0, 0); s = splnet(); mutex_enter(&sc->sc_write_mtx); slottime = (ic->ic_flags & IEEE80211_F_SHSLOT) ? 9 : 20; for (ac = 0; ac < WME_NUM_AC; ac++) { wmep = &ic->ic_wme.wme_chanParams.cap_wmeParams[ac]; /* AIFS[AC] = AIFSN[AC] * aSlotTime + aSIFSTime. */ aifs = wmep->wmep_aifsn * slottime + 10; urtwn_write_4(sc, ac2reg[ac], SM(R92C_EDCA_PARAM_TXOP, wmep->wmep_txopLimit) | SM(R92C_EDCA_PARAM_ECWMIN, wmep->wmep_logcwmin) | SM(R92C_EDCA_PARAM_ECWMAX, wmep->wmep_logcwmax) | SM(R92C_EDCA_PARAM_AIFS, aifs)); } mutex_exit(&sc->sc_write_mtx); splx(s); } static void urtwn_update_avgrssi(struct urtwn_softc *sc, int rate, int8_t rssi) { int pwdb; URTWNHIST_FUNC(); URTWNHIST_CALLARGS("rate=%jd, rsst=%jd", rate, rssi, 0, 0); /* Convert antenna signal to percentage. */ if (rssi <= -100 || rssi >= 20) pwdb = 0; else if (rssi >= 0) pwdb = 100; else pwdb = 100 + rssi; if (!ISSET(sc->chip, URTWN_CHIP_88E)) { if (rate <= 3) { /* CCK gain is smaller than OFDM/MCS gain. */ pwdb += 6; if (pwdb > 100) pwdb = 100; if (pwdb <= 14) pwdb -= 4; else if (pwdb <= 26) pwdb -= 8; else if (pwdb <= 34) pwdb -= 6; else if (pwdb <= 42) pwdb -= 2; } } if (sc->avg_pwdb == -1) /* Init. */ sc->avg_pwdb = pwdb; else if (sc->avg_pwdb < pwdb) sc->avg_pwdb = ((sc->avg_pwdb * 19 + pwdb) / 20) + 1; else sc->avg_pwdb = ((sc->avg_pwdb * 19 + pwdb) / 20); DPRINTFN(DBG_RF, "rate=%jd rssi=%jd PWDB=%jd EMA=%jd", rate, rssi, pwdb, sc->avg_pwdb); } static int8_t urtwn_get_rssi(struct urtwn_softc *sc, int rate, void *physt) { static const int8_t cckoff[] = { 16, -12, -26, -46 }; struct r92c_rx_phystat *phy; struct r92c_rx_cck *cck; uint8_t rpt; int8_t rssi; URTWNHIST_FUNC(); URTWNHIST_CALLARGS("rate=%jd", rate, 0, 0, 0); if (rate <= 3) { cck = (struct r92c_rx_cck *)physt; if (ISSET(sc->sc_flags, URTWN_FLAG_CCK_HIPWR)) { rpt = (cck->agc_rpt >> 5) & 0x3; rssi = (cck->agc_rpt & 0x1f) << 1; } else { rpt = (cck->agc_rpt >> 6) & 0x3; rssi = cck->agc_rpt & 0x3e; } rssi = cckoff[rpt] - rssi; } else { /* OFDM/HT. */ phy = (struct r92c_rx_phystat *)physt; rssi = ((le32toh(phy->phydw1) >> 1) & 0x7f) - 110; } return rssi; } static int8_t urtwn_r88e_get_rssi(struct urtwn_softc *sc, int rate, void *physt) { struct r92c_rx_phystat *phy; struct r88e_rx_cck *cck; uint8_t cck_agc_rpt, lna_idx, vga_idx; int8_t rssi; URTWNHIST_FUNC(); URTWNHIST_CALLARGS("rate=%jd", rate, 0, 0, 0); rssi = 0; if (rate <= 3) { cck = (struct r88e_rx_cck *)physt; cck_agc_rpt = cck->agc_rpt; lna_idx = (cck_agc_rpt & 0xe0) >> 5; vga_idx = cck_agc_rpt & 0x1f; switch (lna_idx) { case 7: if (vga_idx <= 27) rssi = -100 + 2* (27 - vga_idx); else rssi = -100; break; case 6: rssi = -48 + 2 * (2 - vga_idx); break; case 5: rssi = -42 + 2 * (7 - vga_idx); break; case 4: rssi = -36 + 2 * (7 - vga_idx); break; case 3: rssi = -24 + 2 * (7 - vga_idx); break; case 2: rssi = -12 + 2 * (5 - vga_idx); break; case 1: rssi = 8 - (2 * vga_idx); break; case 0: rssi = 14 - (2 * vga_idx); break; } rssi += 6; } else { /* OFDM/HT. */ phy = (struct r92c_rx_phystat *)physt; rssi = ((le32toh(phy->phydw1) >> 1) & 0x7f) - 110; } return rssi; } static void urtwn_rx_frame(struct urtwn_softc *sc, uint8_t *buf, int pktlen) { struct ieee80211com *ic = &sc->sc_ic; struct ifnet *ifp = ic->ic_ifp; struct ieee80211_frame *wh; struct ieee80211_node *ni; struct r92c_rx_desc_usb *stat; uint32_t rxdw0, rxdw3; struct mbuf *m; uint8_t rate; int8_t rssi = 0; int s, infosz; URTWNHIST_FUNC(); URTWNHIST_CALLARGS("buf=%jp, pktlen=%#jd", (uintptr_t)buf, pktlen, 0, 0); stat = (struct r92c_rx_desc_usb *)buf; rxdw0 = le32toh(stat->rxdw0); rxdw3 = le32toh(stat->rxdw3); if (__predict_false(rxdw0 & (R92C_RXDW0_CRCERR | R92C_RXDW0_ICVERR))) { /* * This should not happen since we setup our Rx filter * to not receive these frames. */ DPRINTFN(DBG_RX, "CRC error", 0, 0, 0, 0); if_statinc(ifp, if_ierrors); return; } /* * XXX: This will drop most control packets. Do we really * want this in IEEE80211_M_MONITOR mode? */ // if (__predict_false(pktlen < (int)sizeof(*wh))) { if (__predict_false(pktlen < (int)sizeof(struct ieee80211_frame_ack))) { DPRINTFN(DBG_RX, "packet too short %jd", pktlen, 0, 0, 0); ic->ic_stats.is_rx_tooshort++; if_statinc(ifp, if_ierrors); return; } if (__predict_false(pktlen > MCLBYTES)) { DPRINTFN(DBG_RX, "packet too big %jd", pktlen, 0, 0, 0); if_statinc(ifp, if_ierrors); return; } rate = MS(rxdw3, R92C_RXDW3_RATE); infosz = MS(rxdw0, R92C_RXDW0_INFOSZ) * 8; /* Get RSSI from PHY status descriptor if present. */ if (infosz != 0 && (rxdw0 & R92C_RXDW0_PHYST)) { if (!ISSET(sc->chip, URTWN_CHIP_92C)) rssi = urtwn_r88e_get_rssi(sc, rate, &stat[1]); else rssi = urtwn_get_rssi(sc, rate, &stat[1]); /* Update our average RSSI. */ urtwn_update_avgrssi(sc, rate, rssi); } DPRINTFN(DBG_RX, "Rx frame len=%jd rate=%jd infosz=%jd rssi=%jd", pktlen, rate, infosz, rssi); MGETHDR(m, M_DONTWAIT, MT_DATA); if (__predict_false(m == NULL)) { aprint_error_dev(sc->sc_dev, "couldn't allocate rx mbuf\n"); ic->ic_stats.is_rx_nobuf++; if_statinc(ifp, if_ierrors); return; } if (pktlen > (int)MHLEN) { MCLGET(m, M_DONTWAIT); if (__predict_false(!(m->m_flags & M_EXT))) { aprint_error_dev(sc->sc_dev, "couldn't allocate rx mbuf cluster\n"); m_freem(m); ic->ic_stats.is_rx_nobuf++; if_statinc(ifp, if_ierrors); return; } } /* Finalize mbuf. */ m_set_rcvif(m, ifp); wh = (struct ieee80211_frame *)((uint8_t *)&stat[1] + infosz); memcpy(mtod(m, uint8_t *), wh, pktlen); m->m_pkthdr.len = m->m_len = pktlen; s = splnet(); if (__predict_false(sc->sc_drvbpf != NULL)) { struct urtwn_rx_radiotap_header *tap = &sc->sc_rxtap; tap->wr_flags = 0; if (!(rxdw3 & R92C_RXDW3_HT)) { switch (rate) { /* CCK. */ case 0: tap->wr_rate = 2; break; case 1: tap->wr_rate = 4; break; case 2: tap->wr_rate = 11; break; case 3: tap->wr_rate = 22; break; /* OFDM. */ case 4: tap->wr_rate = 12; break; case 5: tap->wr_rate = 18; break; case 6: tap->wr_rate = 24; break; case 7: tap->wr_rate = 36; break; case 8: tap->wr_rate = 48; break; case 9: tap->wr_rate = 72; break; case 10: tap->wr_rate = 96; break; case 11: tap->wr_rate = 108; break; } } else if (rate >= 12) { /* MCS0~15. */ /* Bit 7 set means HT MCS instead of rate. */ tap->wr_rate = 0x80 | (rate - 12); } tap->wr_dbm_antsignal = rssi; tap->wr_chan_freq = htole16(ic->ic_curchan->ic_freq); tap->wr_chan_flags = htole16(ic->ic_curchan->ic_flags); bpf_mtap2(sc->sc_drvbpf, tap, sc->sc_rxtap_len, m, BPF_D_IN); } ni = ieee80211_find_rxnode(ic, (struct ieee80211_frame_min *)wh); /* push the frame up to the 802.11 stack */ ieee80211_input(ic, m, ni, rssi, 0); /* Node is no longer needed. */ ieee80211_free_node(ni); splx(s); } static void urtwn_rxeof(struct usbd_xfer *xfer, void *priv, usbd_status status) { struct urtwn_rx_data *data = priv; struct urtwn_softc *sc = data->sc; struct r92c_rx_desc_usb *stat; size_t pidx = data->pidx; uint32_t rxdw0; uint8_t *buf; int len, totlen, pktlen, infosz, npkts; URTWNHIST_FUNC(); URTWNHIST_CALLED(); DPRINTFN(DBG_RX, "status=%jd", status, 0, 0, 0); mutex_enter(&sc->sc_rx_mtx); TAILQ_REMOVE(&sc->rx_free_list[pidx], data, next); TAILQ_INSERT_TAIL(&sc->rx_free_list[pidx], data, next); /* Put this Rx buffer back to our free list. */ mutex_exit(&sc->sc_rx_mtx); if (__predict_false(status != USBD_NORMAL_COMPLETION)) { if (status == USBD_STALLED) usbd_clear_endpoint_stall_async(sc->rx_pipe[pidx]); else if (status != USBD_CANCELLED) goto resubmit; return; } usbd_get_xfer_status(xfer, NULL, NULL, &len, NULL); if (__predict_false(len < (int)sizeof(*stat))) { DPRINTFN(DBG_RX, "xfer too short %jd", len, 0, 0, 0); goto resubmit; } buf = data->buf; /* Get the number of encapsulated frames. */ stat = (struct r92c_rx_desc_usb *)buf; if (ISSET(sc->chip, URTWN_CHIP_92EU)) npkts = MS(le32toh(stat->rxdw2), R92E_RXDW2_PKTCNT); else npkts = MS(le32toh(stat->rxdw2), R92C_RXDW2_PKTCNT); DPRINTFN(DBG_RX, "Rx %jd frames in one chunk", npkts, 0, 0, 0); if (npkts != 0) rnd_add_uint32(&sc->rnd_source, npkts); /* Process all of them. */ while (npkts-- > 0) { if (__predict_false(len < (int)sizeof(*stat))) { DPRINTFN(DBG_RX, "len(%jd) is short than header", len, 0, 0, 0); break; } stat = (struct r92c_rx_desc_usb *)buf; rxdw0 = le32toh(stat->rxdw0); pktlen = MS(rxdw0, R92C_RXDW0_PKTLEN); if (__predict_false(pktlen == 0)) { DPRINTFN(DBG_RX, "pktlen is 0 byte", 0, 0, 0, 0); break; } infosz = MS(rxdw0, R92C_RXDW0_INFOSZ) * 8; /* Make sure everything fits in xfer. */ totlen = sizeof(*stat) + infosz + pktlen; if (__predict_false(totlen > len)) { DPRINTFN(DBG_RX, "pktlen (%jd+%jd+%jd) > %jd", (int)sizeof(*stat), infosz, pktlen, len); break; } /* Process 802.11 frame. */ urtwn_rx_frame(sc, buf, pktlen); /* Next chunk is 128-byte aligned. */ totlen = roundup2(totlen, 128); buf += totlen; len -= totlen; } resubmit: /* Setup a new transfer. */ usbd_setup_xfer(xfer, data, data->buf, URTWN_RXBUFSZ, USBD_SHORT_XFER_OK, USBD_NO_TIMEOUT, urtwn_rxeof); (void)usbd_transfer(xfer); } static void urtwn_put_tx_data(struct urtwn_softc *sc, struct urtwn_tx_data *data) { size_t pidx = data->pidx; mutex_enter(&sc->sc_tx_mtx); /* Put this Tx buffer back to our free list. */ TAILQ_INSERT_TAIL(&sc->tx_free_list[pidx], data, next); mutex_exit(&sc->sc_tx_mtx); } static void urtwn_txeof(struct usbd_xfer *xfer, void *priv, usbd_status status) { struct urtwn_tx_data *data = priv; struct urtwn_softc *sc = data->sc; struct ifnet *ifp = &sc->sc_if; size_t pidx = data->pidx; int s; URTWNHIST_FUNC(); URTWNHIST_CALLED(); DPRINTFN(DBG_TX, "status=%jd", status, 0, 0, 0); urtwn_put_tx_data(sc, data); s = splnet(); sc->tx_timer = 0; ifp->if_flags &= ~IFF_OACTIVE; if (__predict_false(status != USBD_NORMAL_COMPLETION)) { if (status != USBD_NOT_STARTED && status != USBD_CANCELLED) { if (status == USBD_STALLED) { struct usbd_pipe *pipe = sc->tx_pipe[pidx]; usbd_clear_endpoint_stall_async(pipe); } device_printf(sc->sc_dev, "transmit failed, %s\n", usbd_errstr(status)); if_statinc(ifp, if_oerrors); } splx(s); return; } if_statinc(ifp, if_opackets); urtwn_start(ifp); splx(s); } static int urtwn_tx(struct urtwn_softc *sc, struct mbuf *m, struct ieee80211_node *ni, struct urtwn_tx_data *data) { struct ieee80211com *ic = &sc->sc_ic; struct ieee80211_frame *wh; struct ieee80211_key *k = NULL; struct r92c_tx_desc_usb *txd; size_t i, padsize, xferlen, txd_len; uint16_t seq, sum; uint8_t raid, type, tid; int s, hasqos, error; URTWNHIST_FUNC(); URTWNHIST_CALLED(); wh = mtod(m, struct ieee80211_frame *); type = wh->i_fc[0] & IEEE80211_FC0_TYPE_MASK; txd_len = sizeof(*txd); if (!ISSET(sc->chip, URTWN_CHIP_92EU)) txd_len = 32; if (wh->i_fc[1] & IEEE80211_FC1_WEP) { k = ieee80211_crypto_encap(ic, ni, m); if (k == NULL) { urtwn_put_tx_data(sc, data); m_free(m); return ENOBUFS; } /* packet header may have moved, reset our local pointer */ wh = mtod(m, struct ieee80211_frame *); } if (__predict_false(sc->sc_drvbpf != NULL)) { struct urtwn_tx_radiotap_header *tap = &sc->sc_txtap; tap->wt_flags = 0; tap->wt_chan_freq = htole16(ic->ic_curchan->ic_freq); tap->wt_chan_flags = htole16(ic->ic_curchan->ic_flags); if (wh->i_fc[1] & IEEE80211_FC1_WEP) tap->wt_flags |= IEEE80211_RADIOTAP_F_WEP; /* XXX: set tap->wt_rate? */ bpf_mtap2(sc->sc_drvbpf, tap, sc->sc_txtap_len, m, BPF_D_OUT); } /* non-qos data frames */ tid = R92C_TXDW1_QSEL_BE; if ((hasqos = ieee80211_has_qos(wh))) { /* data frames in 11n mode */ struct ieee80211_qosframe *qwh = (void *)wh; tid = qwh->i_qos[0] & IEEE80211_QOS_TID; } else if (type != IEEE80211_FC0_TYPE_DATA) { tid = R92C_TXDW1_QSEL_MGNT; } if (((txd_len + m->m_pkthdr.len) % 64) == 0) /* XXX: 64 */ padsize = 8; else padsize = 0; if (ISSET(sc->chip, URTWN_CHIP_92EU)) padsize = 0; /* Fill Tx descriptor. */ txd = (struct r92c_tx_desc_usb *)data->buf; memset(txd, 0, txd_len + padsize); txd->txdw0 |= htole32( SM(R92C_TXDW0_PKTLEN, m->m_pkthdr.len) | SM(R92C_TXDW0_OFFSET, txd_len)); if (!ISSET(sc->chip, URTWN_CHIP_92EU)) { txd->txdw0 |= htole32( R92C_TXDW0_OWN | R92C_TXDW0_FSG | R92C_TXDW0_LSG); } if (IEEE80211_IS_MULTICAST(wh->i_addr1)) txd->txdw0 |= htole32(R92C_TXDW0_BMCAST); /* fix pad field */ if (padsize > 0) { DPRINTFN(DBG_TX, "padding: size=%jd", padsize, 0, 0, 0); txd->txdw1 |= htole32(SM(R92C_TXDW1_PKTOFF, (padsize / 8))); } if (!IEEE80211_IS_MULTICAST(wh->i_addr1) && type == IEEE80211_FC0_TYPE_DATA) { if (ic->ic_curmode == IEEE80211_MODE_11B) raid = R92C_RAID_11B; else raid = R92C_RAID_11BG; DPRINTFN(DBG_TX, "data packet: tid=%jd, raid=%jd", tid, raid, 0, 0); if (!ISSET(sc->chip, URTWN_CHIP_92C)) { txd->txdw1 |= htole32( SM(R88E_TXDW1_MACID, RTWN_MACID_BSS) | SM(R92C_TXDW1_QSEL, tid) | SM(R92C_TXDW1_RAID, raid) | R92C_TXDW1_AGGBK); } else txd->txdw1 |= htole32( SM(R92C_TXDW1_MACID, RTWN_MACID_BSS) | SM(R92C_TXDW1_QSEL, tid) | SM(R92C_TXDW1_RAID, raid) | R92C_TXDW1_AGGBK); if (ISSET(sc->chip, URTWN_CHIP_88E)) txd->txdw2 |= htole32(R88E_TXDW2_AGGBK); if (ISSET(sc->chip, URTWN_CHIP_92EU)) txd->txdw3 |= htole32(R92E_TXDW3_AGGBK); if (hasqos) { txd->txdw4 |= htole32(R92C_TXDW4_QOS); } if (ic->ic_flags & IEEE80211_F_USEPROT) { /* for 11g */ if (ic->ic_protmode == IEEE80211_PROT_CTSONLY) { txd->txdw4 |= htole32(R92C_TXDW4_CTS2SELF | R92C_TXDW4_HWRTSEN); } else if (ic->ic_protmode == IEEE80211_PROT_RTSCTS) { txd->txdw4 |= htole32(R92C_TXDW4_RTSEN | R92C_TXDW4_HWRTSEN); } } /* Send RTS at OFDM24. */ txd->txdw4 |= htole32(SM(R92C_TXDW4_RTSRATE, 8)); txd->txdw5 |= htole32(0x0001ff00); /* Send data at OFDM54. */ if (ISSET(sc->chip, URTWN_CHIP_88E)) txd->txdw5 |= htole32(0x13 & 0x3f); else txd->txdw5 |= htole32(SM(R92C_TXDW5_DATARATE, 11)); } else if (type == IEEE80211_FC0_TYPE_MGT) { DPRINTFN(DBG_TX, "mgmt packet", 0, 0, 0, 0); txd->txdw1 |= htole32( SM(R92C_TXDW1_MACID, RTWN_MACID_BSS) | SM(R92C_TXDW1_QSEL, R92C_TXDW1_QSEL_MGNT) | SM(R92C_TXDW1_RAID, R92C_RAID_11B)); /* Force CCK1. */ txd->txdw4 |= htole32(R92C_TXDW4_DRVRATE); /* Use 1Mbps */ txd->txdw5 |= htole32(SM(R92C_TXDW5_DATARATE, 0)); } else { /* broadcast or multicast packets */ DPRINTFN(DBG_TX, "bc or mc packet", 0, 0, 0, 0); txd->txdw1 |= htole32( SM(R92C_TXDW1_MACID, RTWN_MACID_BC) | SM(R92C_TXDW1_RAID, R92C_RAID_11B)); /* Force CCK1. */ txd->txdw4 |= htole32(R92C_TXDW4_DRVRATE); /* Use 1Mbps */ txd->txdw5 |= htole32(SM(R92C_TXDW5_DATARATE, 0)); } /* Set sequence number */ seq = LE_READ_2(&wh->i_seq[0]) >> IEEE80211_SEQ_SEQ_SHIFT; if (!ISSET(sc->chip, URTWN_CHIP_92EU)) { txd->txdseq |= htole16(seq); if (!hasqos) { /* Use HW sequence numbering for non-QoS frames. */ txd->txdw4 |= htole32(R92C_TXDW4_HWSEQ); txd->txdseq |= htole16(R92C_HWSEQ_EN); } } else { txd->txdseq2 |= htole16((seq & R92E_HWSEQ_MASK) << R92E_HWSEQ_SHIFT); if (!hasqos) { /* Use HW sequence numbering for non-QoS frames. */ txd->txdw4 |= htole32(R92C_TXDW4_HWSEQ); txd->txdw7 |= htole16(R92C_HWSEQ_EN); } } /* Compute Tx descriptor checksum. */ sum = 0; for (i = 0; i < R92C_TXDESC_SUMSIZE / 2; i++) sum ^= ((uint16_t *)txd)[i]; txd->txdsum = sum; /* NB: already little endian. */ xferlen = txd_len + m->m_pkthdr.len + padsize; m_copydata(m, 0, m->m_pkthdr.len, (char *)&txd[0] + txd_len + padsize); s = splnet(); usbd_setup_xfer(data->xfer, data, data->buf, xferlen, USBD_FORCE_SHORT_XFER, URTWN_TX_TIMEOUT, urtwn_txeof); error = usbd_transfer(data->xfer); if (__predict_false(error != USBD_NORMAL_COMPLETION && error != USBD_IN_PROGRESS)) { splx(s); DPRINTFN(DBG_TX, "transfer failed %jd", error, 0, 0, 0); return error; } splx(s); return 0; } struct urtwn_tx_data * urtwn_get_tx_data(struct urtwn_softc *sc, size_t pidx) { struct urtwn_tx_data *data = NULL; mutex_enter(&sc->sc_tx_mtx); if (!TAILQ_EMPTY(&sc->tx_free_list[pidx])) { data = TAILQ_FIRST(&sc->tx_free_list[pidx]); TAILQ_REMOVE(&sc->tx_free_list[pidx], data, next); } mutex_exit(&sc->sc_tx_mtx); return data; } static void urtwn_start(struct ifnet *ifp) { struct urtwn_softc *sc = ifp->if_softc; struct ieee80211com *ic = &sc->sc_ic; struct urtwn_tx_data *data; struct ether_header *eh; struct ieee80211_node *ni; struct mbuf *m; URTWNHIST_FUNC(); URTWNHIST_CALLED(); if ((ifp->if_flags & (IFF_RUNNING | IFF_OACTIVE)) != IFF_RUNNING) return; data = NULL; for (;;) { /* Send pending management frames first. */ IF_POLL(&ic->ic_mgtq, m); if (m != NULL) { /* Use AC_VO for management frames. */ data = urtwn_get_tx_data(sc, sc->ac2idx[WME_AC_VO]); if (data == NULL) { ifp->if_flags |= IFF_OACTIVE; DPRINTFN(DBG_TX, "empty tx_free_list", 0, 0, 0, 0); return; } IF_DEQUEUE(&ic->ic_mgtq, m); ni = M_GETCTX(m, struct ieee80211_node *); M_CLEARCTX(m); goto sendit; } if (ic->ic_state != IEEE80211_S_RUN) break; /* Encapsulate and send data frames. */ IFQ_POLL(&ifp->if_snd, m); if (m == NULL) break; struct ieee80211_frame *wh = mtod(m, struct ieee80211_frame *); uint8_t type = wh->i_fc[0] & IEEE80211_FC0_TYPE_MASK; uint8_t qid = WME_AC_BE; if (ieee80211_has_qos(wh)) { /* data frames in 11n mode */ struct ieee80211_qosframe *qwh = (void *)wh; uint8_t tid = qwh->i_qos[0] & IEEE80211_QOS_TID; qid = TID_TO_WME_AC(tid); } else if (type != IEEE80211_FC0_TYPE_DATA) { qid = WME_AC_VO; } data = urtwn_get_tx_data(sc, sc->ac2idx[qid]); if (data == NULL) { ifp->if_flags |= IFF_OACTIVE; DPRINTFN(DBG_TX, "empty tx_free_list", 0, 0, 0, 0); return; } IFQ_DEQUEUE(&ifp->if_snd, m); if (m->m_len < (int)sizeof(*eh) && (m = m_pullup(m, sizeof(*eh))) == NULL) { device_printf(sc->sc_dev, "m_pullup failed\n"); if_statinc(ifp, if_oerrors); urtwn_put_tx_data(sc, data); m_freem(m); continue; } eh = mtod(m, struct ether_header *); ni = ieee80211_find_txnode(ic, eh->ether_dhost); if (ni == NULL) { device_printf(sc->sc_dev, "unable to find transmit node\n"); if_statinc(ifp, if_oerrors); urtwn_put_tx_data(sc, data); m_freem(m); continue; } bpf_mtap(ifp, m, BPF_D_OUT); if ((m = ieee80211_encap(ic, m, ni)) == NULL) { ieee80211_free_node(ni); device_printf(sc->sc_dev, "unable to encapsulate packet\n"); if_statinc(ifp, if_oerrors); urtwn_put_tx_data(sc, data); m_freem(m); continue; } sendit: bpf_mtap3(ic->ic_rawbpf, m, BPF_D_OUT); if (urtwn_tx(sc, m, ni, data) != 0) { m_freem(m); ieee80211_free_node(ni); device_printf(sc->sc_dev, "unable to transmit packet\n"); if_statinc(ifp, if_oerrors); continue; } m_freem(m); ieee80211_free_node(ni); sc->tx_timer = 5; ifp->if_timer = 1; } } static void urtwn_watchdog(struct ifnet *ifp) { struct urtwn_softc *sc = ifp->if_softc; URTWNHIST_FUNC(); URTWNHIST_CALLED(); ifp->if_timer = 0; if (sc->tx_timer > 0) { if (--sc->tx_timer == 0) { device_printf(sc->sc_dev, "device timeout\n"); /* urtwn_init(ifp); XXX needs a process context! */ if_statinc(ifp, if_oerrors); return; } ifp->if_timer = 1; } ieee80211_watchdog(&sc->sc_ic); } static int urtwn_ioctl(struct ifnet *ifp, u_long cmd, void *data) { struct urtwn_softc *sc = ifp->if_softc; struct ieee80211com *ic = &sc->sc_ic; int s, error = 0; URTWNHIST_FUNC(); URTWNHIST_CALLARGS("cmd=0x%08jx, data=%#jx", cmd, (uintptr_t)data, 0, 0); s = splnet(); switch (cmd) { case SIOCSIFFLAGS: if ((error = ifioctl_common(ifp, cmd, data)) != 0) break; switch (ifp->if_flags & (IFF_UP | IFF_RUNNING)) { case IFF_UP | IFF_RUNNING: break; case IFF_UP: urtwn_init(ifp); break; case IFF_RUNNING: urtwn_stop(ifp, 1); break; case 0: break; } break; case SIOCADDMULTI: case SIOCDELMULTI: if ((error = ether_ioctl(ifp, cmd, data)) == ENETRESET) { /* setup multicast filter, etc */ error = 0; } break; case SIOCS80211CHANNEL: /* * This allows for fast channel switching in monitor mode * (used by kismet). In IBSS mode, we must explicitly reset * the interface to generate a new beacon frame. */ error = ieee80211_ioctl(ic, cmd, data); if (error == ENETRESET && ic->ic_opmode == IEEE80211_M_MONITOR) { urtwn_set_chan(sc, ic->ic_curchan, IEEE80211_HTINFO_2NDCHAN_NONE); error = 0; } break; default: error = ieee80211_ioctl(ic, cmd, data); break; } if (error == ENETRESET) { if ((ifp->if_flags & (IFF_UP | IFF_RUNNING)) == (IFF_UP | IFF_RUNNING) && ic->ic_roaming != IEEE80211_ROAMING_MANUAL) { urtwn_init(ifp); } error = 0; } splx(s); return error; } static __inline int urtwn_power_on(struct urtwn_softc *sc) { return sc->sc_power_on(sc); } static int urtwn_r92c_power_on(struct urtwn_softc *sc) { uint32_t reg; int ntries; URTWNHIST_FUNC(); URTWNHIST_CALLED(); KASSERT(mutex_owned(&sc->sc_write_mtx)); /* Wait for autoload done bit. */ for (ntries = 0; ntries < 1000; ntries++) { if (urtwn_read_1(sc, R92C_APS_FSMCO) & R92C_APS_FSMCO_PFM_ALDN) break; DELAY(5); } if (ntries == 1000) { aprint_error_dev(sc->sc_dev, "timeout waiting for chip autoload\n"); return ETIMEDOUT; } /* Unlock ISO/CLK/Power control register. */ urtwn_write_1(sc, R92C_RSV_CTRL, 0); DELAY(5); /* Move SPS into PWM mode. */ urtwn_write_1(sc, R92C_SPS0_CTRL, 0x2b); DELAY(5); reg = urtwn_read_1(sc, R92C_LDOV12D_CTRL); if (!(reg & R92C_LDOV12D_CTRL_LDV12_EN)) { urtwn_write_1(sc, R92C_LDOV12D_CTRL, reg | R92C_LDOV12D_CTRL_LDV12_EN); DELAY(100); urtwn_write_1(sc, R92C_SYS_ISO_CTRL, urtwn_read_1(sc, R92C_SYS_ISO_CTRL) & ~R92C_SYS_ISO_CTRL_MD2PP); } /* Auto enable WLAN. */ urtwn_write_2(sc, R92C_APS_FSMCO, urtwn_read_2(sc, R92C_APS_FSMCO) | R92C_APS_FSMCO_APFM_ONMAC); for (ntries = 0; ntries < 1000; ntries++) { if (!(urtwn_read_2(sc, R92C_APS_FSMCO) & R92C_APS_FSMCO_APFM_ONMAC)) break; DELAY(100); } if (ntries == 1000) { aprint_error_dev(sc->sc_dev, "timeout waiting for MAC auto ON\n"); return ETIMEDOUT; } /* Enable radio, GPIO and LED functions. */ KASSERT((R92C_APS_FSMCO_AFSM_HSUS | R92C_APS_FSMCO_PDN_EN | R92C_APS_FSMCO_PFM_ALDN) == 0x0812); urtwn_write_2(sc, R92C_APS_FSMCO, R92C_APS_FSMCO_AFSM_HSUS | R92C_APS_FSMCO_PDN_EN | R92C_APS_FSMCO_PFM_ALDN); /* Release RF digital isolation. */ urtwn_write_2(sc, R92C_SYS_ISO_CTRL, urtwn_read_2(sc, R92C_SYS_ISO_CTRL) & ~R92C_SYS_ISO_CTRL_DIOR); /* Initialize MAC. */ urtwn_write_1(sc, R92C_APSD_CTRL, urtwn_read_1(sc, R92C_APSD_CTRL) & ~R92C_APSD_CTRL_OFF); for (ntries = 0; ntries < 200; ntries++) { if (!(urtwn_read_1(sc, R92C_APSD_CTRL) & R92C_APSD_CTRL_OFF_STATUS)) break; DELAY(5); } if (ntries == 200) { aprint_error_dev(sc->sc_dev, "timeout waiting for MAC initialization\n"); return ETIMEDOUT; } /* Enable MAC DMA/WMAC/SCHEDULE/SEC blocks. */ reg = urtwn_read_2(sc, R92C_CR); reg |= R92C_CR_HCI_TXDMA_EN | R92C_CR_HCI_RXDMA_EN | R92C_CR_TXDMA_EN | R92C_CR_RXDMA_EN | R92C_CR_PROTOCOL_EN | R92C_CR_SCHEDULE_EN | R92C_CR_MACTXEN | R92C_CR_MACRXEN | R92C_CR_ENSEC; urtwn_write_2(sc, R92C_CR, reg); urtwn_write_1(sc, 0xfe10, 0x19); urtwn_delay_ms(sc, 1); return 0; } static int urtwn_r92e_power_on(struct urtwn_softc *sc) { uint32_t reg; uint32_t val; int ntries; URTWNHIST_FUNC(); URTWNHIST_CALLED(); KASSERT(mutex_owned(&sc->sc_write_mtx)); /* Enable radio, GPIO and LED functions. */ KASSERT((R92C_APS_FSMCO_AFSM_HSUS | R92C_APS_FSMCO_PDN_EN | R92C_APS_FSMCO_PFM_ALDN) == 0x0812); urtwn_write_2(sc, R92C_APS_FSMCO, R92C_APS_FSMCO_AFSM_HSUS | R92C_APS_FSMCO_PDN_EN | R92C_APS_FSMCO_PFM_ALDN); if (urtwn_read_4(sc, R92E_SYS_CFG1_8192E) & R92E_SPSLDO_SEL){ /* LDO. */ urtwn_write_1(sc, R92E_LDO_SWR_CTRL, 0xc3); } else { urtwn_write_2(sc, R92C_SYS_SWR_CTRL2, urtwn_read_2(sc, R92C_SYS_SWR_CTRL2) & 0xffff); urtwn_write_1(sc, R92E_LDO_SWR_CTRL, 0x83); } for (ntries = 0; ntries < 2; ntries++) { urtwn_write_1(sc, R92C_AFE_PLL_CTRL, urtwn_read_1(sc, R92C_AFE_PLL_CTRL)); urtwn_write_2(sc, R92C_AFE_CTRL4, urtwn_read_2(sc, R92C_AFE_CTRL4)); } /* Reset BB. */ urtwn_write_1(sc, R92C_SYS_FUNC_EN, urtwn_read_1(sc, R92C_SYS_FUNC_EN) & ~(R92C_SYS_FUNC_EN_BBRSTB | R92C_SYS_FUNC_EN_BB_GLB_RST)); urtwn_write_1(sc, R92C_AFE_XTAL_CTRL + 2, urtwn_read_1(sc, R92C_AFE_XTAL_CTRL + 2) | 0x80); /* Disable HWPDN. */ urtwn_write_2(sc, R92C_APS_FSMCO, urtwn_read_2(sc, R92C_APS_FSMCO) & ~R92C_APS_FSMCO_APDM_HPDN); /* Disable WL suspend. */ urtwn_write_2(sc, R92C_APS_FSMCO, urtwn_read_2(sc, R92C_APS_FSMCO) & ~(R92C_APS_FSMCO_AFSM_PCIE | R92C_APS_FSMCO_AFSM_HSUS)); urtwn_write_4(sc, R92C_APS_FSMCO, urtwn_read_4(sc, R92C_APS_FSMCO) | R92C_APS_FSMCO_RDY_MACON); urtwn_write_2(sc, R92C_APS_FSMCO, urtwn_read_2(sc, R92C_APS_FSMCO) | R92C_APS_FSMCO_APFM_ONMAC); for (ntries = 0; ntries < 10000; ntries++) { val = urtwn_read_2(sc, R92C_APS_FSMCO) & R92C_APS_FSMCO_APFM_ONMAC; if (val == 0x0) break; DELAY(10); } if (ntries == 10000) { aprint_error_dev(sc->sc_dev, "timeout waiting for chip power up\n"); return ETIMEDOUT; } urtwn_write_2(sc, R92C_CR, 0x00); reg = urtwn_read_2(sc, R92C_CR); reg |= R92C_CR_HCI_TXDMA_EN | R92C_CR_HCI_RXDMA_EN | R92C_CR_TXDMA_EN | R92C_CR_RXDMA_EN | R92C_CR_PROTOCOL_EN | R92C_CR_SCHEDULE_EN | R92C_CR_ENSEC; urtwn_write_2(sc, R92C_CR, reg); return 0; } static int urtwn_r88e_power_on(struct urtwn_softc *sc) { uint32_t reg; uint8_t val; int ntries; URTWNHIST_FUNC(); URTWNHIST_CALLED(); KASSERT(mutex_owned(&sc->sc_write_mtx)); /* Wait for power ready bit. */ for (ntries = 0; ntries < 5000; ntries++) { val = urtwn_read_1(sc, 0x6) & 0x2; if (val == 0x2) break; DELAY(10); } if (ntries == 5000) { aprint_error_dev(sc->sc_dev, "timeout waiting for chip power up\n"); return ETIMEDOUT; } /* Reset BB. */ urtwn_write_1(sc, R92C_SYS_FUNC_EN, urtwn_read_1(sc, R92C_SYS_FUNC_EN) & ~(R92C_SYS_FUNC_EN_BBRSTB | R92C_SYS_FUNC_EN_BB_GLB_RST)); urtwn_write_1(sc, 0x26, urtwn_read_1(sc, 0x26) | 0x80); /* Disable HWPDN. */ urtwn_write_1(sc, 0x5, urtwn_read_1(sc, 0x5) & ~0x80); /* Disable WL suspend. */ urtwn_write_1(sc, 0x5, urtwn_read_1(sc, 0x5) & ~0x18); urtwn_write_1(sc, 0x5, urtwn_read_1(sc, 0x5) | 0x1); for (ntries = 0; ntries < 5000; ntries++) { if (!(urtwn_read_1(sc, 0x5) & 0x1)) break; DELAY(10); } if (ntries == 5000) return ETIMEDOUT; /* Enable LDO normal mode. */ urtwn_write_1(sc, 0x23, urtwn_read_1(sc, 0x23) & ~0x10); /* Enable MAC DMA/WMAC/SCHEDULE/SEC blocks. */ urtwn_write_2(sc, R92C_CR, 0); reg = urtwn_read_2(sc, R92C_CR); reg |= R92C_CR_HCI_TXDMA_EN | R92C_CR_HCI_RXDMA_EN | R92C_CR_TXDMA_EN | R92C_CR_RXDMA_EN | R92C_CR_PROTOCOL_EN | R92C_CR_SCHEDULE_EN | R92C_CR_ENSEC | R92C_CR_CALTMR_EN; urtwn_write_2(sc, R92C_CR, reg); return 0; } static int __noinline urtwn_llt_init(struct urtwn_softc *sc) { size_t i, page_count, pktbuf_count; uint32_t val; int error; URTWNHIST_FUNC(); URTWNHIST_CALLED(); KASSERT(mutex_owned(&sc->sc_write_mtx)); if (sc->chip & URTWN_CHIP_88E) page_count = R88E_TX_PAGE_COUNT; else if (sc->chip & URTWN_CHIP_92EU) page_count = R92E_TX_PAGE_COUNT; else page_count = R92C_TX_PAGE_COUNT; if (sc->chip & URTWN_CHIP_88E) pktbuf_count = R88E_TXPKTBUF_COUNT; else if (sc->chip & URTWN_CHIP_92EU) pktbuf_count = R88E_TXPKTBUF_COUNT; else pktbuf_count = R92C_TXPKTBUF_COUNT; if (sc->chip & URTWN_CHIP_92EU) { val = urtwn_read_4(sc, R92E_AUTO_LLT) | R92E_AUTO_LLT_EN; urtwn_write_4(sc, R92E_AUTO_LLT, val); DELAY(100); val = urtwn_read_4(sc, R92E_AUTO_LLT); if (val & R92E_AUTO_LLT_EN) return EIO; return 0; } /* Reserve pages [0; page_count]. */ for (i = 0; i < page_count; i++) { if ((error = urtwn_llt_write(sc, i, i + 1)) != 0) return error; } /* NB: 0xff indicates end-of-list. */ if ((error = urtwn_llt_write(sc, i, 0xff)) != 0) return error; /* * Use pages [page_count + 1; pktbuf_count - 1] * as ring buffer. */ for (++i; i < pktbuf_count - 1; i++) { if ((error = urtwn_llt_write(sc, i, i + 1)) != 0) return error; } /* Make the last page point to the beginning of the ring buffer. */ error = urtwn_llt_write(sc, i, pktbuf_count + 1); return error; } static void urtwn_fw_reset(struct urtwn_softc *sc) { uint16_t reg; int ntries; URTWNHIST_FUNC(); URTWNHIST_CALLED(); KASSERT(mutex_owned(&sc->sc_write_mtx)); /* Tell 8051 to reset itself. */ mutex_enter(&sc->sc_fwcmd_mtx); urtwn_write_1(sc, R92C_HMETFR + 3, 0x20); sc->fwcur = 0; mutex_exit(&sc->sc_fwcmd_mtx); /* Wait until 8051 resets by itself. */ for (ntries = 0; ntries < 100; ntries++) { reg = urtwn_read_2(sc, R92C_SYS_FUNC_EN); if (!(reg & R92C_SYS_FUNC_EN_CPUEN)) return; DELAY(50); } /* Force 8051 reset. */ urtwn_write_2(sc, R92C_SYS_FUNC_EN, urtwn_read_2(sc, R92C_SYS_FUNC_EN) & ~R92C_SYS_FUNC_EN_CPUEN); } static void urtwn_r88e_fw_reset(struct urtwn_softc *sc) { uint16_t reg; URTWNHIST_FUNC(); URTWNHIST_CALLED(); KASSERT(mutex_owned(&sc->sc_write_mtx)); if (ISSET(sc->chip, URTWN_CHIP_92EU)) { reg = urtwn_read_2(sc, R92C_RSV_CTRL) & ~R92E_RSV_MIO_EN; urtwn_write_2(sc,R92C_RSV_CTRL, reg); } DELAY(50); reg = urtwn_read_2(sc, R92C_SYS_FUNC_EN); urtwn_write_2(sc, R92C_SYS_FUNC_EN, reg & ~R92C_SYS_FUNC_EN_CPUEN); DELAY(50); urtwn_write_2(sc, R92C_SYS_FUNC_EN, reg | R92C_SYS_FUNC_EN_CPUEN); DELAY(50); if (ISSET(sc->chip, URTWN_CHIP_92EU)) { reg = urtwn_read_2(sc, R92C_RSV_CTRL) | R92E_RSV_MIO_EN; urtwn_write_2(sc,R92C_RSV_CTRL, reg); } DELAY(50); mutex_enter(&sc->sc_fwcmd_mtx); /* Init firmware commands ring. */ sc->fwcur = 0; mutex_exit(&sc->sc_fwcmd_mtx); } static int urtwn_fw_loadpage(struct urtwn_softc *sc, int page, uint8_t *buf, int len) { uint32_t reg; int off, mlen, error = 0; URTWNHIST_FUNC(); URTWNHIST_CALLARGS("page=%jd, buf=%#jx, len=%jd", page, (uintptr_t)buf, len, 0); reg = urtwn_read_4(sc, R92C_MCUFWDL); reg = RW(reg, R92C_MCUFWDL_PAGE, page); urtwn_write_4(sc, R92C_MCUFWDL, reg); off = R92C_FW_START_ADDR; while (len > 0) { if (len > 196) mlen = 196; else if (len > 4) mlen = 4; else mlen = 1; error = urtwn_write_region(sc, off, buf, mlen); if (error != 0) break; off += mlen; buf += mlen; len -= mlen; } return error; } static int __noinline urtwn_load_firmware(struct urtwn_softc *sc) { firmware_handle_t fwh; const struct r92c_fw_hdr *hdr; const char *name; u_char *fw, *ptr; size_t len; uint32_t reg; int mlen, ntries, page, error; URTWNHIST_FUNC(); URTWNHIST_CALLED(); KASSERT(mutex_owned(&sc->sc_write_mtx)); /* Read firmware image from the filesystem. */ if (ISSET(sc->chip, URTWN_CHIP_88E)) name = "rtl8188eufw.bin"; else if (ISSET(sc->chip, URTWN_CHIP_92EU)) name = "rtl8192eefw.bin"; else if ((sc->chip & (URTWN_CHIP_UMC_A_CUT | URTWN_CHIP_92C)) == URTWN_CHIP_UMC_A_CUT) name = "rtl8192cfwU.bin"; else name = "rtl8192cfw.bin"; if ((error = firmware_open("if_urtwn", name, &fwh)) != 0) { aprint_error_dev(sc->sc_dev, "failed load firmware of file %s (error %d)\n", name, error); return error; } const size_t fwlen = len = firmware_get_size(fwh); fw = firmware_malloc(len); if (fw == NULL) { aprint_error_dev(sc->sc_dev, "failed to allocate firmware memory\n"); firmware_close(fwh); return ENOMEM; } error = firmware_read(fwh, 0, fw, len); firmware_close(fwh); if (error != 0) { aprint_error_dev(sc->sc_dev, "failed to read firmware (error %d)\n", error); firmware_free(fw, fwlen); return error; } len = fwlen; ptr = fw; hdr = (const struct r92c_fw_hdr *)ptr; /* Check if there is a valid FW header and skip it. */ if ((le16toh(hdr->signature) >> 4) == 0x88c || (le16toh(hdr->signature) >> 4) == 0x88e || (le16toh(hdr->signature) >> 4) == 0x92e || (le16toh(hdr->signature) >> 4) == 0x92c) { DPRINTFN(DBG_INIT, "FW V%jd.%jd", le16toh(hdr->version), le16toh(hdr->subversion), 0, 0); DPRINTFN(DBG_INIT, "%02jd-%02jd %02jd:%02jd", hdr->month, hdr->date, hdr->hour, hdr->minute); ptr += sizeof(*hdr); len -= sizeof(*hdr); } if (urtwn_read_1(sc, R92C_MCUFWDL) & R92C_MCUFWDL_RAM_DL_SEL) { /* Reset MCU ready status */ urtwn_write_1(sc, R92C_MCUFWDL, 0); if (ISSET(sc->chip, URTWN_CHIP_88E) || ISSET(sc->chip, URTWN_CHIP_92EU)) urtwn_r88e_fw_reset(sc); else urtwn_fw_reset(sc); } if (!ISSET(sc->chip, URTWN_CHIP_88E) && !ISSET(sc->chip, URTWN_CHIP_92EU)) { urtwn_write_2(sc, R92C_SYS_FUNC_EN, urtwn_read_2(sc, R92C_SYS_FUNC_EN) | R92C_SYS_FUNC_EN_CPUEN); } /* download enabled */ urtwn_write_1(sc, R92C_MCUFWDL, urtwn_read_1(sc, R92C_MCUFWDL) | R92C_MCUFWDL_EN); urtwn_write_1(sc, R92C_MCUFWDL + 2, urtwn_read_1(sc, R92C_MCUFWDL + 2) & ~0x08); /* Reset the FWDL checksum. */ urtwn_write_1(sc, R92C_MCUFWDL, urtwn_read_1(sc, R92C_MCUFWDL) | R92C_MCUFWDL_CHKSUM_RPT); DELAY(50); /* download firmware */ for (page = 0; len > 0; page++) { mlen = MIN(len, R92C_FW_PAGE_SIZE); error = urtwn_fw_loadpage(sc, page, ptr, mlen); if (error != 0) { aprint_error_dev(sc->sc_dev, "could not load firmware page %d\n", page); goto fail; } ptr += mlen; len -= mlen; } /* download disable */ urtwn_write_1(sc, R92C_MCUFWDL, urtwn_read_1(sc, R92C_MCUFWDL) & ~R92C_MCUFWDL_EN); urtwn_write_1(sc, R92C_MCUFWDL + 1, 0); /* Wait for checksum report. */ for (ntries = 0; ntries < 1000; ntries++) { if (urtwn_read_4(sc, R92C_MCUFWDL) & R92C_MCUFWDL_CHKSUM_RPT) break; DELAY(5); } if (ntries == 1000) { aprint_error_dev(sc->sc_dev, "timeout waiting for checksum report\n"); error = ETIMEDOUT; goto fail; } /* Wait for firmware readiness. */ reg = urtwn_read_4(sc, R92C_MCUFWDL); reg = (reg & ~R92C_MCUFWDL_WINTINI_RDY) | R92C_MCUFWDL_RDY; urtwn_write_4(sc, R92C_MCUFWDL, reg); if (ISSET(sc->chip, URTWN_CHIP_88E) || ISSET(sc->chip, URTWN_CHIP_92EU)) urtwn_r88e_fw_reset(sc); for (ntries = 0; ntries < 6000; ntries++) { if (urtwn_read_4(sc, R92C_MCUFWDL) & R92C_MCUFWDL_WINTINI_RDY) break; DELAY(5); } if (ntries == 6000) { aprint_error_dev(sc->sc_dev, "timeout waiting for firmware readiness\n"); error = ETIMEDOUT; goto fail; } fail: firmware_free(fw, fwlen); return error; } static __inline int urtwn_dma_init(struct urtwn_softc *sc) { return sc->sc_dma_init(sc); } static int urtwn_r92c_dma_init(struct urtwn_softc *sc) { int hashq, hasnq, haslq, nqueues, nqpages, nrempages; uint32_t reg; int error; URTWNHIST_FUNC(); URTWNHIST_CALLED(); KASSERT(mutex_owned(&sc->sc_write_mtx)); /* Initialize LLT table. */ error = urtwn_llt_init(sc); if (error != 0) return error; /* Get Tx queues to USB endpoints mapping. */ hashq = hasnq = haslq = 0; reg = urtwn_read_2(sc, R92C_USB_EP + 1); DPRINTFN(DBG_INIT, "USB endpoints mapping %#jx", reg, 0, 0, 0); if (MS(reg, R92C_USB_EP_HQ) != 0) hashq = 1; if (MS(reg, R92C_USB_EP_NQ) != 0) hasnq = 1; if (MS(reg, R92C_USB_EP_LQ) != 0) haslq = 1; nqueues = hashq + hasnq + haslq; if (nqueues == 0) return EIO; /* Get the number of pages for each queue. */ nqpages = (R92C_TX_PAGE_COUNT - R92C_PUBQ_NPAGES) / nqueues; /* The remaining pages are assigned to the high priority queue. */ nrempages = (R92C_TX_PAGE_COUNT - R92C_PUBQ_NPAGES) % nqueues; /* Set number of pages for normal priority queue. */ urtwn_write_1(sc, R92C_RQPN_NPQ, hasnq ? nqpages : 0); urtwn_write_4(sc, R92C_RQPN, /* Set number of pages for public queue. */ SM(R92C_RQPN_PUBQ, R92C_PUBQ_NPAGES) | /* Set number of pages for high priority queue. */ SM(R92C_RQPN_HPQ, hashq ? nqpages + nrempages : 0) | /* Set number of pages for low priority queue. */ SM(R92C_RQPN_LPQ, haslq ? nqpages : 0) | /* Load values. */ R92C_RQPN_LD); urtwn_write_1(sc, R92C_TXPKTBUF_BCNQ_BDNY, R92C_TX_PAGE_BOUNDARY); urtwn_write_1(sc, R92C_TXPKTBUF_MGQ_BDNY, R92C_TX_PAGE_BOUNDARY); urtwn_write_1(sc, R92C_TXPKTBUF_WMAC_LBK_BF_HD, R92C_TX_PAGE_BOUNDARY); urtwn_write_1(sc, R92C_TRXFF_BNDY, R92C_TX_PAGE_BOUNDARY); urtwn_write_1(sc, R92C_TDECTRL + 1, R92C_TX_PAGE_BOUNDARY); /* Set queue to USB pipe mapping. */ reg = urtwn_read_2(sc, R92C_TRXDMA_CTRL); reg &= ~R92C_TRXDMA_CTRL_QMAP_M; if (nqueues == 1) { if (hashq) { reg |= R92C_TRXDMA_CTRL_QMAP_HQ; } else if (hasnq) { reg |= R92C_TRXDMA_CTRL_QMAP_NQ; } else { reg |= R92C_TRXDMA_CTRL_QMAP_LQ; } } else if (nqueues == 2) { /* All 2-endpoints configs have a high priority queue. */ if (!hashq) { return EIO; } if (hasnq) { reg |= R92C_TRXDMA_CTRL_QMAP_HQ_NQ; } else { reg |= R92C_TRXDMA_CTRL_QMAP_HQ_LQ; } } else { reg |= R92C_TRXDMA_CTRL_QMAP_3EP; } urtwn_write_2(sc, R92C_TRXDMA_CTRL, reg); /* Set Tx/Rx transfer page boundary. */ urtwn_write_2(sc, R92C_TRXFF_BNDY + 2, 0x27ff); /* Set Tx/Rx transfer page size. */ urtwn_write_1(sc, R92C_PBP, SM(R92C_PBP_PSRX, R92C_PBP_128) | SM(R92C_PBP_PSTX, R92C_PBP_128)); return 0; } static int urtwn_r88e_dma_init(struct urtwn_softc *sc) { usb_interface_descriptor_t *id; uint32_t reg; int nqueues; int error; URTWNHIST_FUNC(); URTWNHIST_CALLED(); KASSERT(mutex_owned(&sc->sc_write_mtx)); /* Initialize LLT table. */ error = urtwn_llt_init(sc); if (error != 0) return error; /* Get Tx queues to USB endpoints mapping. */ id = usbd_get_interface_descriptor(sc->sc_iface); nqueues = id->bNumEndpoints - 1; if (nqueues == 0) return EIO; /* Set number of pages for normal priority queue. */ urtwn_write_2(sc, R92C_RQPN_NPQ, 0); urtwn_write_2(sc, R92C_RQPN_NPQ, 0x000d); urtwn_write_4(sc, R92C_RQPN, 0x808e000d); urtwn_write_1(sc, R92C_TXPKTBUF_BCNQ_BDNY, R88E_TX_PAGE_BOUNDARY); urtwn_write_1(sc, R92C_TXPKTBUF_MGQ_BDNY, R88E_TX_PAGE_BOUNDARY); urtwn_write_1(sc, R92C_TXPKTBUF_WMAC_LBK_BF_HD, R88E_TX_PAGE_BOUNDARY); urtwn_write_1(sc, R92C_TRXFF_BNDY, R88E_TX_PAGE_BOUNDARY); urtwn_write_1(sc, R92C_TDECTRL + 1, R88E_TX_PAGE_BOUNDARY); /* Set queue to USB pipe mapping. */ reg = urtwn_read_2(sc, R92C_TRXDMA_CTRL); reg &= ~R92C_TRXDMA_CTRL_QMAP_M; if (nqueues == 1) reg |= R92C_TRXDMA_CTRL_QMAP_LQ; else if (nqueues == 2) reg |= R92C_TRXDMA_CTRL_QMAP_HQ_NQ; else reg |= R92C_TRXDMA_CTRL_QMAP_3EP; urtwn_write_2(sc, R92C_TRXDMA_CTRL, reg); /* Set Tx/Rx transfer page boundary. */ urtwn_write_2(sc, R92C_TRXFF_BNDY + 2, 0x23ff); /* Set Tx/Rx transfer page size. */ urtwn_write_1(sc, R92C_PBP, SM(R92C_PBP_PSRX, R92C_PBP_128) | SM(R92C_PBP_PSTX, R92C_PBP_128)); return 0; } static void __noinline urtwn_mac_init(struct urtwn_softc *sc) { size_t i; URTWNHIST_FUNC(); URTWNHIST_CALLED(); KASSERT(mutex_owned(&sc->sc_write_mtx)); /* Write MAC initialization values. */ if (ISSET(sc->chip, URTWN_CHIP_88E)) { for (i = 0; i < __arraycount(rtl8188eu_mac); i++) urtwn_write_1(sc, rtl8188eu_mac[i].reg, rtl8188eu_mac[i].val); } else if (ISSET(sc->chip, URTWN_CHIP_92EU)) { for (i = 0; i < __arraycount(rtl8192eu_mac); i++) urtwn_write_1(sc, rtl8192eu_mac[i].reg, rtl8192eu_mac[i].val); } else { for (i = 0; i < __arraycount(rtl8192cu_mac); i++) urtwn_write_1(sc, rtl8192cu_mac[i].reg, rtl8192cu_mac[i].val); } } static void __noinline urtwn_bb_init(struct urtwn_softc *sc) { const struct rtwn_bb_prog *prog; uint32_t reg; uint8_t crystalcap; size_t i; URTWNHIST_FUNC(); URTWNHIST_CALLED(); KASSERT(mutex_owned(&sc->sc_write_mtx)); /* Enable BB and RF. */ urtwn_write_2(sc, R92C_SYS_FUNC_EN, urtwn_read_2(sc, R92C_SYS_FUNC_EN) | R92C_SYS_FUNC_EN_BBRSTB | R92C_SYS_FUNC_EN_BB_GLB_RST | R92C_SYS_FUNC_EN_DIO_RF); if (!ISSET(sc->chip, URTWN_CHIP_88E) && !ISSET(sc->chip, URTWN_CHIP_92EU)) { urtwn_write_1(sc, R92C_AFE_PLL_CTRL, 0x83); urtwn_write_1(sc, R92C_AFE_PLL_CTRL + 1, 0xdb); } urtwn_write_1(sc, R92C_RF_CTRL, R92C_RF_CTRL_EN | R92C_RF_CTRL_RSTB | R92C_RF_CTRL_SDMRSTB); urtwn_write_1(sc, R92C_SYS_FUNC_EN, R92C_SYS_FUNC_EN_USBA | R92C_SYS_FUNC_EN_USBD | R92C_SYS_FUNC_EN_BB_GLB_RST | R92C_SYS_FUNC_EN_BBRSTB); if (!ISSET(sc->chip, URTWN_CHIP_88E) && !ISSET(sc->chip, URTWN_CHIP_92EU)) { urtwn_write_1(sc, R92C_LDOHCI12_CTRL, 0x0f); urtwn_write_1(sc, 0x15, 0xe9); urtwn_write_1(sc, R92C_AFE_XTAL_CTRL + 1, 0x80); } /* Select BB programming based on board type. */ if (ISSET(sc->chip, URTWN_CHIP_88E)) prog = &rtl8188eu_bb_prog; else if (ISSET(sc->chip, URTWN_CHIP_92EU)) prog = &rtl8192eu_bb_prog; else if (!(sc->chip & URTWN_CHIP_92C)) { if (sc->board_type == R92C_BOARD_TYPE_MINICARD) { prog = &rtl8188ce_bb_prog; } else if (sc->board_type == R92C_BOARD_TYPE_HIGHPA) { prog = &rtl8188ru_bb_prog; } else { prog = &rtl8188cu_bb_prog; } } else { if (sc->board_type == R92C_BOARD_TYPE_MINICARD) { prog = &rtl8192ce_bb_prog; } else { prog = &rtl8192cu_bb_prog; } } /* Write BB initialization values. */ for (i = 0; i < prog->count; i++) { /* additional delay depend on registers */ switch (prog->regs[i]) { case 0xfe: urtwn_delay_ms(sc, 50); break; case 0xfd: urtwn_delay_ms(sc, 5); break; case 0xfc: urtwn_delay_ms(sc, 1); break; case 0xfb: DELAY(50); break; case 0xfa: DELAY(5); break; case 0xf9: DELAY(1); break; } urtwn_bb_write(sc, prog->regs[i], prog->vals[i]); DELAY(1); } if (sc->chip & URTWN_CHIP_92C_1T2R) { /* 8192C 1T only configuration. */ reg = urtwn_bb_read(sc, R92C_FPGA0_TXINFO); reg = (reg & ~0x00000003) | 0x2; urtwn_bb_write(sc, R92C_FPGA0_TXINFO, reg); reg = urtwn_bb_read(sc, R92C_FPGA1_TXINFO); reg = (reg & ~0x00300033) | 0x00200022; urtwn_bb_write(sc, R92C_FPGA1_TXINFO, reg); reg = urtwn_bb_read(sc, R92C_CCK0_AFESETTING); reg = (reg & ~0xff000000) | (0x45 << 24); urtwn_bb_write(sc, R92C_CCK0_AFESETTING, reg); reg = urtwn_bb_read(sc, R92C_OFDM0_TRXPATHENA); reg = (reg & ~0x000000ff) | 0x23; urtwn_bb_write(sc, R92C_OFDM0_TRXPATHENA, reg); reg = urtwn_bb_read(sc, R92C_OFDM0_AGCPARAM1); reg = (reg & ~0x00000030) | (1 << 4); urtwn_bb_write(sc, R92C_OFDM0_AGCPARAM1, reg); reg = urtwn_bb_read(sc, 0xe74); reg = (reg & ~0x0c000000) | (2 << 26); urtwn_bb_write(sc, 0xe74, reg); reg = urtwn_bb_read(sc, 0xe78); reg = (reg & ~0x0c000000) | (2 << 26); urtwn_bb_write(sc, 0xe78, reg); reg = urtwn_bb_read(sc, 0xe7c); reg = (reg & ~0x0c000000) | (2 << 26); urtwn_bb_write(sc, 0xe7c, reg); reg = urtwn_bb_read(sc, 0xe80); reg = (reg & ~0x0c000000) | (2 << 26); urtwn_bb_write(sc, 0xe80, reg); reg = urtwn_bb_read(sc, 0xe88); reg = (reg & ~0x0c000000) | (2 << 26); urtwn_bb_write(sc, 0xe88, reg); } /* Write AGC values. */ for (i = 0; i < prog->agccount; i++) { urtwn_bb_write(sc, R92C_OFDM0_AGCRSSITABLE, prog->agcvals[i]); DELAY(1); } if (ISSET(sc->chip, URTWN_CHIP_88E) || ISSET(sc->chip, URTWN_CHIP_92EU)) { urtwn_bb_write(sc, R92C_OFDM0_AGCCORE1(0), 0x69553422); DELAY(1); urtwn_bb_write(sc, R92C_OFDM0_AGCCORE1(0), 0x69553420); DELAY(1); } if (ISSET(sc->chip, URTWN_CHIP_92EU)) { crystalcap = sc->r88e_rom[0xb9]; if (crystalcap == 0x00) crystalcap = 0x20; crystalcap &= 0x3f; reg = urtwn_bb_read(sc, R92C_AFE_CTRL3); urtwn_bb_write(sc, R92C_AFE_CTRL3, RW(reg, R92C_AFE_XTAL_CTRL_ADDR, crystalcap | crystalcap << 6)); urtwn_write_4(sc, R92C_AFE_XTAL_CTRL, 0xf81fb); } else if (ISSET(sc->chip, URTWN_CHIP_88E)) { crystalcap = sc->r88e_rom[0xb9]; if (crystalcap == 0xff) crystalcap = 0x20; crystalcap &= 0x3f; reg = urtwn_bb_read(sc, R92C_AFE_XTAL_CTRL); urtwn_bb_write(sc, R92C_AFE_XTAL_CTRL, RW(reg, R92C_AFE_XTAL_CTRL_ADDR, crystalcap | crystalcap << 6)); } else { if (urtwn_bb_read(sc, R92C_HSSI_PARAM2(0)) & R92C_HSSI_PARAM2_CCK_HIPWR) { SET(sc->sc_flags, URTWN_FLAG_CCK_HIPWR); } } } static void __noinline urtwn_rf_init(struct urtwn_softc *sc) { const struct rtwn_rf_prog *prog; uint32_t reg, mask, saved; size_t i, j, idx; URTWNHIST_FUNC(); URTWNHIST_CALLED(); /* Select RF programming based on board type. */ if (ISSET(sc->chip, URTWN_CHIP_88E)) prog = rtl8188eu_rf_prog; else if (ISSET(sc->chip, URTWN_CHIP_92EU)) prog = rtl8192eu_rf_prog; else if (!(sc->chip & URTWN_CHIP_92C)) { if (sc->board_type == R92C_BOARD_TYPE_MINICARD) { prog = rtl8188ce_rf_prog; } else if (sc->board_type == R92C_BOARD_TYPE_HIGHPA) { prog = rtl8188ru_rf_prog; } else { prog = rtl8188cu_rf_prog; } } else { prog = rtl8192ce_rf_prog; } for (i = 0; i < sc->nrxchains; i++) { /* Save RF_ENV control type. */ idx = i / 2; mask = 0xffffU << ((i % 2) * 16); saved = urtwn_bb_read(sc, R92C_FPGA0_RFIFACESW(idx)) & mask; /* Set RF_ENV enable. */ reg = urtwn_bb_read(sc, R92C_FPGA0_RFIFACEOE(i)); reg |= 0x100000; urtwn_bb_write(sc, R92C_FPGA0_RFIFACEOE(i), reg); DELAY(50); /* Set RF_ENV output high. */ reg = urtwn_bb_read(sc, R92C_FPGA0_RFIFACEOE(i)); reg |= 0x10; urtwn_bb_write(sc, R92C_FPGA0_RFIFACEOE(i), reg); DELAY(50); /* Set address and data lengths of RF registers. */ reg = urtwn_bb_read(sc, R92C_HSSI_PARAM2(i)); reg &= ~R92C_HSSI_PARAM2_ADDR_LENGTH; urtwn_bb_write(sc, R92C_HSSI_PARAM2(i), reg); DELAY(50); reg = urtwn_bb_read(sc, R92C_HSSI_PARAM2(i)); reg &= ~R92C_HSSI_PARAM2_DATA_LENGTH; urtwn_bb_write(sc, R92C_HSSI_PARAM2(i), reg); DELAY(50); /* Write RF initialization values for this chain. */ for (j = 0; j < prog[i].count; j++) { if (prog[i].regs[j] >= 0xf9 && prog[i].regs[j] <= 0xfe) { /* * These are fake RF registers offsets that * indicate a delay is required. */ urtwn_delay_ms(sc, 50); continue; } urtwn_rf_write(sc, i, prog[i].regs[j], prog[i].vals[j]); DELAY(5); } /* Restore RF_ENV control type. */ reg = urtwn_bb_read(sc, R92C_FPGA0_RFIFACESW(idx)) & ~mask; urtwn_bb_write(sc, R92C_FPGA0_RFIFACESW(idx), reg | saved); } if ((sc->chip & (URTWN_CHIP_UMC_A_CUT | URTWN_CHIP_92C)) == URTWN_CHIP_UMC_A_CUT) { urtwn_rf_write(sc, 0, R92C_RF_RX_G1, 0x30255); urtwn_rf_write(sc, 0, R92C_RF_RX_G2, 0x50a00); } /* Cache RF register CHNLBW. */ for (i = 0; i < 2; i++) { sc->rf_chnlbw[i] = urtwn_rf_read(sc, i, R92C_RF_CHNLBW); } } static void __noinline urtwn_cam_init(struct urtwn_softc *sc) { uint32_t content, command; uint8_t idx; size_t i; URTWNHIST_FUNC(); URTWNHIST_CALLED(); KASSERT(mutex_owned(&sc->sc_write_mtx)); if (ISSET(sc->chip, URTWN_CHIP_92EU)) return; for (idx = 0; idx < R92C_CAM_ENTRY_COUNT; idx++) { content = (idx & 3) | (R92C_CAM_ALGO_AES << R92C_CAM_ALGO_S) | R92C_CAM_VALID; command = R92C_CAMCMD_POLLING | R92C_CAMCMD_WRITE | R92C_CAM_CTL0(idx); urtwn_write_4(sc, R92C_CAMWRITE, content); urtwn_write_4(sc, R92C_CAMCMD, command); } for (idx = 0; idx < R92C_CAM_ENTRY_COUNT; idx++) { for (i = 0; i < /* CAM_CONTENT_COUNT */ 8; i++) { if (i == 0) { content = (idx & 3) | (R92C_CAM_ALGO_AES << R92C_CAM_ALGO_S) | R92C_CAM_VALID; } else { content = 0; } command = R92C_CAMCMD_POLLING | R92C_CAMCMD_WRITE | R92C_CAM_CTL0(idx) | i; urtwn_write_4(sc, R92C_CAMWRITE, content); urtwn_write_4(sc, R92C_CAMCMD, command); } } /* Invalidate all CAM entries. */ urtwn_write_4(sc, R92C_CAMCMD, R92C_CAMCMD_POLLING | R92C_CAMCMD_CLR); } static void __noinline urtwn_pa_bias_init(struct urtwn_softc *sc) { uint8_t reg; size_t i; URTWNHIST_FUNC(); URTWNHIST_CALLED(); KASSERT(mutex_owned(&sc->sc_write_mtx)); for (i = 0; i < sc->nrxchains; i++) { if (sc->pa_setting & (1U << i)) continue; urtwn_rf_write(sc, i, R92C_RF_IPA, 0x0f406); urtwn_rf_write(sc, i, R92C_RF_IPA, 0x4f406); urtwn_rf_write(sc, i, R92C_RF_IPA, 0x8f406); urtwn_rf_write(sc, i, R92C_RF_IPA, 0xcf406); } if (!(sc->pa_setting & 0x10)) { reg = urtwn_read_1(sc, 0x16); reg = (reg & ~0xf0) | 0x90; urtwn_write_1(sc, 0x16, reg); } } static void __noinline urtwn_rxfilter_init(struct urtwn_softc *sc) { URTWNHIST_FUNC(); URTWNHIST_CALLED(); KASSERT(mutex_owned(&sc->sc_write_mtx)); /* Initialize Rx filter. */ /* TODO: use better filter for monitor mode. */ urtwn_write_4(sc, R92C_RCR, R92C_RCR_AAP | R92C_RCR_APM | R92C_RCR_AM | R92C_RCR_AB | R92C_RCR_APP_ICV | R92C_RCR_AMF | R92C_RCR_HTC_LOC_CTRL | R92C_RCR_APP_MIC | R92C_RCR_APP_PHYSTS); /* Accept all multicast frames. */ urtwn_write_4(sc, R92C_MAR + 0, 0xffffffff); urtwn_write_4(sc, R92C_MAR + 4, 0xffffffff); /* Accept all management frames. */ urtwn_write_2(sc, R92C_RXFLTMAP0, 0xffff); /* Reject all control frames. */ urtwn_write_2(sc, R92C_RXFLTMAP1, 0x0000); /* Accept all data frames. */ urtwn_write_2(sc, R92C_RXFLTMAP2, 0xffff); } static void __noinline urtwn_edca_init(struct urtwn_softc *sc) { URTWNHIST_FUNC(); URTWNHIST_CALLED(); KASSERT(mutex_owned(&sc->sc_write_mtx)); /* set spec SIFS (used in NAV) */ urtwn_write_2(sc, R92C_SPEC_SIFS, 0x100a); urtwn_write_2(sc, R92C_MAC_SPEC_SIFS, 0x100a); /* set SIFS CCK/OFDM */ urtwn_write_2(sc, R92C_SIFS_CCK, 0x100a); urtwn_write_2(sc, R92C_SIFS_OFDM, 0x100a); /* TXOP */ urtwn_write_4(sc, R92C_EDCA_BE_PARAM, 0x005ea42b); urtwn_write_4(sc, R92C_EDCA_BK_PARAM, 0x0000a44f); urtwn_write_4(sc, R92C_EDCA_VI_PARAM, 0x005ea324); urtwn_write_4(sc, R92C_EDCA_VO_PARAM, 0x002fa226); } static void urtwn_write_txpower(struct urtwn_softc *sc, int chain, uint16_t power[URTWN_RIDX_COUNT]) { uint32_t reg; URTWNHIST_FUNC(); URTWNHIST_CALLARGS("chain=%jd", chain, 0, 0, 0); /* Write per-CCK rate Tx power. */ if (chain == 0) { reg = urtwn_bb_read(sc, R92C_TXAGC_A_CCK1_MCS32); reg = RW(reg, R92C_TXAGC_A_CCK1, power[0]); urtwn_bb_write(sc, R92C_TXAGC_A_CCK1_MCS32, reg); reg = urtwn_bb_read(sc, R92C_TXAGC_B_CCK11_A_CCK2_11); reg = RW(reg, R92C_TXAGC_A_CCK2, power[1]); reg = RW(reg, R92C_TXAGC_A_CCK55, power[2]); reg = RW(reg, R92C_TXAGC_A_CCK11, power[3]); urtwn_bb_write(sc, R92C_TXAGC_B_CCK11_A_CCK2_11, reg); } else { reg = urtwn_bb_read(sc, R92C_TXAGC_B_CCK1_55_MCS32); reg = RW(reg, R92C_TXAGC_B_CCK1, power[0]); reg = RW(reg, R92C_TXAGC_B_CCK2, power[1]); reg = RW(reg, R92C_TXAGC_B_CCK55, power[2]); urtwn_bb_write(sc, R92C_TXAGC_B_CCK1_55_MCS32, reg); reg = urtwn_bb_read(sc, R92C_TXAGC_B_CCK11_A_CCK2_11); reg = RW(reg, R92C_TXAGC_B_CCK11, power[3]); urtwn_bb_write(sc, R92C_TXAGC_B_CCK11_A_CCK2_11, reg); } /* Write per-OFDM rate Tx power. */ urtwn_bb_write(sc, R92C_TXAGC_RATE18_06(chain), SM(R92C_TXAGC_RATE06, power[ 4]) | SM(R92C_TXAGC_RATE09, power[ 5]) | SM(R92C_TXAGC_RATE12, power[ 6]) | SM(R92C_TXAGC_RATE18, power[ 7])); urtwn_bb_write(sc, R92C_TXAGC_RATE54_24(chain), SM(R92C_TXAGC_RATE24, power[ 8]) | SM(R92C_TXAGC_RATE36, power[ 9]) | SM(R92C_TXAGC_RATE48, power[10]) | SM(R92C_TXAGC_RATE54, power[11])); /* Write per-MCS Tx power. */ urtwn_bb_write(sc, R92C_TXAGC_MCS03_MCS00(chain), SM(R92C_TXAGC_MCS00, power[12]) | SM(R92C_TXAGC_MCS01, power[13]) | SM(R92C_TXAGC_MCS02, power[14]) | SM(R92C_TXAGC_MCS03, power[15])); urtwn_bb_write(sc, R92C_TXAGC_MCS07_MCS04(chain), SM(R92C_TXAGC_MCS04, power[16]) | SM(R92C_TXAGC_MCS05, power[17]) | SM(R92C_TXAGC_MCS06, power[18]) | SM(R92C_TXAGC_MCS07, power[19])); urtwn_bb_write(sc, R92C_TXAGC_MCS11_MCS08(chain), SM(R92C_TXAGC_MCS08, power[20]) | SM(R92C_TXAGC_MCS09, power[21]) | SM(R92C_TXAGC_MCS10, power[22]) | SM(R92C_TXAGC_MCS11, power[23])); urtwn_bb_write(sc, R92C_TXAGC_MCS15_MCS12(chain), SM(R92C_TXAGC_MCS12, power[24]) | SM(R92C_TXAGC_MCS13, power[25]) | SM(R92C_TXAGC_MCS14, power[26]) | SM(R92C_TXAGC_MCS15, power[27])); } static void urtwn_get_txpower(struct urtwn_softc *sc, size_t chain, u_int chan, u_int ht40m, uint16_t power[URTWN_RIDX_COUNT]) { struct r92c_rom *rom = &sc->rom; uint16_t cckpow, ofdmpow, htpow, diff, maxpow; const struct rtwn_txpwr *base; int ridx, group; URTWNHIST_FUNC(); URTWNHIST_CALLARGS("chain=%jd, chan=%jd", chain, chan, 0, 0); /* Determine channel group. */ if (chan <= 3) { group = 0; } else if (chan <= 9) { group = 1; } else { group = 2; } /* Get original Tx power based on board type and RF chain. */ if (!(sc->chip & URTWN_CHIP_92C)) { if (sc->board_type == R92C_BOARD_TYPE_HIGHPA) { base = &rtl8188ru_txagc[chain]; } else { base = &rtl8192cu_txagc[chain]; } } else { base = &rtl8192cu_txagc[chain]; } memset(power, 0, URTWN_RIDX_COUNT * sizeof(power[0])); if (sc->regulatory == 0) { for (ridx = 0; ridx <= 3; ridx++) { power[ridx] = base->pwr[0][ridx]; } } for (ridx = 4; ridx < URTWN_RIDX_COUNT; ridx++) { if (sc->regulatory == 3) { power[ridx] = base->pwr[0][ridx]; /* Apply vendor limits. */ if (ht40m != IEEE80211_HTINFO_2NDCHAN_NONE) { maxpow = rom->ht40_max_pwr[group]; } else { maxpow = rom->ht20_max_pwr[group]; } maxpow = (maxpow >> (chain * 4)) & 0xf; if (power[ridx] > maxpow) { power[ridx] = maxpow; } } else if (sc->regulatory == 1) { if (ht40m == IEEE80211_HTINFO_2NDCHAN_NONE) { power[ridx] = base->pwr[group][ridx]; } } else if (sc->regulatory != 2) { power[ridx] = base->pwr[0][ridx]; } } /* Compute per-CCK rate Tx power. */ cckpow = rom->cck_tx_pwr[chain][group]; for (ridx = 0; ridx <= 3; ridx++) { power[ridx] += cckpow; if (power[ridx] > R92C_MAX_TX_PWR) { power[ridx] = R92C_MAX_TX_PWR; } } htpow = rom->ht40_1s_tx_pwr[chain][group]; if (sc->ntxchains > 1) { /* Apply reduction for 2 spatial streams. */ diff = rom->ht40_2s_tx_pwr_diff[group]; diff = (diff >> (chain * 4)) & 0xf; htpow = (htpow > diff) ? htpow - diff : 0; } /* Compute per-OFDM rate Tx power. */ diff = rom->ofdm_tx_pwr_diff[group]; diff = (diff >> (chain * 4)) & 0xf; ofdmpow = htpow + diff; /* HT->OFDM correction. */ for (ridx = 4; ridx <= 11; ridx++) { power[ridx] += ofdmpow; if (power[ridx] > R92C_MAX_TX_PWR) { power[ridx] = R92C_MAX_TX_PWR; } } /* Compute per-MCS Tx power. */ if (ht40m == IEEE80211_HTINFO_2NDCHAN_NONE) { diff = rom->ht20_tx_pwr_diff[group]; diff = (diff >> (chain * 4)) & 0xf; htpow += diff; /* HT40->HT20 correction. */ } for (ridx = 12; ridx < URTWN_RIDX_COUNT; ridx++) { power[ridx] += htpow; if (power[ridx] > R92C_MAX_TX_PWR) { power[ridx] = R92C_MAX_TX_PWR; } } #ifdef URTWN_DEBUG if (urtwn_debug & DBG_RF) { /* Dump per-rate Tx power values. */ DPRINTFN(DBG_RF, "Tx power for chain %jd:", chain, 0, 0, 0); for (ridx = 0; ridx < URTWN_RIDX_COUNT; ridx++) DPRINTFN(DBG_RF, "Rate %jd = %ju", ridx, power[ridx], 0, 0); } #endif } void urtwn_r88e_get_txpower(struct urtwn_softc *sc, size_t chain, u_int chan, u_int ht40m, uint16_t power[URTWN_RIDX_COUNT]) { uint16_t cckpow, ofdmpow, bw20pow, htpow; const struct rtwn_r88e_txpwr *base; int ridx, group; URTWNHIST_FUNC(); URTWNHIST_CALLARGS("chain=%jd, chan=%jd", chain, chan, 0, 0); /* Determine channel group. */ if (chan <= 2) group = 0; else if (chan <= 5) group = 1; else if (chan <= 8) group = 2; else if (chan <= 11) group = 3; else if (chan <= 13) group = 4; else group = 5; /* Get original Tx power based on board type and RF chain. */ base = &rtl8188eu_txagc[chain]; memset(power, 0, URTWN_RIDX_COUNT * sizeof(power[0])); if (sc->regulatory == 0) { for (ridx = 0; ridx <= 3; ridx++) power[ridx] = base->pwr[0][ridx]; } for (ridx = 4; ridx < URTWN_RIDX_COUNT; ridx++) { if (sc->regulatory == 3) power[ridx] = base->pwr[0][ridx]; else if (sc->regulatory == 1) { if (ht40m == IEEE80211_HTINFO_2NDCHAN_NONE) power[ridx] = base->pwr[group][ridx]; } else if (sc->regulatory != 2) power[ridx] = base->pwr[0][ridx]; } /* Compute per-CCK rate Tx power. */ cckpow = sc->cck_tx_pwr[group]; for (ridx = 0; ridx <= 3; ridx++) { power[ridx] += cckpow; if (power[ridx] > R92C_MAX_TX_PWR) power[ridx] = R92C_MAX_TX_PWR; } htpow = sc->ht40_tx_pwr[group]; /* Compute per-OFDM rate Tx power. */ ofdmpow = htpow + sc->ofdm_tx_pwr_diff; for (ridx = 4; ridx <= 11; ridx++) { power[ridx] += ofdmpow; if (power[ridx] > R92C_MAX_TX_PWR) power[ridx] = R92C_MAX_TX_PWR; } bw20pow = htpow + sc->bw20_tx_pwr_diff; for (ridx = 12; ridx <= 27; ridx++) { power[ridx] += bw20pow; if (power[ridx] > R92C_MAX_TX_PWR) power[ridx] = R92C_MAX_TX_PWR; } } static void urtwn_set_txpower(struct urtwn_softc *sc, u_int chan, u_int ht40m) { uint16_t power[URTWN_RIDX_COUNT]; size_t i; URTWNHIST_FUNC(); URTWNHIST_CALLED(); for (i = 0; i < sc->ntxchains; i++) { /* Compute per-rate Tx power values. */ if (ISSET(sc->chip, URTWN_CHIP_88E) || ISSET(sc->chip, URTWN_CHIP_92EU)) urtwn_r88e_get_txpower(sc, i, chan, ht40m, power); else urtwn_get_txpower(sc, i, chan, ht40m, power); /* Write per-rate Tx power values to hardware. */ urtwn_write_txpower(sc, i, power); } } static void __noinline urtwn_set_chan(struct urtwn_softc *sc, struct ieee80211_channel *c, u_int ht40m) { struct ieee80211com *ic = &sc->sc_ic; u_int chan; size_t i; chan = ieee80211_chan2ieee(ic, c); /* XXX center freq! */ URTWNHIST_FUNC(); URTWNHIST_CALLARGS("chan=%jd", chan, 0, 0, 0); KASSERT(mutex_owned(&sc->sc_write_mtx)); if (ht40m == IEEE80211_HTINFO_2NDCHAN_ABOVE) { chan += 2; } else if (ht40m == IEEE80211_HTINFO_2NDCHAN_BELOW){ chan -= 2; } /* Set Tx power for this new channel. */ urtwn_set_txpower(sc, chan, ht40m); for (i = 0; i < sc->nrxchains; i++) { urtwn_rf_write(sc, i, R92C_RF_CHNLBW, RW(sc->rf_chnlbw[i], R92C_RF_CHNLBW_CHNL, chan)); } if (ht40m) { /* Is secondary channel below or above primary? */ int prichlo = (ht40m == IEEE80211_HTINFO_2NDCHAN_ABOVE); uint32_t reg; urtwn_write_1(sc, R92C_BWOPMODE, urtwn_read_1(sc, R92C_BWOPMODE) & ~R92C_BWOPMODE_20MHZ); reg = urtwn_read_1(sc, R92C_RRSR + 2); reg = (reg & ~0x6f) | (prichlo ? 1 : 2) << 5; urtwn_write_1(sc, R92C_RRSR + 2, (uint8_t)reg); urtwn_bb_write(sc, R92C_FPGA0_RFMOD, urtwn_bb_read(sc, R92C_FPGA0_RFMOD) | R92C_RFMOD_40MHZ); urtwn_bb_write(sc, R92C_FPGA1_RFMOD, urtwn_bb_read(sc, R92C_FPGA1_RFMOD) | R92C_RFMOD_40MHZ); /* Set CCK side band. */ reg = urtwn_bb_read(sc, R92C_CCK0_SYSTEM); reg = (reg & ~0x00000010) | (prichlo ? 0 : 1) << 4; urtwn_bb_write(sc, R92C_CCK0_SYSTEM, reg); reg = urtwn_bb_read(sc, R92C_OFDM1_LSTF); reg = (reg & ~0x00000c00) | (prichlo ? 1 : 2) << 10; urtwn_bb_write(sc, R92C_OFDM1_LSTF, reg); urtwn_bb_write(sc, R92C_FPGA0_ANAPARAM2, urtwn_bb_read(sc, R92C_FPGA0_ANAPARAM2) & ~R92C_FPGA0_ANAPARAM2_CBW20); reg = urtwn_bb_read(sc, 0x818); reg = (reg & ~0x0c000000) | (prichlo ? 2 : 1) << 26; urtwn_bb_write(sc, 0x818, reg); /* Select 40MHz bandwidth. */ urtwn_rf_write(sc, 0, R92C_RF_CHNLBW, (sc->rf_chnlbw[0] & ~0xfff) | chan); } else { urtwn_write_1(sc, R92C_BWOPMODE, urtwn_read_1(sc, R92C_BWOPMODE) | R92C_BWOPMODE_20MHZ); urtwn_bb_write(sc, R92C_FPGA0_RFMOD, urtwn_bb_read(sc, R92C_FPGA0_RFMOD) & ~R92C_RFMOD_40MHZ); urtwn_bb_write(sc, R92C_FPGA1_RFMOD, urtwn_bb_read(sc, R92C_FPGA1_RFMOD) & ~R92C_RFMOD_40MHZ); if (!ISSET(sc->chip, URTWN_CHIP_88E) && !ISSET(sc->chip, URTWN_CHIP_92EU)) { urtwn_bb_write(sc, R92C_FPGA0_ANAPARAM2, urtwn_bb_read(sc, R92C_FPGA0_ANAPARAM2) | R92C_FPGA0_ANAPARAM2_CBW20); } /* Select 20MHz bandwidth. */ urtwn_rf_write(sc, 0, R92C_RF_CHNLBW, (sc->rf_chnlbw[0] & ~0xfff) | chan | (ISSET(sc->chip, URTWN_CHIP_88E) || ISSET(sc->chip, URTWN_CHIP_92EU) ? R88E_RF_CHNLBW_BW20 : R92C_RF_CHNLBW_BW20)); } } static void __noinline urtwn_iq_calib(struct urtwn_softc *sc, bool inited) { URTWNHIST_FUNC(); URTWNHIST_CALLARGS("inited=%jd", inited, 0, 0, 0); uint32_t addaBackup[16], iqkBackup[4], piMode; #ifdef notyet uint32_t odfm0_agccore_regs[3]; uint32_t ant_regs[3]; uint32_t rf_regs[8]; #endif uint32_t reg0, reg1, reg2; int i, attempt; #ifdef notyet urtwn_write_1(sc, R92E_STBC_SETTING + 2, urtwn_read_1(sc, R92E_STBC_SETTING + 2)); urtwn_write_1(sc, R92C_ACLK_MON, 0); /* Save AGCCORE regs. */ for (i = 0; i < sc->nrxchains; i++) { odfm0_agccore_regs[i] = urtwn_read_4(sc, R92C_OFDM0_AGCCORE1(i)); } #endif /* Save BB regs. */ reg0 = urtwn_bb_read(sc, R92C_OFDM0_TRXPATHENA); reg1 = urtwn_bb_read(sc, R92C_OFDM0_TRMUXPAR); reg2 = urtwn_bb_read(sc, R92C_FPGA0_RFIFACESW(1)); /* Save adda regs to be restored when finished. */ for (i = 0; i < __arraycount(addaReg); i++) addaBackup[i] = urtwn_bb_read(sc, addaReg[i]); /* Save mac regs. */ iqkBackup[0] = urtwn_read_1(sc, R92C_TXPAUSE); iqkBackup[1] = urtwn_read_1(sc, R92C_BCN_CTRL); iqkBackup[2] = urtwn_read_1(sc, R92C_BCN_CTRL1); iqkBackup[3] = urtwn_read_4(sc, R92C_GPIO_MUXCFG); #ifdef notyet ant_regs[0] = urtwn_read_4(sc, R92C_CONFIG_ANT_A); ant_regs[1] = urtwn_read_4(sc, R92C_CONFIG_ANT_B); rf_regs[0] = urtwn_read_4(sc, R92C_FPGA0_RFIFACESW(0)); for (i = 0; i < sc->nrxchains; i++) rf_regs[i+1] = urtwn_read_4(sc, R92C_FPGA0_RFIFACEOE(i)); reg4 = urtwn_read_4(sc, R92C_CCK0_AFESETTING); #endif piMode = (urtwn_bb_read(sc, R92C_HSSI_PARAM1(0)) & R92C_HSSI_PARAM1_PI); if (piMode == 0) { urtwn_bb_write(sc, R92C_HSSI_PARAM1(0), urtwn_bb_read(sc, R92C_HSSI_PARAM1(0))| R92C_HSSI_PARAM1_PI); urtwn_bb_write(sc, R92C_HSSI_PARAM1(1), urtwn_bb_read(sc, R92C_HSSI_PARAM1(1))| R92C_HSSI_PARAM1_PI); } attempt = 1; next_attempt: /* Set mac regs for calibration. */ for (i = 0; i < __arraycount(addaReg); i++) { urtwn_bb_write(sc, addaReg[i], addaReg[__arraycount(addaReg) - 1]); } urtwn_write_2(sc, R92C_CCK0_AFESETTING, urtwn_read_2(sc, R92C_CCK0_AFESETTING)); urtwn_write_2(sc, R92C_OFDM0_TRXPATHENA, R92C_IQK_TRXPATHENA); urtwn_write_2(sc, R92C_OFDM0_TRMUXPAR, R92C_IQK_TRMUXPAR); urtwn_write_2(sc, R92C_FPGA0_RFIFACESW(1), R92C_IQK_RFIFACESW1); urtwn_write_4(sc, R92C_LSSI_PARAM(0), R92C_IQK_LSSI_PARAM); if (sc->ntxchains > 1) urtwn_bb_write(sc, R92C_LSSI_PARAM(1), R92C_IQK_LSSI_PARAM); urtwn_write_1(sc, R92C_TXPAUSE, (~R92C_TXPAUSE_BCN) & R92C_TXPAUSE_ALL); urtwn_write_1(sc, R92C_BCN_CTRL, (iqkBackup[1] & ~R92C_BCN_CTRL_EN_BCN)); urtwn_write_1(sc, R92C_BCN_CTRL1, (iqkBackup[2] & ~R92C_BCN_CTRL_EN_BCN)); urtwn_write_1(sc, R92C_GPIO_MUXCFG, (iqkBackup[3] & ~R92C_GPIO_MUXCFG_ENBT)); urtwn_bb_write(sc, R92C_CONFIG_ANT_A, R92C_IQK_CONFIG_ANT); if (sc->ntxchains > 1) urtwn_bb_write(sc, R92C_CONFIG_ANT_B, R92C_IQK_CONFIG_ANT); urtwn_bb_write(sc, R92C_FPGA0_IQK, R92C_FPGA0_IQK_SETTING); urtwn_bb_write(sc, R92C_TX_IQK, R92C_TX_IQK_SETTING); urtwn_bb_write(sc, R92C_RX_IQK, R92C_RX_IQK_SETTING); /* Restore BB regs. */ urtwn_bb_write(sc, R92C_OFDM0_TRXPATHENA, reg0); urtwn_bb_write(sc, R92C_FPGA0_RFIFACESW(1), reg2); urtwn_bb_write(sc, R92C_OFDM0_TRMUXPAR, reg1); urtwn_bb_write(sc, R92C_FPGA0_IQK, 0x0); urtwn_bb_write(sc, R92C_LSSI_PARAM(0), R92C_IQK_LSSI_RESTORE); if (sc->nrxchains > 1) urtwn_bb_write(sc, R92C_LSSI_PARAM(1), R92C_IQK_LSSI_RESTORE); if (attempt-- > 0) goto next_attempt; /* Restore mode. */ if (piMode == 0) { urtwn_bb_write(sc, R92C_HSSI_PARAM1(0), urtwn_bb_read(sc, R92C_HSSI_PARAM1(0)) & ~R92C_HSSI_PARAM1_PI); urtwn_bb_write(sc, R92C_HSSI_PARAM1(1), urtwn_bb_read(sc, R92C_HSSI_PARAM1(1)) & ~R92C_HSSI_PARAM1_PI); } #ifdef notyet for (i = 0; i < sc->nrxchains; i++) { urtwn_write_4(sc, R92C_OFDM0_AGCCORE1(i), odfm0_agccore_regs[i]); } #endif /* Restore adda regs. */ for (i = 0; i < __arraycount(addaReg); i++) urtwn_bb_write(sc, addaReg[i], addaBackup[i]); /* Restore mac regs. */ urtwn_write_1(sc, R92C_TXPAUSE, iqkBackup[0]); urtwn_write_1(sc, R92C_BCN_CTRL, iqkBackup[1]); urtwn_write_1(sc, R92C_USTIME_TSF, iqkBackup[2]); urtwn_write_4(sc, R92C_GPIO_MUXCFG, iqkBackup[3]); #ifdef notyet urtwn_write_4(sc, R92C_CONFIG_ANT_A, ant_regs[0]); urtwn_write_4(sc, R92C_CONFIG_ANT_B, ant_regs[1]); urtwn_write_4(sc, R92C_FPGA0_RFIFACESW(0), rf_regs[0]); for (i = 0; i < sc->nrxchains; i++) urtwn_write_4(sc, R92C_FPGA0_RFIFACEOE(i), rf_regs[i+1]); urtwn_write_4(sc, R92C_CCK0_AFESETTING, reg4); #endif } static void urtwn_lc_calib(struct urtwn_softc *sc) { uint32_t rf_ac[2]; uint8_t txmode; size_t i; URTWNHIST_FUNC(); URTWNHIST_CALLED(); KASSERT(mutex_owned(&sc->sc_write_mtx)); txmode = urtwn_read_1(sc, R92C_OFDM1_LSTF + 3); if ((txmode & 0x70) != 0) { /* Disable all continuous Tx. */ urtwn_write_1(sc, R92C_OFDM1_LSTF + 3, txmode & ~0x70); /* Set RF mode to standby mode. */ for (i = 0; i < sc->nrxchains; i++) { rf_ac[i] = urtwn_rf_read(sc, i, R92C_RF_AC); urtwn_rf_write(sc, i, R92C_RF_AC, RW(rf_ac[i], R92C_RF_AC_MODE, R92C_RF_AC_MODE_STANDBY)); } } else { /* Block all Tx queues. */ urtwn_write_1(sc, R92C_TXPAUSE, 0xff); } /* Start calibration. */ urtwn_rf_write(sc, 0, R92C_RF_CHNLBW, urtwn_rf_read(sc, 0, R92C_RF_CHNLBW) | R92C_RF_CHNLBW_LCSTART); /* Give calibration the time to complete. */ urtwn_delay_ms(sc, 100); /* Restore configuration. */ if ((txmode & 0x70) != 0) { /* Restore Tx mode. */ urtwn_write_1(sc, R92C_OFDM1_LSTF + 3, txmode); /* Restore RF mode. */ for (i = 0; i < sc->nrxchains; i++) { urtwn_rf_write(sc, i, R92C_RF_AC, rf_ac[i]); } } else { /* Unblock all Tx queues. */ urtwn_write_1(sc, R92C_TXPAUSE, 0x00); } } static void urtwn_temp_calib(struct urtwn_softc *sc) { int temp, t_meter_reg; URTWNHIST_FUNC(); URTWNHIST_CALLED(); KASSERT(mutex_owned(&sc->sc_write_mtx)); if (!ISSET(sc->chip, URTWN_CHIP_92EU)) t_meter_reg = R92C_RF_T_METER; else t_meter_reg = R92E_RF_T_METER; if (sc->thcal_state == 0) { /* Start measuring temperature. */ DPRINTFN(DBG_RF, "start measuring temperature", 0, 0, 0, 0); urtwn_rf_write(sc, 0, t_meter_reg, 0x60); sc->thcal_state = 1; return; } sc->thcal_state = 0; /* Read measured temperature. */ temp = urtwn_rf_read(sc, 0, R92C_RF_T_METER) & 0x1f; DPRINTFN(DBG_RF, "temperature=%jd", temp, 0, 0, 0); if (temp == 0) /* Read failed, skip. */ return; /* * Redo LC calibration if temperature changed significantly since * last calibration. */ if (sc->thcal_lctemp == 0) { /* First LC calibration is performed in urtwn_init(). */ sc->thcal_lctemp = temp; } else if (abs(temp - sc->thcal_lctemp) > 1) { DPRINTFN(DBG_RF, "LC calib triggered by temp: %jd -> %jd", sc->thcal_lctemp, temp, 0, 0); urtwn_lc_calib(sc); /* Record temperature of last LC calibration. */ sc->thcal_lctemp = temp; } } static int urtwn_init(struct ifnet *ifp) { struct urtwn_softc *sc = ifp->if_softc; struct ieee80211com *ic = &sc->sc_ic; struct urtwn_rx_data *data; uint32_t reg; size_t i; int error; URTWNHIST_FUNC(); URTWNHIST_CALLED(); urtwn_stop(ifp, 0); mutex_enter(&sc->sc_write_mtx); mutex_enter(&sc->sc_task_mtx); /* Init host async commands ring. */ sc->cmdq.cur = sc->cmdq.next = sc->cmdq.queued = 0; mutex_exit(&sc->sc_task_mtx); mutex_enter(&sc->sc_fwcmd_mtx); /* Init firmware commands ring. */ sc->fwcur = 0; mutex_exit(&sc->sc_fwcmd_mtx); /* Allocate Tx/Rx buffers. */ error = urtwn_alloc_rx_list(sc); if (error != 0) { aprint_error_dev(sc->sc_dev, "could not allocate Rx buffers\n"); goto fail; } error = urtwn_alloc_tx_list(sc); if (error != 0) { aprint_error_dev(sc->sc_dev, "could not allocate Tx buffers\n"); goto fail; } /* Power on adapter. */ error = urtwn_power_on(sc); if (error != 0) goto fail; /* Initialize DMA. */ error = urtwn_dma_init(sc); if (error != 0) goto fail; /* Set info size in Rx descriptors (in 64-bit words). */ urtwn_write_1(sc, R92C_RX_DRVINFO_SZ, 4); /* Init interrupts. */ if (ISSET(sc->chip, URTWN_CHIP_88E) || ISSET(sc->chip, URTWN_CHIP_92EU)) { urtwn_write_4(sc, R88E_HISR, 0xffffffff); urtwn_write_4(sc, R88E_HIMR, R88E_HIMR_CPWM | R88E_HIMR_CPWM2 | R88E_HIMR_TBDER | R88E_HIMR_PSTIMEOUT); urtwn_write_4(sc, R88E_HIMRE, R88E_HIMRE_RXFOVW | R88E_HIMRE_TXFOVW | R88E_HIMRE_RXERR | R88E_HIMRE_TXERR); if (ISSET(sc->chip, URTWN_CHIP_88E)) { urtwn_write_1(sc, R92C_USB_SPECIAL_OPTION, urtwn_read_1(sc, R92C_USB_SPECIAL_OPTION) | R92C_USB_SPECIAL_OPTION_INT_BULK_SEL); } if (ISSET(sc->chip, URTWN_CHIP_92EU)) urtwn_write_1(sc, R92C_USB_HRPWM, 0); } else { urtwn_write_4(sc, R92C_HISR, 0xffffffff); urtwn_write_4(sc, R92C_HIMR, 0xffffffff); } /* Set MAC address. */ IEEE80211_ADDR_COPY(ic->ic_myaddr, CLLADDR(ifp->if_sadl)); urtwn_write_region(sc, R92C_MACID, ic->ic_myaddr, IEEE80211_ADDR_LEN); /* Set initial network type. */ reg = urtwn_read_4(sc, R92C_CR); switch (ic->ic_opmode) { case IEEE80211_M_STA: default: reg = RW(reg, R92C_CR_NETTYPE, R92C_CR_NETTYPE_INFRA); break; case IEEE80211_M_IBSS: reg = RW(reg, R92C_CR_NETTYPE, R92C_CR_NETTYPE_ADHOC); break; } urtwn_write_4(sc, R92C_CR, reg); /* Set response rate */ reg = urtwn_read_4(sc, R92C_RRSR); reg = RW(reg, R92C_RRSR_RATE_BITMAP, R92C_RRSR_RATE_CCK_ONLY_1M); urtwn_write_4(sc, R92C_RRSR, reg); /* SIFS (used in NAV) */ urtwn_write_2(sc, R92C_SPEC_SIFS, SM(R92C_SPEC_SIFS_CCK, 0x10) | SM(R92C_SPEC_SIFS_OFDM, 0x10)); /* Set short/long retry limits. */ urtwn_write_2(sc, R92C_RL, SM(R92C_RL_SRL, 0x30) | SM(R92C_RL_LRL, 0x30)); /* Initialize EDCA parameters. */ urtwn_edca_init(sc); /* Setup rate fallback. */ if (!ISSET(sc->chip, URTWN_CHIP_88E) && !ISSET(sc->chip, URTWN_CHIP_92EU)) { urtwn_write_4(sc, R92C_DARFRC + 0, 0x00000000); urtwn_write_4(sc, R92C_DARFRC + 4, 0x10080404); urtwn_write_4(sc, R92C_RARFRC + 0, 0x04030201); urtwn_write_4(sc, R92C_RARFRC + 4, 0x08070605); } urtwn_write_1(sc, R92C_FWHW_TXQ_CTRL, urtwn_read_1(sc, R92C_FWHW_TXQ_CTRL) | R92C_FWHW_TXQ_CTRL_AMPDU_RTY_NEW); /* Set ACK timeout. */ urtwn_write_1(sc, R92C_ACKTO, 0x40); /* Setup USB aggregation. */ /* Tx */ reg = urtwn_read_4(sc, R92C_TDECTRL); reg = RW(reg, R92C_TDECTRL_BLK_DESC_NUM, 6); urtwn_write_4(sc, R92C_TDECTRL, reg); /* Rx */ urtwn_write_1(sc, R92C_TRXDMA_CTRL, urtwn_read_1(sc, R92C_TRXDMA_CTRL) | R92C_TRXDMA_CTRL_RXDMA_AGG_EN); urtwn_write_1(sc, R92C_USB_SPECIAL_OPTION, urtwn_read_1(sc, R92C_USB_SPECIAL_OPTION) & ~R92C_USB_SPECIAL_OPTION_AGG_EN); urtwn_write_1(sc, R92C_RXDMA_AGG_PG_TH, 48); if (ISSET(sc->chip, URTWN_CHIP_88E) || ISSET(sc->chip, URTWN_CHIP_92EU)) urtwn_write_1(sc, R92C_RXDMA_AGG_PG_TH + 1, 4); else urtwn_write_1(sc, R92C_USB_DMA_AGG_TO, 4); /* Initialize beacon parameters. */ urtwn_write_2(sc, R92C_BCN_CTRL, 0x1010); urtwn_write_2(sc, R92C_TBTT_PROHIBIT, 0x6404); urtwn_write_1(sc, R92C_DRVERLYINT, R92C_DRVERLYINT_INIT_TIME); urtwn_write_1(sc, R92C_BCNDMATIM, R92C_BCNDMATIM_INIT_TIME); urtwn_write_2(sc, R92C_BCNTCFG, 0x660f); if (!ISSET(sc->chip, URTWN_CHIP_88E) && !ISSET(sc->chip, URTWN_CHIP_92EU)) { /* Setup AMPDU aggregation. */ urtwn_write_4(sc, R92C_AGGLEN_LMT, 0x99997631); /* MCS7~0 */ urtwn_write_1(sc, R92C_AGGR_BREAK_TIME, 0x16); urtwn_write_2(sc, 0x4ca, 0x0708); urtwn_write_1(sc, R92C_BCN_MAX_ERR, 0xff); urtwn_write_1(sc, R92C_BCN_CTRL, R92C_BCN_CTRL_DIS_TSF_UDT0); } /* Load 8051 microcode. */ error = urtwn_load_firmware(sc); if (error != 0) goto fail; SET(sc->sc_flags, URTWN_FLAG_FWREADY); /* Initialize MAC/BB/RF blocks. */ /* * XXX: urtwn_mac_init() sets R92C_RCR[0:15] = R92C_RCR_APM | * R92C_RCR_AM | R92C_RCR_AB | R92C_RCR_AICV | R92C_RCR_AMF. * XXX: This setting should be removed from rtl8192cu_mac[]. */ urtwn_mac_init(sc); // sets R92C_RCR[0:15] urtwn_rxfilter_init(sc); // reset R92C_RCR urtwn_bb_init(sc); urtwn_rf_init(sc); if (ISSET(sc->chip, URTWN_CHIP_88E) || ISSET(sc->chip, URTWN_CHIP_92EU)) { urtwn_write_2(sc, R92C_CR, urtwn_read_2(sc, R92C_CR) | R92C_CR_MACTXEN | R92C_CR_MACRXEN); } /* Turn CCK and OFDM blocks on. */ reg = urtwn_bb_read(sc, R92C_FPGA0_RFMOD); reg |= R92C_RFMOD_CCK_EN; urtwn_bb_write(sc, R92C_FPGA0_RFMOD, reg); reg = urtwn_bb_read(sc, R92C_FPGA0_RFMOD); reg |= R92C_RFMOD_OFDM_EN; urtwn_bb_write(sc, R92C_FPGA0_RFMOD, reg); /* Clear per-station keys table. */ urtwn_cam_init(sc); /* Enable hardware sequence numbering. */ urtwn_write_1(sc, R92C_HWSEQ_CTRL, 0xff); /* Perform LO and IQ calibrations. */ urtwn_iq_calib(sc, sc->iqk_inited); sc->iqk_inited = true; /* Perform LC calibration. */ urtwn_lc_calib(sc); if (!ISSET(sc->chip, URTWN_CHIP_88E) && !ISSET(sc->chip, URTWN_CHIP_92EU)) { /* Fix USB interference issue. */ urtwn_write_1(sc, 0xfe40, 0xe0); urtwn_write_1(sc, 0xfe41, 0x8d); urtwn_write_1(sc, 0xfe42, 0x80); urtwn_write_4(sc, 0x20c, 0xfd0320); urtwn_pa_bias_init(sc); } if (!(sc->chip & (URTWN_CHIP_92C | URTWN_CHIP_92C_1T2R)) || !(sc->chip & URTWN_CHIP_92EU)) { /* 1T1R */ urtwn_bb_write(sc, R92C_FPGA0_RFPARAM(0), urtwn_bb_read(sc, R92C_FPGA0_RFPARAM(0)) | __BIT(13)); } /* Initialize GPIO setting. */ urtwn_write_1(sc, R92C_GPIO_MUXCFG, urtwn_read_1(sc, R92C_GPIO_MUXCFG) & ~R92C_GPIO_MUXCFG_ENBT); /* Fix for lower temperature. */ if (!ISSET(sc->chip, URTWN_CHIP_88E) && !ISSET(sc->chip, URTWN_CHIP_92EU)) urtwn_write_1(sc, 0x15, 0xe9); /* Set default channel. */ urtwn_set_chan(sc, ic->ic_curchan, IEEE80211_HTINFO_2NDCHAN_NONE); /* Queue Rx xfers. */ for (size_t j = 0; j < sc->rx_npipe; j++) { for (i = 0; i < URTWN_RX_LIST_COUNT; i++) { data = &sc->rx_data[j][i]; usbd_setup_xfer(data->xfer, data, data->buf, URTWN_RXBUFSZ, USBD_SHORT_XFER_OK, USBD_NO_TIMEOUT, urtwn_rxeof); error = usbd_transfer(data->xfer); if (__predict_false(error != USBD_NORMAL_COMPLETION && error != USBD_IN_PROGRESS)) goto fail; } } /* We're ready to go. */ ifp->if_flags &= ~IFF_OACTIVE; ifp->if_flags |= IFF_RUNNING; sc->sc_running = true; mutex_exit(&sc->sc_write_mtx); if (ic->ic_opmode == IEEE80211_M_MONITOR) ieee80211_new_state(ic, IEEE80211_S_RUN, -1); else if (ic->ic_roaming != IEEE80211_ROAMING_MANUAL) ieee80211_new_state(ic, IEEE80211_S_SCAN, -1); urtwn_wait_async(sc); return 0; fail: mutex_exit(&sc->sc_write_mtx); urtwn_stop(ifp, 1); return error; } static void __noinline urtwn_stop(struct ifnet *ifp, int disable) { struct urtwn_softc *sc = ifp->if_softc; struct ieee80211com *ic = &sc->sc_ic; size_t i; int s; URTWNHIST_FUNC(); URTWNHIST_CALLED(); s = splusb(); ieee80211_new_state(ic, IEEE80211_S_INIT, -1); urtwn_wait_async(sc); splx(s); sc->tx_timer = 0; ifp->if_timer = 0; ifp->if_flags &= ~(IFF_RUNNING | IFF_OACTIVE); callout_stop(&sc->sc_scan_to); callout_stop(&sc->sc_calib_to); /* Abort Tx. */ for (i = 0; i < sc->tx_npipe; i++) { if (sc->tx_pipe[i] != NULL) usbd_abort_pipe(sc->tx_pipe[i]); } /* Stop Rx pipe. */ for (i = 0; i < sc->rx_npipe; i++) { if (sc->rx_pipe[i] != NULL) usbd_abort_pipe(sc->rx_pipe[i]); } /* Free Tx/Rx buffers. */ urtwn_free_tx_list(sc); urtwn_free_rx_list(sc); sc->sc_running = false; if (disable) urtwn_chip_stop(sc); } static int urtwn_reset(struct ifnet *ifp) { struct urtwn_softc *sc = ifp->if_softc; struct ieee80211com *ic = &sc->sc_ic; if (ic->ic_opmode != IEEE80211_M_MONITOR) return ENETRESET; urtwn_set_chan(sc, ic->ic_curchan, IEEE80211_HTINFO_2NDCHAN_NONE); return 0; } static void urtwn_chip_stop(struct urtwn_softc *sc) { uint32_t reg; bool disabled = true; URTWNHIST_FUNC(); URTWNHIST_CALLED(); if (ISSET(sc->chip, URTWN_CHIP_88E) || ISSET(sc->chip, URTWN_CHIP_92EU)) return; mutex_enter(&sc->sc_write_mtx); /* * RF Off Sequence */ /* Pause MAC TX queue */ urtwn_write_1(sc, R92C_TXPAUSE, 0xFF); /* Disable RF */ urtwn_rf_write(sc, 0, 0, 0); urtwn_write_1(sc, R92C_APSD_CTRL, R92C_APSD_CTRL_OFF); /* Reset BB state machine */ urtwn_write_1(sc, R92C_SYS_FUNC_EN, R92C_SYS_FUNC_EN_USBD | R92C_SYS_FUNC_EN_USBA | R92C_SYS_FUNC_EN_BB_GLB_RST); urtwn_write_1(sc, R92C_SYS_FUNC_EN, R92C_SYS_FUNC_EN_USBD | R92C_SYS_FUNC_EN_USBA); /* * Reset digital sequence */ if (urtwn_read_1(sc, R92C_MCUFWDL) & R92C_MCUFWDL_RDY) { /* Reset MCU ready status */ urtwn_write_1(sc, R92C_MCUFWDL, 0); /* If firmware in ram code, do reset */ if (ISSET(sc->sc_flags, URTWN_FLAG_FWREADY)) { if (ISSET(sc->chip, URTWN_CHIP_88E) || ISSET(sc->chip, URTWN_CHIP_92EU)) urtwn_r88e_fw_reset(sc); else urtwn_fw_reset(sc); CLR(sc->sc_flags, URTWN_FLAG_FWREADY); } } /* Reset MAC and Enable 8051 */ urtwn_write_1(sc, R92C_SYS_FUNC_EN + 1, 0x54); /* Reset MCU ready status */ urtwn_write_1(sc, R92C_MCUFWDL, 0); if (disabled) { /* Disable MAC clock */ urtwn_write_2(sc, R92C_SYS_CLKR, 0x70A3); /* Disable AFE PLL */ urtwn_write_1(sc, R92C_AFE_PLL_CTRL, 0x80); /* Gated AFE DIG_CLOCK */ urtwn_write_2(sc, R92C_AFE_XTAL_CTRL, 0x880F); /* Isolated digital to PON */ urtwn_write_1(sc, R92C_SYS_ISO_CTRL, 0xF9); } /* * Pull GPIO PIN to balance level and LED control */ /* 1. Disable GPIO[7:0] */ urtwn_write_2(sc, R92C_GPIO_PIN_CTRL + 2, 0x0000); reg = urtwn_read_4(sc, R92C_GPIO_PIN_CTRL) & ~0x0000ff00; reg |= ((reg << 8) & 0x0000ff00) | 0x00ff0000; urtwn_write_4(sc, R92C_GPIO_PIN_CTRL, reg); /* Disable GPIO[10:8] */ urtwn_write_1(sc, R92C_GPIO_MUXCFG + 3, 0x00); reg = urtwn_read_2(sc, R92C_GPIO_MUXCFG + 2) & ~0x00f0; reg |= (((reg & 0x000f) << 4) | 0x0780); urtwn_write_2(sc, R92C_GPIO_MUXCFG + 2, reg); /* Disable LED0 & 1 */ urtwn_write_2(sc, R92C_LEDCFG0, 0x8080); /* * Reset digital sequence */ if (disabled) { /* Disable ELDR clock */ urtwn_write_2(sc, R92C_SYS_CLKR, 0x70A3); /* Isolated ELDR to PON */ urtwn_write_1(sc, R92C_SYS_ISO_CTRL + 1, 0x82); } /* * Disable analog sequence */ if (disabled) { /* Disable A15 power */ urtwn_write_1(sc, R92C_LDOA15_CTRL, 0x04); /* Disable digital core power */ urtwn_write_1(sc, R92C_LDOV12D_CTRL, urtwn_read_1(sc, R92C_LDOV12D_CTRL) & ~R92C_LDOV12D_CTRL_LDV12_EN); } /* Enter PFM mode */ urtwn_write_1(sc, R92C_SPS0_CTRL, 0x23); /* Set USB suspend */ urtwn_write_2(sc, R92C_APS_FSMCO, R92C_APS_FSMCO_APDM_HOST | R92C_APS_FSMCO_AFSM_HSUS | R92C_APS_FSMCO_PFM_ALDN); urtwn_write_1(sc, R92C_RSV_CTRL, 0x0E); mutex_exit(&sc->sc_write_mtx); } static void urtwn_delay_ms(struct urtwn_softc *sc, int ms) { if (sc->sc_running == false) DELAY(ms * 1000); else usbd_delay_ms(sc->sc_udev, ms); } MODULE(MODULE_CLASS_DRIVER, if_urtwn, NULL); #ifdef _MODULE #include "ioconf.c" #endif static int if_urtwn_modcmd(modcmd_t cmd, void *aux) { int error = 0; switch (cmd) { case MODULE_CMD_INIT: #ifdef _MODULE error = config_init_component(cfdriver_ioconf_urtwn, cfattach_ioconf_urtwn, cfdata_ioconf_urtwn); #endif return error; case MODULE_CMD_FINI: #ifdef _MODULE error = config_fini_component(cfdriver_ioconf_urtwn, cfattach_ioconf_urtwn, cfdata_ioconf_urtwn); #endif return error; default: return ENOTTY; } } |
| 1 1 3 3 3 3 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 | /* $NetBSD: cpu_ucode.c,v 1.13 2020/04/25 15:26:18 bouyer Exp $ */ /* * Copyright (c) 2012 The NetBSD Foundation, Inc. * All rights reserved. * * This code is derived from software contributed to The NetBSD Foundation * by Christoph Egger. * * Redistribution and use in source and binary forms, with or without * modification, are permitted provided that the following conditions * are met: * 1. Redistributions of source code must retain the above copyright * notice, this list of conditions and the following disclaimer. * 2. Redistributions in binary form must reproduce the above copyright * notice, this list of conditions and the following disclaimer in the * documentation and/or other materials provided with the distribution. * * THIS SOFTWARE IS PROVIDED BY THE NETBSD FOUNDATION, INC. AND CONTRIBUTORS * ``AS IS'' AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED * TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR * PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE FOUNDATION OR CONTRIBUTORS * BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR * CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF * SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS * INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN * CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) * ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE * POSSIBILITY OF SUCH DAMAGE. */ #include <sys/cdefs.h> __KERNEL_RCSID(0, "$NetBSD: cpu_ucode.c,v 1.13 2020/04/25 15:26:18 bouyer Exp $"); #if defined(_KERNEL_OPT) #include "opt_cpu_ucode.h" #include "opt_xen.h" #endif #include <sys/param.h> #include <sys/cpuio.h> #include <sys/cpu.h> #include <dev/firmload.h> #include <machine/cpuvar.h> #include <machine/cputypes.h> #include <x86/cpu_ucode.h> #ifdef XEN #include <xen/include/public/xen.h> #include <xen/hypervisor.h> #endif static struct cpu_ucode_softc ucode_softc; int cpu_ucode_get_version(struct cpu_ucode_version *data) { union { struct cpu_ucode_version_amd a; struct cpu_ucode_version_intel1 i; } v; size_t l; int error; if (!data->data) return 0; switch (cpu_vendor) { case CPUVENDOR_AMD: l = sizeof(v.a); error = cpu_ucode_amd_get_version(data, &v, l); break; case CPUVENDOR_INTEL: l = sizeof(v.i); error = cpu_ucode_intel_get_version(data, &v, l); break; default: return EOPNOTSUPP; } if (error) return error; return copyout(&v, data->data, l); } int cpu_ucode_md_open(firmware_handle_t *fwh, int loader_version, const char *fwname) { switch (cpu_vendor) { case CPUVENDOR_AMD: return cpu_ucode_amd_firmware_open(fwh, fwname); case CPUVENDOR_INTEL: return cpu_ucode_intel_firmware_open(fwh, fwname); default: return EOPNOTSUPP; } } #ifndef XENPV int cpu_ucode_apply(const struct cpu_ucode *data) { struct cpu_ucode_softc *sc = &ucode_softc; int error; sc->loader_version = data->loader_version; error = cpu_ucode_load(sc, data->fwname); if (error) return error; switch (cpu_vendor) { case CPUVENDOR_AMD: error = cpu_ucode_amd_apply(sc, data->cpu_nr); break; case CPUVENDOR_INTEL: error = cpu_ucode_intel_apply(sc, data->cpu_nr); break; default: error = EOPNOTSUPP; } if (sc->sc_blob != NULL) firmware_free(sc->sc_blob, sc->sc_blobsize); sc->sc_blob = NULL; sc->sc_blobsize = 0; return error; } #else int cpu_ucode_apply(const struct cpu_ucode *data) { struct cpu_ucode_softc *sc = &ucode_softc; struct xen_platform_op op; int error; /* Xen updates all??? */ if (data->cpu_nr != CPU_UCODE_ALL_CPUS) return EOPNOTSUPP; sc->loader_version = data->loader_version; error = cpu_ucode_load(sc, data->fwname); if (error) return error; op.cmd = XENPF_microcode_update; set_xen_guest_handle(op.u.microcode.data, sc->sc_blob); op.u.microcode.length = sc->sc_blobsize; error = -HYPERVISOR_platform_op(&op); if (sc->sc_blob) firmware_free(sc->sc_blob, sc->sc_blobsize); sc->sc_blob = NULL; sc->sc_blobsize = 0; return error; } #endif /* XEN */ |
| 5 16 5 5 5 5 5 5 5 13 13 13 13 13 13 13 13 13 13 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 197 198 199 200 201 202 203 204 205 206 207 208 209 210 211 212 213 214 215 216 217 218 219 220 221 222 223 224 225 226 227 228 229 230 231 232 233 234 235 236 237 238 239 240 241 242 243 244 245 246 247 248 249 250 251 252 253 254 255 256 257 258 259 260 261 262 263 264 265 266 267 268 269 270 271 272 273 274 275 276 277 278 279 280 281 282 283 284 285 286 287 288 289 290 291 292 293 294 295 296 297 298 299 300 301 302 303 304 305 306 307 308 309 310 311 312 313 314 315 316 317 318 319 320 321 322 323 324 325 326 327 328 329 330 331 332 333 334 335 336 337 338 339 340 341 342 343 344 345 346 347 348 349 350 351 352 353 354 355 356 357 358 359 360 361 362 363 364 365 366 367 368 369 370 371 372 373 374 375 376 377 378 379 380 381 382 383 384 385 386 387 388 389 390 391 392 393 394 395 396 397 398 399 400 401 402 403 404 405 406 407 408 409 410 411 412 413 414 415 416 417 418 419 420 421 422 423 424 425 426 427 428 429 430 431 432 433 434 435 436 437 438 439 440 441 442 443 444 445 446 447 448 449 450 451 452 453 454 455 456 457 458 459 460 461 462 463 464 465 466 467 468 469 470 471 472 473 474 475 476 477 478 479 480 481 482 483 484 485 486 487 488 489 490 491 492 493 494 495 496 497 498 499 500 501 502 503 504 505 506 507 508 509 510 511 512 513 514 515 516 517 518 519 520 521 522 523 524 525 526 527 528 529 530 531 532 533 534 535 536 537 538 539 540 541 542 543 544 545 546 547 548 549 550 551 552 553 554 555 556 557 558 559 560 561 562 563 564 565 566 567 568 569 570 571 572 573 574 575 576 577 578 579 580 581 582 583 584 585 586 587 588 589 590 591 592 593 594 595 596 597 598 599 600 601 602 603 604 605 606 607 | /* $NetBSD: pckbport.c,v 1.20 2021/08/07 16:19:15 thorpej Exp $ */ /* * Copyright (c) 2004 Ben Harris * Copyright (c) 1998 * Matthias Drochner. All rights reserved. * * Redistribution and use in source and binary forms, with or without * modification, are permitted provided that the following conditions * are met: * 1. Redistributions of source code must retain the above copyright * notice, this list of conditions and the following disclaimer. * 2. Redistributions in binary form must reproduce the above copyright * notice, this list of conditions and the following disclaimer in the * documentation and/or other materials provided with the distribution. * * THIS SOFTWARE IS PROVIDED BY THE AUTHOR ``AS IS'' AND ANY EXPRESS OR * IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES * OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. * IN NO EVENT SHALL THE AUTHOR BE LIABLE FOR ANY DIRECT, INDIRECT, * INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT * NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, * DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY * THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT * (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF * THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. */ #include <sys/cdefs.h> __KERNEL_RCSID(0, "$NetBSD: pckbport.c,v 1.20 2021/08/07 16:19:15 thorpej Exp $"); #include <sys/param.h> #include <sys/systm.h> #include <sys/callout.h> #include <sys/kernel.h> #include <sys/proc.h> #include <sys/device.h> #include <sys/malloc.h> #include <sys/errno.h> #include <sys/queue.h> #include <dev/pckbport/pckbdreg.h> #include <dev/pckbport/pckbportvar.h> #include "locators.h" #include "pckbd.h" #if (NPCKBD > 0) #include <dev/pckbport/pckbdvar.h> #endif /* descriptor for one device command */ struct pckbport_devcmd { TAILQ_ENTRY(pckbport_devcmd) next; int flags; #define KBC_CMDFLAG_SYNC 1 /* give descriptor back to caller */ #define KBC_CMDFLAG_SLOW 2 u_char cmd[4]; int cmdlen, cmdidx, retries; u_char response[4]; int status, responselen, responseidx; }; /* data per slave device */ struct pckbport_slotdata { int polling; /* don't process data in interrupt handler */ TAILQ_HEAD(, pckbport_devcmd) cmdqueue; /* active commands */ TAILQ_HEAD(, pckbport_devcmd) freequeue; /* free commands */ #define NCMD 5 struct pckbport_devcmd cmds[NCMD]; }; #define CMD_IN_QUEUE(q) (TAILQ_FIRST(&(q)->cmdqueue) != NULL) static void pckbport_init_slotdata(struct pckbport_slotdata *); static int pckbportprint(void *, const char *); static struct pckbport_slotdata pckbport_cons_slotdata; static int pckbport_poll_data1(pckbport_tag_t, pckbport_slot_t); static int pckbport_send_devcmd(struct pckbport_tag *, pckbport_slot_t, u_char); static void pckbport_poll_cmd1(struct pckbport_tag *, pckbport_slot_t, struct pckbport_devcmd *); static void pckbport_cleanqueue(struct pckbport_slotdata *); static void pckbport_cleanup(void *); static int pckbport_cmdresponse(struct pckbport_tag *, pckbport_slot_t, u_char); static void pckbport_start(struct pckbport_tag *, pckbport_slot_t); static const char * const pckbport_slot_names[] = { "kbd", "aux" }; static struct pckbport_tag pckbport_cntag; #define KBD_DELAY DELAY(8) #ifdef PCKBPORTDEBUG #define DPRINTF(a) printf a #else #define DPRINTF(a) #endif static int pckbport_poll_data1(pckbport_tag_t t, pckbport_slot_t slot) { return t->t_ops->t_poll_data1(t->t_cookie, slot); } static int pckbport_send_devcmd(struct pckbport_tag *t, pckbport_slot_t slot, u_char val) { return t->t_ops->t_send_devcmd(t->t_cookie, slot, val); } pckbport_tag_t pckbport_attach(void *cookie, struct pckbport_accessops const *ops) { pckbport_tag_t t; if (cookie == pckbport_cntag.t_cookie && ops == pckbport_cntag.t_ops) return &pckbport_cntag; t = malloc(sizeof(struct pckbport_tag), M_DEVBUF, M_WAITOK | M_ZERO); callout_init(&t->t_cleanup, 0); t->t_cookie = cookie; t->t_ops = ops; return t; } device_t pckbport_attach_slot(device_t dev, pckbport_tag_t t, pckbport_slot_t slot) { struct pckbport_attach_args pa; void *sdata; device_t found; int alloced = 0; int locs[PCKBPORTCF_NLOCS]; pa.pa_tag = t; pa.pa_slot = slot; if (t->t_slotdata[slot] == NULL) { sdata = malloc(sizeof(struct pckbport_slotdata), M_DEVBUF, M_WAITOK); t->t_slotdata[slot] = sdata; pckbport_init_slotdata(t->t_slotdata[slot]); alloced++; } locs[PCKBPORTCF_SLOT] = slot; found = config_found(dev, &pa, pckbportprint, CFARGS(.submatch = config_stdsubmatch, .iattr = "pckbport", .locators = locs)); if (found == NULL && alloced) { free(t->t_slotdata[slot], M_DEVBUF); t->t_slotdata[slot] = NULL; } return found; } int pckbportprint(void *aux, const char *pnp) { struct pckbport_attach_args *pa = aux; if (!pnp) aprint_normal(" (%s slot)", pckbport_slot_names[pa->pa_slot]); return QUIET; } void pckbport_init_slotdata(struct pckbport_slotdata *q) { int i; TAILQ_INIT(&q->cmdqueue); TAILQ_INIT(&q->freequeue); for (i = 0; i < NCMD; i++) TAILQ_INSERT_TAIL(&q->freequeue, &(q->cmds[i]), next); q->polling = 0; } void pckbport_flush(pckbport_tag_t t, pckbport_slot_t slot) { (void)pckbport_poll_data1(t, slot); } int pckbport_poll_data(pckbport_tag_t t, pckbport_slot_t slot) { struct pckbport_slotdata *q = t->t_slotdata[slot]; int c; c = pckbport_poll_data1(t, slot); if (c != -1 && q && CMD_IN_QUEUE(q)) /* * we jumped into a running command - try to deliver * the response */ if (pckbport_cmdresponse(t, slot, c)) return -1; return c; } /* * switch scancode translation on / off * return nonzero on success */ int pckbport_xt_translation(pckbport_tag_t t, pckbport_slot_t slot, int on) { return t->t_ops->t_xt_translation(t->t_cookie, slot, on); } void pckbport_slot_enable(pckbport_tag_t t, pckbport_slot_t slot, int on) { t->t_ops->t_slot_enable(t->t_cookie, slot, on); } void pckbport_set_poll(pckbport_tag_t t, pckbport_slot_t slot, int on) { t->t_slotdata[slot]->polling = on; t->t_ops->t_set_poll(t->t_cookie, slot, on); } /* * Pass command to device, poll for ACK and data. * to be called at spltty() */ static void pckbport_poll_cmd1(struct pckbport_tag *t, pckbport_slot_t slot, struct pckbport_devcmd *cmd) { int i, c = 0; while (cmd->cmdidx < cmd->cmdlen) { if (!pckbport_send_devcmd(t, slot, cmd->cmd[cmd->cmdidx])) { printf("pckbport_cmd: send error\n"); cmd->status = EIO; return; } for (i = 10; i; i--) { /* 1s ??? */ c = pckbport_poll_data1(t, slot); if (c != -1) break; } switch (c) { case KBR_ACK: cmd->cmdidx++; continue; case KBR_BAT_DONE: case KBR_BAT_FAIL: case KBR_RESEND: DPRINTF(("%s: %s\n", __func__, c == KBR_RESEND ? "RESEND" : (c == KBR_BAT_DONE ? "BAT_DONE" : "BAT_FAIL"))); if (cmd->retries++ < 5) continue; else { DPRINTF(("%s: cmd failed\n", __func__)); cmd->status = EIO; return; } case -1: DPRINTF(("%s: timeout\n", __func__)); cmd->status = EIO; return; } DPRINTF(("%s: lost 0x%x\n", __func__, c)); } while (cmd->responseidx < cmd->responselen) { if (cmd->flags & KBC_CMDFLAG_SLOW) i = 100; /* 10s ??? */ else i = 10; /* 1s ??? */ while (i--) { c = pckbport_poll_data1(t, slot); if (c != -1) break; } if (c == -1) { DPRINTF(("%s: no data\n", __func__)); cmd->status = ETIMEDOUT; return; } else cmd->response[cmd->responseidx++] = c; } } /* for use in autoconfiguration */ int pckbport_poll_cmd(pckbport_tag_t t, pckbport_slot_t slot, const u_char *cmd, int len, int responselen, u_char *respbuf, int slow) { struct pckbport_devcmd nc; if ((len > 4) || (responselen > 4)) return (EINVAL); memset(&nc, 0, sizeof(nc)); memcpy(nc.cmd, cmd, len); nc.cmdlen = len; nc.responselen = responselen; nc.flags = (slow ? KBC_CMDFLAG_SLOW : 0); pckbport_poll_cmd1(t, slot, &nc); if (nc.status == 0 && respbuf) memcpy(respbuf, nc.response, responselen); return nc.status; } /* * Clean up a command queue, throw away everything. */ void pckbport_cleanqueue(struct pckbport_slotdata *q) { struct pckbport_devcmd *cmd; while ((cmd = TAILQ_FIRST(&q->cmdqueue))) { TAILQ_REMOVE(&q->cmdqueue, cmd, next); #ifdef PCKBPORTDEBUG printf("%s: removing", __func__); for (int i = 0; i < cmd->cmdlen; i++) printf(" %02x", cmd->cmd[i]); printf("\n"); #endif TAILQ_INSERT_TAIL(&q->freequeue, cmd, next); } } /* * Timeout error handler: clean queues and data port. * XXX could be less invasive. */ void pckbport_cleanup(void *self) { struct pckbport_tag *t = self; int s; u_char cmd[1], resp[2]; printf("pckbport: command timeout\n"); s = spltty(); if (t->t_slotdata[PCKBPORT_KBD_SLOT]) pckbport_cleanqueue(t->t_slotdata[PCKBPORT_KBD_SLOT]); if (t->t_slotdata[PCKBPORT_AUX_SLOT]) pckbport_cleanqueue(t->t_slotdata[PCKBPORT_AUX_SLOT]); #if 0 /* XXXBJH Move to controller driver? */ while (bus_space_read_1(t->t_iot, t->t_ioh_c, 0) & KBS_DIB) { KBD_DELAY; (void) bus_space_read_1(t->t_iot, t->t_ioh_d, 0); } #endif cmd[0] = KBC_RESET; (void)pckbport_poll_cmd(t, PCKBPORT_KBD_SLOT, cmd, 1, 2, resp, 1); pckbport_flush(t, PCKBPORT_KBD_SLOT); splx(s); } /* * Pass command to device during normal operation. * to be called at spltty() */ void pckbport_start(struct pckbport_tag *t, pckbport_slot_t slot) { struct pckbport_slotdata *q = t->t_slotdata[slot]; struct pckbport_devcmd *cmd = TAILQ_FIRST(&q->cmdqueue); KASSERT(cmd != NULL); if (q->polling) { do { pckbport_poll_cmd1(t, slot, cmd); if (cmd->status) printf("pckbport_start: command error\n"); TAILQ_REMOVE(&q->cmdqueue, cmd, next); if (cmd->flags & KBC_CMDFLAG_SYNC) wakeup(cmd); else { callout_stop(&t->t_cleanup); TAILQ_INSERT_TAIL(&q->freequeue, cmd, next); } cmd = TAILQ_FIRST(&q->cmdqueue); } while (cmd); return; } if (!pckbport_send_devcmd(t, slot, cmd->cmd[cmd->cmdidx])) { printf("pckbport_start: send error\n"); /* XXX what now? */ return; } } /* * Handle command responses coming in asynchronously, * return nonzero if valid response. * to be called at spltty() */ int pckbport_cmdresponse(struct pckbport_tag *t, pckbport_slot_t slot, u_char data) { struct pckbport_slotdata *q = t->t_slotdata[slot]; struct pckbport_devcmd *cmd = TAILQ_FIRST(&q->cmdqueue); KASSERT(cmd != NULL); if (cmd->cmdidx < cmd->cmdlen) { if (data != KBR_ACK && data != KBR_RESEND) return 0; if (data == KBR_RESEND) { if (cmd->retries++ < 5) /* try again last command */ goto restart; else { DPRINTF(("%s: cmd failed\n", __func__)); cmd->status = EIO; /* dequeue */ } } else { if (++cmd->cmdidx < cmd->cmdlen) goto restart; if (cmd->responselen) return 1; /* else dequeue */ } } else if (cmd->responseidx < cmd->responselen) { cmd->response[cmd->responseidx++] = data; if (cmd->responseidx < cmd->responselen) return 1; /* else dequeue */ } else return 0; /* dequeue: */ TAILQ_REMOVE(&q->cmdqueue, cmd, next); if (cmd->flags & KBC_CMDFLAG_SYNC) wakeup(cmd); else { callout_stop(&t->t_cleanup); TAILQ_INSERT_TAIL(&q->freequeue, cmd, next); } if (!CMD_IN_QUEUE(q)) return 1; restart: pckbport_start(t, slot); return 1; } /* * Put command into the device's command queue, return zero or errno. */ int pckbport_enqueue_cmd(pckbport_tag_t t, pckbport_slot_t slot, const u_char *cmd, int len, int responselen, int sync, u_char *respbuf) { struct pckbport_slotdata *q = t->t_slotdata[slot]; struct pckbport_devcmd *nc; int s, isactive, res = 0; if ((len > 4) || (responselen > 4)) return EINVAL; s = spltty(); nc = TAILQ_FIRST(&q->freequeue); if (nc) TAILQ_REMOVE(&q->freequeue, nc, next); splx(s); if (!nc) return ENOMEM; memset(nc, 0, sizeof(*nc)); memcpy(nc->cmd, cmd, len); nc->cmdlen = len; nc->responselen = responselen; nc->flags = (sync ? KBC_CMDFLAG_SYNC : 0); s = spltty(); if (q->polling && sync) /* * XXX We should poll until the queue is empty. * But we don't come here normally, so make * it simple and throw away everything. */ pckbport_cleanqueue(q); isactive = CMD_IN_QUEUE(q); TAILQ_INSERT_TAIL(&q->cmdqueue, nc, next); if (!isactive) pckbport_start(t, slot); if (q->polling) res = (sync ? nc->status : 0); else if (sync) { if ((res = tsleep(nc, 0, "kbccmd", 1*hz))) { TAILQ_REMOVE(&q->cmdqueue, nc, next); pckbport_cleanup(t); } else res = nc->status; } else callout_reset(&t->t_cleanup, hz, pckbport_cleanup, t); if (sync) { if (respbuf) memcpy(respbuf, nc->response, responselen); TAILQ_INSERT_TAIL(&q->freequeue, nc, next); } splx(s); return res; } void pckbport_set_inputhandler(pckbport_tag_t t, pckbport_slot_t slot, pckbport_inputfcn func, void *arg, const char *name) { if (slot >= PCKBPORT_NSLOTS) panic("pckbport_set_inputhandler: bad slot %d", slot); t->t_ops->t_intr_establish(t->t_cookie, slot); t->t_inputhandler[slot] = func; t->t_inputarg[slot] = arg; t->t_subname[slot] = name; } void pckbportintr(pckbport_tag_t t, pckbport_slot_t slot, int data) { struct pckbport_slotdata *q; q = t->t_slotdata[slot]; if (!q) { /* XXX do something for live insertion? */ printf("pckbportintr: no dev for slot %d\n", slot); return; } if (CMD_IN_QUEUE(q) && pckbport_cmdresponse(t, slot, data)) return; if (t->t_inputhandler[slot]) { (*t->t_inputhandler[slot])(t->t_inputarg[slot], data); return; } DPRINTF(("%s: slot %d lost %d\n", __func__, slot, data)); } int pckbport_cnattach(void *cookie, struct pckbport_accessops const *ops, pckbport_slot_t slot) { int res = 0; pckbport_tag_t t = &pckbport_cntag; callout_init(&t->t_cleanup, 0); t->t_cookie = cookie; t->t_ops = ops; /* flush */ pckbport_flush(t, slot); #if (NPCKBD > 0) res = pckbd_cnattach(t, slot); #elif (NPCKBPORT_MACHDEP_CNATTACH > 0) res = pckbport_machdep_cnattach(t, slot); #else res = ENXIO; #endif /* NPCKBPORT_MACHDEP_CNATTACH > 0 */ if (res == 0) { t->t_slotdata[slot] = &pckbport_cons_slotdata; pckbport_init_slotdata(&pckbport_cons_slotdata); } return res; } |
| 4 4 6 6 6 6 4 4 7 6 6 4 4 4 4 4 4 4 4 11 7 1 4 4 7 7 4 4 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 197 198 199 200 201 202 203 204 205 206 207 208 209 210 211 212 213 214 215 216 217 218 219 220 221 222 223 224 225 226 227 228 229 230 231 232 233 234 235 236 237 238 239 240 241 242 243 244 245 246 247 248 249 250 251 252 253 254 255 256 257 258 259 260 261 262 263 264 265 266 267 268 269 270 271 272 273 274 275 276 277 278 279 280 281 282 283 284 285 286 287 288 289 290 291 292 293 294 295 296 297 298 299 300 301 302 303 304 305 306 307 308 309 310 311 312 313 314 315 316 317 318 319 320 321 322 323 324 325 326 327 328 329 330 331 332 333 334 335 336 337 338 339 340 341 342 343 344 345 346 347 348 349 350 351 352 353 354 355 356 357 358 359 360 361 362 363 364 365 366 367 368 369 370 371 372 373 374 375 376 377 378 379 380 381 382 383 384 385 386 387 388 389 390 391 392 393 394 395 396 397 398 399 400 401 402 403 404 405 406 407 408 409 410 411 412 413 414 415 416 417 418 419 420 421 422 423 424 425 426 427 428 429 430 431 432 433 434 435 436 437 438 439 440 441 442 443 444 445 446 447 448 449 450 451 452 453 454 455 456 457 458 459 460 461 462 463 464 465 466 467 468 469 470 471 472 473 474 475 476 477 478 479 480 481 482 483 484 485 486 487 488 489 490 491 492 493 494 495 496 497 498 499 500 501 502 503 504 505 506 507 508 509 510 511 512 513 514 515 516 517 518 519 520 521 522 523 524 525 526 527 528 529 530 531 532 533 534 535 536 537 538 539 540 541 542 543 544 545 546 547 548 549 550 551 552 553 554 555 556 557 558 559 560 561 562 563 564 565 566 567 568 569 570 571 572 573 574 575 576 577 578 579 580 581 582 583 584 585 586 587 588 589 590 591 592 593 594 595 596 597 598 599 600 601 602 603 604 605 606 607 608 609 610 611 612 613 614 615 616 617 618 619 620 621 622 623 624 625 626 627 628 629 630 631 632 633 634 635 636 637 638 639 640 641 642 643 644 645 646 647 648 649 650 651 652 653 654 655 656 657 658 659 660 661 662 663 664 665 666 667 668 669 670 671 672 673 674 675 676 677 678 679 680 681 682 683 684 685 686 687 688 689 690 691 692 693 694 695 696 697 698 699 700 701 702 703 704 705 706 707 708 709 710 711 712 713 714 715 716 717 718 719 720 721 722 723 724 725 726 727 728 729 730 731 732 733 734 735 736 737 738 739 740 741 742 743 744 745 746 747 748 749 750 751 752 753 754 755 756 757 758 759 760 761 762 763 764 765 766 767 768 769 770 771 772 773 774 775 776 777 778 779 780 781 782 783 784 785 786 787 788 789 790 791 792 793 794 795 796 797 798 799 800 801 802 803 804 805 806 807 808 809 810 811 812 813 814 815 816 817 818 819 820 821 822 823 824 825 826 827 828 829 830 831 832 833 834 835 836 837 838 839 840 841 842 843 844 845 846 847 848 849 850 851 852 853 854 855 856 857 858 859 860 861 862 863 864 865 866 867 868 869 870 871 872 873 874 875 876 877 878 879 880 881 882 883 884 885 886 887 888 889 890 891 892 893 894 895 896 897 898 899 900 901 902 903 904 905 906 907 908 909 910 911 912 913 914 915 916 917 918 919 920 921 922 923 924 925 926 927 928 929 930 931 932 933 934 935 936 937 938 939 940 941 942 943 944 945 946 947 948 949 950 951 952 953 954 955 956 957 958 959 960 961 962 963 964 965 966 967 968 969 970 971 972 973 974 975 976 977 978 979 980 981 982 983 984 985 986 987 988 989 990 991 992 993 994 995 996 997 998 999 1000 1001 1002 1003 1004 1005 1006 1007 1008 1009 1010 1011 1012 1013 1014 1015 1016 1017 1018 1019 1020 1021 1022 1023 1024 1025 1026 1027 1028 1029 1030 1031 1032 1033 1034 1035 1036 1037 1038 1039 1040 1041 1042 1043 1044 1045 1046 1047 1048 1049 1050 1051 1052 1053 1054 1055 1056 1057 1058 1059 1060 1061 1062 1063 1064 1065 1066 1067 1068 1069 1070 1071 1072 1073 1074 1075 1076 1077 1078 1079 1080 1081 1082 1083 1084 1085 1086 1087 1088 1089 1090 1091 1092 1093 1094 1095 1096 1097 1098 1099 1100 1101 1102 1103 1104 1105 1106 1107 1108 1109 1110 1111 1112 1113 | /* $NetBSD: mld6.c,v 1.101 2019/09/25 09:53:38 ozaki-r Exp $ */ /* $KAME: mld6.c,v 1.25 2001/01/16 14:14:18 itojun Exp $ */ /* * Copyright (C) 1998 WIDE Project. * All rights reserved. * * Redistribution and use in source and binary forms, with or without * modification, are permitted provided that the following conditions * are met: * 1. Redistributions of source code must retain the above copyright * notice, this list of conditions and the following disclaimer. * 2. Redistributions in binary form must reproduce the above copyright * notice, this list of conditions and the following disclaimer in the * documentation and/or other materials provided with the distribution. * 3. Neither the name of the project nor the names of its contributors * may be used to endorse or promote products derived from this software * without specific prior written permission. * * THIS SOFTWARE IS PROVIDED BY THE PROJECT AND CONTRIBUTORS ``AS IS'' AND * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE * ARE DISCLAIMED. IN NO EVENT SHALL THE PROJECT OR CONTRIBUTORS BE LIABLE * FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL * DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS * OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) * HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT * LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY * OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF * SUCH DAMAGE. */ /* * Copyright (c) 1992, 1993 * The Regents of the University of California. All rights reserved. * * This code is derived from software contributed to Berkeley by * Stephen Deering of Stanford University. * * Redistribution and use in source and binary forms, with or without * modification, are permitted provided that the following conditions * are met: * 1. Redistributions of source code must retain the above copyright * notice, this list of conditions and the following disclaimer. * 2. Redistributions in binary form must reproduce the above copyright * notice, this list of conditions and the following disclaimer in the * documentation and/or other materials provided with the distribution. * 3. Neither the name of the University nor the names of its contributors * may be used to endorse or promote products derived from this software * without specific prior written permission. * * THIS SOFTWARE IS PROVIDED BY THE REGENTS AND CONTRIBUTORS ``AS IS'' AND * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE * ARE DISCLAIMED. IN NO EVENT SHALL THE REGENTS OR CONTRIBUTORS BE LIABLE * FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL * DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS * OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) * HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT * LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY * OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF * SUCH DAMAGE. * * @(#)igmp.c 8.1 (Berkeley) 7/19/93 */ /* * Copyright (c) 1988 Stephen Deering. * * This code is derived from software contributed to Berkeley by * Stephen Deering of Stanford University. * * Redistribution and use in source and binary forms, with or without * modification, are permitted provided that the following conditions * are met: * 1. Redistributions of source code must retain the above copyright * notice, this list of conditions and the following disclaimer. * 2. Redistributions in binary form must reproduce the above copyright * notice, this list of conditions and the following disclaimer in the * documentation and/or other materials provided with the distribution. * 3. All advertising materials mentioning features or use of this software * must display the following acknowledgement: * This product includes software developed by the University of * California, Berkeley and its contributors. * 4. Neither the name of the University nor the names of its contributors * may be used to endorse or promote products derived from this software * without specific prior written permission. * * THIS SOFTWARE IS PROVIDED BY THE REGENTS AND CONTRIBUTORS ``AS IS'' AND * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE * ARE DISCLAIMED. IN NO EVENT SHALL THE REGENTS OR CONTRIBUTORS BE LIABLE * FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL * DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS * OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) * HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT * LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY * OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF * SUCH DAMAGE. * * @(#)igmp.c 8.1 (Berkeley) 7/19/93 */ #include <sys/cdefs.h> __KERNEL_RCSID(0, "$NetBSD: mld6.c,v 1.101 2019/09/25 09:53:38 ozaki-r Exp $"); #ifdef _KERNEL_OPT #include "opt_inet.h" #include "opt_net_mpsafe.h" #endif #include <sys/param.h> #include <sys/systm.h> #include <sys/mbuf.h> #include <sys/socket.h> #include <sys/socketvar.h> #include <sys/syslog.h> #include <sys/sysctl.h> #include <sys/kernel.h> #include <sys/callout.h> #include <sys/cprng.h> #include <sys/rwlock.h> #include <net/if.h> #include <netinet/in.h> #include <netinet/in_var.h> #include <netinet6/in6_var.h> #include <netinet/ip6.h> #include <netinet6/ip6_var.h> #include <netinet6/scope6_var.h> #include <netinet/icmp6.h> #include <netinet6/icmp6_private.h> #include <netinet6/mld6_var.h> static krwlock_t in6_multilock __cacheline_aligned; /* * Protocol constants */ /* * time between repetitions of a node's initial report of interest in a * multicast address(in seconds) */ #define MLD_UNSOLICITED_REPORT_INTERVAL 10 static struct ip6_pktopts ip6_opts; static void mld_start_listening(struct in6_multi *); static void mld_stop_listening(struct in6_multi *); static struct mld_hdr *mld_allocbuf(struct mbuf **, struct in6_multi *, int); static void mld_sendpkt(struct in6_multi *, int, const struct in6_addr *); static void mld_starttimer(struct in6_multi *); static void mld_stoptimer(struct in6_multi *); static u_long mld_timerresid(struct in6_multi *); static void in6m_ref(struct in6_multi *); static void in6m_unref(struct in6_multi *); static void in6m_destroy(struct in6_multi *); void mld_init(void) { static u_int8_t hbh_buf[8]; struct ip6_hbh *hbh = (struct ip6_hbh *)hbh_buf; u_int16_t rtalert_code = htons((u_int16_t)IP6OPT_RTALERT_MLD); /* ip6h_nxt will be fill in later */ hbh->ip6h_len = 0; /* (8 >> 3) - 1 */ /* XXX: grotty hard coding... */ hbh_buf[2] = IP6OPT_PADN; /* 2 byte padding */ hbh_buf[3] = 0; hbh_buf[4] = IP6OPT_RTALERT; hbh_buf[5] = IP6OPT_RTALERT_LEN - 2; memcpy(&hbh_buf[6], (void *)&rtalert_code, sizeof(u_int16_t)); ip6_opts.ip6po_hbh = hbh; /* We will specify the hoplimit by a multicast option. */ ip6_opts.ip6po_hlim = -1; ip6_opts.ip6po_prefer_tempaddr = IP6PO_TEMPADDR_NOTPREFER; rw_init(&in6_multilock); } static void mld_starttimer(struct in6_multi *in6m) { struct timeval now; KASSERT(rw_write_held(&in6_multilock)); KASSERTMSG(in6m->in6m_timer != IN6M_TIMER_UNDEF, "in6m_timer=%d", in6m->in6m_timer); microtime(&now); in6m->in6m_timer_expire.tv_sec = now.tv_sec + in6m->in6m_timer / hz; in6m->in6m_timer_expire.tv_usec = now.tv_usec + (in6m->in6m_timer % hz) * (1000000 / hz); if (in6m->in6m_timer_expire.tv_usec > 1000000) { in6m->in6m_timer_expire.tv_sec++; in6m->in6m_timer_expire.tv_usec -= 1000000; } /* start or restart the timer */ callout_schedule(&in6m->in6m_timer_ch, in6m->in6m_timer); } /* * mld_stoptimer releases in6_multilock when calling callout_halt. * The caller must ensure in6m won't be freed while releasing the lock. */ static void mld_stoptimer(struct in6_multi *in6m) { KASSERT(rw_write_held(&in6_multilock)); if (in6m->in6m_timer == IN6M_TIMER_UNDEF) return; rw_exit(&in6_multilock); callout_halt(&in6m->in6m_timer_ch, NULL); rw_enter(&in6_multilock, RW_WRITER); in6m->in6m_timer = IN6M_TIMER_UNDEF; } static void mld_timeo(void *arg) { struct in6_multi *in6m = arg; KASSERTMSG(in6m->in6m_refcount > 0, "in6m_refcount=%d", in6m->in6m_refcount); KERNEL_LOCK_UNLESS_NET_MPSAFE(); rw_enter(&in6_multilock, RW_WRITER); if (in6m->in6m_timer == IN6M_TIMER_UNDEF) goto out; in6m->in6m_timer = IN6M_TIMER_UNDEF; switch (in6m->in6m_state) { case MLD_REPORTPENDING: mld_start_listening(in6m); break; default: mld_sendpkt(in6m, MLD_LISTENER_REPORT, NULL); break; } out: rw_exit(&in6_multilock); KERNEL_UNLOCK_UNLESS_NET_MPSAFE(); } static u_long mld_timerresid(struct in6_multi *in6m) { struct timeval now, diff; microtime(&now); if (now.tv_sec > in6m->in6m_timer_expire.tv_sec || (now.tv_sec == in6m->in6m_timer_expire.tv_sec && now.tv_usec > in6m->in6m_timer_expire.tv_usec)) { return (0); } diff = in6m->in6m_timer_expire; diff.tv_sec -= now.tv_sec; diff.tv_usec -= now.tv_usec; if (diff.tv_usec < 0) { diff.tv_sec--; diff.tv_usec += 1000000; } /* return the remaining time in milliseconds */ return diff.tv_sec * 1000 + diff.tv_usec / 1000; } static void mld_start_listening(struct in6_multi *in6m) { struct in6_addr all_in6; KASSERT(rw_write_held(&in6_multilock)); /* * RFC2710 page 10: * The node never sends a Report or Done for the link-scope all-nodes * address. * MLD messages are never sent for multicast addresses whose scope is 0 * (reserved) or 1 (node-local). */ all_in6 = in6addr_linklocal_allnodes; if (in6_setscope(&all_in6, in6m->in6m_ifp, NULL)) { /* XXX: this should not happen! */ in6m->in6m_timer = 0; in6m->in6m_state = MLD_OTHERLISTENER; } if (IN6_ARE_ADDR_EQUAL(&in6m->in6m_addr, &all_in6) || IPV6_ADDR_MC_SCOPE(&in6m->in6m_addr) < IPV6_ADDR_SCOPE_LINKLOCAL) { in6m->in6m_timer = IN6M_TIMER_UNDEF; in6m->in6m_state = MLD_OTHERLISTENER; } else { mld_sendpkt(in6m, MLD_LISTENER_REPORT, NULL); in6m->in6m_timer = cprng_fast32() % (MLD_UNSOLICITED_REPORT_INTERVAL * hz); in6m->in6m_state = MLD_IREPORTEDLAST; mld_starttimer(in6m); } } static void mld_stop_listening(struct in6_multi *in6m) { struct in6_addr allnode, allrouter; KASSERT(rw_lock_held(&in6_multilock)); allnode = in6addr_linklocal_allnodes; if (in6_setscope(&allnode, in6m->in6m_ifp, NULL)) { /* XXX: this should not happen! */ return; } allrouter = in6addr_linklocal_allrouters; if (in6_setscope(&allrouter, in6m->in6m_ifp, NULL)) { /* XXX impossible */ return; } if (in6m->in6m_state == MLD_IREPORTEDLAST && (!IN6_ARE_ADDR_EQUAL(&in6m->in6m_addr, &allnode)) && IPV6_ADDR_MC_SCOPE(&in6m->in6m_addr) > IPV6_ADDR_SCOPE_INTFACELOCAL) { mld_sendpkt(in6m, MLD_LISTENER_DONE, &allrouter); } } void mld_input(struct mbuf *m, int off) { struct ip6_hdr *ip6; struct mld_hdr *mldh; struct ifnet *ifp; struct in6_multi *in6m = NULL; struct in6_addr mld_addr, all_in6; u_long timer = 0; /* timer value in the MLD query header */ struct psref psref; ifp = m_get_rcvif_psref(m, &psref); if (__predict_false(ifp == NULL)) goto out; IP6_EXTHDR_GET(mldh, struct mld_hdr *, m, off, sizeof(*mldh)); if (mldh == NULL) { ICMP6_STATINC(ICMP6_STAT_TOOSHORT); goto out_nodrop; } ip6 = mtod(m, struct ip6_hdr *); /* source address validation */ if (!IN6_IS_ADDR_LINKLOCAL(&ip6->ip6_src)) { /* * RFC3590 allows the IPv6 unspecified address as the source * address of MLD report and done messages. However, as this * same document says, this special rule is for snooping * switches and the RFC requires routers to discard MLD packets * with the unspecified source address. The RFC only talks * about hosts receiving an MLD query or report in Security * Considerations, but this is probably the correct intention. * RFC3590 does not talk about other cases than link-local and * the unspecified source addresses, but we believe the same * rule should be applied. * As a result, we only allow link-local addresses as the * source address; otherwise, simply discard the packet. */ #if 0 /* * XXX: do not log in an input path to avoid log flooding, * though RFC3590 says "SHOULD log" if the source of a query * is the unspecified address. */ char ip6bufs[INET6_ADDRSTRLEN]; char ip6bufm[INET6_ADDRSTRLEN]; log(LOG_INFO, "mld_input: src %s is not link-local (grp=%s)\n", IN6_PRINT(ip6bufs,&ip6->ip6_src), IN6_PRINT(ip6bufm, &mldh->mld_addr)); #endif goto out; } /* * make a copy for local work (in6_setscope() may modify the 1st arg) */ mld_addr = mldh->mld_addr; if (in6_setscope(&mld_addr, ifp, NULL)) { /* XXX: this should not happen! */ goto out; } /* * In the MLD specification, there are 3 states and a flag. * * In Non-Listener state, we simply don't have a membership record. * In Delaying Listener state, our timer is running (in6m->in6m_timer) * In Idle Listener state, our timer is not running * (in6m->in6m_timer==IN6M_TIMER_UNDEF) * * The flag is in6m->in6m_state, it is set to MLD_OTHERLISTENER if * we have heard a report from another member, or MLD_IREPORTEDLAST * if we sent the last report. */ switch (mldh->mld_type) { case MLD_LISTENER_QUERY: { struct in6_multi *next; if (ifp->if_flags & IFF_LOOPBACK) break; if (!IN6_IS_ADDR_UNSPECIFIED(&mld_addr) && !IN6_IS_ADDR_MULTICAST(&mld_addr)) break; /* print error or log stat? */ all_in6 = in6addr_linklocal_allnodes; if (in6_setscope(&all_in6, ifp, NULL)) { /* XXX: this should not happen! */ break; } /* * - Start the timers in all of our membership records * that the query applies to for the interface on * which the query arrived excl. those that belong * to the "all-nodes" group (ff02::1). * - Restart any timer that is already running but has * a value longer than the requested timeout. * - Use the value specified in the query message as * the maximum timeout. */ timer = ntohs(mldh->mld_maxdelay); rw_enter(&in6_multilock, RW_WRITER); /* * mld_stoptimer and mld_sendpkt release in6_multilock * temporarily, so we have to prevent in6m from being freed * while releasing the lock by having an extra reference to it. * * Also in6_purge_multi might remove items from the list of the * ifp while releasing the lock. Fortunately in6_purge_multi is * never executed as long as we have a psref of the ifp. */ LIST_FOREACH_SAFE(in6m, &ifp->if_multiaddrs, in6m_entry, next) { if (IN6_ARE_ADDR_EQUAL(&in6m->in6m_addr, &all_in6) || IPV6_ADDR_MC_SCOPE(&in6m->in6m_addr) < IPV6_ADDR_SCOPE_LINKLOCAL) continue; if (in6m->in6m_state == MLD_REPORTPENDING) continue; /* we are not yet ready */ if (!IN6_IS_ADDR_UNSPECIFIED(&mld_addr) && !IN6_ARE_ADDR_EQUAL(&mld_addr, &in6m->in6m_addr)) continue; if (timer == 0) { in6m_ref(in6m); /* send a report immediately */ mld_stoptimer(in6m); mld_sendpkt(in6m, MLD_LISTENER_REPORT, NULL); in6m->in6m_state = MLD_IREPORTEDLAST; in6m_unref(in6m); /* May free in6m */ } else if (in6m->in6m_timer == IN6M_TIMER_UNDEF || mld_timerresid(in6m) > timer) { in6m->in6m_timer = 1 + (cprng_fast32() % timer) * hz / 1000; mld_starttimer(in6m); } } rw_exit(&in6_multilock); break; } case MLD_LISTENER_REPORT: /* * For fast leave to work, we have to know that we are the * last person to send a report for this group. Reports * can potentially get looped back if we are a multicast * router, so discard reports sourced by me. * Note that it is impossible to check IFF_LOOPBACK flag of * ifp for this purpose, since ip6_mloopback pass the physical * interface to looutput. */ if (m->m_flags & M_LOOP) /* XXX: grotty flag, but efficient */ break; if (!IN6_IS_ADDR_MULTICAST(&mldh->mld_addr)) break; /* * If we belong to the group being reported, stop * our timer for that group. */ rw_enter(&in6_multilock, RW_WRITER); in6m = in6_lookup_multi(&mld_addr, ifp); if (in6m) { in6m_ref(in6m); mld_stoptimer(in6m); /* transit to idle state */ in6m->in6m_state = MLD_OTHERLISTENER; /* clear flag */ in6m_unref(in6m); in6m = NULL; /* in6m might be freed */ } rw_exit(&in6_multilock); break; default: /* this is impossible */ #if 0 /* * this case should be impossible because of filtering in * icmp6_input(). But we explicitly disabled this part * just in case. */ log(LOG_ERR, "mld_input: illegal type(%d)", mldh->mld_type); #endif break; } out: m_freem(m); out_nodrop: m_put_rcvif_psref(ifp, &psref); } /* * XXX mld_sendpkt must be called with in6_multilock held and * will release in6_multilock before calling ip6_output and * returning to avoid locking against myself in ip6_output. */ static void mld_sendpkt(struct in6_multi *in6m, int type, const struct in6_addr *dst) { struct mbuf *mh; struct mld_hdr *mldh; struct ip6_hdr *ip6 = NULL; struct ip6_moptions im6o; struct in6_ifaddr *ia = NULL; struct ifnet *ifp = in6m->in6m_ifp; int ignflags; struct psref psref; int bound; KASSERT(rw_write_held(&in6_multilock)); /* * At first, find a link local address on the outgoing interface * to use as the source address of the MLD packet. * We do not reject tentative addresses for MLD report to deal with * the case where we first join a link-local address. */ ignflags = (IN6_IFF_NOTREADY|IN6_IFF_ANYCAST) & ~IN6_IFF_TENTATIVE; bound = curlwp_bind(); ia = in6ifa_ifpforlinklocal_psref(ifp, ignflags, &psref); if (ia == NULL) { curlwp_bindx(bound); return; } if ((ia->ia6_flags & IN6_IFF_TENTATIVE)) { ia6_release(ia, &psref); ia = NULL; } /* Allocate two mbufs to store IPv6 header and MLD header */ mldh = mld_allocbuf(&mh, in6m, type); if (mldh == NULL) { ia6_release(ia, &psref); curlwp_bindx(bound); return; } /* fill src/dst here */ ip6 = mtod(mh, struct ip6_hdr *); ip6->ip6_src = ia ? ia->ia_addr.sin6_addr : in6addr_any; ip6->ip6_dst = dst ? *dst : in6m->in6m_addr; ia6_release(ia, &psref); curlwp_bindx(bound); mldh->mld_addr = in6m->in6m_addr; in6_clearscope(&mldh->mld_addr); /* XXX */ mldh->mld_cksum = in6_cksum(mh, IPPROTO_ICMPV6, sizeof(struct ip6_hdr), sizeof(struct mld_hdr)); /* construct multicast option */ memset(&im6o, 0, sizeof(im6o)); im6o.im6o_multicast_if_index = if_get_index(ifp); im6o.im6o_multicast_hlim = 1; /* * Request loopback of the report if we are acting as a multicast * router, so that the process-level routing daemon can hear it. */ im6o.im6o_multicast_loop = (ip6_mrouter != NULL); /* increment output statistics */ ICMP6_STATINC(ICMP6_STAT_OUTHIST + type); icmp6_ifstat_inc(ifp, ifs6_out_msg); switch (type) { case MLD_LISTENER_QUERY: icmp6_ifstat_inc(ifp, ifs6_out_mldquery); break; case MLD_LISTENER_REPORT: icmp6_ifstat_inc(ifp, ifs6_out_mldreport); break; case MLD_LISTENER_DONE: icmp6_ifstat_inc(ifp, ifs6_out_mlddone); break; } /* XXX we cannot call ip6_output with holding in6_multilock */ rw_exit(&in6_multilock); ip6_output(mh, &ip6_opts, NULL, ia ? 0 : IPV6_UNSPECSRC, &im6o, NULL, NULL); rw_enter(&in6_multilock, RW_WRITER); } static struct mld_hdr * mld_allocbuf(struct mbuf **mh, struct in6_multi *in6m, int type) { struct mbuf *md; struct mld_hdr *mldh; struct ip6_hdr *ip6; /* * Allocate mbufs to store ip6 header and MLD header. * We allocate 2 mbufs and make chain in advance because * it is more convenient when inserting the hop-by-hop option later. */ MGETHDR(*mh, M_DONTWAIT, MT_HEADER); if (*mh == NULL) return NULL; MGET(md, M_DONTWAIT, MT_DATA); if (md == NULL) { m_free(*mh); *mh = NULL; return NULL; } (*mh)->m_next = md; md->m_next = NULL; m_reset_rcvif((*mh)); (*mh)->m_pkthdr.len = sizeof(struct ip6_hdr) + sizeof(struct mld_hdr); (*mh)->m_len = sizeof(struct ip6_hdr); m_align(*mh, sizeof(struct ip6_hdr)); /* fill in the ip6 header */ ip6 = mtod(*mh, struct ip6_hdr *); memset(ip6, 0, sizeof(*ip6)); ip6->ip6_flow = 0; ip6->ip6_vfc &= ~IPV6_VERSION_MASK; ip6->ip6_vfc |= IPV6_VERSION; /* ip6_plen will be set later */ ip6->ip6_nxt = IPPROTO_ICMPV6; /* ip6_hlim will be set by im6o.im6o_multicast_hlim */ /* ip6_src/dst will be set by mld_sendpkt() or mld_sendbuf() */ /* fill in the MLD header as much as possible */ md->m_len = sizeof(struct mld_hdr); mldh = mtod(md, struct mld_hdr *); memset(mldh, 0, sizeof(struct mld_hdr)); mldh->mld_type = type; return mldh; } static void in6m_ref(struct in6_multi *in6m) { KASSERT(rw_write_held(&in6_multilock)); in6m->in6m_refcount++; } static void in6m_unref(struct in6_multi *in6m) { KASSERT(rw_write_held(&in6_multilock)); if (--in6m->in6m_refcount == 0) in6m_destroy(in6m); } /* * Add an address to the list of IP6 multicast addresses for a given interface. */ struct in6_multi * in6_addmulti(struct in6_addr *maddr6, struct ifnet *ifp, int *errorp, int timer) { struct sockaddr_in6 sin6; struct in6_multi *in6m; *errorp = 0; rw_enter(&in6_multilock, RW_WRITER); /* * See if address already in list. */ in6m = in6_lookup_multi(maddr6, ifp); if (in6m != NULL) { /* * Found it; just increment the reference count. */ in6m->in6m_refcount++; } else { /* * New address; allocate a new multicast record * and link it into the interface's multicast list. */ in6m = malloc(sizeof(*in6m), M_IPMADDR, M_NOWAIT|M_ZERO); if (in6m == NULL) { *errorp = ENOBUFS; goto out; } in6m->in6m_addr = *maddr6; in6m->in6m_ifp = ifp; in6m->in6m_refcount = 1; in6m->in6m_timer = IN6M_TIMER_UNDEF; callout_init(&in6m->in6m_timer_ch, CALLOUT_MPSAFE); callout_setfunc(&in6m->in6m_timer_ch, mld_timeo, in6m); LIST_INSERT_HEAD(&ifp->if_multiaddrs, in6m, in6m_entry); /* * Ask the network driver to update its multicast reception * filter appropriately for the new address. */ sockaddr_in6_init(&sin6, maddr6, 0, 0, 0); *errorp = if_mcast_op(ifp, SIOCADDMULTI, sin6tosa(&sin6)); if (*errorp) { callout_destroy(&in6m->in6m_timer_ch); LIST_REMOVE(in6m, in6m_entry); free(in6m, M_IPMADDR); in6m = NULL; goto out; } in6m->in6m_timer = timer; if (in6m->in6m_timer > 0) { in6m->in6m_state = MLD_REPORTPENDING; mld_starttimer(in6m); goto out; } /* * Let MLD6 know that we have joined a new IP6 multicast * group. */ mld_start_listening(in6m); } out: rw_exit(&in6_multilock); return in6m; } static void in6m_destroy(struct in6_multi *in6m) { struct sockaddr_in6 sin6; KASSERT(rw_write_held(&in6_multilock)); KASSERTMSG(in6m->in6m_refcount == 0, "in6m_refcount=%d", in6m->in6m_refcount); /* * Unlink from list if it's listed. This must be done before * mld_stop_listening because it releases in6_multilock and that allows * someone to look up the removing in6m from the list and add a * reference to the entry unexpectedly. */ if (in6_lookup_multi(&in6m->in6m_addr, in6m->in6m_ifp) != NULL) LIST_REMOVE(in6m, in6m_entry); /* * No remaining claims to this record; let MLD6 know * that we are leaving the multicast group. */ mld_stop_listening(in6m); /* * Delete all references of this multicasting group from * the membership arrays */ in6_purge_mcast_references(in6m); /* * Notify the network driver to update its multicast * reception filter. */ sockaddr_in6_init(&sin6, &in6m->in6m_addr, 0, 0, 0); if_mcast_op(in6m->in6m_ifp, SIOCDELMULTI, sin6tosa(&sin6)); /* Tell mld_timeo we're halting the timer */ in6m->in6m_timer = IN6M_TIMER_UNDEF; rw_exit(&in6_multilock); callout_halt(&in6m->in6m_timer_ch, NULL); callout_destroy(&in6m->in6m_timer_ch); free(in6m, M_IPMADDR); rw_enter(&in6_multilock, RW_WRITER); } /* * Delete a multicast address record. */ void in6_delmulti_locked(struct in6_multi *in6m) { KASSERT(rw_write_held(&in6_multilock)); KASSERTMSG(in6m->in6m_refcount > 0, "in6m_refcount=%d", in6m->in6m_refcount); /* * The caller should have a reference to in6m. So we don't need to care * of releasing the lock in mld_stoptimer. */ mld_stoptimer(in6m); if (--in6m->in6m_refcount == 0) in6m_destroy(in6m); } void in6_delmulti(struct in6_multi *in6m) { rw_enter(&in6_multilock, RW_WRITER); in6_delmulti_locked(in6m); rw_exit(&in6_multilock); } /* * Look up the in6_multi record for a given IP6 multicast address * on a given interface. If no matching record is found, "in6m" * returns NULL. */ struct in6_multi * in6_lookup_multi(const struct in6_addr *addr, const struct ifnet *ifp) { struct in6_multi *in6m; KASSERT(rw_lock_held(&in6_multilock)); LIST_FOREACH(in6m, &ifp->if_multiaddrs, in6m_entry) { if (IN6_ARE_ADDR_EQUAL(&in6m->in6m_addr, addr)) break; } return in6m; } void in6_lookup_and_delete_multi(const struct in6_addr *addr, const struct ifnet *ifp) { struct in6_multi *in6m; rw_enter(&in6_multilock, RW_WRITER); in6m = in6_lookup_multi(addr, ifp); if (in6m != NULL) in6_delmulti_locked(in6m); rw_exit(&in6_multilock); } bool in6_multi_group(const struct in6_addr *addr, const struct ifnet *ifp) { bool ingroup; rw_enter(&in6_multilock, RW_READER); ingroup = in6_lookup_multi(addr, ifp) != NULL; rw_exit(&in6_multilock); return ingroup; } /* * Purge in6_multi records associated to the interface. */ void in6_purge_multi(struct ifnet *ifp) { struct in6_multi *in6m, *next; rw_enter(&in6_multilock, RW_WRITER); LIST_FOREACH_SAFE(in6m, &ifp->if_multiaddrs, in6m_entry, next) { LIST_REMOVE(in6m, in6m_entry); /* * Normally multicast addresses are already purged at this * point. Remaining references aren't accessible via ifp, * so what we can do here is to prevent ifp from being * accessed via in6m by removing it from the list of ifp. */ mld_stoptimer(in6m); } rw_exit(&in6_multilock); } void in6_multi_lock(int op) { rw_enter(&in6_multilock, op); } void in6_multi_unlock(void) { rw_exit(&in6_multilock); } bool in6_multi_locked(int op) { switch (op) { case RW_READER: return rw_read_held(&in6_multilock); case RW_WRITER: return rw_write_held(&in6_multilock); default: return rw_lock_held(&in6_multilock); } } struct in6_multi_mship * in6_joingroup(struct ifnet *ifp, struct in6_addr *addr, int *errorp, int timer) { struct in6_multi_mship *imm; imm = malloc(sizeof(*imm), M_IPMADDR, M_NOWAIT|M_ZERO); if (imm == NULL) { *errorp = ENOBUFS; return NULL; } imm->i6mm_maddr = in6_addmulti(addr, ifp, errorp, timer); if (!imm->i6mm_maddr) { /* *errorp is already set */ free(imm, M_IPMADDR); return NULL; } return imm; } int in6_leavegroup(struct in6_multi_mship *imm) { struct in6_multi *in6m; rw_enter(&in6_multilock, RW_WRITER); in6m = imm->i6mm_maddr; imm->i6mm_maddr = NULL; if (in6m != NULL) { in6_delmulti_locked(in6m); } rw_exit(&in6_multilock); free(imm, M_IPMADDR); return 0; } /* * DEPRECATED: keep it just to avoid breaking old sysctl users. */ static int in6_mkludge_sysctl(SYSCTLFN_ARGS) { if (namelen != 1) return EINVAL; *oldlenp = 0; return 0; } static int in6_multicast_sysctl(SYSCTLFN_ARGS) { struct ifnet *ifp; struct ifaddr *ifa; struct in6_ifaddr *ia6; struct in6_multi *in6m; uint32_t tmp; int error; size_t written; struct psref psref, psref_ia; int bound, s; if (namelen != 1) return EINVAL; rw_enter(&in6_multilock, RW_READER); bound = curlwp_bind(); ifp = if_get_byindex(name[0], &psref); if (ifp == NULL) { curlwp_bindx(bound); rw_exit(&in6_multilock); return ENODEV; } if (oldp == NULL) { *oldlenp = 0; s = pserialize_read_enter(); IFADDR_READER_FOREACH(ifa, ifp) { LIST_FOREACH(in6m, &ifp->if_multiaddrs, in6m_entry) { *oldlenp += 2 * sizeof(struct in6_addr) + sizeof(uint32_t); } } pserialize_read_exit(s); if_put(ifp, &psref); curlwp_bindx(bound); rw_exit(&in6_multilock); return 0; } error = 0; written = 0; s = pserialize_read_enter(); IFADDR_READER_FOREACH(ifa, ifp) { if (ifa->ifa_addr->sa_family != AF_INET6) continue; ifa_acquire(ifa, &psref_ia); pserialize_read_exit(s); ia6 = ifatoia6(ifa); LIST_FOREACH(in6m, &ifp->if_multiaddrs, in6m_entry) { if (written + 2 * sizeof(struct in6_addr) + sizeof(uint32_t) > *oldlenp) goto done; /* * XXX return the first IPv6 address to keep backward * compatibility, however now multicast addresses * don't belong to any IPv6 addresses so it should be * unnecessary. */ error = sysctl_copyout(l, &ia6->ia_addr.sin6_addr, oldp, sizeof(struct in6_addr)); if (error) goto done; oldp = (char *)oldp + sizeof(struct in6_addr); written += sizeof(struct in6_addr); error = sysctl_copyout(l, &in6m->in6m_addr, oldp, sizeof(struct in6_addr)); if (error) goto done; oldp = (char *)oldp + sizeof(struct in6_addr); written += sizeof(struct in6_addr); tmp = in6m->in6m_refcount; error = sysctl_copyout(l, &tmp, oldp, sizeof(tmp)); if (error) goto done; oldp = (char *)oldp + sizeof(tmp); written += sizeof(tmp); } s = pserialize_read_enter(); break; } pserialize_read_exit(s); done: ifa_release(ifa, &psref_ia); if_put(ifp, &psref); curlwp_bindx(bound); rw_exit(&in6_multilock); *oldlenp = written; return error; } void in6_sysctl_multicast_setup(struct sysctllog **clog) { sysctl_createv(clog, 0, NULL, NULL, CTLFLAG_PERMANENT, CTLTYPE_NODE, "inet6", NULL, NULL, 0, NULL, 0, CTL_NET, PF_INET6, CTL_EOL); sysctl_createv(clog, 0, NULL, NULL, CTLFLAG_PERMANENT, CTLTYPE_NODE, "multicast", SYSCTL_DESCR("Multicast information"), in6_multicast_sysctl, 0, NULL, 0, CTL_NET, PF_INET6, CTL_CREATE, CTL_EOL); sysctl_createv(clog, 0, NULL, NULL, CTLFLAG_PERMANENT, CTLTYPE_NODE, "multicast_kludge", SYSCTL_DESCR("multicast kludge information"), in6_mkludge_sysctl, 0, NULL, 0, CTL_NET, PF_INET6, CTL_CREATE, CTL_EOL); } |
| 134 133 133 133 133 96 96 96 28 28 26 28 6 23 24 6 6 28 28 28 28 28 28 28 6 23 28 28 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 197 198 199 200 201 202 203 204 205 206 207 208 209 210 211 212 213 214 215 216 217 218 219 220 221 222 223 224 225 226 227 228 229 230 231 232 233 234 235 236 237 238 239 240 241 242 243 244 245 246 247 248 249 250 251 252 253 254 255 256 257 258 259 260 261 262 263 264 265 266 267 268 269 270 271 272 273 274 275 276 277 278 279 280 281 282 283 284 285 286 287 288 289 290 291 292 293 294 295 296 297 298 299 300 301 302 303 304 305 306 307 308 309 310 311 312 313 314 315 316 317 318 319 320 321 322 323 324 325 326 327 328 329 330 331 332 333 334 335 336 337 338 339 340 341 342 343 344 345 346 347 348 349 350 351 352 353 354 355 356 357 358 359 360 361 362 363 364 365 366 367 368 369 370 371 372 373 374 375 376 377 378 379 380 381 382 383 384 385 386 387 388 389 390 391 392 393 394 395 396 397 398 399 400 401 402 403 404 405 406 407 408 409 410 411 412 413 414 415 416 417 418 419 420 421 422 423 424 425 426 427 428 429 430 431 432 433 434 435 436 437 438 439 440 441 442 443 444 445 446 447 448 449 450 451 452 453 454 455 456 457 458 459 460 461 462 463 464 465 466 467 468 469 470 471 472 473 474 475 476 477 478 479 480 481 482 483 484 485 486 487 488 489 490 491 492 493 494 495 496 497 498 499 500 501 502 503 504 505 506 507 508 509 510 511 512 513 514 515 516 517 518 519 520 521 522 523 524 525 526 527 528 529 530 531 532 533 534 535 536 537 538 539 540 541 542 543 544 545 546 547 548 549 550 551 552 553 | /* $NetBSD: uvm_pager.c,v 1.130 2020/10/18 18:22:29 chs Exp $ */ /* * Copyright (c) 1997 Charles D. Cranor and Washington University. * All rights reserved. * * Redistribution and use in source and binary forms, with or without * modification, are permitted provided that the following conditions * are met: * 1. Redistributions of source code must retain the above copyright * notice, this list of conditions and the following disclaimer. * 2. Redistributions in binary form must reproduce the above copyright * notice, this list of conditions and the following disclaimer in the * documentation and/or other materials provided with the distribution. * * THIS SOFTWARE IS PROVIDED BY THE AUTHOR ``AS IS'' AND ANY EXPRESS OR * IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES * OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. * IN NO EVENT SHALL THE AUTHOR BE LIABLE FOR ANY DIRECT, INDIRECT, * INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT * NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, * DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY * THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT * (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF * THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. * * from: Id: uvm_pager.c,v 1.1.2.23 1998/02/02 20:38:06 chuck Exp */ /* * uvm_pager.c: generic functions used to assist the pagers. */ #include <sys/cdefs.h> __KERNEL_RCSID(0, "$NetBSD: uvm_pager.c,v 1.130 2020/10/18 18:22:29 chs Exp $"); #include "opt_uvmhist.h" #include "opt_readahead.h" #include "opt_pagermap.h" #include <sys/param.h> #include <sys/systm.h> #include <sys/atomic.h> #include <sys/vnode.h> #include <sys/buf.h> #include <uvm/uvm.h> /* * XXX * this is needed until the device strategy interface * is changed to do physically-addressed i/o. */ #ifndef PAGER_MAP_DEFAULT_SIZE #define PAGER_MAP_DEFAULT_SIZE (16 * 1024 * 1024) #endif #ifndef PAGER_MAP_SIZE #define PAGER_MAP_SIZE PAGER_MAP_DEFAULT_SIZE #endif size_t pager_map_size = PAGER_MAP_SIZE; /* * list of uvm pagers in the system */ const struct uvm_pagerops * const uvmpagerops[] = { &aobj_pager, &uvm_deviceops, &uvm_vnodeops, &ubc_pager, }; /* * the pager map: provides KVA for I/O */ struct vm_map *pager_map; /* XXX */ kmutex_t pager_map_wanted_lock __cacheline_aligned; bool pager_map_wanted; /* locked by pager map */ static vaddr_t emergva; static int emerg_ncolors; static bool emerginuse; void uvm_pager_realloc_emerg(void) { vaddr_t new_emergva, old_emergva; int old_emerg_ncolors; if (__predict_true(emergva != 0 && emerg_ncolors >= uvmexp.ncolors)) return; KASSERT(!emerginuse); new_emergva = uvm_km_alloc(kernel_map, round_page(MAXPHYS) + ptoa(uvmexp.ncolors), ptoa(uvmexp.ncolors), UVM_KMF_VAONLY); KASSERT(new_emergva != 0); old_emergva = emergva; old_emerg_ncolors = emerg_ncolors; /* * don't support re-color in late boot anyway. */ if (0) /* XXX */ mutex_enter(&pager_map_wanted_lock); emergva = new_emergva; emerg_ncolors = uvmexp.ncolors; wakeup(&old_emergva); if (0) /* XXX */ mutex_exit(&pager_map_wanted_lock); if (old_emergva) uvm_km_free(kernel_map, old_emergva, round_page(MAXPHYS) + ptoa(old_emerg_ncolors), UVM_KMF_VAONLY); } /* * uvm_pager_init: init pagers (at boot time) */ void uvm_pager_init(void) { u_int lcv; vaddr_t sva, eva; /* * init pager map */ sva = 0; pager_map = uvm_km_suballoc(kernel_map, &sva, &eva, pager_map_size, 0, false, NULL); mutex_init(&pager_map_wanted_lock, MUTEX_DEFAULT, IPL_NONE); pager_map_wanted = false; uvm_pager_realloc_emerg(); /* * call pager init functions */ for (lcv = 0 ; lcv < __arraycount(uvmpagerops); lcv++) { if (uvmpagerops[lcv]->pgo_init) uvmpagerops[lcv]->pgo_init(); } } #ifdef PMAP_DIRECT /* * uvm_pagermapdirect: map a single page via the pmap's direct segment * * this is an abuse of pmap_direct_process(), since the kva is being grabbed * and no processing is taking place, but for now.. */ static int uvm_pagermapdirect(void *kva, size_t sz, void *cookie) { KASSERT(sz == PAGE_SIZE); *(vaddr_t *)cookie = (vaddr_t)kva; return 0; } #endif /* * uvm_pagermapin: map pages into KVA (pager_map) for I/O that needs mappings * * we basically just map in a blank map entry to reserve the space in the * map and then use pmap_enter() to put the mappings in by hand. */ vaddr_t uvm_pagermapin(struct vm_page **pps, int npages, int flags) { vsize_t size; vaddr_t kva; vaddr_t cva; struct vm_page *pp; vm_prot_t prot; const bool pdaemon = (curlwp == uvm.pagedaemon_lwp); const u_int first_color = VM_PGCOLOR(*pps); UVMHIST_FUNC(__func__); UVMHIST_CALLARGS(maphist,"(pps=%#jx, npages=%jd, first_color=%ju)", (uintptr_t)pps, npages, first_color, 0); #ifdef PMAP_DIRECT /* * for a single page the direct mapped segment can be used. */ if (npages == 1) { int error __diagused; KASSERT((pps[0]->flags & PG_BUSY) != 0); error = pmap_direct_process(VM_PAGE_TO_PHYS(pps[0]), 0, PAGE_SIZE, uvm_pagermapdirect, &kva); KASSERT(error == 0); UVMHIST_LOG(maphist, "<- done, direct (KVA=%#jx)", kva,0,0,0); return kva; } #endif /* * compute protection. outgoing I/O only needs read * access to the page, whereas incoming needs read/write. */ prot = VM_PROT_READ; if (flags & UVMPAGER_MAPIN_READ) prot |= VM_PROT_WRITE; ReStart: size = ptoa(npages); kva = 0; /* let system choose VA */ if (uvm_map(pager_map, &kva, size, NULL, UVM_UNKNOWN_OFFSET, first_color, UVM_FLAG_COLORMATCH | UVM_FLAG_NOMERGE | (pdaemon ? UVM_FLAG_NOWAIT : 0)) != 0) { if (pdaemon) { mutex_enter(&pager_map_wanted_lock); if (emerginuse) { UVM_UNLOCK_AND_WAIT(&emergva, &pager_map_wanted_lock, false, "emergva", 0); goto ReStart; } emerginuse = true; mutex_exit(&pager_map_wanted_lock); kva = emergva + ptoa(first_color); /* The shift implicitly truncates to PAGE_SIZE */ KASSERT(npages <= (MAXPHYS >> PAGE_SHIFT)); goto enter; } if ((flags & UVMPAGER_MAPIN_WAITOK) == 0) { UVMHIST_LOG(maphist,"<- NOWAIT failed", 0,0,0,0); return(0); } mutex_enter(&pager_map_wanted_lock); pager_map_wanted = true; UVMHIST_LOG(maphist, " SLEEPING on pager_map",0,0,0,0); UVM_UNLOCK_AND_WAIT(pager_map, &pager_map_wanted_lock, false, "pager_map", 0); goto ReStart; } enter: /* got it */ for (cva = kva; npages != 0; npages--, cva += PAGE_SIZE) { pp = *pps++; KASSERT(pp); // KASSERT(!((VM_PAGE_TO_PHYS(pp) ^ cva) & uvmexp.colormask)); KASSERT(pp->flags & PG_BUSY); pmap_kenter_pa(cva, VM_PAGE_TO_PHYS(pp), prot, 0); } pmap_update(vm_map_pmap(pager_map)); UVMHIST_LOG(maphist, "<- done (KVA=%#jx)", kva,0,0,0); return(kva); } /* * uvm_pagermapout: remove pager_map mapping * * we remove our mappings by hand and then remove the mapping (waking * up anyone wanting space). */ void uvm_pagermapout(vaddr_t kva, int npages) { vsize_t size = ptoa(npages); struct vm_map_entry *entries; UVMHIST_FUNC(__func__); UVMHIST_CALLARGS(maphist, " (kva=%#jx, npages=%jd)", kva, npages,0,0); #ifdef PMAP_DIRECT /* * solitary pages are mapped directly. */ if (npages == 1) { UVMHIST_LOG(maphist,"<- done, direct", 0,0,0,0); return; } #endif /* * duplicate uvm_unmap, but add in pager_map_wanted handling. */ pmap_kremove(kva, size); pmap_update(pmap_kernel()); if ((kva & ~ptoa(uvmexp.colormask)) == emergva) { mutex_enter(&pager_map_wanted_lock); KASSERT(emerginuse); emerginuse = false; wakeup(&emergva); mutex_exit(&pager_map_wanted_lock); return; } vm_map_lock(pager_map); uvm_unmap_remove(pager_map, kva, kva + size, &entries, 0); mutex_enter(&pager_map_wanted_lock); if (pager_map_wanted) { pager_map_wanted = false; wakeup(pager_map); } mutex_exit(&pager_map_wanted_lock); vm_map_unlock(pager_map); if (entries) uvm_unmap_detach(entries, 0); UVMHIST_LOG(maphist,"<- done",0,0,0,0); } void uvm_aio_aiodone_pages(struct vm_page **pgs, int npages, bool write, int error) { struct uvm_object *uobj; struct vm_page *pg; krwlock_t *slock; int pageout_done; /* number of PG_PAGEOUT pages processed */ int swslot; int i; bool swap; UVMHIST_FUNC(__func__); UVMHIST_CALLED(ubchist); swslot = 0; pageout_done = 0; slock = NULL; uobj = NULL; pg = pgs[0]; swap = (pg->uanon != NULL && pg->uobject == NULL) || (pg->flags & PG_AOBJ) != 0; if (!swap) { uobj = pg->uobject; slock = uobj->vmobjlock; rw_enter(slock, RW_WRITER); } else { #if defined(VMSWAP) if (error) { if (pg->uobject != NULL) { swslot = uao_find_swslot(pg->uobject, pg->offset >> PAGE_SHIFT); } else { KASSERT(pg->uanon != NULL); swslot = pg->uanon->an_swslot; } KASSERT(swslot); } #else /* defined(VMSWAP) */ panic("%s: swap", __func__); #endif /* defined(VMSWAP) */ } for (i = 0; i < npages; i++) { #if defined(VMSWAP) bool anon_disposed = false; /* XXX gcc */ #endif /* defined(VMSWAP) */ pg = pgs[i]; KASSERT(swap || pg->uobject == uobj); UVMHIST_LOG(ubchist, "pg %#jx", (uintptr_t)pg, 0,0,0); #if defined(VMSWAP) /* * for swap i/os, lock each page's object (or anon) * individually since each page may need a different lock. */ if (swap) { if (pg->uobject != NULL) { slock = pg->uobject->vmobjlock; } else { slock = pg->uanon->an_lock; } rw_enter(slock, RW_WRITER); anon_disposed = (pg->flags & PG_RELEASED) != 0; KASSERT(!anon_disposed || pg->uobject != NULL || pg->uanon->an_ref == 0); } #endif /* defined(VMSWAP) */ if (write && uobj != NULL) { KASSERT(uvm_obj_page_writeback_p(pg)); uvm_obj_page_clear_writeback(pg); } /* * process errors. for reads, just mark the page to be freed. * for writes, if the error was ENOMEM, we assume this was * a transient failure so we mark the page dirty so that * we'll try to write it again later. for all other write * errors, we assume the error is permanent, thus the data * in the page is lost. bummer. */ if (error) { int slot; if (!write) { pg->flags |= PG_RELEASED; continue; } else if (error == ENOMEM) { if (pg->flags & PG_PAGEOUT) { pg->flags &= ~PG_PAGEOUT; pageout_done++; } uvm_pagemarkdirty(pg, UVM_PAGE_STATUS_DIRTY); uvm_pagelock(pg); uvm_pageactivate(pg); uvm_pageunlock(pg); slot = 0; } else slot = SWSLOT_BAD; #if defined(VMSWAP) if (swap) { if (pg->uobject != NULL) { int oldslot __diagused; oldslot = uao_set_swslot(pg->uobject, pg->offset >> PAGE_SHIFT, slot); KASSERT(oldslot == swslot + i); } else { KASSERT(pg->uanon->an_swslot == swslot + i); pg->uanon->an_swslot = slot; } } #endif /* defined(VMSWAP) */ } /* * if the page is PG_FAKE, this must have been a read to * initialize the page. clear PG_FAKE and activate the page. */ if (pg->flags & PG_FAKE) { KASSERT(!write); pg->flags &= ~PG_FAKE; #if defined(READAHEAD_STATS) pg->flags |= PG_READAHEAD; uvm_ra_total.ev_count++; #endif /* defined(READAHEAD_STATS) */ KASSERT(uvm_pagegetdirty(pg) == UVM_PAGE_STATUS_CLEAN); uvm_pagelock(pg); uvm_pageenqueue(pg); uvm_pageunlock(pg); } #if defined(VMSWAP) /* * for swap pages, unlock everything for this page now. */ if (swap) { if (pg->uobject == NULL && anon_disposed) { uvm_anon_release(pg->uanon); } else { uvm_page_unbusy(&pg, 1); rw_exit(slock); } } #endif /* defined(VMSWAP) */ } if (pageout_done != 0) { uvm_pageout_done(pageout_done); } if (!swap) { uvm_page_unbusy(pgs, npages); rw_exit(slock); } else { #if defined(VMSWAP) KASSERT(write); /* these pages are now only in swap. */ if (error != ENOMEM) { atomic_add_int(&uvmexp.swpgonly, npages); } if (error) { if (error != ENOMEM) uvm_swap_markbad(swslot, npages); else uvm_swap_free(swslot, npages); } atomic_dec_uint(&uvmexp.pdpending); #endif /* defined(VMSWAP) */ } } /* * uvm_aio_aiodone: do iodone processing for async i/os. * this should be called in thread context, not interrupt context. */ void uvm_aio_aiodone(struct buf *bp) { const int npages = bp->b_bufsize >> PAGE_SHIFT; struct vm_page *pgs[howmany(MAXPHYS, MIN_PAGE_SIZE)]; int i, error; bool write; UVMHIST_FUNC(__func__); UVMHIST_CALLARGS(ubchist, "bp %#jx", (uintptr_t)bp, 0,0,0); KASSERT(bp->b_bufsize <= MAXPHYS); KASSERT(npages <= __arraycount(pgs)); error = bp->b_error; write = (bp->b_flags & B_READ) == 0; for (i = 0; i < npages; i++) { pgs[i] = uvm_pageratop((vaddr_t)bp->b_data + (i << PAGE_SHIFT)); UVMHIST_LOG(ubchist, "pgs[%jd] = %#jx", i, (uintptr_t)pgs[i], 0, 0); } uvm_pagermapout((vaddr_t)bp->b_data, npages); uvm_aio_aiodone_pages(pgs, npages, write, error); if (write && (bp->b_cflags & BC_AGE) != 0) { mutex_enter(bp->b_objlock); vwakeup(bp); mutex_exit(bp->b_objlock); } putiobuf(bp); } /* * uvm_pageratop: convert KVAs in the pager map back to their page * structures. */ struct vm_page * uvm_pageratop(vaddr_t kva) { struct vm_page *pg; paddr_t pa; bool rv __diagused; rv = pmap_extract(pmap_kernel(), kva, &pa); KASSERT(rv); pg = PHYS_TO_VM_PAGE(pa); KASSERT(pg != NULL); return (pg); } |
| 18 5 5 2 5 5 2 18 7 5 5 16 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 197 198 199 200 201 202 203 204 205 206 207 208 209 210 211 212 213 214 215 216 217 218 219 220 221 222 223 224 225 226 227 228 229 230 231 232 233 234 235 236 237 238 239 240 241 242 243 244 245 246 247 248 249 250 251 252 253 254 255 256 257 258 259 260 261 262 263 264 265 266 267 268 269 270 271 272 273 274 275 276 277 278 279 280 281 282 283 284 285 286 287 288 289 290 291 292 293 294 295 296 297 298 299 300 301 302 303 304 305 306 307 308 309 310 311 312 313 314 315 316 317 318 319 320 321 322 323 324 325 326 327 328 329 330 331 332 333 334 335 336 337 338 339 340 341 342 343 344 345 346 347 348 349 350 351 352 353 354 355 356 357 358 359 360 361 362 363 364 365 366 367 368 369 370 371 372 373 374 375 376 377 378 379 380 381 382 383 384 385 386 387 388 389 390 391 392 393 394 395 396 397 398 399 400 401 402 403 404 405 406 407 408 409 410 411 412 413 414 415 416 417 418 419 420 421 422 423 424 425 426 427 428 429 430 431 432 433 434 435 436 437 438 439 440 441 442 443 444 445 446 447 448 449 450 451 452 453 454 455 456 457 458 459 460 461 462 463 464 465 466 467 468 469 470 471 472 473 474 475 476 477 478 479 480 481 482 483 484 485 486 487 488 489 490 491 492 493 494 495 496 497 498 499 500 501 502 503 504 505 506 507 508 509 510 511 512 513 514 515 516 517 518 519 520 521 522 523 524 525 526 527 528 529 530 531 532 533 534 535 536 537 538 539 540 541 542 543 544 545 546 547 548 549 550 551 552 553 554 555 556 557 558 559 560 561 562 563 564 565 566 567 568 569 570 571 572 573 574 575 576 577 578 579 580 581 582 583 584 585 586 587 588 589 590 591 592 593 594 595 596 597 598 599 600 601 602 603 604 605 606 607 608 609 610 611 612 613 614 615 616 617 618 619 620 621 622 623 624 625 626 627 628 629 630 631 632 633 634 635 636 637 638 639 640 641 642 643 644 645 646 647 648 649 650 651 652 653 654 655 656 657 658 659 660 661 662 663 664 665 666 667 668 669 670 671 672 673 674 675 676 677 678 679 680 681 682 683 684 685 686 687 688 689 690 691 692 693 694 695 696 697 698 699 700 701 702 703 704 | /* $NetBSD: pckbc.c,v 1.62 2020/05/01 01:34:57 riastradh Exp $ */ /* * Copyright (c) 2004 Ben Harris. * Copyright (c) 1998 * Matthias Drochner. All rights reserved. * * Redistribution and use in source and binary forms, with or without * modification, are permitted provided that the following conditions * are met: * 1. Redistributions of source code must retain the above copyright * notice, this list of conditions and the following disclaimer. * 2. Redistributions in binary form must reproduce the above copyright * notice, this list of conditions and the following disclaimer in the * documentation and/or other materials provided with the distribution. * * THIS SOFTWARE IS PROVIDED BY THE AUTHOR ``AS IS'' AND ANY EXPRESS OR * IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES * OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. * IN NO EVENT SHALL THE AUTHOR BE LIABLE FOR ANY DIRECT, INDIRECT, * INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT * NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, * DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY * THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT * (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF * THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. */ #include <sys/cdefs.h> __KERNEL_RCSID(0, "$NetBSD: pckbc.c,v 1.62 2020/05/01 01:34:57 riastradh Exp $"); #include <sys/param.h> #include <sys/systm.h> #include <sys/callout.h> #include <sys/kernel.h> #include <sys/proc.h> #include <sys/device.h> #include <sys/malloc.h> #include <sys/errno.h> #include <sys/queue.h> #include <sys/bus.h> #include <dev/ic/i8042reg.h> #include <dev/ic/pckbcvar.h> #include <dev/pckbport/pckbportvar.h> #include "locators.h" #include <sys/rndsource.h> /* data per slave device */ struct pckbc_slotdata { int polling; /* don't process data in interrupt handler */ int poll_data; /* data read from inr handler if polling */ int poll_stat; /* status read from inr handler if polling */ krndsource_t rnd_source; }; static void pckbc_init_slotdata(struct pckbc_slotdata *); static int pckbc_attach_slot(struct pckbc_softc *, pckbc_slot_t); struct pckbc_internal pckbc_consdata; int pckbc_console_attached; static int pckbc_console; static struct pckbc_slotdata pckbc_cons_slotdata; static int pckbc_xt_translation(void *, pckbport_slot_t, int); static int pckbc_send_devcmd(void *, pckbport_slot_t, u_char); static void pckbc_slot_enable(void *, pckbport_slot_t, int); static void pckbc_intr_establish(void *, pckbport_slot_t); static void pckbc_set_poll(void *, pckbc_slot_t, int on); static int pckbc_wait_output(bus_space_tag_t, bus_space_handle_t); static int pckbc_get8042cmd(struct pckbc_internal *); static int pckbc_put8042cmd(struct pckbc_internal *); void pckbc_cleanqueue(struct pckbc_slotdata *); void pckbc_cleanup(void *); int pckbc_cmdresponse(struct pckbc_internal *, pckbc_slot_t, u_char); void pckbc_start(struct pckbc_internal *, pckbc_slot_t); const char * const pckbc_slot_names[] = { "kbd", "aux" }; static struct pckbport_accessops const pckbc_ops = { pckbc_xt_translation, pckbc_send_devcmd, pckbc_poll_data1, pckbc_slot_enable, pckbc_intr_establish, pckbc_set_poll }; #define KBD_DELAY DELAY(8) static inline int pckbc_wait_output(bus_space_tag_t iot, bus_space_handle_t ioh_c) { u_int i; for (i = 100000; i; i--) if (!(bus_space_read_1(iot, ioh_c, 0) & KBS_IBF)) { KBD_DELAY; return (1); } return (0); } int pckbc_send_cmd(bus_space_tag_t iot, bus_space_handle_t ioh_c, u_char val) { if (!pckbc_wait_output(iot, ioh_c)) return (0); bus_space_write_1(iot, ioh_c, 0, val); return (1); } /* * Note: the spl games here are to deal with some strange PC kbd controllers * in some system configurations. * This is not canonical way to handle polling input. */ int pckbc_poll_data1(void *pt, pckbc_slot_t slot) { struct pckbc_internal *t = pt; struct pckbc_slotdata *q = t->t_slotdata[slot]; int s; u_char stat, c; int i = 100; /* polls for ~100ms */ int checkaux = t->t_haveaux; s = splhigh(); if (q && q->polling && q->poll_data != -1 && q->poll_stat != -1) { stat = q->poll_stat; c = q->poll_data; q->poll_data = -1; q->poll_stat = -1; goto process; } for (; i; i--, delay(1000)) { stat = bus_space_read_1(t->t_iot, t->t_ioh_c, 0); if (stat & KBS_DIB) { KBD_DELAY; c = bus_space_read_1(t->t_iot, t->t_ioh_d, 0); process: if (checkaux && (stat & 0x20)) { /* aux data */ if (slot != PCKBC_AUX_SLOT) { #ifdef PCKBCDEBUG printf("pckbc: lost aux 0x%x\n", c); #endif continue; } } else { if (slot == PCKBC_AUX_SLOT) { #ifdef PCKBCDEBUG printf("pckbc: lost kbd 0x%x\n", c); #endif continue; } } splx(s); return (c); } } splx(s); return (-1); } /* * Get the current command byte. */ static int pckbc_get8042cmd(struct pckbc_internal *t) { bus_space_tag_t iot = t->t_iot; bus_space_handle_t ioh_c = t->t_ioh_c; int data; if (!pckbc_send_cmd(iot, ioh_c, K_RDCMDBYTE)) return (0); data = pckbc_poll_data1(t, PCKBC_KBD_SLOT); if (data == -1) return (0); t->t_cmdbyte = data; return (1); } /* * Pass command byte to keyboard controller (8042). */ static int pckbc_put8042cmd(struct pckbc_internal *t) { bus_space_tag_t iot = t->t_iot; bus_space_handle_t ioh_d = t->t_ioh_d; bus_space_handle_t ioh_c = t->t_ioh_c; if (!pckbc_send_cmd(iot, ioh_c, K_LDCMDBYTE)) return (0); if (!pckbc_wait_output(iot, ioh_c)) return (0); bus_space_write_1(iot, ioh_d, 0, t->t_cmdbyte); return (1); } static int pckbc_send_devcmd(void *pt, pckbc_slot_t slot, u_char val) { struct pckbc_internal *t = pt; bus_space_tag_t iot = t->t_iot; bus_space_handle_t ioh_d = t->t_ioh_d; bus_space_handle_t ioh_c = t->t_ioh_c; if (slot == PCKBC_AUX_SLOT) { if (!pckbc_send_cmd(iot, ioh_c, KBC_AUXWRITE)) return (0); } if (!pckbc_wait_output(iot, ioh_c)) return (0); bus_space_write_1(iot, ioh_d, 0, val); return (1); } int pckbc_is_console(bus_space_tag_t iot, bus_addr_t addr) { if (pckbc_console && !pckbc_console_attached && bus_space_is_equal(pckbc_consdata.t_iot, iot) && pckbc_consdata.t_addr == addr) return (1); return (0); } static int pckbc_attach_slot(struct pckbc_softc *sc, pckbc_slot_t slot) { struct pckbc_internal *t = sc->id; void *sdata; device_t child; int alloced = 0; if (t->t_slotdata[slot] == NULL) { sdata = malloc(sizeof(struct pckbc_slotdata), M_DEVBUF, M_WAITOK); t->t_slotdata[slot] = sdata; pckbc_init_slotdata(t->t_slotdata[slot]); alloced++; } child = pckbport_attach_slot(sc->sc_dv, t->t_pt, slot); if (child == NULL && alloced) { free(t->t_slotdata[slot], M_DEVBUF); t->t_slotdata[slot] = NULL; } if (child != NULL && t->t_slotdata[slot] != NULL) { memset(&t->t_slotdata[slot]->rnd_source, 0, sizeof(t->t_slotdata[slot]->rnd_source)); rnd_attach_source(&t->t_slotdata[slot]->rnd_source, device_xname(child), RND_TYPE_TTY, RND_FLAG_DEFAULT); } return child != NULL; } void pckbc_attach(struct pckbc_softc *sc) { struct pckbc_internal *t; bus_space_tag_t iot; bus_space_handle_t ioh_d, ioh_c; int res; u_char cmdbits = 0; t = sc->id; iot = t->t_iot; ioh_d = t->t_ioh_d; ioh_c = t->t_ioh_c; t->t_pt = pckbport_attach(t, &pckbc_ops); if (t->t_pt == NULL) { aprint_error(": attach failed\n"); return; } /* flush */ (void) pckbc_poll_data1(t, PCKBC_KBD_SLOT); /* set initial cmd byte */ if (!pckbc_put8042cmd(t)) { aprint_error("pckbc: cmd word write error\n"); return; } /* * XXX Don't check the keyboard port. There are broken keyboard controllers * which don't pass the test but work normally otherwise. */ #if 0 /* * check kbd port ok */ if (!pckbc_send_cmd(iot, ioh_c, KBC_KBDTEST)) return; res = pckbc_poll_data1(t, PCKBC_KBD_SLOT, 0); /* * Normally, we should get a "0" here. * But there are keyboard controllers behaving differently. */ if (res == 0 || res == 0xfa || res == 0x01 || res == 0xab) { #ifdef PCKBCDEBUG if (res != 0) printf("pckbc: returned %x on kbd slot test\n", res); #endif if (pckbc_attach_slot(sc, PCKBC_KBD_SLOT)) cmdbits |= KC8_KENABLE; } else { printf("pckbc: kbd port test: %x\n", res); return; } #else if (pckbc_attach_slot(sc, PCKBC_KBD_SLOT)) cmdbits |= KC8_KENABLE; #endif /* 0 */ /* * Check aux port ok. * Avoid KBC_AUXTEST because it hangs some older controllers * (eg UMC880?). */ if (!pckbc_send_cmd(iot, ioh_c, KBC_AUXECHO)) { aprint_error("pckbc: aux echo error 1\n"); goto nomouse; } if (!pckbc_wait_output(iot, ioh_c)) { aprint_error("pckbc: aux echo error 2\n"); goto nomouse; } t->t_haveaux = 1; bus_space_write_1(iot, ioh_d, 0, 0x5a); /* a random value */ res = pckbc_poll_data1(t, PCKBC_AUX_SLOT); /* * The following is needed to find the aux port on the Tadpole * SPARCle. */ if (res == -1 && ISSET(t->t_flags, PCKBC_NEED_AUXWRITE)) { /* Read of aux echo timed out, try again */ if (!pckbc_send_cmd(iot, ioh_c, KBC_AUXWRITE)) goto nomouse; if (!pckbc_wait_output(iot, ioh_c)) goto nomouse; bus_space_write_1(iot, ioh_d, 0, 0x5a); res = pckbc_poll_data1(t, PCKBC_AUX_SLOT); } if (res != -1) { /* * In most cases, the 0x5a gets echoed. * Some older controllers (Gateway 2000 circa 1993) * return 0xfe here. * We are satisfied if there is anything in the * aux output buffer. */ if (pckbc_attach_slot(sc, PCKBC_AUX_SLOT)) cmdbits |= KC8_MENABLE; } else { #ifdef PCKBCDEBUG printf("pckbc: aux echo test failed\n"); #endif t->t_haveaux = 0; } nomouse: /* enable needed interrupts */ t->t_cmdbyte |= cmdbits; if (!pckbc_put8042cmd(t)) aprint_error("pckbc: cmd word write error\n"); } static void pckbc_init_slotdata(struct pckbc_slotdata *q) { q->polling = 0; } /* * switch scancode translation on / off * return nonzero on success */ static int pckbc_xt_translation(void *self, pckbc_slot_t slot, int on) { struct pckbc_internal *t = self; int ison; if (ISSET(t->t_flags, PCKBC_CANT_TRANSLATE)) return (-1); if (slot != PCKBC_KBD_SLOT) { /* translation only for kbd slot */ if (on) return (0); else return (1); } ison = t->t_cmdbyte & KC8_TRANS; if ((on && ison) || (!on && !ison)) return (1); t->t_cmdbyte ^= KC8_TRANS; if (!pckbc_put8042cmd(t)) return (0); /* read back to be sure */ if (!pckbc_get8042cmd(t)) return (0); ison = t->t_cmdbyte & KC8_TRANS; if ((on && ison) || (!on && !ison)) return (1); return (0); } static const struct pckbc_portcmd { u_char cmd_en, cmd_dis; } pckbc_portcmd[2] = { { KBC_KBDENABLE, KBC_KBDDISABLE, }, { KBC_AUXENABLE, KBC_AUXDISABLE, } }; void pckbc_slot_enable(void *self, pckbc_slot_t slot, int on) { struct pckbc_internal *t = (struct pckbc_internal *)self; const struct pckbc_portcmd *cmd; cmd = &pckbc_portcmd[slot]; if (!pckbc_send_cmd(t->t_iot, t->t_ioh_c, on ? cmd->cmd_en : cmd->cmd_dis)) printf("pckbc: pckbc_slot_enable(%d) failed\n", on); } static void pckbc_set_poll(void *self, pckbc_slot_t slot, int on) { struct pckbc_internal *t = (struct pckbc_internal *)self; t->t_slotdata[slot]->polling = on; if (on) { t->t_slotdata[slot]->poll_data = -1; t->t_slotdata[slot]->poll_stat = -1; } else { int s; /* * If disabling polling on a device that's been configured, * make sure there are no bytes left in the FIFO, holding up * the interrupt line. Otherwise we won't get any further * interrupts. */ if (t->t_sc) { s = spltty(); pckbcintr(t->t_sc); splx(s); } } } static void pckbc_intr_establish(void *pt, pckbport_slot_t slot) { struct pckbc_internal *t = pt; (*t->t_sc->intr_establish)(t->t_sc, slot); } int pckbcintr_hard(void *vsc) { struct pckbc_softc *sc = (struct pckbc_softc *)vsc; struct pckbc_internal *t = sc->id; u_char stat; pckbc_slot_t slot; struct pckbc_slotdata *q; int served = 0, data, next, s; for(;;) { stat = bus_space_read_1(t->t_iot, t->t_ioh_c, 0); if (!(stat & KBS_DIB)) break; served = 1; slot = (t->t_haveaux && (stat & 0x20)) ? PCKBC_AUX_SLOT : PCKBC_KBD_SLOT; q = t->t_slotdata[slot]; if (!q) { /* XXX do something for live insertion? */ printf("pckbc: no dev for slot %d\n", slot); KBD_DELAY; (void) bus_space_read_1(t->t_iot, t->t_ioh_d, 0); continue; } KBD_DELAY; data = bus_space_read_1(t->t_iot, t->t_ioh_d, 0); rnd_add_uint32(&q->rnd_source, (stat<<8)|data); if (q->polling) { q->poll_data = data; q->poll_stat = stat; break; /* pckbc_poll_data() will get it */ } #if 0 /* XXXBJH */ if (CMD_IN_QUEUE(q) && pckbc_cmdresponse(t, slot, data)) continue; #endif s = splhigh(); next = (t->rbuf_write+1) % PCKBC_RBUF_SIZE; if (next == t->rbuf_read) { splx(s); break; } t->rbuf[t->rbuf_write].data = data; t->rbuf[t->rbuf_write].slot = slot; t->rbuf_write = next; splx(s); } return (served); } void pckbcintr_soft(void *vsc) { struct pckbc_softc *sc = vsc; struct pckbc_internal *t = sc->id; int data, slot, s; #ifndef __GENERIC_SOFT_INTERRUPTS_ALL_LEVELS int st; st = spltty(); #endif s = splhigh(); while (t->rbuf_read != t->rbuf_write) { slot = t->rbuf[t->rbuf_read].slot; data = t->rbuf[t->rbuf_read].data; t->rbuf_read = (t->rbuf_read+1) % PCKBC_RBUF_SIZE; splx(s); pckbportintr(t->t_pt, slot, data); s = splhigh(); } splx(s); #ifndef __GENERIC_SOFT_INTERRUPTS_ALL_LEVELS splx(st); #endif } int pckbcintr(void *vsc) { struct pckbc_softc *sc = (struct pckbc_softc *)vsc; struct pckbc_internal *t = sc->id; u_char stat; pckbc_slot_t slot; struct pckbc_slotdata *q; int served = 0, data; for(;;) { stat = bus_space_read_1(t->t_iot, t->t_ioh_c, 0); if (!(stat & KBS_DIB)) break; slot = (t->t_haveaux && (stat & 0x20)) ? PCKBC_AUX_SLOT : PCKBC_KBD_SLOT; q = t->t_slotdata[slot]; if (q != NULL && q->polling) return 0; served = 1; KBD_DELAY; data = bus_space_read_1(t->t_iot, t->t_ioh_d, 0); if (q != NULL) rnd_add_uint32(&q->rnd_source, (stat<<8)|data); pckbportintr(t->t_pt, slot, data); } return (served); } int pckbc_cnattach(bus_space_tag_t iot, bus_addr_t addr, bus_size_t cmd_offset, pckbc_slot_t slot, int flags) { bus_space_handle_t ioh_d, ioh_c; #ifdef PCKBC_CNATTACH_SELFTEST int reply; #endif int res = 0; if (bus_space_map(iot, addr + KBDATAP, 1, 0, &ioh_d)) return (ENXIO); if (bus_space_map(iot, addr + cmd_offset, 1, 0, &ioh_c)) { bus_space_unmap(iot, ioh_d, 1); return (ENXIO); } memset(&pckbc_consdata, 0, sizeof(pckbc_consdata)); pckbc_consdata.t_iot = iot; pckbc_consdata.t_ioh_d = ioh_d; pckbc_consdata.t_ioh_c = ioh_c; pckbc_consdata.t_addr = addr; pckbc_consdata.t_flags = flags; callout_init(&pckbc_consdata.t_cleanup, 0); /* flush */ (void) pckbc_poll_data1(&pckbc_consdata, PCKBC_KBD_SLOT); #ifdef PCKBC_CNATTACH_SELFTEST /* * In some machines (e.g. netwinder) pckbc refuses to talk at * all until we request a self-test. */ if (!pckbc_send_cmd(iot, ioh_c, KBC_SELFTEST)) { printf("pckbc: unable to request selftest\n"); res = EIO; goto out; } reply = pckbc_poll_data1(&pckbc_consdata, PCKBC_KBD_SLOT); if (reply != 0x55) { printf("pckbc: selftest returned 0x%02x\n", reply); res = EIO; goto out; } #endif /* PCKBC_CNATTACH_SELFTEST */ /* init cmd byte, enable ports */ pckbc_consdata.t_cmdbyte = KC8_CPU; if (!pckbc_put8042cmd(&pckbc_consdata)) { printf("pckbc: cmd word write error\n"); res = EIO; goto out; } res = pckbport_cnattach(&pckbc_consdata, &pckbc_ops, slot); out: if (res) { bus_space_unmap(iot, pckbc_consdata.t_ioh_d, 1); bus_space_unmap(iot, pckbc_consdata.t_ioh_c, 1); } else { pckbc_consdata.t_slotdata[slot] = &pckbc_cons_slotdata; pckbc_init_slotdata(&pckbc_cons_slotdata); pckbc_console = 1; } return (res); } bool pckbc_resume(device_t dv, const pmf_qual_t *qual) { struct pckbc_softc *sc = device_private(dv); struct pckbc_internal *t; t = sc->id; (void)pckbc_poll_data1(t, PCKBC_KBD_SLOT); if (!pckbc_send_cmd(t->t_iot, t->t_ioh_c, KBC_SELFTEST)) return false; (void)pckbc_poll_data1(t, PCKBC_KBD_SLOT); (void)pckbc_put8042cmd(t); pckbcintr(t->t_sc); return true; } |
| 8 8 8 8 4 8 35 41 1 39 8 8 39 39 7 35 2 1 1 2 1 2 2 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 197 198 199 200 201 202 203 204 205 206 207 208 209 210 211 212 213 214 215 216 217 218 219 220 221 222 223 224 225 226 227 228 229 230 231 232 233 234 235 236 237 238 239 240 241 242 243 244 245 246 247 248 249 250 251 252 253 254 255 256 257 258 259 260 261 262 263 264 265 266 267 268 269 270 271 272 273 274 275 276 277 278 279 280 281 282 283 284 285 286 287 288 289 290 291 292 293 294 295 296 297 298 299 300 301 302 303 304 305 306 307 308 309 310 311 312 313 314 315 316 317 318 319 320 321 322 323 324 325 326 327 328 329 330 331 332 333 334 335 336 337 338 339 340 341 342 343 344 345 346 347 348 349 350 351 352 353 354 355 356 357 358 359 360 361 362 363 364 365 366 367 368 369 370 371 372 373 374 375 376 377 378 379 380 381 382 383 384 385 386 387 388 389 390 391 392 393 394 395 396 397 398 399 400 401 402 403 404 405 406 407 408 409 410 411 412 413 414 415 416 417 418 419 420 421 422 423 424 425 426 427 428 429 430 431 432 433 434 435 436 437 438 439 440 441 442 443 444 445 446 447 448 449 450 451 452 453 454 455 456 457 458 459 460 461 462 463 464 465 466 467 468 469 470 471 472 473 474 475 476 477 478 479 480 481 482 483 484 485 486 487 488 489 490 491 492 493 494 495 496 497 498 499 500 501 502 503 504 505 506 507 508 509 510 511 512 513 514 515 516 517 518 519 520 521 522 523 524 525 526 527 528 529 530 531 532 533 534 535 536 537 538 539 540 541 542 543 544 545 546 547 548 549 550 551 552 553 554 555 556 557 558 559 560 561 562 563 564 565 566 567 568 569 570 571 572 573 574 575 576 577 578 579 580 581 582 583 584 585 586 587 588 589 590 591 592 593 594 595 596 597 598 599 600 601 602 603 604 605 606 607 608 609 610 611 612 613 614 615 616 617 618 619 620 621 622 623 624 625 626 627 628 629 630 631 632 633 634 635 636 637 638 639 640 641 642 643 644 645 646 647 648 649 650 651 652 653 654 655 656 657 658 659 660 661 662 663 664 665 666 667 668 669 670 671 672 673 674 675 676 677 678 679 680 681 682 683 684 685 686 687 688 689 690 691 692 693 694 695 696 697 698 699 700 701 702 703 704 705 706 707 708 709 710 711 712 713 714 715 716 717 718 719 720 721 722 723 724 725 726 727 728 729 730 731 732 733 734 735 736 737 738 739 740 741 742 743 744 745 746 747 748 749 750 751 752 753 754 755 756 757 758 759 760 761 762 763 764 765 766 767 768 769 770 771 772 773 774 775 776 777 778 779 780 781 782 783 784 785 786 787 788 789 790 791 792 793 794 795 796 797 798 799 800 801 802 803 804 805 806 807 808 809 810 811 812 813 814 815 816 817 818 819 820 821 822 823 824 825 826 827 828 829 830 831 832 833 834 835 836 837 838 839 840 841 842 843 844 845 846 847 | /* $NetBSD: usbdi_util.c,v 1.87 2022/04/17 13:16:52 riastradh Exp $ */ /* * Copyright (c) 1998, 2012 The NetBSD Foundation, Inc. * All rights reserved. * * This code is derived from software contributed to The NetBSD Foundation * by Lennart Augustsson (lennart@augustsson.net) at * Carlstedt Research & Technology and Matthew R. Green (mrg@eterna.com.au). * * Redistribution and use in source and binary forms, with or without * modification, are permitted provided that the following conditions * are met: * 1. Redistributions of source code must retain the above copyright * notice, this list of conditions and the following disclaimer. * 2. Redistributions in binary form must reproduce the above copyright * notice, this list of conditions and the following disclaimer in the * documentation and/or other materials provided with the distribution. * * THIS SOFTWARE IS PROVIDED BY THE NETBSD FOUNDATION, INC. AND CONTRIBUTORS * ``AS IS'' AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED * TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR * PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE FOUNDATION OR CONTRIBUTORS * BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR * CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF * SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS * INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN * CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) * ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE * POSSIBILITY OF SUCH DAMAGE. */ #include <sys/cdefs.h> __KERNEL_RCSID(0, "$NetBSD: usbdi_util.c,v 1.87 2022/04/17 13:16:52 riastradh Exp $"); #ifdef _KERNEL_OPT #include "opt_usb.h" #endif #include <sys/param.h> #include <sys/systm.h> #include <sys/kernel.h> #include <sys/kmem.h> #include <sys/proc.h> #include <sys/device.h> #include <sys/bus.h> #include <dev/usb/usb.h> #include <dev/usb/usbhid.h> #include <dev/usb/usbdi.h> #include <dev/usb/usbdivar.h> #include <dev/usb/usbdi_util.h> #include <dev/usb/usb_quirks.h> #include <dev/usb/usbhist.h> #define DPRINTF(FMT,A,B,C,D) USBHIST_LOGN(usbdebug,1,FMT,A,B,C,D) #define DPRINTFN(N,FMT,A,B,C,D) USBHIST_LOGN(usbdebug,N,FMT,A,B,C,D) usbd_status usbd_get_desc(struct usbd_device *dev, int type, int index, int len, void *desc) { usb_device_request_t req; usbd_status err; USBHIST_FUNC(); USBHIST_CALLARGS(usbdebug, "type=%jd, index=%jd, len=%jd", type, index, len, 0); /* * Provide hard-coded configuration descriptors * for devices that may corrupt it. This cannot * be done for device descriptors which are used * to identify the device. */ if (type != UDESC_DEVICE && dev->ud_quirks->uq_flags & UQ_DESC_CORRUPT) { err = usbd_get_desc_fake(dev, type, index, len, desc); goto out; } req.bmRequestType = UT_READ_DEVICE; req.bRequest = UR_GET_DESCRIPTOR; USETW2(req.wValue, type, index); USETW(req.wIndex, 0); USETW(req.wLength, len); err = usbd_do_request(dev, &req, desc); out: return err; } usbd_status usbd_get_config_desc(struct usbd_device *dev, int confidx, usb_config_descriptor_t *d) { USBHIST_FUNC(); USBHIST_CALLARGS(usbdebug, "confidx=%jd", confidx, 0, 0, 0); usbd_status err; err = usbd_get_desc(dev, UDESC_CONFIG, confidx, USB_CONFIG_DESCRIPTOR_SIZE, d); if (err) return err; if (d->bDescriptorType != UDESC_CONFIG) { DPRINTFN(1, "confidx=%jd, bad desc len=%jd type=%jd", confidx, d->bLength, d->bDescriptorType, 0); return USBD_INVAL; } return USBD_NORMAL_COMPLETION; } usbd_status usbd_get_config_desc_full(struct usbd_device *dev, int conf, void *d, int size) { USBHIST_FUNC(); USBHIST_CALLARGS(usbdebug, "conf=%jd", conf, 0, 0, 0); return usbd_get_desc(dev, UDESC_CONFIG, conf, size, d); } usbd_status usbd_get_bos_desc(struct usbd_device *dev, int confidx, usb_bos_descriptor_t *d) { USBHIST_FUNC(); USBHIST_CALLARGS(usbdebug, "confidx=%jd", confidx, 0, 0, 0); usbd_status err; err = usbd_get_desc(dev, UDESC_BOS, confidx, USB_BOS_DESCRIPTOR_SIZE, d); if (err) return err; if (d->bDescriptorType != UDESC_BOS) { DPRINTFN(1, "confidx=%jd, bad desc len=%jd type=%jd", confidx, d->bLength, d->bDescriptorType, 0); return USBD_INVAL; } return USBD_NORMAL_COMPLETION; } usbd_status usbd_get_device_desc(struct usbd_device *dev, usb_device_descriptor_t *d) { USBHIST_FUNC(); USBHIST_CALLED(usbdebug); return usbd_get_desc(dev, UDESC_DEVICE, 0, USB_DEVICE_DESCRIPTOR_SIZE, d); } /* * Get the first 8 bytes of the device descriptor. * Do as Windows does: try to read 64 bytes -- there are devices which * recognize the initial descriptor fetch (before the control endpoint's * MaxPacketSize is known by the host) by exactly this length. */ usbd_status usbd_get_initial_ddesc(struct usbd_device *dev, usb_device_descriptor_t *desc) { USBHIST_FUNC(); USBHIST_CALLARGS(usbdebug, "dev %#jx", (uintptr_t)dev, 0, 0, 0); usb_device_request_t req; char buf[64]; int res, actlen; req.bmRequestType = UT_READ_DEVICE; req.bRequest = UR_GET_DESCRIPTOR; USETW2(req.wValue, UDESC_DEVICE, 0); USETW(req.wIndex, 0); USETW(req.wLength, 8); res = usbd_do_request_flags(dev, &req, buf, USBD_SHORT_XFER_OK, &actlen, USBD_DEFAULT_TIMEOUT); if (res) return res; if (actlen < 8) return USBD_SHORT_XFER; memcpy(desc, buf, 8); return USBD_NORMAL_COMPLETION; } usbd_status usbd_get_string_desc(struct usbd_device *dev, int sindex, int langid, usb_string_descriptor_t *sdesc, int *sizep) { USBHIST_FUNC(); USBHIST_CALLED(usbdebug); usb_device_request_t req; usbd_status err; int actlen; /* * Pass a full-sized buffer to usbd_do_request_len(). At least * one device has been seen returning additional data beyond the * provided buffers (2-bytes written shortly after the request * claims to have completed and returned the 2 byte header, * corrupting other memory.) */ req.bmRequestType = UT_READ_DEVICE; req.bRequest = UR_GET_DESCRIPTOR; USETW2(req.wValue, UDESC_STRING, sindex); USETW(req.wIndex, langid); USETW(req.wLength, 2); /* only size byte first */ err = usbd_do_request_len(dev, &req, sizeof(*sdesc), sdesc, USBD_SHORT_XFER_OK, &actlen, USBD_DEFAULT_TIMEOUT); if (err) return err; if (actlen < 2) return USBD_SHORT_XFER; if (sdesc->bLength > sizeof(*sdesc)) return USBD_INVAL; USETW(req.wLength, sdesc->bLength); /* the whole string */ err = usbd_do_request_len(dev, &req, sizeof(*sdesc), sdesc, USBD_SHORT_XFER_OK, &actlen, USBD_DEFAULT_TIMEOUT); if (err) return err; if (actlen != sdesc->bLength) { DPRINTF("expected %jd, got %jd", sdesc->bLength, actlen, 0, 0); } *sizep = actlen; return USBD_NORMAL_COMPLETION; } /* -------------------------------------------------------------------------- */ usbd_status usbd_get_device_status(struct usbd_device *dev, usb_status_t *st) { USBHIST_FUNC(); USBHIST_CALLED(usbdebug); usb_device_request_t req; req.bmRequestType = UT_READ_DEVICE; req.bRequest = UR_GET_STATUS; USETW(req.wValue, 0); USETW(req.wIndex, 0); USETW(req.wLength, sizeof(usb_status_t)); return usbd_do_request(dev, &req, st); } usbd_status usbd_get_hub_status(struct usbd_device *dev, usb_hub_status_t *st) { USBHIST_FUNC(); USBHIST_CALLARGS(usbdebug, "dev %#jx", (uintptr_t)dev, 0, 0, 0); usb_device_request_t req; req.bmRequestType = UT_READ_CLASS_DEVICE; req.bRequest = UR_GET_STATUS; USETW(req.wValue, 0); USETW(req.wIndex, 0); USETW(req.wLength, sizeof(usb_hub_status_t)); return usbd_do_request(dev, &req, st); } usbd_status usbd_get_port_status(struct usbd_device *dev, int port, usb_port_status_t *ps) { USBHIST_FUNC(); USBHIST_CALLARGS(usbdebug, "dev %#jx port %jd", (uintptr_t)dev, port, 0, 0); usb_device_request_t req; req.bmRequestType = UT_READ_CLASS_OTHER; req.bRequest = UR_GET_STATUS; USETW(req.wValue, 0); USETW(req.wIndex, port); USETW(req.wLength, sizeof(*ps)); return usbd_do_request(dev, &req, ps); } /* USB 3.1 10.16.2.6, 10.16.2.6.3 */ usbd_status usbd_get_port_status_ext(struct usbd_device *dev, int port, usb_port_status_ext_t *pse) { USBHIST_FUNC(); USBHIST_CALLARGS(usbdebug, "dev %#jx port %jd", (uintptr_t)dev, port, 0, 0); usb_device_request_t req; req.bmRequestType = UT_READ_CLASS_OTHER; req.bRequest = UR_GET_STATUS; USETW2(req.wValue, 0, UR_PST_EXT_PORT_STATUS); USETW(req.wIndex, port); USETW(req.wLength, sizeof(*pse)); return usbd_do_request(dev, &req, pse); } /* -------------------------------------------------------------------------- */ usbd_status usbd_clear_hub_feature(struct usbd_device *dev, int sel) { USBHIST_FUNC(); USBHIST_CALLARGS(usbdebug, "dev %#jx sel %jd", (uintptr_t)dev, sel, 0, 0); usb_device_request_t req; req.bmRequestType = UT_WRITE_CLASS_DEVICE; req.bRequest = UR_CLEAR_FEATURE; USETW(req.wValue, sel); USETW(req.wIndex, 0); USETW(req.wLength, 0); return usbd_do_request(dev, &req, 0); } usbd_status usbd_set_hub_feature(struct usbd_device *dev, int sel) { USBHIST_FUNC(); USBHIST_CALLARGS(usbdebug, "dev %#jx sel %jd", (uintptr_t)dev, sel, 0, 0); usb_device_request_t req; req.bmRequestType = UT_WRITE_CLASS_DEVICE; req.bRequest = UR_SET_FEATURE; USETW(req.wValue, sel); USETW(req.wIndex, 0); USETW(req.wLength, 0); return usbd_do_request(dev, &req, 0); } usbd_status usbd_clear_port_feature(struct usbd_device *dev, int port, int sel) { USBHIST_FUNC(); USBHIST_CALLARGS(usbdebug, "dev %#jx port %jd sel %jd", (uintptr_t)dev, port, sel, 0); usb_device_request_t req; req.bmRequestType = UT_WRITE_CLASS_OTHER; req.bRequest = UR_CLEAR_FEATURE; USETW(req.wValue, sel); USETW(req.wIndex, port); USETW(req.wLength, 0); return usbd_do_request(dev, &req, 0); } usbd_status usbd_set_port_feature(struct usbd_device *dev, int port, int sel) { USBHIST_FUNC(); USBHIST_CALLARGS(usbdebug, "dev %#jx port %jd sel %.d", (uintptr_t)dev, sel, 0, 0); usb_device_request_t req; req.bmRequestType = UT_WRITE_CLASS_OTHER; req.bRequest = UR_SET_FEATURE; USETW(req.wValue, sel); USETW(req.wIndex, port); USETW(req.wLength, 0); return usbd_do_request(dev, &req, 0); } usbd_status usbd_set_port_u1_timeout(struct usbd_device *dev, int port, int timeout) { USBHIST_FUNC(); USBHIST_CALLARGS(usbdebug, "dev %#jx port %jd timeout %.d", (uintptr_t)dev, port, timeout, 0); usb_device_request_t req; req.bmRequestType = UT_WRITE_CLASS_OTHER; req.bRequest = UR_SET_FEATURE; USETW(req.wValue, UHF_PORT_U1_TIMEOUT); USETW2(req.wIndex, timeout, port); USETW(req.wLength, 0); return usbd_do_request(dev, &req, 0); } usbd_status usbd_set_port_u2_timeout(struct usbd_device *dev, int port, int timeout) { USBHIST_FUNC(); USBHIST_CALLARGS(usbdebug, "dev %#jx port %jd timeout %jd", (uintptr_t)dev, port, timeout, 0); usb_device_request_t req; req.bmRequestType = UT_WRITE_CLASS_OTHER; req.bRequest = UR_SET_FEATURE; USETW(req.wValue, UHF_PORT_U2_TIMEOUT); USETW2(req.wIndex, timeout, port); USETW(req.wLength, 0); return usbd_do_request(dev, &req, 0); } usbd_status usbd_clear_endpoint_feature(struct usbd_device *dev, int epaddr, int sel) { USBHIST_FUNC(); USBHIST_CALLARGS(usbdebug, "dev %#jx epaddr %jd sel %jd", (uintptr_t)dev, epaddr, sel, 0); usb_device_request_t req; req.bmRequestType = UT_WRITE_ENDPOINT; req.bRequest = UR_CLEAR_FEATURE; USETW(req.wValue, sel); USETW(req.wIndex, epaddr); USETW(req.wLength, 0); return usbd_do_request(dev, &req, 0); } /* -------------------------------------------------------------------------- */ usbd_status usbd_get_config(struct usbd_device *dev, uint8_t *conf) { USBHIST_FUNC(); USBHIST_CALLARGS(usbdebug, "dev %#jx", (uintptr_t)dev, 0, 0, 0); usb_device_request_t req; req.bmRequestType = UT_READ_DEVICE; req.bRequest = UR_GET_CONFIG; USETW(req.wValue, 0); USETW(req.wIndex, 0); USETW(req.wLength, 1); return usbd_do_request(dev, &req, conf); } usbd_status usbd_set_config(struct usbd_device *dev, int conf) { USBHIST_FUNC(); USBHIST_CALLARGS(usbdebug, "dev %#jx conf %jd", (uintptr_t)dev, conf, 0, 0); usb_device_request_t req; req.bmRequestType = UT_WRITE_DEVICE; req.bRequest = UR_SET_CONFIG; USETW(req.wValue, conf); USETW(req.wIndex, 0); USETW(req.wLength, 0); return usbd_do_request(dev, &req, 0); } usbd_status usbd_set_address(struct usbd_device *dev, int addr) { USBHIST_FUNC(); USBHIST_CALLARGS(usbdebug, "dev %#jx addr %jd", (uintptr_t)dev, addr, 0, 0); usb_device_request_t req; req.bmRequestType = UT_WRITE_DEVICE; req.bRequest = UR_SET_ADDRESS; USETW(req.wValue, addr); USETW(req.wIndex, 0); USETW(req.wLength, 0); return usbd_do_request(dev, &req, 0); } usbd_status usbd_set_idle(struct usbd_interface *iface, int duration, int id) { usb_interface_descriptor_t *ifd = usbd_get_interface_descriptor(iface); struct usbd_device *dev; usb_device_request_t req; USBHIST_FUNC(); USBHIST_CALLARGS(usbdebug, "duration %jd id %jd", duration, id, 0, 0); if (ifd == NULL) return USBD_IOERROR; usbd_interface2device_handle(iface, &dev); req.bmRequestType = UT_WRITE_CLASS_INTERFACE; req.bRequest = UR_SET_IDLE; USETW2(req.wValue, duration, id); USETW(req.wIndex, ifd->bInterfaceNumber); USETW(req.wLength, 0); return usbd_do_request(dev, &req, 0); } /* -------------------------------------------------------------------------- */ usbd_status usbd_get_protocol(struct usbd_interface *iface, uint8_t *report) { usb_interface_descriptor_t *id = usbd_get_interface_descriptor(iface); struct usbd_device *dev; usb_device_request_t req; USBHIST_FUNC(); USBHIST_CALLARGS(usbdebug, "iface=%#jx, endpt=%jd", (uintptr_t)iface, id->bInterfaceNumber, 0, 0); if (id == NULL) return USBD_IOERROR; usbd_interface2device_handle(iface, &dev); req.bmRequestType = UT_READ_CLASS_INTERFACE; req.bRequest = UR_GET_PROTOCOL; USETW(req.wValue, 0); USETW(req.wIndex, id->bInterfaceNumber); USETW(req.wLength, 1); return usbd_do_request(dev, &req, report); } usbd_status usbd_set_protocol(struct usbd_interface *iface, int report) { usb_interface_descriptor_t *id = usbd_get_interface_descriptor(iface); struct usbd_device *dev; usb_device_request_t req; USBHIST_FUNC(); USBHIST_CALLARGS(usbdebug, "iface=%#jx, report=%jd, endpt=%jd", (uintptr_t)iface, report, id->bInterfaceNumber, 0); if (id == NULL) return USBD_IOERROR; usbd_interface2device_handle(iface, &dev); req.bmRequestType = UT_WRITE_CLASS_INTERFACE; req.bRequest = UR_SET_PROTOCOL; USETW(req.wValue, report); USETW(req.wIndex, id->bInterfaceNumber); USETW(req.wLength, 0); return usbd_do_request(dev, &req, 0); } /* -------------------------------------------------------------------------- */ usbd_status usbd_set_report(struct usbd_interface *iface, int type, int id, void *data, int len) { usb_interface_descriptor_t *ifd = usbd_get_interface_descriptor(iface); struct usbd_device *dev; usb_device_request_t req; USBHIST_FUNC(); USBHIST_CALLARGS(usbdebug, "len=%jd", len, 0, 0, 0); if (ifd == NULL) return USBD_IOERROR; usbd_interface2device_handle(iface, &dev); req.bmRequestType = UT_WRITE_CLASS_INTERFACE; req.bRequest = UR_SET_REPORT; USETW2(req.wValue, type, id); USETW(req.wIndex, ifd->bInterfaceNumber); USETW(req.wLength, len); return usbd_do_request(dev, &req, data); } usbd_status usbd_get_report(struct usbd_interface *iface, int type, int id, void *data, int len) { usb_interface_descriptor_t *ifd = usbd_get_interface_descriptor(iface); struct usbd_device *dev; usb_device_request_t req; USBHIST_FUNC(); USBHIST_CALLARGS(usbdebug, "len=%jd", len, 0, 0, 0); if (ifd == NULL) return USBD_IOERROR; usbd_interface2device_handle(iface, &dev); req.bmRequestType = UT_READ_CLASS_INTERFACE; req.bRequest = UR_GET_REPORT; USETW2(req.wValue, type, id); USETW(req.wIndex, ifd->bInterfaceNumber); USETW(req.wLength, len); return usbd_do_request(dev, &req, data); } usbd_status usbd_get_report_descriptor(struct usbd_device *dev, int ifcno, int size, void *d) { USBHIST_FUNC(); USBHIST_CALLARGS(usbdebug, "dev %#jx ifcno %jd size %jd", (uintptr_t)dev, ifcno, size, 0); usb_device_request_t req; req.bmRequestType = UT_READ_INTERFACE; req.bRequest = UR_GET_DESCRIPTOR; USETW2(req.wValue, UDESC_REPORT, 0); /* report id should be 0 */ USETW(req.wIndex, ifcno); USETW(req.wLength, size); return usbd_do_request(dev, &req, d); } /* -------------------------------------------------------------------------- */ usb_hid_descriptor_t * usbd_get_hid_descriptor(struct usbd_interface *ifc) { usb_interface_descriptor_t *idesc = usbd_get_interface_descriptor(ifc); struct usbd_device *dev; usb_config_descriptor_t *cdesc; usb_hid_descriptor_t *hd; char *p, *end; if (idesc == NULL) return NULL; usbd_interface2device_handle(ifc, &dev); cdesc = usbd_get_config_descriptor(dev); p = (char *)idesc + idesc->bLength; end = (char *)cdesc + UGETW(cdesc->wTotalLength); for (; end - p >= sizeof(*hd); p += hd->bLength) { hd = (usb_hid_descriptor_t *)p; if (hd->bLength < sizeof(*hd) || hd->bLength > end - p) break; if (hd->bLength >= USB_HID_DESCRIPTOR_SIZE(0) && hd->bDescriptorType == UDESC_HID) return hd; if (hd->bDescriptorType == UDESC_INTERFACE) break; } return NULL; } usbd_status usbd_read_report_desc(struct usbd_interface *ifc, void **descp, int *sizep) { usb_interface_descriptor_t *id; usb_hid_descriptor_t *hid; struct usbd_device *dev; usbd_status err; usbd_interface2device_handle(ifc, &dev); id = usbd_get_interface_descriptor(ifc); if (id == NULL) return USBD_INVAL; hid = usbd_get_hid_descriptor(ifc); if (hid == NULL) return USBD_IOERROR; *sizep = UGETW(hid->descrs[0].wDescriptorLength); if (*sizep == 0) return USBD_INVAL; *descp = kmem_alloc(*sizep, KM_SLEEP); err = usbd_get_report_descriptor(dev, id->bInterfaceNumber, *sizep, *descp); if (err) { kmem_free(*descp, *sizep); *descp = NULL; return err; } return USBD_NORMAL_COMPLETION; } usbd_status usbd_bulk_transfer(struct usbd_xfer *xfer, struct usbd_pipe *pipe, uint16_t flags, uint32_t timeout, void *buf, uint32_t *size) { usbd_status err; USBHIST_FUNC(); USBHIST_CALLARGS(usbdebug, "start transfer %jd bytes", *size, 0, 0, 0); usbd_setup_xfer(xfer, 0, buf, *size, flags, timeout, NULL); err = usbd_sync_transfer_sig(xfer); usbd_get_xfer_status(xfer, NULL, NULL, size, NULL); DPRINTFN(1, "transferred %jd", *size, 0, 0, 0); if (err) { usbd_clear_endpoint_stall(pipe); } USBHIST_LOG(usbdebug, "<- done xfer %#jx err %jd", (uintptr_t)xfer, err, 0, 0); return err; } usbd_status usbd_intr_transfer(struct usbd_xfer *xfer, struct usbd_pipe *pipe, uint16_t flags, uint32_t timeout, void *buf, uint32_t *size) { usbd_status err; USBHIST_FUNC(); USBHIST_CALLARGS(usbdebug, "start transfer %jd bytes", *size, 0, 0, 0); usbd_setup_xfer(xfer, 0, buf, *size, flags, timeout, NULL); err = usbd_sync_transfer_sig(xfer); usbd_get_xfer_status(xfer, NULL, NULL, size, NULL); DPRINTFN(1, "transferred %jd", *size, 0, 0, 0); if (err) { usbd_clear_endpoint_stall(pipe); } USBHIST_LOG(usbdebug, "<- done xfer %#jx err %jd", (uintptr_t)xfer, err, 0, 0); return err; } void usb_detach_waitold(device_t dv) { USBHIST_FUNC(); USBHIST_CALLARGS(usbdebug, "waiting for dv %#jx", (uintptr_t)dv, 0, 0, 0); if (tsleep(dv, PZERO, "usbdet", hz * 60)) /* XXXSMP ok */ aprint_error_dev(dv, "usb_detach_waitold: didn't detach\n"); DPRINTFN(1, "done", 0, 0, 0, 0); } void usb_detach_wakeupold(device_t dv) { USBHIST_FUNC(); USBHIST_CALLARGS(usbdebug, "for dv %#jx", (uintptr_t)dv, 0, 0, 0); wakeup(dv); /* XXXSMP ok */ } /* -------------------------------------------------------------------------- */ void usb_desc_iter_init(struct usbd_device *dev, usbd_desc_iter_t *iter) { const usb_config_descriptor_t *cd = usbd_get_config_descriptor(dev); iter->cur = (const uByte *)cd; iter->end = (const uByte *)cd + UGETW(cd->wTotalLength); } const usb_descriptor_t * usb_desc_iter_peek(usbd_desc_iter_t *iter) { const usb_descriptor_t *desc; if (iter->end - iter->cur < sizeof(usb_descriptor_t)) { if (iter->cur != iter->end) printf("%s: bad descriptor\n", __func__); return NULL; } desc = (const usb_descriptor_t *)iter->cur; if (desc->bLength < USB_DESCRIPTOR_SIZE) { printf("%s: descriptor length too small\n", __func__); return NULL; } if (desc->bLength > iter->end - iter->cur) { printf("%s: descriptor length too large\n", __func__); return NULL; } return desc; } const usb_descriptor_t * usb_desc_iter_next(usbd_desc_iter_t *iter) { const usb_descriptor_t *desc = usb_desc_iter_peek(iter); if (desc == NULL) return NULL; KASSERT(desc->bLength <= iter->end - iter->cur); iter->cur += desc->bLength; return desc; } /* * Return the next interface descriptor, skipping over any other * descriptors. Returns NULL at the end or on error. */ const usb_interface_descriptor_t * usb_desc_iter_next_interface(usbd_desc_iter_t *iter) { const usb_descriptor_t *desc; while ((desc = usb_desc_iter_peek(iter)) != NULL && desc->bDescriptorType != UDESC_INTERFACE) { usb_desc_iter_next(iter); } if ((desc = usb_desc_iter_next(iter)) == NULL || desc->bLength < sizeof(usb_interface_descriptor_t)) return NULL; KASSERT(desc->bDescriptorType == UDESC_INTERFACE); return (const usb_interface_descriptor_t *)desc; } /* * Returns the next non-interface descriptor, returning NULL when the * next descriptor would be an interface descriptor. */ const usb_descriptor_t * usb_desc_iter_next_non_interface(usbd_desc_iter_t *iter) { const usb_descriptor_t *desc; if ((desc = usb_desc_iter_peek(iter)) != NULL && desc->bDescriptorType != UDESC_INTERFACE) { return usb_desc_iter_next(iter); } else { return NULL; } } const usb_cdc_descriptor_t * usb_find_desc(struct usbd_device *dev, int type, int subtype) { usbd_desc_iter_t iter; const usb_cdc_descriptor_t *desc; usb_desc_iter_init(dev, &iter); for (;;) { desc = (const usb_cdc_descriptor_t *)usb_desc_iter_next(&iter); if (desc == NULL) break; if (desc->bDescriptorType != type) continue; if (subtype == USBD_CDCSUBTYPE_ANY || subtype == desc->bDescriptorSubtype) break; } return desc; } /* * Same as usb_find_desc(), but searches only in the specified * interface. */ const usb_cdc_descriptor_t * usb_find_desc_if(struct usbd_device *dev, int type, int subtype, usb_interface_descriptor_t *id) { usbd_desc_iter_t iter; const usb_cdc_descriptor_t *desc; if (id == NULL) return usb_find_desc(dev, type, subtype); usb_desc_iter_init(dev, &iter); iter.cur = (void *)id; /* start from the interface desc */ usb_desc_iter_next(&iter); /* and skip it */ while ((desc = (const usb_cdc_descriptor_t *)usb_desc_iter_next(&iter)) != NULL) { if (desc->bDescriptorType == UDESC_INTERFACE) { /* we ran into the next interface --- not found */ return NULL; } if (desc->bDescriptorType == type && (subtype == USBD_CDCSUBTYPE_ANY || subtype == desc->bDescriptorSubtype)) break; } return desc; } |
| 6 1 5 5 3 1 1 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 197 198 199 200 201 202 203 204 205 206 207 208 209 210 211 212 213 214 215 216 217 218 219 220 221 222 223 224 225 226 227 228 229 230 231 232 233 234 235 236 237 238 239 240 241 242 243 244 245 246 247 248 249 250 251 252 253 254 255 256 257 258 259 260 261 262 263 264 265 266 267 268 269 270 271 272 273 274 275 276 277 278 279 280 281 282 283 284 285 286 287 288 289 290 291 292 293 294 295 296 297 298 299 300 301 302 303 304 305 306 307 308 309 310 311 312 313 314 315 316 317 318 319 320 321 322 323 324 325 326 327 328 329 330 331 332 333 334 335 336 337 338 339 340 341 342 343 344 345 346 347 348 349 350 351 352 353 354 355 356 357 358 359 360 361 362 363 364 365 366 367 368 369 370 371 372 373 374 375 376 377 378 379 380 381 382 383 384 385 386 387 388 389 390 391 392 393 394 395 396 397 398 399 400 401 402 403 404 405 406 407 408 409 410 411 412 413 414 415 416 417 418 419 420 421 422 423 424 425 426 427 428 429 430 431 432 433 434 435 436 437 438 439 440 441 442 443 444 445 446 447 448 449 450 451 452 453 454 455 456 457 458 459 460 461 462 463 464 465 466 467 468 469 470 471 472 473 474 475 476 477 478 479 480 481 482 483 484 485 486 487 488 489 490 491 492 493 494 495 496 497 498 499 500 501 502 503 504 505 506 507 508 509 510 511 512 513 514 515 516 517 518 519 520 521 522 523 524 525 526 527 528 529 530 531 532 533 534 535 536 537 538 539 540 541 542 543 544 545 546 547 548 549 550 551 552 553 554 555 556 557 558 559 560 561 562 563 564 565 566 567 568 569 570 571 572 573 574 575 576 577 578 579 580 581 582 583 584 585 586 587 588 589 590 591 592 593 594 595 596 597 598 599 600 601 602 603 604 605 606 607 608 609 610 611 612 613 614 615 616 617 618 619 620 621 622 623 624 625 626 627 628 629 630 631 632 633 634 635 636 637 638 639 640 641 642 643 644 645 646 647 | /* $NetBSD: coda_vfsops.c,v 1.90 2022/03/28 12:37:46 riastradh Exp $ */ /* * * Coda: an Experimental Distributed File System * Release 3.1 * * Copyright (c) 1987-1998 Carnegie Mellon University * All Rights Reserved * * Permission to use, copy, modify and distribute this software and its * documentation is hereby granted, provided that both the copyright * notice and this permission notice appear in all copies of the * software, derivative works or modified versions, and any portions * thereof, and that both notices appear in supporting documentation, and * that credit is given to Carnegie Mellon University in all documents * and publicity pertaining to direct or indirect use of this code or its * derivatives. * * CODA IS AN EXPERIMENTAL SOFTWARE SYSTEM AND IS KNOWN TO HAVE BUGS, * SOME OF WHICH MAY HAVE SERIOUS CONSEQUENCES. CARNEGIE MELLON ALLOWS * FREE USE OF THIS SOFTWARE IN ITS "AS IS" CONDITION. CARNEGIE MELLON * DISCLAIMS ANY LIABILITY OF ANY KIND FOR ANY DAMAGES WHATSOEVER * RESULTING DIRECTLY OR INDIRECTLY FROM THE USE OF THIS SOFTWARE OR OF * ANY DERIVATIVE WORK. * * Carnegie Mellon encourages users of this software to return any * improvements or extensions that they make, and to grant Carnegie * Mellon the rights to redistribute these changes without encumbrance. * * @(#) cfs/coda_vfsops.c,v 1.1.1.1 1998/08/29 21:26:45 rvb Exp $ */ /* * Mach Operating System * Copyright (c) 1989 Carnegie-Mellon University * All rights reserved. The CMU software License Agreement specifies * the terms and conditions for use and redistribution. */ /* * This code was written for the Coda file system at Carnegie Mellon * University. Contributers include David Steere, James Kistler, and * M. Satyanarayanan. */ #include <sys/cdefs.h> __KERNEL_RCSID(0, "$NetBSD: coda_vfsops.c,v 1.90 2022/03/28 12:37:46 riastradh Exp $"); #include <sys/param.h> #include <sys/systm.h> #include <sys/sysctl.h> #include <sys/malloc.h> #include <sys/conf.h> #include <sys/namei.h> #include <sys/dirent.h> #include <sys/mount.h> #include <sys/proc.h> #include <sys/select.h> #include <sys/kauth.h> #include <sys/module.h> #include <coda/coda.h> #include <coda/cnode.h> #include <coda/coda_vfsops.h> #include <coda/coda_venus.h> #include <coda/coda_subr.h> #include <coda/coda_opstats.h> /* for VN_RDEV */ #include <miscfs/specfs/specdev.h> #include <miscfs/genfs/genfs.h> MODULE(MODULE_CLASS_VFS, coda, "vcoda"); #define ENTRY if(coda_vfsop_print_entry) myprintf(("Entered %s\n",__func__)) extern struct vnode *coda_ctlvp; extern struct coda_mntinfo coda_mnttbl[NVCODA]; /* indexed by minor device number */ /* structure to keep statistics of internally generated/satisfied calls */ struct coda_op_stats coda_vfsopstats[CODA_VFSOPS_SIZE]; #define MARK_ENTRY(op) (coda_vfsopstats[op].entries++) #define MARK_INT_SAT(op) (coda_vfsopstats[op].sat_intrn++) #define MARK_INT_FAIL(op) (coda_vfsopstats[op].unsat_intrn++) #define MRAK_INT_GEN(op) (coda_vfsopstats[op].gen_intrn++) extern const struct cdevsw vcoda_cdevsw; extern const struct vnodeopv_desc coda_vnodeop_opv_desc; const struct vnodeopv_desc * const coda_vnodeopv_descs[] = { &coda_vnodeop_opv_desc, NULL, }; struct vfsops coda_vfsops = { .vfs_name = MOUNT_CODA, .vfs_min_mount_data = 256, /* This is the pathname, unlike every other fs */ .vfs_mount = coda_mount, .vfs_start = coda_start, .vfs_unmount = coda_unmount, .vfs_root = coda_root, .vfs_quotactl = (void *)eopnotsupp, .vfs_statvfs = coda_nb_statvfs, .vfs_sync = coda_sync, .vfs_vget = coda_vget, .vfs_loadvnode = coda_loadvnode, .vfs_fhtovp = (void *)eopnotsupp, .vfs_vptofh = (void *)eopnotsupp, .vfs_init = coda_init, .vfs_done = coda_done, .vfs_mountroot = (void *)eopnotsupp, .vfs_snapshot = (void *)eopnotsupp, .vfs_extattrctl = vfs_stdextattrctl, .vfs_suspendctl = genfs_suspendctl, .vfs_renamelock_enter = genfs_renamelock_enter, .vfs_renamelock_exit = genfs_renamelock_exit, .vfs_fsync = (void *)eopnotsupp, .vfs_opv_descs = coda_vnodeopv_descs }; static int coda_modcmd(modcmd_t cmd, void *arg) { switch (cmd) { case MODULE_CMD_INIT: return vfs_attach(&coda_vfsops); case MODULE_CMD_FINI: return vfs_detach(&coda_vfsops); default: return ENOTTY; } } int coda_vfsopstats_init(void) { int i; for (i=0;i<CODA_VFSOPS_SIZE;i++) { coda_vfsopstats[i].opcode = i; coda_vfsopstats[i].entries = 0; coda_vfsopstats[i].sat_intrn = 0; coda_vfsopstats[i].unsat_intrn = 0; coda_vfsopstats[i].gen_intrn = 0; } return 0; } /* * cfs mount vfsop * Set up mount info record and attach it to vfs struct. */ /*ARGSUSED*/ int coda_mount(struct mount *vfsp, /* Allocated and initialized by mount(2) */ const char *path, /* path covered: ignored by the fs-layer */ void *data, /* Need to define a data type for this in netbsd? */ size_t *data_len) { struct lwp *l = curlwp; struct vnode *dvp; struct cnode *cp; dev_t dev; struct coda_mntinfo *mi; struct vnode *rtvp; const struct cdevsw *cdev; CodaFid rootfid = INVAL_FID; CodaFid ctlfid = CTL_FID; int error; if (data == NULL) return EINVAL; if (vfsp->mnt_flag & MNT_GETARGS) return EINVAL; ENTRY; coda_vfsopstats_init(); coda_vnodeopstats_init(); MARK_ENTRY(CODA_MOUNT_STATS); if (CODA_MOUNTED(vfsp)) { MARK_INT_FAIL(CODA_MOUNT_STATS); return(EBUSY); } /* Validate mount device. Similar to getmdev(). */ /* * XXX: coda passes the mount device as the entire mount args, * All other fs pass a structure contining a pointer. * In order to get sys_mount() to do the copyin() we've set a * fixed default size for the filename buffer. */ /* Ensure that namei() doesn't run off the filename buffer */ if (*data_len < 1 || *data_len > PATH_MAX || strnlen(data, *data_len) >= *data_len) { MARK_INT_FAIL(CODA_MOUNT_STATS); return EINVAL; } error = namei_simple_kernel((char *)data, NSM_FOLLOW_NOEMULROOT, &dvp); if (error) { MARK_INT_FAIL(CODA_MOUNT_STATS); return (error); } if (dvp->v_type != VCHR) { MARK_INT_FAIL(CODA_MOUNT_STATS); vrele(dvp); return(ENXIO); } dev = dvp->v_rdev; vrele(dvp); cdev = cdevsw_lookup(dev); if (cdev == NULL) { MARK_INT_FAIL(CODA_MOUNT_STATS); return(ENXIO); } /* * See if the device table matches our expectations. */ if (cdev != &vcoda_cdevsw) { MARK_INT_FAIL(CODA_MOUNT_STATS); return(ENXIO); } if (minor(dev) >= NVCODA) { MARK_INT_FAIL(CODA_MOUNT_STATS); return(ENXIO); } /* * Initialize the mount record and link it to the vfs struct */ mi = &coda_mnttbl[minor(dev)]; if (!VC_OPEN(&mi->mi_vcomm)) { MARK_INT_FAIL(CODA_MOUNT_STATS); return(ENODEV); } /* No initialization (here) of mi_vcomm! */ vfsp->mnt_data = mi; vfsp->mnt_stat.f_fsidx.__fsid_val[0] = 0; vfsp->mnt_stat.f_fsidx.__fsid_val[1] = makefstype(MOUNT_CODA); vfsp->mnt_stat.f_fsid = vfsp->mnt_stat.f_fsidx.__fsid_val[0]; vfsp->mnt_stat.f_namemax = CODA_MAXNAMLEN; mi->mi_vfsp = vfsp; /* * Make a root vnode to placate the Vnode interface, but don't * actually make the CODA_ROOT call to venus until the first call * to coda_root in case a server is down while venus is starting. */ cp = make_coda_node(&rootfid, vfsp, VDIR); rtvp = CTOV(cp); rtvp->v_vflag |= VV_ROOT; cp = make_coda_node(&ctlfid, vfsp, VCHR); coda_ctlvp = CTOV(cp); /* Add vfs and rootvp to chain of vfs hanging off mntinfo */ mi->mi_vfsp = vfsp; mi->mi_rootvp = rtvp; /* set filesystem block size */ vfsp->mnt_stat.f_bsize = 8192; /* XXX -JJK */ vfsp->mnt_stat.f_frsize = 8192; /* XXX -JJK */ /* error is currently guaranteed to be zero, but in case some code changes... */ CODADEBUG(1, myprintf(("coda_mount returned %d\n",error));); if (error) MARK_INT_FAIL(CODA_MOUNT_STATS); else MARK_INT_SAT(CODA_MOUNT_STATS); return set_statvfs_info("/coda", UIO_SYSSPACE, "CODA", UIO_SYSSPACE, vfsp->mnt_op->vfs_name, vfsp, l); } int coda_start(struct mount *vfsp, int flags) { ENTRY; vftomi(vfsp)->mi_started = 1; return (0); } int coda_unmount(struct mount *vfsp, int mntflags) { struct coda_mntinfo *mi = vftomi(vfsp); int active, error = 0; ENTRY; MARK_ENTRY(CODA_UMOUNT_STATS); if (!CODA_MOUNTED(vfsp)) { MARK_INT_FAIL(CODA_UMOUNT_STATS); return(EINVAL); } if (mi->mi_vfsp == vfsp) { /* We found the victim */ if (!IS_UNMOUNTING(VTOC(mi->mi_rootvp))) return (EBUSY); /* Venus is still running */ #ifdef DEBUG printf("coda_unmount: ROOT: vp %p, cp %p\n", mi->mi_rootvp, VTOC(mi->mi_rootvp)); #endif mi->mi_started = 0; vrele(mi->mi_rootvp); vrele(coda_ctlvp); active = coda_kill(vfsp, NOT_DOWNCALL); mi->mi_rootvp->v_vflag &= ~VV_ROOT; error = vflush(mi->mi_vfsp, NULLVP, FORCECLOSE); printf("coda_unmount: active = %d, vflush active %d\n", active, error); error = 0; /* I'm going to take this out to allow lookups to go through. I'm * not sure it's important anyway. -- DCS 2/2/94 */ /* vfsp->VFS_DATA = NULL; */ /* No more vfsp's to hold onto */ mi->mi_vfsp = NULL; mi->mi_rootvp = NULL; if (error) MARK_INT_FAIL(CODA_UMOUNT_STATS); else MARK_INT_SAT(CODA_UMOUNT_STATS); return(error); } return (EINVAL); } /* * find root of cfs */ int coda_root(struct mount *vfsp, int lktype, struct vnode **vpp) { struct coda_mntinfo *mi = vftomi(vfsp); int error; struct lwp *l = curlwp; /* XXX - bnoble */ CodaFid VFid; static const CodaFid invalfid = INVAL_FID; ENTRY; MARK_ENTRY(CODA_ROOT_STATS); if (vfsp == mi->mi_vfsp) { if (memcmp(&VTOC(mi->mi_rootvp)->c_fid, &invalfid, sizeof(CodaFid))) { /* Found valid root. */ *vpp = mi->mi_rootvp; /* On Mach, this is vref. On NetBSD, VOP_LOCK */ vref(*vpp); vn_lock(*vpp, lktype); MARK_INT_SAT(CODA_ROOT_STATS); return(0); } } error = venus_root(vftomi(vfsp), l->l_cred, l->l_proc, &VFid); if (!error) { struct cnode *cp = VTOC(mi->mi_rootvp); /* * Save the new rootfid in the cnode, and rekey the cnode * with the new fid key. */ error = vcache_rekey_enter(vfsp, mi->mi_rootvp, &invalfid, sizeof(CodaFid), &VFid, sizeof(CodaFid)); if (error) goto exit; cp->c_fid = VFid; vcache_rekey_exit(vfsp, mi->mi_rootvp, &invalfid, sizeof(CodaFid), &cp->c_fid, sizeof(CodaFid)); *vpp = mi->mi_rootvp; vref(*vpp); vn_lock(*vpp, lktype); MARK_INT_SAT(CODA_ROOT_STATS); goto exit; } else if (error == ENODEV || error == EINTR) { /* Gross hack here! */ /* * If Venus fails to respond to the CODA_ROOT call, coda_call returns * ENODEV. Return the uninitialized root vnode to allow vfs * operations such as unmount to continue. Without this hack, * there is no way to do an unmount if Venus dies before a * successful CODA_ROOT call is done. All vnode operations * will fail. */ *vpp = mi->mi_rootvp; vref(*vpp); vn_lock(*vpp, lktype); MARK_INT_FAIL(CODA_ROOT_STATS); error = 0; goto exit; } else { CODADEBUG( CODA_ROOT, myprintf(("error %d in CODA_ROOT\n", error)); ); MARK_INT_FAIL(CODA_ROOT_STATS); goto exit; } exit: return(error); } /* * Get file system statistics. */ int coda_nb_statvfs(struct mount *vfsp, struct statvfs *sbp) { struct lwp *l = curlwp; struct coda_statfs fsstat; int error; ENTRY; MARK_ENTRY(CODA_STATFS_STATS); if (!CODA_MOUNTED(vfsp)) { /* MARK_INT_FAIL(CODA_STATFS_STATS); */ return(EINVAL); } /* XXX - what to do about f_flags, others? --bnoble */ /* Below This is what AFS does #define NB_SFS_SIZ 0x895440 */ /* Note: Normal fs's have a bsize of 0x400 == 1024 */ error = venus_statfs(vftomi(vfsp), l->l_cred, l, &fsstat); if (!error) { sbp->f_bsize = 8192; /* XXX */ sbp->f_frsize = 8192; /* XXX */ sbp->f_iosize = 8192; /* XXX */ sbp->f_blocks = fsstat.f_blocks; sbp->f_bfree = fsstat.f_bfree; sbp->f_bavail = fsstat.f_bavail; sbp->f_bresvd = 0; sbp->f_files = fsstat.f_files; sbp->f_ffree = fsstat.f_ffree; sbp->f_favail = fsstat.f_ffree; sbp->f_fresvd = 0; copy_statvfs_info(sbp, vfsp); } MARK_INT_SAT(CODA_STATFS_STATS); return(error); } /* * Flush any pending I/O. */ int coda_sync(struct mount *vfsp, int waitfor, kauth_cred_t cred) { ENTRY; MARK_ENTRY(CODA_SYNC_STATS); MARK_INT_SAT(CODA_SYNC_STATS); return(0); } int coda_vget(struct mount *vfsp, ino_t ino, int lktype, struct vnode **vpp) { ENTRY; return (EOPNOTSUPP); } int coda_loadvnode(struct mount *mp, struct vnode *vp, const void *key, size_t key_len, const void **new_key) { CodaFid fid; struct cnode *cp; extern int (**coda_vnodeop_p)(void *); KASSERT(key_len == sizeof(CodaFid)); memcpy(&fid, key, key_len); cp = kmem_zalloc(sizeof(*cp), KM_SLEEP); mutex_init(&cp->c_lock, MUTEX_DEFAULT, IPL_NONE); cp->c_fid = fid; cp->c_vnode = vp; vp->v_op = coda_vnodeop_p; vp->v_tag = VT_CODA; vp->v_type = VNON; vp->v_data = cp; *new_key = &cp->c_fid; return 0; } /* * fhtovp is now what vget used to be in 4.3-derived systems. For * some silly reason, vget is now keyed by a 32 bit ino_t, rather than * a type-specific fid. */ int coda_fhtovp(struct mount *vfsp, struct fid *fhp, struct mbuf *nam, struct vnode **vpp, int *exflagsp, kauth_cred_t *creadanonp, int lktype) { struct cfid *cfid = (struct cfid *)fhp; struct cnode *cp = 0; int error; struct lwp *l = curlwp; /* XXX -mach */ CodaFid VFid; int vtype; ENTRY; MARK_ENTRY(CODA_VGET_STATS); /* Check for vget of control object. */ if (IS_CTL_FID(&cfid->cfid_fid)) { *vpp = coda_ctlvp; vref(coda_ctlvp); MARK_INT_SAT(CODA_VGET_STATS); return(0); } error = venus_fhtovp(vftomi(vfsp), &cfid->cfid_fid, l->l_cred, l->l_proc, &VFid, &vtype); if (error) { CODADEBUG(CODA_VGET, myprintf(("vget error %d\n",error));) *vpp = (struct vnode *)0; } else { CODADEBUG(CODA_VGET, myprintf(("vget: %s type %d result %d\n", coda_f2s(&VFid), vtype, error)); ) cp = make_coda_node(&VFid, vfsp, vtype); *vpp = CTOV(cp); } return(error); } int coda_vptofh(struct vnode *vnp, struct fid *fidp) { ENTRY; return (EOPNOTSUPP); } void coda_init(void) { ENTRY; } void coda_done(void) { ENTRY; } SYSCTL_SETUP(sysctl_vfs_coda_setup, "sysctl vfs.coda subtree setup") { sysctl_createv(clog, 0, NULL, NULL, CTLFLAG_PERMANENT, CTLTYPE_NODE, "coda", SYSCTL_DESCR("code vfs options"), NULL, 0, NULL, 0, CTL_VFS, 18, CTL_EOL); /* * XXX the "18" above could be dynamic, thereby eliminating * one more instance of the "number to vfs" mapping problem, * but "18" is the order as taken from sys/mount.h */ /* sysctl_createv(clog, 0, NULL, NULL, CTLFLAG_PERMANENT|CTLFLAG_READWRITE, CTLTYPE_INT, "clusterread", SYSCTL_DESCR( anyone? ), NULL, 0, &doclusterread, 0, CTL_VFS, 18, FFS_CLUSTERREAD, CTL_EOL); */ } /* * To allow for greater ease of use, some vnodes may be orphaned when * Venus dies. Certain operations should still be allowed to go * through, but without propagating orphan-ness. So this function will * get a new vnode for the file from the current run of Venus. */ int getNewVnode(struct vnode **vpp) { struct cfid cfid; struct coda_mntinfo *mi = vftomi((*vpp)->v_mount); ENTRY; cfid.cfid_len = (short)sizeof(CodaFid); cfid.cfid_fid = VTOC(*vpp)->c_fid; /* Structure assignment. */ /* XXX ? */ /* We're guessing that if set, the 1st element on the list is a * valid vnode to use. If not, return ENODEV as venus is dead. */ if (mi->mi_vfsp == NULL) return ENODEV; return coda_fhtovp(mi->mi_vfsp, (struct fid*)&cfid, NULL, vpp, NULL, NULL, LK_EXCLUSIVE); } /* Get the mount structure corresponding to a given device. * Return NULL if no device is found or the device is not mounted. */ struct mount *devtomp(dev_t dev) { struct mount *mp; struct vnode *vp; if (spec_node_lookup_by_dev(VBLK, dev, VDEAD_NOWAIT, &vp) == 0) { mp = spec_node_getmountedfs(vp); vrele(vp); } else { mp = NULL; } return mp; } |
| 5 1 1 2 1 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 197 198 199 200 201 202 203 204 205 206 207 208 209 210 211 212 213 214 215 216 217 218 219 220 221 222 223 224 225 226 227 228 229 230 231 232 233 234 235 236 237 238 239 240 241 242 243 244 245 246 247 248 249 250 251 252 253 254 255 256 257 258 259 260 261 262 263 264 265 266 267 268 269 270 271 272 273 274 275 276 277 278 279 280 281 282 283 284 285 286 287 288 289 290 291 292 293 294 295 296 297 298 299 300 301 302 303 304 305 306 307 308 309 310 311 312 313 314 315 316 317 318 319 320 321 322 323 324 325 326 327 328 329 330 331 332 333 334 335 336 337 338 339 340 341 342 343 344 345 346 347 348 349 350 351 352 353 354 355 356 357 358 359 360 361 362 363 364 365 366 367 368 369 370 371 372 373 374 375 376 377 378 379 380 381 382 383 384 385 386 387 388 389 390 391 392 393 394 395 396 397 398 399 400 401 402 403 404 405 406 407 408 409 410 411 412 413 414 415 416 417 418 419 420 421 422 423 424 425 426 427 428 429 430 431 432 433 434 435 436 437 438 439 440 441 442 443 444 445 446 447 448 449 450 451 452 453 454 455 456 457 458 459 460 461 462 463 464 465 466 467 468 469 470 471 472 473 474 475 476 477 478 479 480 481 482 483 484 485 486 487 488 489 490 491 492 493 494 495 496 497 498 499 500 501 502 503 504 505 506 507 508 509 510 511 512 513 514 515 516 517 518 519 520 521 522 523 524 525 526 527 528 529 530 531 532 533 534 535 536 537 538 539 540 541 542 543 544 545 546 547 548 549 550 551 552 553 554 555 556 557 558 559 560 561 562 563 564 565 566 567 568 569 570 571 572 573 574 575 576 577 578 579 580 581 582 583 584 585 586 587 588 589 590 591 592 593 594 595 596 597 598 599 600 601 602 603 604 605 606 607 608 609 610 611 612 613 614 615 616 617 618 619 620 621 622 623 624 625 626 627 628 629 630 631 632 633 634 635 636 637 638 639 640 641 642 643 644 645 646 647 648 649 650 651 652 653 654 655 656 657 658 659 660 661 662 663 664 665 666 667 668 669 670 671 672 673 674 675 676 677 678 679 680 681 682 683 684 685 686 687 688 689 690 691 692 693 694 695 696 697 698 699 700 701 702 703 704 705 706 707 708 709 710 711 712 713 714 715 716 717 718 719 720 721 722 723 724 725 726 727 728 729 730 731 732 733 734 735 736 737 738 739 740 741 742 743 744 745 746 747 748 749 750 751 752 753 754 755 756 757 758 759 760 761 762 763 764 765 766 767 768 769 770 771 772 773 774 775 776 777 778 779 780 781 782 783 784 785 786 787 788 789 790 791 792 793 794 795 796 797 798 799 800 801 802 803 804 805 806 807 808 809 810 811 812 813 814 815 816 817 818 819 820 821 822 823 824 825 826 827 828 829 830 831 832 833 834 835 836 837 838 839 840 841 842 843 844 845 846 847 848 849 850 851 852 853 854 855 856 857 858 859 860 861 862 863 864 865 866 867 868 869 870 871 872 873 874 875 876 877 878 879 880 881 882 883 884 885 886 887 888 889 890 891 892 893 894 895 896 897 898 899 900 901 902 903 904 905 906 907 908 909 910 911 912 913 914 915 916 917 918 919 920 921 922 923 924 925 926 927 928 929 930 931 932 933 934 935 936 937 938 939 940 941 942 943 944 945 946 947 948 949 950 951 952 953 954 955 956 957 958 959 960 961 962 963 964 965 966 967 968 969 970 971 972 973 974 975 976 977 978 979 980 981 982 983 984 985 986 987 988 989 990 991 992 993 994 995 996 997 998 999 1000 1001 1002 1003 1004 1005 1006 1007 1008 1009 1010 1011 1012 1013 1014 1015 1016 1017 1018 1019 1020 1021 1022 1023 1024 1025 1026 1027 1028 1029 1030 1031 1032 1033 1034 1035 1036 1037 1038 1039 1040 1041 1042 1043 1044 1045 1046 1047 1048 1049 1050 1051 1052 1053 1054 1055 1056 1057 1058 1059 1060 1061 1062 1063 1064 1065 1066 1067 1068 1069 1070 1071 1072 1073 1074 1075 1076 1077 1078 1079 1080 1081 1082 1083 1084 1085 1086 1087 1088 1089 1090 1091 1092 1093 1094 1095 1096 1097 1098 1099 1100 1101 1102 1103 1104 1105 1106 1107 1108 1109 1110 1111 1112 1113 1114 1115 1116 1117 1118 1119 1120 1121 1122 1123 1124 1125 1126 1127 1128 1129 1130 1131 1132 1133 1134 1135 1136 1137 1138 1139 1140 1141 1142 1143 1144 1145 1146 1147 1148 1149 1150 1151 1152 1153 1154 1155 1156 1157 1158 1159 1160 1161 1162 1163 1164 1165 1166 1167 1168 1169 1170 1171 1172 1173 1174 1175 1176 1177 1178 1179 1180 1181 1182 1183 1184 1185 1186 1187 1188 1189 1190 1191 1192 1193 1194 1195 1196 1197 1198 1199 1200 1201 1202 1203 1204 1205 1206 1207 1208 1209 1210 1211 1212 1213 1214 1215 1216 1217 1218 1219 1220 1221 1222 1223 1224 1225 1226 1227 1228 1229 1230 1231 1232 1233 1234 1235 1236 1237 1238 1239 1240 1241 1242 1243 1244 1245 1246 1247 1248 1249 1250 1251 1252 1253 1254 1255 1256 1257 1258 1259 1260 1261 1262 1263 1264 1265 1266 1267 1268 1269 1270 1271 1272 1273 1274 1275 1276 1277 1278 1279 1280 1281 1282 1283 1284 1285 1286 1287 1288 1289 1290 1291 1292 1293 1294 1295 1296 1297 1298 1299 1300 1301 1302 1303 1304 1305 1306 1307 1308 1309 1310 1311 1312 1313 1314 1315 1316 1317 1318 1319 1320 1321 1322 1323 1324 1325 1326 1327 1328 1329 1330 1331 1332 1333 1334 1335 1336 1337 1338 1339 1340 1341 1342 1343 1344 1345 1346 1347 1348 1349 1350 1351 1352 1353 1354 1355 1356 1357 1358 1359 1360 1361 1362 1363 1364 1365 1366 1367 1368 1369 1370 1371 1372 1373 1374 1375 1376 1377 1378 1379 1380 1381 1382 1383 1384 1385 1386 1387 1388 1389 1390 1391 1392 1393 1394 1395 1396 1397 1398 1399 1400 1401 1402 1403 1404 1405 1406 1407 1408 1409 1410 1411 1412 1413 1414 1415 1416 1417 1418 1419 1420 1421 1422 1423 1424 1425 1426 1427 1428 1429 1430 1431 1432 1433 1434 1435 1436 1437 1438 1439 1440 1441 1442 1443 1444 1445 1446 1447 1448 1449 1450 1451 1452 1453 1454 1455 1456 1457 1458 1459 1460 1461 1462 1463 1464 1465 1466 1467 1468 1469 1470 1471 1472 1473 1474 1475 1476 1477 1478 1479 1480 1481 1482 1483 1484 1485 1486 1487 1488 1489 1490 1491 1492 1493 1494 1495 1496 1497 1498 1499 1500 1501 1502 1503 1504 1505 1506 1507 1508 1509 1510 1511 1512 1513 1514 1515 1516 1517 1518 1519 1520 1521 1522 1523 1524 1525 1526 1527 1528 1529 1530 1531 1532 1533 1534 1535 1536 1537 1538 1539 1540 1541 1542 1543 1544 1545 1546 1547 1548 1549 1550 1551 1552 1553 1554 1555 1556 1557 1558 1559 1560 1561 1562 1563 1564 1565 1566 1567 1568 1569 1570 1571 1572 1573 1574 1575 1576 1577 1578 1579 1580 1581 1582 1583 1584 1585 1586 1587 1588 1589 1590 1591 1592 1593 1594 1595 1596 1597 1598 1599 1600 1601 1602 1603 1604 1605 1606 1607 1608 1609 1610 1611 1612 1613 1614 1615 1616 1617 1618 1619 1620 1621 1622 1623 1624 1625 1626 1627 1628 1629 1630 1631 1632 1633 1634 1635 1636 1637 1638 1639 1640 1641 1642 1643 1644 1645 1646 1647 1648 1649 1650 1651 1652 1653 1654 1655 1656 1657 1658 1659 1660 1661 1662 1663 1664 1665 1666 1667 1668 1669 1670 1671 1672 1673 1674 1675 1676 1677 1678 1679 1680 1681 1682 1683 1684 1685 1686 1687 1688 1689 1690 1691 1692 1693 1694 1695 1696 1697 1698 1699 1700 1701 1702 1703 1704 1705 1706 1707 1708 1709 1710 1711 1712 1713 1714 1715 1716 1717 1718 1719 1720 1721 1722 1723 1724 1725 1726 1727 1728 1729 1730 1731 1732 1733 1734 1735 1736 1737 1738 1739 1740 1741 1742 1743 1744 1745 1746 1747 1748 1749 1750 1751 1752 1753 1754 1755 1756 1757 1758 1759 1760 1761 1762 1763 1764 1765 1766 1767 1768 1769 1770 1771 1772 1773 1774 1775 1776 1777 1778 1779 1780 1781 1782 1783 1784 1785 1786 1787 1788 1789 1790 1791 1792 1793 1794 1795 1796 1797 1798 1799 1800 1801 1802 1803 1804 1805 1806 1807 1808 1809 1810 1811 1812 1813 1814 1815 1816 1817 1818 1819 1820 1821 1822 1823 1824 1825 1826 1827 1828 1829 1830 1831 1832 1833 1834 1835 1836 1837 1838 1839 1840 1841 1842 1843 1844 1845 1846 1847 1848 1849 1850 1851 1852 1853 1854 1855 1856 1857 1858 1859 1860 1861 1862 1863 1864 1865 1866 1867 1868 1869 1870 1871 1872 1873 1874 1875 1876 1877 1878 1879 1880 1881 1882 1883 1884 1885 1886 1887 1888 1889 1890 1891 1892 1893 1894 1895 1896 1897 1898 1899 1900 1901 1902 1903 1904 1905 1906 1907 1908 1909 1910 1911 1912 1913 1914 1915 1916 1917 1918 1919 1920 1921 1922 1923 1924 1925 1926 1927 1928 1929 1930 1931 1932 1933 1934 1935 1936 1937 1938 1939 1940 1941 1942 1943 1944 1945 1946 1947 1948 1949 1950 1951 1952 1953 1954 1955 1956 1957 1958 1959 1960 1961 1962 1963 1964 1965 1966 1967 1968 1969 1970 1971 1972 1973 1974 1975 1976 1977 1978 1979 1980 1981 1982 1983 1984 1985 1986 1987 1988 1989 1990 1991 1992 1993 1994 1995 1996 1997 1998 1999 2000 2001 2002 2003 2004 2005 2006 2007 2008 2009 2010 2011 2012 2013 2014 2015 2016 2017 2018 2019 2020 2021 2022 2023 2024 2025 2026 2027 2028 2029 2030 2031 2032 2033 2034 2035 2036 2037 2038 2039 2040 2041 2042 2043 2044 2045 2046 2047 2048 2049 2050 2051 2052 2053 2054 2055 2056 2057 2058 2059 2060 2061 2062 2063 2064 2065 2066 2067 2068 2069 2070 2071 2072 2073 2074 2075 2076 2077 2078 2079 2080 2081 2082 2083 2084 2085 2086 2087 2088 2089 2090 2091 2092 2093 2094 2095 2096 2097 2098 2099 2100 2101 2102 2103 2104 2105 2106 2107 2108 2109 2110 2111 2112 2113 2114 2115 2116 2117 2118 2119 2120 2121 2122 2123 2124 2125 2126 2127 2128 2129 2130 2131 2132 2133 2134 2135 2136 2137 2138 2139 2140 2141 2142 2143 2144 2145 2146 2147 2148 2149 2150 2151 2152 2153 2154 2155 2156 2157 2158 2159 2160 2161 2162 2163 2164 2165 2166 2167 2168 2169 2170 2171 2172 2173 2174 2175 2176 2177 2178 2179 2180 2181 2182 2183 2184 2185 2186 2187 2188 2189 2190 2191 2192 2193 2194 2195 2196 2197 2198 2199 2200 2201 2202 2203 2204 2205 2206 2207 2208 2209 2210 2211 2212 2213 2214 2215 2216 2217 2218 2219 2220 2221 2222 2223 2224 2225 2226 2227 2228 2229 2230 2231 2232 2233 2234 2235 2236 2237 2238 2239 2240 2241 2242 2243 2244 2245 2246 2247 2248 2249 2250 2251 2252 2253 2254 2255 2256 2257 2258 2259 2260 2261 2262 2263 2264 2265 2266 2267 2268 2269 2270 2271 2272 2273 2274 2275 2276 2277 2278 2279 2280 2281 2282 2283 2284 2285 2286 2287 2288 2289 2290 2291 2292 2293 2294 2295 2296 2297 2298 2299 2300 2301 2302 2303 2304 2305 2306 2307 2308 2309 2310 2311 2312 2313 2314 2315 2316 2317 2318 2319 2320 2321 2322 2323 2324 2325 2326 2327 2328 2329 2330 2331 2332 2333 2334 2335 2336 2337 2338 2339 2340 2341 2342 2343 2344 2345 2346 2347 2348 2349 2350 2351 2352 2353 2354 2355 2356 2357 2358 2359 2360 2361 2362 2363 2364 2365 2366 2367 2368 2369 2370 2371 2372 2373 2374 2375 2376 2377 2378 2379 2380 2381 2382 2383 2384 2385 2386 2387 2388 2389 2390 2391 2392 2393 2394 2395 2396 2397 2398 2399 2400 2401 2402 2403 2404 2405 2406 2407 2408 2409 2410 2411 2412 2413 2414 2415 2416 2417 2418 2419 2420 2421 2422 2423 2424 2425 2426 2427 2428 2429 2430 2431 2432 2433 2434 2435 2436 2437 2438 2439 2440 2441 2442 2443 2444 2445 2446 2447 2448 2449 2450 2451 2452 2453 2454 2455 2456 2457 2458 2459 2460 2461 2462 2463 2464 2465 2466 2467 2468 2469 2470 2471 2472 2473 2474 2475 2476 2477 2478 2479 2480 2481 2482 2483 2484 2485 2486 2487 2488 2489 2490 2491 2492 2493 2494 2495 2496 2497 2498 2499 2500 2501 2502 2503 2504 2505 2506 2507 2508 2509 2510 2511 2512 2513 2514 2515 2516 2517 2518 2519 2520 2521 2522 2523 2524 2525 2526 2527 2528 2529 2530 2531 2532 2533 2534 2535 2536 2537 2538 2539 2540 2541 2542 2543 2544 2545 2546 2547 2548 2549 2550 2551 2552 2553 2554 2555 2556 2557 2558 2559 2560 2561 2562 2563 2564 2565 2566 2567 2568 2569 2570 2571 2572 2573 2574 2575 2576 2577 | /* $NetBSD: lfs_vfsops.c,v 1.382 2022/03/19 13:53:33 hannken Exp $ */ /*- * Copyright (c) 1999, 2000, 2001, 2002, 2003, 2007, 2007 * The NetBSD Foundation, Inc. * All rights reserved. * * This code is derived from software contributed to The NetBSD Foundation * by Konrad E. Schroder <perseant@hhhh.org>. * * Redistribution and use in source and binary forms, with or without * modification, are permitted provided that the following conditions * are met: * 1. Redistributions of source code must retain the above copyright * notice, this list of conditions and the following disclaimer. * 2. Redistributions in binary form must reproduce the above copyright * notice, this list of conditions and the following disclaimer in the * documentation and/or other materials provided with the distribution. * * THIS SOFTWARE IS PROVIDED BY THE NETBSD FOUNDATION, INC. AND CONTRIBUTORS * ``AS IS'' AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED * TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR * PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE FOUNDATION OR CONTRIBUTORS * BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR * CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF * SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS * INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN * CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) * ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE * POSSIBILITY OF SUCH DAMAGE. */ /*- * Copyright (c) 1989, 1991, 1993, 1994 * The Regents of the University of California. All rights reserved. * * Redistribution and use in source and binary forms, with or without * modification, are permitted provided that the following conditions * are met: * 1. Redistributions of source code must retain the above copyright * notice, this list of conditions and the following disclaimer. * 2. Redistributions in binary form must reproduce the above copyright * notice, this list of conditions and the following disclaimer in the * documentation and/or other materials provided with the distribution. * 3. Neither the name of the University nor the names of its contributors * may be used to endorse or promote products derived from this software * without specific prior written permission. * * THIS SOFTWARE IS PROVIDED BY THE REGENTS AND CONTRIBUTORS ``AS IS'' AND * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE * ARE DISCLAIMED. IN NO EVENT SHALL THE REGENTS OR CONTRIBUTORS BE LIABLE * FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL * DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS * OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) * HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT * LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY * OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF * SUCH DAMAGE. * * @(#)lfs_vfsops.c 8.20 (Berkeley) 6/10/95 */ #include <sys/cdefs.h> __KERNEL_RCSID(0, "$NetBSD: lfs_vfsops.c,v 1.382 2022/03/19 13:53:33 hannken Exp $"); #if defined(_KERNEL_OPT) #include "opt_lfs.h" #include "opt_quota.h" #include "opt_uvmhist.h" #endif #include <sys/param.h> #include <sys/systm.h> #include <sys/namei.h> #include <sys/proc.h> #include <sys/kernel.h> #include <sys/vnode.h> #include <sys/mount.h> #include <sys/kthread.h> #include <sys/buf.h> #include <sys/device.h> #include <sys/file.h> #include <sys/disklabel.h> #include <sys/ioctl.h> #include <sys/errno.h> #include <sys/malloc.h> #include <sys/pool.h> #include <sys/socket.h> #include <sys/syslog.h> #include <sys/sysctl.h> #include <sys/conf.h> #include <sys/kauth.h> #include <sys/module.h> #include <sys/syscallvar.h> #include <sys/syscall.h> #include <sys/syscallargs.h> #include <miscfs/specfs/specdev.h> #include <ufs/lfs/ulfs_quotacommon.h> #include <ufs/lfs/ulfs_inode.h> #include <ufs/lfs/ulfsmount.h> #include <ufs/lfs/ulfs_bswap.h> #include <ufs/lfs/ulfs_extern.h> #ifdef UVMHIST #include <uvm/uvm.h> #endif #include <uvm/uvm_extern.h> #include <uvm/uvm_object.h> #include <uvm/uvm_page.h> #include <uvm/uvm_stat.h> #include <ufs/lfs/lfs.h> #include <ufs/lfs/lfs_accessors.h> #include <ufs/lfs/lfs_kernel.h> #include <ufs/lfs/lfs_extern.h> #include <miscfs/genfs/genfs.h> #include <miscfs/genfs/genfs_node.h> MODULE(MODULE_CLASS_VFS, lfs, NULL); static int lfs_gop_write(struct vnode *, struct vm_page **, int, int); static int lfs_mountfs(struct vnode *, struct mount *, struct lwp *); static int lfs_flushfiles(struct mount *, int); extern const struct vnodeopv_desc lfs_vnodeop_opv_desc; extern const struct vnodeopv_desc lfs_specop_opv_desc; extern const struct vnodeopv_desc lfs_fifoop_opv_desc; struct lwp * lfs_writer_daemon = NULL; kcondvar_t lfs_writerd_cv; int lfs_do_flush = 0; #ifdef LFS_KERNEL_RFW int lfs_do_rfw = 0; #endif const struct vnodeopv_desc * const lfs_vnodeopv_descs[] = { &lfs_vnodeop_opv_desc, &lfs_specop_opv_desc, &lfs_fifoop_opv_desc, NULL, }; struct vfsops lfs_vfsops = { .vfs_name = MOUNT_LFS, .vfs_min_mount_data = sizeof (struct ulfs_args), .vfs_mount = lfs_mount, .vfs_start = ulfs_start, .vfs_unmount = lfs_unmount, .vfs_root = ulfs_root, .vfs_quotactl = ulfs_quotactl, .vfs_statvfs = lfs_statvfs, .vfs_sync = lfs_sync, .vfs_vget = lfs_vget, .vfs_loadvnode = lfs_loadvnode, .vfs_newvnode = lfs_newvnode, .vfs_fhtovp = lfs_fhtovp, .vfs_vptofh = lfs_vptofh, .vfs_init = lfs_init, .vfs_reinit = lfs_reinit, .vfs_done = lfs_done, .vfs_mountroot = lfs_mountroot, .vfs_snapshot = (void *)eopnotsupp, .vfs_extattrctl = lfs_extattrctl, .vfs_suspendctl = genfs_suspendctl, .vfs_renamelock_enter = genfs_renamelock_enter, .vfs_renamelock_exit = genfs_renamelock_exit, .vfs_fsync = (void *)eopnotsupp, .vfs_opv_descs = lfs_vnodeopv_descs }; const struct genfs_ops lfs_genfsops = { .gop_size = lfs_gop_size, .gop_alloc = ulfs_gop_alloc, .gop_write = lfs_gop_write, .gop_markupdate = ulfs_gop_markupdate, .gop_putrange = genfs_gop_putrange, }; struct shortlong { const char *sname; const char *lname; }; static int sysctl_lfs_dostats(SYSCTLFN_ARGS) { extern struct lfs_stats lfs_stats; extern int lfs_dostats; int error; error = sysctl_lookup(SYSCTLFN_CALL(rnode)); if (error || newp == NULL) return (error); if (lfs_dostats == 0) memset(&lfs_stats, 0, sizeof(lfs_stats)); return (0); } SYSCTL_SETUP(lfs_sysctl_setup, "lfs sysctl") { int i; extern int lfs_writeindir, lfs_dostats, lfs_clean_vnhead, lfs_fs_pagetrip, lfs_ignore_lazy_sync; #ifdef DEBUG extern int lfs_debug_log_subsys[DLOG_MAX]; struct shortlong dlog_names[DLOG_MAX] = { /* Must match lfs.h ! */ { "rollforward", "Debug roll-forward code" }, { "alloc", "Debug inode allocation and free list" }, { "avail", "Debug space-available-now accounting" }, { "flush", "Debug flush triggers" }, { "lockedlist", "Debug locked list accounting" }, { "vnode_verbose", "Verbose per-vnode-written debugging" }, { "vnode", "Debug vnode use during segment write" }, { "segment", "Debug segment writing" }, { "seguse", "Debug segment used-bytes accounting" }, { "cleaner", "Debug cleaning routines" }, { "mount", "Debug mount/unmount routines" }, { "pagecache", "Debug UBC interactions" }, { "dirop", "Debug directory-operation accounting" }, { "malloc", "Debug private malloc accounting" }, }; #endif /* DEBUG */ struct shortlong stat_names[] = { /* Must match lfs.h! */ { "segsused", "Number of new segments allocated" }, { "psegwrites", "Number of partial-segment writes" }, { "psyncwrites", "Number of synchronous partial-segment" " writes" }, { "pcleanwrites", "Number of partial-segment writes by the" " cleaner" }, { "blocktot", "Number of blocks written" }, { "cleanblocks", "Number of blocks written by the cleaner" }, { "ncheckpoints", "Number of checkpoints made" }, { "nwrites", "Number of whole writes" }, { "nsync_writes", "Number of synchronous writes" }, { "wait_exceeded", "Number of times writer waited for" " cleaner" }, { "write_exceeded", "Number of times writer invoked flush" }, { "flush_invoked", "Number of times flush was invoked" }, { "vflush_invoked", "Number of time vflush was called" }, { "clean_inlocked", "Number of vnodes skipped for being dead" }, { "clean_vnlocked", "Number of vnodes skipped for vget failure" }, { "segs_reclaimed", "Number of segments reclaimed" }, }; sysctl_createv(clog, 0, NULL, NULL, CTLFLAG_PERMANENT, CTLTYPE_NODE, "lfs", SYSCTL_DESCR("Log-structured file system"), NULL, 0, NULL, 0, CTL_VFS, 5, CTL_EOL); /* * XXX the "5" above could be dynamic, thereby eliminating one * more instance of the "number to vfs" mapping problem, but * "5" is the order as taken from sys/mount.h */ sysctl_createv(clog, 0, NULL, NULL, CTLFLAG_PERMANENT|CTLFLAG_READWRITE, CTLTYPE_INT, "flushindir", NULL, NULL, 0, &lfs_writeindir, 0, CTL_VFS, 5, LFS_WRITEINDIR, CTL_EOL); sysctl_createv(clog, 0, NULL, NULL, CTLFLAG_PERMANENT|CTLFLAG_READWRITE, CTLTYPE_INT, "clean_vnhead", NULL, NULL, 0, &lfs_clean_vnhead, 0, CTL_VFS, 5, LFS_CLEAN_VNHEAD, CTL_EOL); sysctl_createv(clog, 0, NULL, NULL, CTLFLAG_PERMANENT|CTLFLAG_READWRITE, CTLTYPE_INT, "dostats", SYSCTL_DESCR("Maintain statistics on LFS operations"), sysctl_lfs_dostats, 0, &lfs_dostats, 0, CTL_VFS, 5, LFS_DOSTATS, CTL_EOL); sysctl_createv(clog, 0, NULL, NULL, CTLFLAG_PERMANENT|CTLFLAG_READWRITE, CTLTYPE_INT, "pagetrip", SYSCTL_DESCR("How many dirty pages in fs triggers" " a flush"), NULL, 0, &lfs_fs_pagetrip, 0, CTL_VFS, 5, LFS_FS_PAGETRIP, CTL_EOL); sysctl_createv(clog, 0, NULL, NULL, CTLFLAG_PERMANENT|CTLFLAG_READWRITE, CTLTYPE_INT, "ignore_lazy_sync", SYSCTL_DESCR("Lazy Sync is ignored entirely"), NULL, 0, &lfs_ignore_lazy_sync, 0, CTL_VFS, 5, LFS_IGNORE_LAZY_SYNC, CTL_EOL); #ifdef LFS_KERNEL_RFW sysctl_createv(clog, 0, NULL, NULL, CTLFLAG_PERMANENT|CTLFLAG_READWRITE, CTLTYPE_INT, "rfw", SYSCTL_DESCR("Use in-kernel roll-forward on mount"), NULL, 0, &lfs_do_rfw, 0, CTL_VFS, 5, LFS_DO_RFW, CTL_EOL); #endif sysctl_createv(clog, 0, NULL, NULL, CTLFLAG_PERMANENT, CTLTYPE_NODE, "stats", SYSCTL_DESCR("Debugging options"), NULL, 0, NULL, 0, CTL_VFS, 5, LFS_STATS, CTL_EOL); for (i = 0; i < sizeof(struct lfs_stats) / sizeof(u_int); i++) { sysctl_createv(clog, 0, NULL, NULL, CTLFLAG_PERMANENT|CTLFLAG_READONLY, CTLTYPE_INT, stat_names[i].sname, SYSCTL_DESCR(stat_names[i].lname), NULL, 0, &(((u_int *)&lfs_stats.segsused)[i]), 0, CTL_VFS, 5, LFS_STATS, i, CTL_EOL); } #ifdef DEBUG sysctl_createv(clog, 0, NULL, NULL, CTLFLAG_PERMANENT, CTLTYPE_NODE, "debug", SYSCTL_DESCR("Debugging options"), NULL, 0, NULL, 0, CTL_VFS, 5, LFS_DEBUGLOG, CTL_EOL); for (i = 0; i < DLOG_MAX; i++) { sysctl_createv(clog, 0, NULL, NULL, CTLFLAG_PERMANENT|CTLFLAG_READWRITE, CTLTYPE_INT, dlog_names[i].sname, SYSCTL_DESCR(dlog_names[i].lname), NULL, 0, &(lfs_debug_log_subsys[i]), 0, CTL_VFS, 5, LFS_DEBUGLOG, i, CTL_EOL); } #endif } /* old cleaner syscall interface. see VOP_FCNTL() */ static const struct syscall_package lfs_syscalls[] = { { SYS_lfs_bmapv, 0, (sy_call_t *)sys_lfs_bmapv }, { SYS_lfs_markv, 0, (sy_call_t *)sys_lfs_markv }, { SYS___lfs_segwait50, 0, (sy_call_t *)sys___lfs_segwait50 }, { SYS_lfs_segclean, 0, (sy_call_t *)sys_lfs_segclean }, { 0, 0, NULL }, }; static int lfs_modcmd(modcmd_t cmd, void *arg) { int error; switch (cmd) { case MODULE_CMD_INIT: error = syscall_establish(NULL, lfs_syscalls); if (error) return error; error = vfs_attach(&lfs_vfsops); if (error != 0) { syscall_disestablish(NULL, lfs_syscalls); break; } cv_init(&lfs_allclean_wakeup, "segment"); break; case MODULE_CMD_FINI: error = vfs_detach(&lfs_vfsops); if (error != 0) break; syscall_disestablish(NULL, lfs_syscalls); cv_destroy(&lfs_allclean_wakeup); break; default: error = ENOTTY; break; } return (error); } /* * XXX Same structure as FFS inodes? Should we share a common pool? */ struct pool lfs_inode_pool; struct pool lfs_dinode_pool; struct pool lfs_inoext_pool; struct pool lfs_lbnentry_pool; /* * The writer daemon. UVM keeps track of how many dirty pages we are holding * in lfs_subsys_pages; the daemon flushes the filesystem when this value * crosses the (user-defined) threshold LFS_MAX_PAGES. */ static void lfs_writerd(void *arg) { mount_iterator_t *iter; struct mount *mp; struct lfs *fs; struct vfsops *vfs = NULL; int fsflags; int lfsc; int wrote_something = 0; mutex_enter(&lfs_lock); KASSERTMSG(lfs_writer_daemon == NULL, "more than one LFS writer daemon"); lfs_writer_daemon = curlwp; mutex_exit(&lfs_lock); /* Take an extra reference to the LFS vfsops. */ vfs = vfs_getopsbyname(MOUNT_LFS); mutex_enter(&lfs_lock); for (;;) { KASSERT(mutex_owned(&lfs_lock)); if (wrote_something == 0) cv_timedwait(&lfs_writerd_cv, &lfs_lock, hz/10 + 1); KASSERT(mutex_owned(&lfs_lock)); wrote_something = 0; /* * If global state wants a flush, flush everything. */ if (lfs_do_flush || locked_queue_count > LFS_MAX_BUFS || locked_queue_bytes > LFS_MAX_BYTES || lfs_subsys_pages > LFS_MAX_PAGES) { if (lfs_do_flush) { DLOG((DLOG_FLUSH, "lfs_writerd: lfs_do_flush\n")); } if (locked_queue_count > LFS_MAX_BUFS) { DLOG((DLOG_FLUSH, "lfs_writerd: lqc = %d, max %d\n", locked_queue_count, LFS_MAX_BUFS)); } if (locked_queue_bytes > LFS_MAX_BYTES) { DLOG((DLOG_FLUSH, "lfs_writerd: lqb = %ld, max %ld\n", locked_queue_bytes, LFS_MAX_BYTES)); } if (lfs_subsys_pages > LFS_MAX_PAGES) { DLOG((DLOG_FLUSH, "lfs_writerd: lssp = %d, max %d\n", lfs_subsys_pages, LFS_MAX_PAGES)); } lfs_flush(NULL, SEGM_WRITERD, 0); lfs_do_flush = 0; KASSERT(mutex_owned(&lfs_lock)); continue; } KASSERT(mutex_owned(&lfs_lock)); mutex_exit(&lfs_lock); /* * Look through the list of LFSs to see if any of them * have requested pageouts. */ mountlist_iterator_init(&iter); lfsc = 0; while ((mp = mountlist_iterator_next(iter)) != NULL) { KASSERT(!mutex_owned(&lfs_lock)); if (strncmp(mp->mnt_stat.f_fstypename, MOUNT_LFS, sizeof(mp->mnt_stat.f_fstypename)) == 0) { ++lfsc; fs = VFSTOULFS(mp)->um_lfs; daddr_t ooffset = 0; fsflags = SEGM_SINGLE; mutex_enter(&lfs_lock); ooffset = lfs_sb_getoffset(fs); if (lfs_sb_getnextseg(fs) < lfs_sb_getcurseg(fs) && fs->lfs_nowrap) { /* Don't try to write if we're suspended */ mutex_exit(&lfs_lock); continue; } if (LFS_STARVED_FOR_SEGS(fs)) { mutex_exit(&lfs_lock); DLOG((DLOG_FLUSH, "lfs_writerd: need cleaning before writing possible\n")); lfs_wakeup_cleaner(fs); continue; } if ((fs->lfs_dirvcount > LFS_MAX_FSDIROP(fs) || lfs_dirvcount > LFS_MAX_DIROP) && fs->lfs_dirops == 0) { fsflags &= ~SEGM_SINGLE; fsflags |= SEGM_CKP; DLOG((DLOG_FLUSH, "lfs_writerd: checkpoint\n")); lfs_flush_fs(fs, fsflags); } else if (fs->lfs_pdflush) { DLOG((DLOG_FLUSH, "lfs_writerd: pdflush set\n")); lfs_flush_fs(fs, fsflags); } else if (!TAILQ_EMPTY(&fs->lfs_pchainhd)) { DLOG((DLOG_FLUSH, "lfs_writerd: pchain non-empty\n")); mutex_exit(&lfs_lock); lfs_writer_enter(fs, "wrdirop"); lfs_flush_pchain(fs); lfs_writer_leave(fs); mutex_enter(&lfs_lock); } if (lfs_sb_getoffset(fs) != ooffset) ++wrote_something; mutex_exit(&lfs_lock); } KASSERT(!mutex_owned(&lfs_lock)); } if (lfsc == 0) { mutex_enter(&lfs_lock); lfs_writer_daemon = NULL; mutex_exit(&lfs_lock); mountlist_iterator_destroy(iter); break; } mountlist_iterator_destroy(iter); mutex_enter(&lfs_lock); } KASSERT(!mutex_owned(&lfs_lock)); /* Give up our extra reference so the module can be unloaded. */ mutex_enter(&vfs_list_lock); if (vfs != NULL) vfs->vfs_refcount--; mutex_exit(&vfs_list_lock); /* Done! */ kthread_exit(0); } /* * Initialize the filesystem, most work done by ulfs_init. */ void lfs_init(void) { /* * XXX: should we use separate pools for 32-bit and 64-bit * dinodes? */ malloc_type_attach(M_SEGMENT); pool_init(&lfs_inode_pool, sizeof(struct inode), 0, 0, 0, "lfsinopl", &pool_allocator_nointr, IPL_NONE); pool_init(&lfs_dinode_pool, sizeof(union lfs_dinode), 0, 0, 0, "lfsdinopl", &pool_allocator_nointr, IPL_NONE); pool_init(&lfs_inoext_pool, sizeof(struct lfs_inode_ext), 8, 0, 0, "lfsinoextpl", &pool_allocator_nointr, IPL_NONE); pool_init(&lfs_lbnentry_pool, sizeof(struct lbnentry), 0, 0, 0, "lfslbnpool", &pool_allocator_nointr, IPL_NONE); ulfs_init(); #ifdef DEBUG memset(lfs_log, 0, sizeof(lfs_log)); #endif mutex_init(&lfs_lock, MUTEX_DEFAULT, IPL_NONE); cv_init(&lfs_writerd_cv, "lfswrite"); cv_init(&locked_queue_cv, "lfsbuf"); cv_init(&lfs_writing_cv, "lfsflush"); } void lfs_reinit(void) { ulfs_reinit(); } void lfs_done(void) { ulfs_done(); mutex_destroy(&lfs_lock); cv_destroy(&lfs_writerd_cv); cv_destroy(&locked_queue_cv); cv_destroy(&lfs_writing_cv); pool_destroy(&lfs_inode_pool); pool_destroy(&lfs_dinode_pool); pool_destroy(&lfs_inoext_pool); pool_destroy(&lfs_lbnentry_pool); malloc_type_detach(M_SEGMENT); } /* * Called by main() when ulfs is going to be mounted as root. */ int lfs_mountroot(void) { extern struct vnode *rootvp; struct lfs *fs = NULL; /* LFS */ struct mount *mp; struct lwp *l = curlwp; struct ulfsmount *ump; int error; if (device_class(root_device) != DV_DISK) return (ENODEV); if (rootdev == NODEV) return (ENODEV); if ((error = vfs_rootmountalloc(MOUNT_LFS, "root_device", &mp))) { vrele(rootvp); return (error); } if ((error = lfs_mountfs(rootvp, mp, l))) { vfs_unbusy(mp); vfs_rele(mp); return (error); } mountlist_append(mp); ump = VFSTOULFS(mp); fs = ump->um_lfs; lfs_sb_setfsmnt(fs, mp->mnt_stat.f_mntonname); (void)lfs_statvfs(mp, &mp->mnt_stat); vfs_unbusy(mp); setrootfstime((time_t)lfs_sb_gettstamp(VFSTOULFS(mp)->um_lfs)); return (0); } /* * VFS Operations. * * mount system call */ int lfs_mount(struct mount *mp, const char *path, void *data, size_t *data_len) { struct lwp *l = curlwp; struct vnode *devvp; struct ulfs_args *args = data; struct ulfsmount *ump = NULL; struct lfs *fs = NULL; /* LFS */ int error = 0, update; mode_t accessmode; if (args == NULL) return EINVAL; if (*data_len < sizeof *args) return EINVAL; if (mp->mnt_flag & MNT_GETARGS) { ump = VFSTOULFS(mp); if (ump == NULL) return EIO; args->fspec = NULL; *data_len = sizeof *args; return 0; } update = mp->mnt_flag & MNT_UPDATE; /* Check arguments */ if (args->fspec != NULL) { /* * Look up the name and verify that it's sane. */ error = namei_simple_user(args->fspec, NSM_FOLLOW_NOEMULROOT, &devvp); if (error != 0) return (error); if (!update) { /* * Be sure this is a valid block device */ if (devvp->v_type != VBLK) error = ENOTBLK; else if (bdevsw_lookup(devvp->v_rdev) == NULL) error = ENXIO; } else { /* * Be sure we're still naming the same device * used for our initial mount * * XXX dholland 20151010: if namei gives us a * different vnode for the same device, * wouldn't it be better to use it going * forward rather than ignore it in favor of * the old one? */ ump = VFSTOULFS(mp); fs = ump->um_lfs; if (devvp != fs->lfs_devvp) { if (devvp->v_rdev != fs->lfs_devvp->v_rdev) error = EINVAL; else { vrele(devvp); devvp = fs->lfs_devvp; vref(devvp); } } } } else { if (!update) { /* New mounts must have a filename for the device */ return (EINVAL); } else { /* Use the extant mount */ ump = VFSTOULFS(mp); fs = ump->um_lfs; devvp = fs->lfs_devvp; vref(devvp); } } /* * If mount by non-root, then verify that user has necessary * permissions on the device. */ if (error == 0) { accessmode = VREAD; if (update ? (mp->mnt_iflag & IMNT_WANTRDWR) != 0 : (mp->mnt_flag & MNT_RDONLY) == 0) accessmode |= VWRITE; vn_lock(devvp, LK_EXCLUSIVE | LK_RETRY); error = kauth_authorize_system(l->l_cred, KAUTH_SYSTEM_MOUNT, KAUTH_REQ_SYSTEM_MOUNT_DEVICE, mp, devvp, KAUTH_ARG(accessmode)); VOP_UNLOCK(devvp); } if (error) { vrele(devvp); return (error); } if (!update) { int flags; if (mp->mnt_flag & MNT_RDONLY) flags = FREAD; else flags = FREAD|FWRITE; vn_lock(devvp, LK_EXCLUSIVE | LK_RETRY); error = VOP_OPEN(devvp, flags, FSCRED); VOP_UNLOCK(devvp); if (error) goto fail; error = lfs_mountfs(devvp, mp, l); /* LFS */ if (error) { vn_lock(devvp, LK_EXCLUSIVE | LK_RETRY); (void)VOP_CLOSE(devvp, flags, NOCRED); VOP_UNLOCK(devvp); goto fail; } ump = VFSTOULFS(mp); fs = ump->um_lfs; } else { /* * Update the mount. */ /* * The initial mount got a reference on this * device, so drop the one obtained via * namei(), above. */ vrele(devvp); ump = VFSTOULFS(mp); fs = ump->um_lfs; if (!fs->lfs_ronly && (mp->mnt_iflag & IMNT_WANTRDONLY)) { /* * Changing from read/write to read-only. */ int flags = WRITECLOSE; if (mp->mnt_flag & MNT_FORCE) flags |= FORCECLOSE; error = lfs_flushfiles(mp, flags); if (error) return error; fs->lfs_ronly = 1; } else if (fs->lfs_ronly && (mp->mnt_iflag & IMNT_WANTRDWR)) { /* * Changing from read-only to read/write. * Note in the superblocks that we're writing. */ /* XXX: quotas should have been on even if readonly */ if (fs->lfs_use_quota2) { #ifdef LFS_QUOTA2 error = lfs_quota2_mount(mp); #else uprintf("%s: no kernel support for this " "filesystem's quotas\n", mp->mnt_stat.f_mntonname); if (mp->mnt_flag & MNT_FORCE) { uprintf("%s: mounting anyway; " "fsck afterwards\n", mp->mnt_stat.f_mntonname); } else { error = EINVAL; } #endif if (error) { return error; } } fs->lfs_ronly = 0; if (lfs_sb_getpflags(fs) & LFS_PF_CLEAN) { lfs_sb_setpflags(fs, lfs_sb_getpflags(fs) & ~LFS_PF_CLEAN); lfs_writesuper(fs, lfs_sb_getsboff(fs, 0)); lfs_writesuper(fs, lfs_sb_getsboff(fs, 1)); } } if (args->fspec == NULL) return 0; } error = set_statvfs_info(path, UIO_USERSPACE, args->fspec, UIO_USERSPACE, mp->mnt_op->vfs_name, mp, l); if (error == 0) lfs_sb_setfsmnt(fs, mp->mnt_stat.f_mntonname); return error; fail: vrele(devvp); return (error); } /* * Helper for mountfs. Note that the fs pointer may be a dummy one * pointing into a superblock buffer. (Which is gross; see below.) */ static int lfs_checkmagic(struct lfs *fs) { switch (fs->lfs_dlfs_u.u_32.dlfs_magic) { case LFS_MAGIC: fs->lfs_is64 = false; fs->lfs_dobyteswap = false; break; case LFS64_MAGIC: fs->lfs_is64 = true; fs->lfs_dobyteswap = false; break; #ifdef LFS_EI case LFS_MAGIC_SWAPPED: fs->lfs_is64 = false; fs->lfs_dobyteswap = true; break; case LFS64_MAGIC_SWAPPED: fs->lfs_is64 = true; fs->lfs_dobyteswap = true; break; #endif default: /* XXX needs translation */ return EINVAL; } return 0; } /* * Common code for mount and mountroot * LFS specific */ int lfs_mountfs(struct vnode *devvp, struct mount *mp, struct lwp *l) { struct lfs *primarysb, *altsb, *thesb; struct buf *primarybuf, *altbuf; struct lfs *fs; struct ulfsmount *ump; struct vnode *vp; dev_t dev; int error, i, ronly, fsbsize; kauth_cred_t cred; CLEANERINFO *cip; SEGUSE *sup; daddr_t sb_addr; ino_t *orphan; size_t norphan; cred = l ? l->l_cred : NOCRED; /* The superblock is supposed to be 512 bytes. */ __CTASSERT(sizeof(struct dlfs) == DEV_BSIZE); /* * Flush out any old buffers remaining from a previous use. */ vn_lock(devvp, LK_EXCLUSIVE | LK_RETRY); error = vinvalbuf(devvp, V_SAVE, cred, l, 0, 0); VOP_UNLOCK(devvp); if (error) return (error); ronly = (mp->mnt_flag & MNT_RDONLY) != 0; /* Don't free random space on error. */ primarybuf = NULL; altbuf = NULL; ump = NULL; sb_addr = LFS_LABELPAD / DEV_BSIZE; while (1) { /* * Read in the superblock. * * Note that because LFS_SBPAD is substantially larger * (8K) than the actual on-disk superblock (512 bytes) * the buffer contains enough space to be used as a * whole struct lfs (in-memory superblock) - we do this * only so we can set and use the is64 and dobyteswap * members. XXX this is gross and the logic here should * be reworked. */ error = bread(devvp, sb_addr, LFS_SBPAD, 0, &primarybuf); if (error) goto out; primarysb = (struct lfs *)primarybuf->b_data; /* Check the basics. */ error = lfs_checkmagic(primarysb); if (error) { DLOG((DLOG_MOUNT, "lfs_mountfs: primary superblock wrong magic\n")); goto out; } if (lfs_sb_getbsize(primarysb) > MAXBSIZE || lfs_sb_getversion(primarysb) > LFS_VERSION || lfs_sb_getbsize(primarysb) < sizeof(struct dlfs)) { DLOG((DLOG_MOUNT, "lfs_mountfs: primary superblock sanity failed\n")); /* XXX needs translation */ error = EINVAL; goto out; } if (lfs_sb_getinodefmt(primarysb) > LFS_MAXINODEFMT) { DLOG((DLOG_MOUNT, "lfs_mountfs: unknown inode format %d\n", lfs_sb_getinodefmt(primarysb))); error = EINVAL; goto out; } if (lfs_sb_getversion(primarysb) == 1) fsbsize = DEV_BSIZE; else { fsbsize = 1 << lfs_sb_getffshift(primarysb); /* * Could be, if the frag size is large enough, that we * don't have the "real" primary superblock. If that's * the case, get the real one, and try again. */ if (sb_addr != (lfs_sb_getsboff(primarysb, 0) << (lfs_sb_getffshift(primarysb) - DEV_BSHIFT))) { DLOG((DLOG_MOUNT, "lfs_mountfs: sb daddr" " 0x%llx is not right, trying 0x%llx\n", (long long)sb_addr, (long long)(lfs_sb_getsboff(primarysb, 0) << (lfs_sb_getffshift(primarysb) - DEV_BSHIFT)))); sb_addr = lfs_sb_getsboff(primarysb, 0) << (lfs_sb_getffshift(primarysb) - DEV_BSHIFT); brelse(primarybuf, BC_INVAL); continue; } } break; } /* * Check the second superblock to see which is newer; then mount * using the older of the two. This is necessary to ensure that * the filesystem is valid if it was not unmounted cleanly. */ if (lfs_sb_getsboff(primarysb, 1) && lfs_sb_getsboff(primarysb, 1) - LFS_LABELPAD / fsbsize > LFS_SBPAD / fsbsize) { error = bread(devvp, lfs_sb_getsboff(primarysb, 1) * (fsbsize / DEV_BSIZE), LFS_SBPAD, 0, &altbuf); if (error) goto out; altsb = (struct lfs *)altbuf->b_data; /* * Note: this used to do the sanity check only if the * timestamp/serial comparison required use of altsb; * this way is less tolerant, but if altsb is corrupted * enough that the magic number, version, and blocksize * are bogus, why would the timestamp or serial fields * mean anything either? If this kind of thing happens, * you need to fsck anyway. */ error = lfs_checkmagic(altsb); if (error) goto out; /* Check the basics. */ if (lfs_sb_getbsize(altsb) > MAXBSIZE || lfs_sb_getversion(altsb) > LFS_VERSION || lfs_sb_getbsize(altsb) < sizeof(struct dlfs)) { DLOG((DLOG_MOUNT, "lfs_mountfs: alt superblock" " sanity failed\n")); error = EINVAL; /* XXX needs translation */ goto out; } if (lfs_sb_getversion(primarysb) == 1) { /* 1s resolution comparison */ if (lfs_sb_gettstamp(altsb) < lfs_sb_gettstamp(primarysb)) thesb = altsb; else thesb = primarysb; } else { /* monotonic infinite-resolution comparison */ if (lfs_sb_getserial(altsb) < lfs_sb_getserial(primarysb)) thesb = altsb; else thesb = primarysb; } } else { DLOG((DLOG_MOUNT, "lfs_mountfs: invalid alt superblock location" " daddr=0x%x\n", lfs_sb_getsboff(primarysb, 1))); error = EINVAL; goto out; } /* * Allocate the mount structure, copy the superblock into it. * Note that the 32-bit and 64-bit superblocks are the same size. */ fs = kmem_zalloc(sizeof(struct lfs), KM_SLEEP); memcpy(&fs->lfs_dlfs_u.u_32, &thesb->lfs_dlfs_u.u_32, sizeof(struct dlfs)); fs->lfs_is64 = thesb->lfs_is64; fs->lfs_dobyteswap = thesb->lfs_dobyteswap; fs->lfs_hasolddirfmt = false; /* set for real below */ /* Compatibility */ if (lfs_sb_getversion(fs) < 2) { lfs_sb_setsumsize(fs, LFS_V1_SUMMARY_SIZE); lfs_sb_setibsize(fs, lfs_sb_getbsize(fs)); lfs_sb_sets0addr(fs, lfs_sb_getsboff(fs, 0)); lfs_sb_settstamp(fs, lfs_sb_getotstamp(fs)); lfs_sb_setfsbtodb(fs, 0); } if (lfs_sb_getresvseg(fs) == 0) lfs_sb_setresvseg(fs, MIN(lfs_sb_getminfreeseg(fs) - 1, \ MAX(MIN_RESV_SEGS, lfs_sb_getminfreeseg(fs) / 2 + 1))); /* * If we aren't going to be able to write meaningfully to this * filesystem, and were not mounted readonly, bomb out now. */ if (lfs_fsbtob(fs, LFS_NRESERVE(fs)) > LFS_MAX_BYTES && !ronly) { DLOG((DLOG_MOUNT, "lfs_mount: to mount this filesystem read/write," " we need BUFPAGES >= %lld\n", (long long)((bufmem_hiwater / bufmem_lowater) * LFS_INVERSE_MAX_BYTES( lfs_fsbtob(fs, LFS_NRESERVE(fs))) >> PAGE_SHIFT))); kmem_free(fs, sizeof(struct lfs)); error = EFBIG; /* XXX needs translation */ goto out; } /* Before rolling forward, lock so vget will sleep for other procs */ if (l != NULL) { fs->lfs_flags = LFS_NOTYET; fs->lfs_rfpid = l->l_proc->p_pid; } ump = kmem_zalloc(sizeof(*ump), KM_SLEEP); ump->um_lfs = fs; ump->um_fstype = fs->lfs_is64 ? ULFS2 : ULFS1; /* ump->um_cleaner_thread = NULL; */ brelse(primarybuf, BC_INVAL); brelse(altbuf, BC_INVAL); primarybuf = NULL; altbuf = NULL; /* Set up the I/O information */ fs->lfs_devbsize = DEV_BSIZE; fs->lfs_iocount = 0; fs->lfs_diropwait = 0; fs->lfs_activesb = 0; lfs_sb_setuinodes(fs, 0); fs->lfs_ravail = 0; fs->lfs_favail = 0; fs->lfs_sbactive = 0; /* Set up the ifile and lock aflags */ fs->lfs_doifile = 0; fs->lfs_writer = 0; fs->lfs_dirops = 0; fs->lfs_nadirop = 0; fs->lfs_seglock = 0; fs->lfs_pdflush = 0; fs->lfs_sleepers = 0; fs->lfs_pages = 0; rw_init(&fs->lfs_fraglock); rw_init(&fs->lfs_iflock); cv_init(&fs->lfs_sleeperscv, "lfs_slp"); cv_init(&fs->lfs_diropscv, "lfs_dirop"); cv_init(&fs->lfs_stopcv, "lfsstop"); cv_init(&fs->lfs_nextsegsleep, "segment"); /* Set the file system readonly/modify bits. */ fs->lfs_ronly = ronly; if (ronly == 0) fs->lfs_fmod = 1; /* Device we're using */ dev = devvp->v_rdev; fs->lfs_dev = dev; fs->lfs_devvp = devvp; /* ulfs-level information */ fs->um_flags = 0; fs->um_bptrtodb = lfs_sb_getffshift(fs) - DEV_BSHIFT; fs->um_seqinc = lfs_sb_getfrag(fs); fs->um_nindir = lfs_sb_getnindir(fs); fs->um_lognindir = ffs(lfs_sb_getnindir(fs)) - 1; fs->um_maxsymlinklen = lfs_sb_getmaxsymlinklen(fs); fs->um_dirblksiz = LFS_DIRBLKSIZ; fs->um_maxfilesize = lfs_sb_getmaxfilesize(fs); /* quota stuff */ /* XXX: these need to come from the on-disk superblock to be used */ fs->lfs_use_quota2 = 0; fs->lfs_quota_magic = 0; fs->lfs_quota_flags = 0; fs->lfs_quotaino[0] = 0; fs->lfs_quotaino[1] = 0; /* Initialize the mount structure. */ mp->mnt_data = ump; mp->mnt_stat.f_fsidx.__fsid_val[0] = (long)dev; mp->mnt_stat.f_fsidx.__fsid_val[1] = makefstype(MOUNT_LFS); mp->mnt_stat.f_fsid = mp->mnt_stat.f_fsidx.__fsid_val[0]; mp->mnt_stat.f_namemax = LFS_MAXNAMLEN; mp->mnt_stat.f_iosize = lfs_sb_getbsize(fs); mp->mnt_flag |= MNT_LOCAL; mp->mnt_iflag |= IMNT_SHRLOOKUP; mp->mnt_fs_bshift = lfs_sb_getbshift(fs); mp->mnt_iflag |= IMNT_CAN_RWTORO; if (fs->um_maxsymlinklen > 0) mp->mnt_iflag |= IMNT_DTYPE; else fs->lfs_hasolddirfmt = true; ump->um_mountp = mp; for (i = 0; i < ULFS_MAXQUOTAS; i++) ump->um_quotas[i] = NULLVP; spec_node_setmountedfs(devvp, mp); /* Set up reserved memory for pageout */ lfs_setup_resblks(fs); /* Set up vdirop tailq */ TAILQ_INIT(&fs->lfs_dchainhd); /* and paging tailq */ TAILQ_INIT(&fs->lfs_pchainhd); /* and delayed segment accounting for truncation list */ LIST_INIT(&fs->lfs_segdhd); /* * We use the ifile vnode for almost every operation. Instead of * retrieving it from the hash table each time we retrieve it here, * artificially increment the reference count and keep a pointer * to it in the incore copy of the superblock. */ if ((error = VFS_VGET(mp, LFS_IFILE_INUM, LK_EXCLUSIVE, &vp)) != 0) { DLOG((DLOG_MOUNT, "lfs_mountfs: ifile vget failed, error=%d\n", error)); goto out; } fs->lfs_ivnode = vp; vref(vp); /* Set up inode bitmap, order free list, and gather orphans. */ lfs_order_freelist(fs, &orphan, &norphan); /* Set up segment usage flags for the autocleaner. */ fs->lfs_nactive = 0; fs->lfs_suflags = malloc(2 * sizeof(u_int32_t *), M_SEGMENT, M_WAITOK); fs->lfs_suflags[0] = malloc(lfs_sb_getnseg(fs) * sizeof(u_int32_t), M_SEGMENT, M_WAITOK); fs->lfs_suflags[1] = malloc(lfs_sb_getnseg(fs) * sizeof(u_int32_t), M_SEGMENT, M_WAITOK); memset(fs->lfs_suflags[1], 0, lfs_sb_getnseg(fs) * sizeof(u_int32_t)); for (i = 0; i < lfs_sb_getnseg(fs); i++) { int changed; struct buf *bp; LFS_SEGENTRY(sup, fs, i, bp); changed = 0; if (!ronly) { if (sup->su_nbytes == 0 && !(sup->su_flags & SEGUSE_EMPTY)) { sup->su_flags |= SEGUSE_EMPTY; ++changed; } else if (!(sup->su_nbytes == 0) && (sup->su_flags & SEGUSE_EMPTY)) { sup->su_flags &= ~SEGUSE_EMPTY; ++changed; } if (sup->su_flags & (SEGUSE_ACTIVE|SEGUSE_INVAL)) { sup->su_flags &= ~(SEGUSE_ACTIVE|SEGUSE_INVAL); ++changed; } } fs->lfs_suflags[0][i] = sup->su_flags; if (changed) LFS_WRITESEGENTRY(sup, fs, i, bp); else brelse(bp, 0); } /* Free the orphans we discovered while ordering the freelist. */ lfs_free_orphans(fs, orphan, norphan); /* * XXX: if the fs has quotas, quotas should be on even if * readonly. Otherwise you can't query the quota info! * However, that's not how the quota2 code got written and I * don't know if it'll behave itself if enabled while * readonly, so for now use the same enable logic as ffs. * * XXX: also, if you use the -f behavior allowed here (and * equivalently above for remount) it will corrupt the fs. It * ought not to allow that. It should allow mounting readonly * if there are quotas and the kernel doesn't have the quota * code, but only readonly. * * XXX: and if you use the -f behavior allowed here it will * likely crash at unmount time (or remount time) because we * think quotas are active. * * Although none of this applies until there's a way to set * lfs_use_quota2 and have quotas in the fs at all. */ if (!ronly && fs->lfs_use_quota2) { #ifdef LFS_QUOTA2 error = lfs_quota2_mount(mp); #else uprintf("%s: no kernel support for this filesystem's quotas\n", mp->mnt_stat.f_mntonname); if (mp->mnt_flag & MNT_FORCE) { uprintf("%s: mounting anyway; fsck afterwards\n", mp->mnt_stat.f_mntonname); } else { error = EINVAL; } #endif if (error) { /* XXX XXX must clean up the stuff immediately above */ printf("lfs_mountfs: sorry, leaking some memory\n"); goto out; } } #ifdef LFS_KERNEL_RFW lfs_roll_forward(fs, mp, l); #endif /* If writing, sb is not clean; record in case of immediate crash */ if (!fs->lfs_ronly) { lfs_sb_setpflags(fs, lfs_sb_getpflags(fs) & ~LFS_PF_CLEAN); lfs_writesuper(fs, lfs_sb_getsboff(fs, 0)); lfs_writesuper(fs, lfs_sb_getsboff(fs, 1)); } /* Allow vget now that roll-forward is complete */ fs->lfs_flags &= ~(LFS_NOTYET); wakeup(&fs->lfs_flags); /* * Initialize the ifile cleaner info with information from * the superblock. */ { struct buf *bp; LFS_CLEANERINFO(cip, fs, bp); lfs_ci_setclean(fs, cip, lfs_sb_getnclean(fs)); lfs_ci_setdirty(fs, cip, lfs_sb_getnseg(fs) - lfs_sb_getnclean(fs)); lfs_ci_setavail(fs, cip, lfs_sb_getavail(fs)); lfs_ci_setbfree(fs, cip, lfs_sb_getbfree(fs)); (void) LFS_BWRITE_LOG(bp); /* Ifile */ } /* * Mark the current segment as ACTIVE, since we're going to * be writing to it. */ { struct buf *bp; LFS_SEGENTRY(sup, fs, lfs_dtosn(fs, lfs_sb_getoffset(fs)), bp); sup->su_flags |= SEGUSE_DIRTY | SEGUSE_ACTIVE; fs->lfs_nactive++; LFS_WRITESEGENTRY(sup, fs, lfs_dtosn(fs, lfs_sb_getoffset(fs)), bp); /* Ifile */ } /* Now that roll-forward is done, unlock the Ifile */ vput(vp); /* Start the pagedaemon-anticipating daemon */ mutex_enter(&lfs_lock); if (lfs_writer_daemon == NULL && kthread_create(PRI_BIO, 0, NULL, lfs_writerd, NULL, NULL, "lfs_writer") != 0) panic("fork lfs_writer"); mutex_exit(&lfs_lock); printf("WARNING: the log-structured file system is experimental\n" "WARNING: it may cause system crashes and/or corrupt data\n"); return (0); out: if (primarybuf) brelse(primarybuf, BC_INVAL); if (altbuf) brelse(altbuf, BC_INVAL); if (ump) { kmem_free(ump->um_lfs, sizeof(struct lfs)); kmem_free(ump, sizeof(*ump)); mp->mnt_data = NULL; } return (error); } /* * unmount system call */ int lfs_unmount(struct mount *mp, int mntflags) { struct ulfsmount *ump; struct lfs *fs; int error, ronly; ump = VFSTOULFS(mp); fs = ump->um_lfs; error = lfs_flushfiles(mp, mntflags & MNT_FORCE ? FORCECLOSE : 0); if (error) return error; /* Finish with the Ifile, now that we're done with it */ vgone(fs->lfs_ivnode); ronly = !fs->lfs_ronly; if (fs->lfs_devvp->v_type != VBAD) spec_node_setmountedfs(fs->lfs_devvp, NULL); vn_lock(fs->lfs_devvp, LK_EXCLUSIVE | LK_RETRY); error = VOP_CLOSE(fs->lfs_devvp, ronly ? FREAD : FREAD|FWRITE, NOCRED); vput(fs->lfs_devvp); /* Complain about page leakage */ if (fs->lfs_pages > 0) printf("lfs_unmount: still claim %d pages (%d in subsystem)\n", fs->lfs_pages, lfs_subsys_pages); /* Free per-mount data structures */ free(fs->lfs_ino_bitmap, M_SEGMENT); free(fs->lfs_suflags[0], M_SEGMENT); free(fs->lfs_suflags[1], M_SEGMENT); free(fs->lfs_suflags, M_SEGMENT); lfs_free_resblks(fs); cv_destroy(&fs->lfs_sleeperscv); cv_destroy(&fs->lfs_diropscv); cv_destroy(&fs->lfs_stopcv); cv_destroy(&fs->lfs_nextsegsleep); rw_destroy(&fs->lfs_fraglock); rw_destroy(&fs->lfs_iflock); kmem_free(fs, sizeof(struct lfs)); kmem_free(ump, sizeof(*ump)); mp->mnt_data = NULL; mp->mnt_flag &= ~MNT_LOCAL; return (error); } static int lfs_flushfiles(struct mount *mp, int flags) { struct lwp *l = curlwp; struct ulfsmount *ump; struct lfs *fs; struct vnode *vp; int error; ump = VFSTOULFS(mp); fs = ump->um_lfs; /* Two checkpoints */ if (!fs->lfs_ronly) { lfs_segwrite(mp, SEGM_CKP | SEGM_SYNC); lfs_segwrite(mp, SEGM_CKP | SEGM_SYNC); } /* wake up the cleaner so it can die */ /* XXX: shouldn't this be *after* the error cases below? */ lfs_wakeup_cleaner(fs); mutex_enter(&lfs_lock); while (fs->lfs_sleepers) cv_wait(&fs->lfs_sleeperscv, &lfs_lock); mutex_exit(&lfs_lock); #ifdef LFS_EXTATTR if (ump->um_fstype == ULFS1) { if (ump->um_extattr.uepm_flags & ULFS_EXTATTR_UEPM_STARTED) { ulfs_extattr_stop(mp, curlwp); } if (ump->um_extattr.uepm_flags & ULFS_EXTATTR_UEPM_INITIALIZED) { ulfs_extattr_uepm_destroy(&ump->um_extattr); mp->mnt_flag &= ~MNT_EXTATTR; } } #endif #ifdef LFS_QUOTA if ((error = lfsquota1_umount(mp, flags)) != 0) return (error); #endif #ifdef LFS_QUOTA2 if ((error = lfsquota2_umount(mp, flags)) != 0) return (error); #endif if ((error = vflush(mp, fs->lfs_ivnode, flags)) != 0) return (error); if ((error = VFS_SYNC(mp, 1, l->l_cred)) != 0) return (error); vp = fs->lfs_ivnode; mutex_enter(vp->v_interlock); if (LIST_FIRST(&vp->v_dirtyblkhd)) panic("lfs_unmount: still dirty blocks on ifile vnode"); mutex_exit(vp->v_interlock); /* Explicitly write the superblock, to update serial and pflags */ if (!fs->lfs_ronly) { lfs_sb_setpflags(fs, lfs_sb_getpflags(fs) | LFS_PF_CLEAN); lfs_writesuper(fs, lfs_sb_getsboff(fs, 0)); lfs_writesuper(fs, lfs_sb_getsboff(fs, 1)); } mutex_enter(&lfs_lock); while (fs->lfs_iocount) mtsleep(&fs->lfs_iocount, PRIBIO + 1, "lfs_umount", 0, &lfs_lock); mutex_exit(&lfs_lock); return 0; } /* * Get file system statistics. * * NB: We don't lock to access the superblock here, because it's not * really that important if we get it wrong. */ int lfs_statvfs(struct mount *mp, struct statvfs *sbp) { struct lfs *fs; struct ulfsmount *ump; ump = VFSTOULFS(mp); fs = ump->um_lfs; sbp->f_bsize = lfs_sb_getbsize(fs); sbp->f_frsize = lfs_sb_getfsize(fs); sbp->f_iosize = lfs_sb_getbsize(fs); sbp->f_blocks = LFS_EST_NONMETA(fs) - VTOI(fs->lfs_ivnode)->i_lfs_effnblks; sbp->f_bfree = LFS_EST_BFREE(fs); /* * XXX this should be lfs_sb_getsize (measured in frags) * rather than dsize (measured in diskblocks). However, * getsize needs a format version check (for version 1 it * needs to be blockstofrags'd) so for the moment I'm going to * leave this... it won't fire wrongly as frags are at least * as big as diskblocks. */ KASSERT(sbp->f_bfree <= lfs_sb_getdsize(fs)); #if 0 if (sbp->f_bfree < 0) sbp->f_bfree = 0; #endif sbp->f_bresvd = LFS_EST_RSVD(fs); if (sbp->f_bfree > sbp->f_bresvd) sbp->f_bavail = sbp->f_bfree - sbp->f_bresvd; else sbp->f_bavail = 0; /* XXX: huh? - dholland 20150728 */ sbp->f_files = lfs_sb_getbfree(fs) / lfs_btofsb(fs, lfs_sb_getibsize(fs)) * LFS_INOPB(fs); sbp->f_ffree = sbp->f_files - lfs_sb_getnfiles(fs); sbp->f_favail = sbp->f_ffree; sbp->f_fresvd = 0; copy_statvfs_info(sbp, mp); return (0); } /* * Go through the disk queues to initiate sandbagged IO; * go through the inodes to write those that have been modified; * initiate the writing of the super block if it has been modified. * * Note: we are always called with the filesystem marked `MPBUSY'. */ int lfs_sync(struct mount *mp, int waitfor, kauth_cred_t cred) { int error; struct lfs *fs; fs = VFSTOULFS(mp)->um_lfs; if (fs->lfs_ronly) return 0; /* Snapshots should not hose the syncer */ /* * XXX Sync can block here anyway, since we don't have a very * XXX good idea of how much data is pending. If it's more * XXX than a segment and lfs_nextseg is close to the end of * XXX the log, we'll likely block. */ mutex_enter(&lfs_lock); if (fs->lfs_nowrap && lfs_sb_getnextseg(fs) < lfs_sb_getcurseg(fs)) { mutex_exit(&lfs_lock); return 0; } mutex_exit(&lfs_lock); lfs_writer_enter(fs, "lfs_dirops"); /* All syncs must be checkpoints until roll-forward is implemented. */ DLOG((DLOG_FLUSH, "lfs_sync at 0x%jx\n", (uintmax_t)lfs_sb_getoffset(fs))); error = lfs_segwrite(mp, SEGM_CKP | (waitfor ? SEGM_SYNC : 0)); lfs_writer_leave(fs); #ifdef LFS_QUOTA lfs_qsync(mp); #endif return (error); } /* * Look up an LFS dinode number to find its incore vnode. If not already * in core, read it in from the specified device. Return the inode locked. * Detection and handling of mount points must be done by the calling routine. */ int lfs_vget(struct mount *mp, ino_t ino, int lktype, struct vnode **vpp) { int error; error = vcache_get(mp, &ino, sizeof(ino), vpp); if (error) return error; error = vn_lock(*vpp, lktype); if (error) { vrele(*vpp); *vpp = NULL; return error; } return 0; } /* * Create a new vnode/inode pair and initialize what fields we can. */ static void lfs_init_vnode(struct ulfsmount *ump, ino_t ino, struct vnode *vp) { struct lfs *fs = ump->um_lfs; struct inode *ip; union lfs_dinode *dp; ASSERT_NO_SEGLOCK(fs); /* Initialize the inode. */ ip = pool_get(&lfs_inode_pool, PR_WAITOK); memset(ip, 0, sizeof(*ip)); dp = pool_get(&lfs_dinode_pool, PR_WAITOK); memset(dp, 0, sizeof(*dp)); ip->inode_ext.lfs = pool_get(&lfs_inoext_pool, PR_WAITOK); memset(ip->inode_ext.lfs, 0, sizeof(*ip->inode_ext.lfs)); ip->i_din = dp; ip->i_ump = ump; ip->i_vnode = vp; ip->i_dev = fs->lfs_dev; lfs_dino_setinumber(fs, dp, ino); ip->i_number = ino; ip->i_lfs = fs; ip->i_lfs_effnblks = 0; SPLAY_INIT(&ip->i_lfs_lbtree); ip->i_lfs_nbtree = 0; LIST_INIT(&ip->i_lfs_segdhd); vp->v_tag = VT_LFS; vp->v_op = lfs_vnodeop_p; vp->v_data = ip; } /* * Undo lfs_init_vnode(). */ static void lfs_deinit_vnode(struct ulfsmount *ump, struct vnode *vp) { struct inode *ip = VTOI(vp); pool_put(&lfs_inoext_pool, ip->inode_ext.lfs); pool_put(&lfs_dinode_pool, ip->i_din); pool_put(&lfs_inode_pool, ip); vp->v_data = NULL; } /* * Read an inode from disk and initialize this vnode / inode pair. * Caller assures no other thread will try to load this inode. */ int lfs_loadvnode(struct mount *mp, struct vnode *vp, const void *key, size_t key_len, const void **new_key) { struct lfs *fs; union lfs_dinode *dip; struct inode *ip; struct buf *bp; IFILE *ifp; struct ulfsmount *ump; ino_t ino; daddr_t daddr; int error, retries; struct timespec ts; KASSERT(key_len == sizeof(ino)); memcpy(&ino, key, key_len); memset(&ts, 0, sizeof ts); /* XXX gcc */ ump = VFSTOULFS(mp); fs = ump->um_lfs; /* * If the filesystem is not completely mounted yet, suspend * any access requests (wait for roll-forward to complete). */ mutex_enter(&lfs_lock); while ((fs->lfs_flags & LFS_NOTYET) && curproc->p_pid != fs->lfs_rfpid) mtsleep(&fs->lfs_flags, PRIBIO+1, "lfs_notyet", 0, &lfs_lock); mutex_exit(&lfs_lock); /* Translate the inode number to a disk address. */ if (ino == LFS_IFILE_INUM) daddr = lfs_sb_getidaddr(fs); else { /* XXX bounds-check this too */ LFS_IENTRY(ifp, fs, ino, bp); daddr = lfs_if_getdaddr(fs, ifp); if (lfs_sb_getversion(fs) > 1) { ts.tv_sec = lfs_if_getatime_sec(fs, ifp); ts.tv_nsec = lfs_if_getatime_nsec(fs, ifp); } brelse(bp, 0); if (daddr == LFS_UNUSED_DADDR) return (ENOENT); } /* Allocate/init new vnode/inode. */ lfs_init_vnode(ump, ino, vp); ip = VTOI(vp); /* If the cleaner supplied the inode, use it. */ if (curlwp == fs->lfs_cleaner_thread && fs->lfs_cleaner_hint != NULL && fs->lfs_cleaner_hint->bi_lbn == LFS_UNUSED_LBN) { dip = fs->lfs_cleaner_hint->bi_bp; if (fs->lfs_is64) { error = copyin(dip, &ip->i_din->u_64, sizeof(struct lfs64_dinode)); } else { error = copyin(dip, &ip->i_din->u_32, sizeof(struct lfs32_dinode)); } if (error) { lfs_deinit_vnode(ump, vp); return error; } KASSERT(ip->i_number == ino); goto out; } /* Read in the disk contents for the inode, copy into the inode. */ retries = 0; again: error = bread(fs->lfs_devvp, LFS_FSBTODB(fs, daddr), (lfs_sb_getversion(fs) == 1 ? lfs_sb_getbsize(fs) : lfs_sb_getibsize(fs)), 0, &bp); if (error) { lfs_deinit_vnode(ump, vp); return error; } dip = lfs_ifind(fs, ino, bp); if (dip == NULL) { /* Assume write has not completed yet; try again */ brelse(bp, BC_INVAL); ++retries; if (retries <= LFS_IFIND_RETRIES) { mutex_enter(&lfs_lock); if (fs->lfs_iocount) { DLOG((DLOG_VNODE, "%s: dinode %d not found, retrying...\n", __func__, ino)); (void)mtsleep(&fs->lfs_iocount, PRIBIO + 1, "lfs ifind", 1, &lfs_lock); } else retries = LFS_IFIND_RETRIES; mutex_exit(&lfs_lock); goto again; } #ifdef DEBUG /* If the seglock is held look at the bpp to see what is there anyway */ mutex_enter(&lfs_lock); if (fs->lfs_seglock > 0) { struct buf **bpp; union lfs_dinode *dp; int i; for (bpp = fs->lfs_sp->bpp; bpp != fs->lfs_sp->cbpp; ++bpp) { if ((*bpp)->b_vp == fs->lfs_ivnode && bpp != fs->lfs_sp->bpp) { /* Inode block */ printf("%s: block 0x%" PRIx64 ": ", __func__, (*bpp)->b_blkno); for (i = 0; i < LFS_INOPB(fs); i++) { dp = DINO_IN_BLOCK(fs, (*bpp)->b_data, i); if (lfs_dino_getinumber(fs, dp)) printf("%ju ", (uintmax_t)lfs_dino_getinumber(fs, dp)); } printf("\n"); } } } mutex_exit(&lfs_lock); #endif /* DEBUG */ panic("lfs_loadvnode: dinode not found"); } lfs_copy_dinode(fs, ip->i_din, dip); brelse(bp, 0); out: if (lfs_sb_getversion(fs) > 1) { lfs_dino_setatime(fs, ip->i_din, ts.tv_sec); lfs_dino_setatimensec(fs, ip->i_din, ts.tv_nsec); } lfs_vinit(mp, &vp); *new_key = &ip->i_number; return 0; } /* * Create a new inode and initialize this vnode / inode pair. */ int lfs_newvnode(struct mount *mp, struct vnode *dvp, struct vnode *vp, struct vattr *vap, kauth_cred_t cred, void *extra, size_t *key_len, const void **new_key) { ino_t ino; struct inode *ip; struct ulfsmount *ump; struct lfs *fs; int error, mode, gen; KASSERT(dvp != NULL || vap->va_fileid > 0); KASSERT(dvp != NULL && dvp->v_mount == mp); KASSERT(vap->va_type != VNON); *key_len = sizeof(ino); ump = VFSTOULFS(mp); fs = ump->um_lfs; mode = MAKEIMODE(vap->va_type, vap->va_mode); /* * Allocate fresh inode. With "dvp == NULL" take the inode number * and version from "vap". */ if (dvp == NULL) { ino = vap->va_fileid; gen = vap->va_gen; error = lfs_valloc_fixed(fs, ino, gen); } else { error = lfs_valloc(dvp, mode, cred, &ino, &gen); } if (error) return error; /* Attach inode to vnode. */ lfs_init_vnode(ump, ino, vp); ip = VTOI(vp); mutex_enter(&lfs_lock); LFS_SET_UINO(ip, IN_CHANGE); mutex_exit(&lfs_lock); /* Note no blocks yet */ ip->i_lfs_hiblk = -1; /* Set a new generation number for this inode. */ ip->i_gen = gen; lfs_dino_setgen(fs, ip->i_din, gen); memset(ip->i_lfs_fragsize, 0, ULFS_NDADDR * sizeof(*ip->i_lfs_fragsize)); /* Set uid / gid. */ if (cred == NOCRED || cred == FSCRED) { ip->i_gid = 0; ip->i_uid = 0; } else { ip->i_gid = VTOI(dvp)->i_gid; ip->i_uid = kauth_cred_geteuid(cred); } DIP_ASSIGN(ip, gid, ip->i_gid); DIP_ASSIGN(ip, uid, ip->i_uid); #if defined(LFS_QUOTA) || defined(LFS_QUOTA2) error = lfs_chkiq(ip, 1, cred, 0); if (error) { lfs_vfree(dvp, ino, mode); lfs_deinit_vnode(ump, vp); return error; } #endif /* Set type and finalize. */ ip->i_flags = 0; DIP_ASSIGN(ip, flags, 0); ip->i_mode = mode; DIP_ASSIGN(ip, mode, mode); if (vap->va_rdev != VNOVAL) { /* * Want to be able to use this to make badblock * inodes, so don't truncate the dev number. */ // XXX clean this up if (ump->um_fstype == ULFS1) ip->i_din->u_32.di_rdev = ulfs_rw32(vap->va_rdev, ULFS_MPNEEDSWAP(fs)); else ip->i_din->u_64.di_rdev = ulfs_rw64(vap->va_rdev, ULFS_MPNEEDSWAP(fs)); } lfs_vinit(mp, &vp); *new_key = &ip->i_number; return 0; } /* * File handle to vnode */ int lfs_fhtovp(struct mount *mp, struct fid *fhp, int lktype, struct vnode **vpp) { struct lfid lfh; struct lfs *fs; if (fhp->fid_len != sizeof(struct lfid)) return EINVAL; memcpy(&lfh, fhp, sizeof(lfh)); if (lfh.lfid_ino < LFS_IFILE_INUM) return ESTALE; fs = VFSTOULFS(mp)->um_lfs; if (lfh.lfid_ident != lfs_sb_getident(fs)) return ESTALE; if (lfh.lfid_ino > ((lfs_dino_getsize(fs, VTOI(fs->lfs_ivnode)->i_din) >> lfs_sb_getbshift(fs)) - lfs_sb_getcleansz(fs) - lfs_sb_getsegtabsz(fs)) * lfs_sb_getifpb(fs)) return ESTALE; return (ulfs_fhtovp(mp, &lfh.lfid_ufid, lktype, vpp)); } /* * Vnode pointer to File handle */ /* ARGSUSED */ int lfs_vptofh(struct vnode *vp, struct fid *fhp, size_t *fh_size) { struct inode *ip; struct lfid lfh; if (*fh_size < sizeof(struct lfid)) { *fh_size = sizeof(struct lfid); return E2BIG; } *fh_size = sizeof(struct lfid); ip = VTOI(vp); memset(&lfh, 0, sizeof(lfh)); lfh.lfid_len = sizeof(struct lfid); lfh.lfid_ino = ip->i_number; lfh.lfid_gen = ip->i_gen; lfh.lfid_ident = lfs_sb_getident(ip->i_lfs); memcpy(fhp, &lfh, sizeof(lfh)); return (0); } /* * ulfs_bmaparray callback function for writing. * * Since blocks will be written to the new segment anyway, * we don't care about current daddr of them. */ static bool lfs_issequential_hole(const struct lfs *fs, daddr_t daddr0, daddr_t daddr1) { (void)fs; /* not used */ KASSERT(daddr0 == UNWRITTEN || (0 <= daddr0 && daddr0 <= LFS_MAX_DADDR(fs))); KASSERT(daddr1 == UNWRITTEN || (0 <= daddr1 && daddr1 <= LFS_MAX_DADDR(fs))); /* NOTE: all we want to know here is 'hole or not'. */ /* NOTE: UNASSIGNED is converted to 0 by ulfs_bmaparray. */ /* * treat UNWRITTENs and all resident blocks as 'contiguous' */ if (daddr0 != 0 && daddr1 != 0) return true; /* * both are in hole? */ if (daddr0 == 0 && daddr1 == 0) return true; /* all holes are 'contiguous' for us. */ return false; } /* * lfs_gop_write functions exactly like genfs_gop_write, except that * (1) it requires the seglock to be held by its caller, and sp->fip * to be properly initialized (it will return without re-initializing * sp->fip, and without calling lfs_writeseg). * (2) it uses the remaining space in the segment, rather than VOP_BMAP, * to determine how large a block it can write at once (though it does * still use VOP_BMAP to find holes in the file); * (3) it calls lfs_gatherblock instead of VOP_STRATEGY on its blocks * (leaving lfs_writeseg to deal with the cluster blocks, so we might * now have clusters of clusters, ick.) */ static int lfs_gop_write(struct vnode *vp, struct vm_page **pgs, int npages, int flags) { int i, error, run, haveeof = 0; int fs_bshift; vaddr_t kva; off_t eof, offset, startoffset = 0; size_t bytes, iobytes, skipbytes; bool async = (flags & PGO_SYNCIO) == 0; daddr_t lbn, blkno; struct vm_page *pg; struct buf *mbp, *bp; struct vnode *devvp = VTOI(vp)->i_devvp; struct inode *ip = VTOI(vp); struct lfs *fs = ip->i_lfs; struct segment *sp = fs->lfs_sp; SEGSUM *ssp; UVMHIST_FUNC("lfs_gop_write"); UVMHIST_CALLED(ubchist); const char * failreason = NULL; ASSERT_SEGLOCK(fs); /* The Ifile lives in the buffer cache */ KASSERT(vp != fs->lfs_ivnode); /* * We don't want to fill the disk before the cleaner has a chance * to make room for us. If we're in danger of doing that, fail * with EAGAIN. The caller will have to notice this, unlock * so the cleaner can run, relock and try again. * * We must write everything, however, if our vnode is being * reclaimed. */ mutex_enter(vp->v_interlock); if (LFS_STARVED_FOR_SEGS(fs) && vdead_check(vp, VDEAD_NOWAIT) == 0) { mutex_exit(vp->v_interlock); failreason = "Starved for segs and not flushing vp"; goto tryagain; } mutex_exit(vp->v_interlock); /* * Sometimes things slip past the filters in lfs_putpages, * and the pagedaemon tries to write pages---problem is * that the pagedaemon never acquires the segment lock. * * Alternatively, pages that were clean when we called * genfs_putpages may have become dirty in the meantime. In this * case the segment header is not properly set up for blocks * to be added to it. * * Unbusy and unclean the pages, and put them on the ACTIVE * queue under the hypothesis that they couldn't have got here * unless they were modified *quite* recently. * * XXXUBC that last statement is an oversimplification of course. */ if (!LFS_SEGLOCK_HELD(fs)) { failreason = "Seglock not held"; goto tryagain; } if (ip->i_lfs_iflags & LFSI_NO_GOP_WRITE) { failreason = "Inode with no_gop_write"; goto tryagain; } if ((pgs[0]->offset & lfs_sb_getbmask(fs)) != 0) { failreason = "Bad page offset"; goto tryagain; } UVMHIST_LOG(ubchist, "vp %#jx pgs %#jx npages %jd flags 0x%jx", (uintptr_t)vp, (uintptr_t)pgs, npages, flags); GOP_SIZE(vp, vp->v_size, &eof, 0); haveeof = 1; if (vp->v_type == VREG) fs_bshift = vp->v_mount->mnt_fs_bshift; else fs_bshift = DEV_BSHIFT; error = 0; pg = pgs[0]; startoffset = pg->offset; KASSERT(eof >= 0); if (startoffset >= eof) { failreason = "Offset beyond EOF"; goto tryagain; } else bytes = MIN(npages << PAGE_SHIFT, eof - startoffset); skipbytes = 0; KASSERT(bytes != 0); /* Swap PG_DELWRI for PG_PAGEOUT */ for (i = 0; i < npages; i++) { if (pgs[i]->flags & PG_DELWRI) { KASSERT(!(pgs[i]->flags & PG_PAGEOUT)); pgs[i]->flags &= ~PG_DELWRI; pgs[i]->flags |= PG_PAGEOUT; uvm_pageout_start(1); rw_enter(vp->v_uobj.vmobjlock, RW_WRITER); uvm_pagelock(pgs[i]); uvm_pageunwire(pgs[i]); uvm_pageunlock(pgs[i]); rw_exit(vp->v_uobj.vmobjlock); } } /* * Check to make sure we're starting on a block boundary. * We'll check later to make sure we always write entire * blocks (or fragments). */ if (startoffset & lfs_sb_getbmask(fs)) printf("%" PRId64 " & %" PRIu64 " = %" PRId64 "\n", startoffset, lfs_sb_getbmask(fs), startoffset & lfs_sb_getbmask(fs)); KASSERT((startoffset & lfs_sb_getbmask(fs)) == 0); if (bytes & lfs_sb_getffmask(fs)) { printf("lfs_gop_write: asked to write %ld bytes\n", (long)bytes); panic("lfs_gop_write: non-integer blocks"); } /* * We could deadlock here on pager_map with UVMPAGER_MAPIN_WAITOK. * If we would, write what we have and try again. If we don't * have anything to write, we'll have to sleep. */ ssp = (SEGSUM *)sp->segsum; if ((kva = uvm_pagermapin(pgs, npages, UVMPAGER_MAPIN_WRITE | (lfs_ss_getnfinfo(fs, ssp) < 1 ? UVMPAGER_MAPIN_WAITOK : 0))) == 0x0) { DLOG((DLOG_PAGE, "lfs_gop_write: forcing write\n")); #if 0 " with nfinfo=%d at offset 0x%jx\n", (int)lfs_ss_getnfinfo(fs, ssp), (uintmax_t)lfs_sb_getoffset(fs))); #endif lfs_updatemeta(sp); lfs_release_finfo(fs); (void) lfs_writeseg(fs, sp); lfs_acquire_finfo(fs, ip->i_number, ip->i_gen); /* * Having given up all of the pager_map we were holding, * we can now wait for aiodoned to reclaim it for us * without fear of deadlock. */ kva = uvm_pagermapin(pgs, npages, UVMPAGER_MAPIN_WRITE | UVMPAGER_MAPIN_WAITOK); } mbp = getiobuf(NULL, true); UVMHIST_LOG(ubchist, "vp %#jx mbp %#jx num now %jd bytes 0x%jx", (uintptr_t)vp, (uintptr_t)mbp, vp->v_numoutput, bytes); mbp->b_bufsize = npages << PAGE_SHIFT; mbp->b_data = (void *)kva; mbp->b_resid = mbp->b_bcount = bytes; mbp->b_cflags |= BC_BUSY|BC_AGE; mbp->b_iodone = uvm_aio_aiodone; bp = NULL; for (offset = startoffset; bytes > 0; offset += iobytes, bytes -= iobytes) { lbn = offset >> fs_bshift; error = ulfs_bmaparray(vp, lbn, &blkno, NULL, NULL, &run, lfs_issequential_hole); if (error) { UVMHIST_LOG(ubchist, "ulfs_bmaparray() -> %jd", error,0,0,0); skipbytes += bytes; bytes = 0; break; } iobytes = MIN((((off_t)lbn + 1 + run) << fs_bshift) - offset, bytes); if (blkno == (daddr_t)-1) { skipbytes += iobytes; continue; } /* * Discover how much we can really pack into this buffer. */ /* If no room in the current segment, finish it up */ if (sp->sum_bytes_left < sizeof(int32_t) || sp->seg_bytes_left < (1 << lfs_sb_getbshift(fs))) { int vers; lfs_updatemeta(sp); vers = lfs_fi_getversion(fs, sp->fip); lfs_release_finfo(fs); (void) lfs_writeseg(fs, sp); lfs_acquire_finfo(fs, ip->i_number, vers); } /* Check both for space in segment and space in segsum */ iobytes = MIN(iobytes, (sp->seg_bytes_left >> fs_bshift) << fs_bshift); iobytes = MIN(iobytes, (sp->sum_bytes_left / sizeof(int32_t)) << fs_bshift); KASSERT(iobytes > 0); /* if it's really one i/o, don't make a second buf */ if (offset == startoffset && iobytes == bytes) { bp = mbp; /* * All the LFS output is done by the segwriter. It * will increment numoutput by one for all the bufs it * receives. However this buffer needs one extra to * account for aiodone. */ mutex_enter(vp->v_interlock); vp->v_numoutput++; mutex_exit(vp->v_interlock); } else { bp = getiobuf(NULL, true); UVMHIST_LOG(ubchist, "vp %#jx bp %#jx num now %jd", (uintptr_t)vp, (uintptr_t)bp, vp->v_numoutput, 0); nestiobuf_setup(mbp, bp, offset - pg->offset, iobytes); /* * LFS doesn't like async I/O here, dies with * an assert in lfs_bwrite(). Is that assert * valid? I retained non-async behaviour when * converted this to use nestiobuf --pooka */ bp->b_flags &= ~B_ASYNC; } /* XXX This is silly ... is this necessary? */ mutex_enter(&bufcache_lock); mutex_enter(vp->v_interlock); bgetvp(vp, bp); mutex_exit(vp->v_interlock); mutex_exit(&bufcache_lock); bp->b_lblkno = lfs_lblkno(fs, offset); bp->b_private = mbp; if (devvp->v_type == VBLK) { bp->b_dev = devvp->v_rdev; } VOP_BWRITE(bp->b_vp, bp); while (lfs_gatherblock(sp, bp, NULL)) continue; } nestiobuf_done(mbp, skipbytes, error); if (skipbytes) { UVMHIST_LOG(ubchist, "skipbytes %jd", skipbytes, 0,0,0); } UVMHIST_LOG(ubchist, "returning 0", 0,0,0,0); if (!async) { /* Start a segment write. */ UVMHIST_LOG(ubchist, "flushing", 0,0,0,0); mutex_enter(&lfs_lock); lfs_flush(fs, 0, 1); mutex_exit(&lfs_lock); } if ((sp->seg_flags & SEGM_SINGLE) && lfs_sb_getcurseg(fs) != fs->lfs_startseg) return EAGAIN; return (0); tryagain: /* * We can't write the pages, for whatever reason. * Clean up after ourselves, and make the caller try again. */ mutex_enter(vp->v_interlock); /* Tell why we're here, if we know */ if (failreason != NULL) { DLOG((DLOG_PAGE, "lfs_gop_write: %s\n", failreason)); } if (haveeof && startoffset >= eof) { DLOG((DLOG_PAGE, "lfs_gop_write: ino %d start 0x%" PRIx64 " eof 0x%" PRIx64 " npages=%d\n", VTOI(vp)->i_number, pgs[0]->offset, eof, npages)); } for (i = 0; i < npages; i++) { pg = pgs[i]; if (pg->flags & PG_PAGEOUT) uvm_pageout_done(1); uvm_pagelock(pg); if (pg->flags & PG_DELWRI) { uvm_pageunwire(pg); } uvm_pageactivate(pg); uvm_pageunlock(pg); pg->flags &= ~(PG_DELWRI|PG_PAGEOUT|PG_RELEASED); uvm_pagemarkdirty(pg, UVM_PAGE_STATUS_DIRTY); DLOG((DLOG_PAGE, "pg[%d] = %p (vp %p off %" PRIx64 ")\n", i, pg, vp, pg->offset)); DLOG((DLOG_PAGE, "pg[%d]->flags = %x\n", i, pg->flags)); DLOG((DLOG_PAGE, "pg[%d]->pqflags = %x\n", i, pg->pqflags)); DLOG((DLOG_PAGE, "pg[%d]->uanon = %p\n", i, pg->uanon)); DLOG((DLOG_PAGE, "pg[%d]->uobject = %p\n", i, pg->uobject)); DLOG((DLOG_PAGE, "pg[%d]->wire_count = %d\n", i, pg->wire_count)); DLOG((DLOG_PAGE, "pg[%d]->loan_count = %d\n", i, pg->loan_count)); } uvm_page_unbusy(pgs, npages); mutex_exit(vp->v_interlock); return EAGAIN; } /* * finish vnode/inode initialization. * used by lfs_vget. */ void lfs_vinit(struct mount *mp, struct vnode **vpp) { struct vnode *vp = *vpp; struct inode *ip = VTOI(vp); struct ulfsmount *ump = VFSTOULFS(mp); struct lfs *fs = ump->um_lfs; int i; ip->i_mode = lfs_dino_getmode(fs, ip->i_din); ip->i_nlink = lfs_dino_getnlink(fs, ip->i_din); ip->i_lfs_osize = ip->i_size = lfs_dino_getsize(fs, ip->i_din); ip->i_flags = lfs_dino_getflags(fs, ip->i_din); ip->i_gen = lfs_dino_getgen(fs, ip->i_din); ip->i_uid = lfs_dino_getuid(fs, ip->i_din); ip->i_gid = lfs_dino_getgid(fs, ip->i_din); ip->i_lfs_effnblks = lfs_dino_getblocks(fs, ip->i_din); ip->i_lfs_odnlink = lfs_dino_getnlink(fs, ip->i_din); /* * Initialize the vnode from the inode, check for aliases. In all * cases re-init ip, the underlying vnode/inode may have changed. */ ulfs_vinit(mp, lfs_specop_p, lfs_fifoop_p, &vp); ip = VTOI(vp); memset(ip->i_lfs_fragsize, 0, ULFS_NDADDR * sizeof(*ip->i_lfs_fragsize)); if (vp->v_type != VLNK || ip->i_size >= ip->i_lfs->um_maxsymlinklen) { #ifdef DEBUG for (i = (ip->i_size + lfs_sb_getbsize(fs) - 1) >> lfs_sb_getbshift(fs); i < ULFS_NDADDR; i++) { if ((vp->v_type == VBLK || vp->v_type == VCHR) && i == 0) continue; if (lfs_dino_getdb(fs, ip->i_din, i) != 0) { lfs_dump_dinode(fs, ip->i_din); panic("inconsistent inode (direct)"); } } for ( ; i < ULFS_NDADDR + ULFS_NIADDR; i++) { if (lfs_dino_getib(fs, ip->i_din, i - ULFS_NDADDR) != 0) { lfs_dump_dinode(fs, ip->i_din); panic("inconsistent inode (indirect)"); } } #endif /* DEBUG */ for (i = 0; i < ULFS_NDADDR; i++) if (lfs_dino_getdb(fs, ip->i_din, i) != 0) ip->i_lfs_fragsize[i] = lfs_blksize(fs, ip, i); } KASSERTMSG((vp->v_type != VNON), "lfs_vinit: ino %llu is type VNON! (ifmt=%o)\n", (unsigned long long)ip->i_number, (ip->i_mode & LFS_IFMT) >> 12); /* * Finish inode initialization now that aliasing has been resolved. */ ip->i_devvp = fs->lfs_devvp; vref(ip->i_devvp); #if defined(LFS_QUOTA) || defined(LFS_QUOTA2) ulfsquota_init(ip); #endif genfs_node_init(vp, &lfs_genfsops); uvm_vnp_setsize(vp, ip->i_size); /* Initialize hiblk from file size */ ip->i_lfs_hiblk = lfs_lblkno(ip->i_lfs, ip->i_size + lfs_sb_getbsize(ip->i_lfs) - 1) - 1; *vpp = vp; } /* * Resize the filesystem to contain the specified number of segments. */ int lfs_resize_fs(struct lfs *fs, int newnsegs) { SEGUSE *sup; CLEANERINFO *cip; struct buf *bp, *obp; daddr_t olast, nlast, ilast, noff, start, end; struct vnode *ivp; struct inode *ip; int error, badnews, inc, oldnsegs; int sbbytes, csbbytes, gain, cgain; int i; /* Only support v2 and up */ if (lfs_sb_getversion(fs) < 2) return EOPNOTSUPP; /* If we're doing nothing, do it fast */ oldnsegs = lfs_sb_getnseg(fs); if (newnsegs == oldnsegs) return 0; /* We always have to have two superblocks */ if (newnsegs <= lfs_dtosn(fs, lfs_sb_getsboff(fs, 1))) /* XXX this error code is rather nonsense */ return EFBIG; ivp = fs->lfs_ivnode; ip = VTOI(ivp); error = 0; /* Take the segment lock so no one else calls lfs_newseg() */ lfs_seglock(fs, SEGM_PROT); /* * Make sure the segments we're going to be losing, if any, * are in fact empty. We hold the seglock, so their status * cannot change underneath us. Count the superblocks we lose, * while we're at it. */ sbbytes = csbbytes = 0; cgain = 0; for (i = newnsegs; i < oldnsegs; i++) { LFS_SEGENTRY(sup, fs, i, bp); badnews = sup->su_nbytes || !(sup->su_flags & SEGUSE_INVAL); if (sup->su_flags & SEGUSE_SUPERBLOCK) sbbytes += LFS_SBPAD; if (!(sup->su_flags & SEGUSE_DIRTY)) { ++cgain; if (sup->su_flags & SEGUSE_SUPERBLOCK) csbbytes += LFS_SBPAD; } brelse(bp, 0); if (badnews) { error = EBUSY; goto out; } } /* Note old and new segment table endpoints, and old ifile size */ olast = lfs_sb_getcleansz(fs) + lfs_sb_getsegtabsz(fs); nlast = howmany(newnsegs, lfs_sb_getsepb(fs)) + lfs_sb_getcleansz(fs); ilast = ivp->v_size >> lfs_sb_getbshift(fs); noff = nlast - olast; /* * Make sure no one can use the Ifile while we change it around. * Even after taking the iflock we need to make sure no one still * is holding Ifile buffers, so we get each one, to drain them. * (XXX this could be done better.) */ rw_enter(&fs->lfs_iflock, RW_WRITER); for (i = 0; i < ilast; i++) { /* XXX what to do if bread fails? */ bread(ivp, i, lfs_sb_getbsize(fs), 0, &bp); brelse(bp, 0); } /* Allocate new Ifile blocks */ for (i = ilast; i < ilast + noff; i++) { if (lfs_balloc(ivp, i * lfs_sb_getbsize(fs), lfs_sb_getbsize(fs), NOCRED, 0, &bp) != 0) panic("balloc extending ifile"); memset(bp->b_data, 0, lfs_sb_getbsize(fs)); VOP_BWRITE(bp->b_vp, bp); } /* Register new ifile size */ ip->i_size += noff * lfs_sb_getbsize(fs); lfs_dino_setsize(fs, ip->i_din, ip->i_size); uvm_vnp_setsize(ivp, ip->i_size); /* Copy the inode table to its new position */ if (noff != 0) { if (noff < 0) { start = nlast; end = ilast + noff; inc = 1; } else { start = ilast + noff - 1; end = nlast - 1; inc = -1; } for (i = start; i != end; i += inc) { if (bread(ivp, i, lfs_sb_getbsize(fs), B_MODIFY, &bp) != 0) panic("resize: bread dst blk failed"); if (bread(ivp, i - noff, lfs_sb_getbsize(fs), 0, &obp)) panic("resize: bread src blk failed"); memcpy(bp->b_data, obp->b_data, lfs_sb_getbsize(fs)); VOP_BWRITE(bp->b_vp, bp); brelse(obp, 0); } } /* If we are expanding, write the new empty SEGUSE entries */ if (newnsegs > oldnsegs) { for (i = oldnsegs; i < newnsegs; i++) { if ((error = bread(ivp, i / lfs_sb_getsepb(fs) + lfs_sb_getcleansz(fs), lfs_sb_getbsize(fs), B_MODIFY, &bp)) != 0) panic("lfs: ifile read: %d", error); while ((i + 1) % lfs_sb_getsepb(fs) && i < newnsegs) { sup = &((SEGUSE *)bp->b_data)[i % lfs_sb_getsepb(fs)]; memset(sup, 0, sizeof(*sup)); i++; } VOP_BWRITE(bp->b_vp, bp); } } /* Zero out unused superblock offsets */ for (i = 2; i < LFS_MAXNUMSB; i++) if (lfs_dtosn(fs, lfs_sb_getsboff(fs, i)) >= newnsegs) lfs_sb_setsboff(fs, i, 0x0); /* * Correct superblock entries that depend on fs size. * The computations of these are as follows: * * size = lfs_segtod(fs, nseg) * dsize = lfs_segtod(fs, nseg - minfreeseg) - lfs_btofsb(#super * LFS_SBPAD) * bfree = dsize - lfs_btofsb(fs, bsize * nseg / 2) - blocks_actually_used * avail = lfs_segtod(fs, nclean) - lfs_btofsb(#clean_super * LFS_SBPAD) * + (lfs_segtod(fs, 1) - (offset - curseg)) * - lfs_segtod(fs, minfreeseg - (minfreeseg / 2)) * * XXX - we should probably adjust minfreeseg as well. */ gain = (newnsegs - oldnsegs); lfs_sb_setnseg(fs, newnsegs); lfs_sb_setsegtabsz(fs, nlast - lfs_sb_getcleansz(fs)); lfs_sb_addsize(fs, gain * lfs_btofsb(fs, lfs_sb_getssize(fs))); lfs_sb_adddsize(fs, gain * lfs_btofsb(fs, lfs_sb_getssize(fs)) - lfs_btofsb(fs, sbbytes)); lfs_sb_addbfree(fs, gain * lfs_btofsb(fs, lfs_sb_getssize(fs)) - lfs_btofsb(fs, sbbytes) - gain * lfs_btofsb(fs, lfs_sb_getbsize(fs) / 2)); if (gain > 0) { lfs_sb_addnclean(fs, gain); lfs_sb_addavail(fs, gain * lfs_btofsb(fs, lfs_sb_getssize(fs))); } else { lfs_sb_subnclean(fs, cgain); lfs_sb_subavail(fs, cgain * lfs_btofsb(fs, lfs_sb_getssize(fs)) - lfs_btofsb(fs, csbbytes)); } /* Resize segment flag cache */ fs->lfs_suflags[0] = realloc(fs->lfs_suflags[0], lfs_sb_getnseg(fs) * sizeof(u_int32_t), M_SEGMENT, M_WAITOK); fs->lfs_suflags[1] = realloc(fs->lfs_suflags[1], lfs_sb_getnseg(fs) * sizeof(u_int32_t), M_SEGMENT, M_WAITOK); for (i = oldnsegs; i < newnsegs; i++) fs->lfs_suflags[0][i] = fs->lfs_suflags[1][i] = 0x0; /* Truncate Ifile if necessary */ if (noff < 0) lfs_truncate(ivp, ivp->v_size + (noff << lfs_sb_getbshift(fs)), 0, NOCRED); /* Update cleaner info so the cleaner can die */ /* XXX what to do if bread fails? */ bread(ivp, 0, lfs_sb_getbsize(fs), B_MODIFY, &bp); cip = bp->b_data; lfs_ci_setclean(fs, cip, lfs_sb_getnclean(fs)); lfs_ci_setdirty(fs, cip, lfs_sb_getnseg(fs) - lfs_sb_getnclean(fs)); VOP_BWRITE(bp->b_vp, bp); /* Let Ifile accesses proceed */ rw_exit(&fs->lfs_iflock); out: lfs_segunlock(fs); return error; } /* * Extended attribute dispatch */ int lfs_extattrctl(struct mount *mp, int cmd, struct vnode *vp, int attrnamespace, const char *attrname) { #ifdef LFS_EXTATTR struct ulfsmount *ump; ump = VFSTOULFS(mp); if (ump->um_fstype == ULFS1) { return ulfs_extattrctl(mp, cmd, vp, attrnamespace, attrname); } #endif return vfs_stdextattrctl(mp, cmd, vp, attrnamespace, attrname); } |
| 6 2 4 8 3 5 5 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 197 198 199 200 201 202 203 204 205 206 207 208 209 210 211 | /* $NetBSD: vfs_syscalls_20.c,v 1.46 2020/06/28 14:37:53 christos Exp $ */ /* * Copyright (c) 1989, 1993 * The Regents of the University of California. All rights reserved. * (c) UNIX System Laboratories, Inc. * All or some portions of this file are derived from material licensed * to the University of California by American Telephone and Telegraph * Co. or Unix System Laboratories, Inc. and are reproduced herein with * the permission of UNIX System Laboratories, Inc. * * Redistribution and use in source and binary forms, with or without * modification, are permitted provided that the following conditions * are met: * 1. Redistributions of source code must retain the above copyright * notice, this list of conditions and the following disclaimer. * 2. Redistributions in binary form must reproduce the above copyright * notice, this list of conditions and the following disclaimer in the * documentation and/or other materials provided with the distribution. * 3. Neither the name of the University nor the names of its contributors * may be used to endorse or promote products derived from this software * without specific prior written permission. * * THIS SOFTWARE IS PROVIDED BY THE REGENTS AND CONTRIBUTORS ``AS IS'' AND * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE * ARE DISCLAIMED. IN NO EVENT SHALL THE REGENTS OR CONTRIBUTORS BE LIABLE * FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL * DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS * OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) * HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT * LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY * OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF * SUCH DAMAGE. * * @(#)vfs_syscalls.c 8.42 (Berkeley) 7/31/95 */ #include <sys/cdefs.h> __KERNEL_RCSID(0, "$NetBSD: vfs_syscalls_20.c,v 1.46 2020/06/28 14:37:53 christos Exp $"); #ifdef _KERNEL_OPT #include "opt_compat_netbsd.h" #endif #include <sys/param.h> #include <sys/systm.h> #include <sys/namei.h> #include <sys/filedesc.h> #include <sys/kernel.h> #include <sys/file.h> #include <sys/stat.h> #include <sys/vnode.h> #include <sys/mount.h> #include <sys/proc.h> #include <sys/uio.h> #include <sys/dirent.h> #include <sys/sysctl.h> #include <sys/syscall.h> #include <sys/syscallvar.h> #include <sys/syscallargs.h> #include <sys/kauth.h> #include <sys/vfs_syscalls.h> #include <compat/common/compat_mod.h> #include <compat/sys/mount.h> #include <compat/sys/statvfs.h> static const struct syscall_package vfs_syscalls_20_syscalls[] = { { SYS_compat_20_fhstatfs, 0, (sy_call_t *)compat_20_sys_fhstatfs }, { SYS_compat_20_fstatfs, 0, (sy_call_t *)compat_20_sys_fstatfs }, { SYS_compat_20_getfsstat, 0, (sy_call_t *)compat_20_sys_getfsstat }, { SYS_compat_20_statfs, 0, (sy_call_t *)compat_20_sys_statfs }, { 0, 0, NULL } }; /* * Get filesystem statistics. */ /* ARGSUSED */ int compat_20_sys_statfs(struct lwp *l, const struct compat_20_sys_statfs_args *uap, register_t *retval) { /* { syscallarg(const char *) path; syscallarg(struct statfs12 *) buf; } */ struct mount *mp; struct statvfs *sbuf; int error; struct vnode *vp; error = namei_simple_user(SCARG(uap, path), NSM_FOLLOW_TRYEMULROOT, &vp); if (error != 0) return error; mp = vp->v_mount; sbuf = STATVFSBUF_GET(); if ((error = dostatvfs(mp, sbuf, l, 0, 1)) != 0) goto done; error = statvfs_to_statfs12_copy(sbuf, SCARG(uap, buf), 0); done: vrele(vp); STATVFSBUF_PUT(sbuf); return error; } /* * Get filesystem statistics. */ /* ARGSUSED */ int compat_20_sys_fstatfs(struct lwp *l, const struct compat_20_sys_fstatfs_args *uap, register_t *retval) { /* { syscallarg(int) fd; syscallarg(struct statfs12 *) buf; } */ struct file *fp; struct mount *mp; struct statvfs *sbuf; int error; /* fd_getvnode() will use the descriptor for us */ if ((error = fd_getvnode(SCARG(uap, fd), &fp)) != 0) return (error); mp = fp->f_vnode->v_mount; sbuf = STATVFSBUF_GET(); if ((error = dostatvfs(mp, sbuf, l, 0, 1)) != 0) goto out; error = statvfs_to_statfs12_copy(sbuf, SCARG(uap, buf), 0); out: fd_putfile(SCARG(uap, fd)); STATVFSBUF_PUT(sbuf); return error; } /* * Get statistics on all filesystems. */ int compat_20_sys_getfsstat(struct lwp *l, const struct compat_20_sys_getfsstat_args *uap, register_t *retval) { /* { syscallarg(struct statfs12 *) buf; syscallarg(long) bufsize; syscallarg(int) flags; } */ return do_sys_getvfsstat(l, SCARG(uap, buf), SCARG(uap, bufsize), SCARG(uap, flags), statvfs_to_statfs12_copy, sizeof(struct statfs12), retval); } int compat_20_sys_fhstatfs(struct lwp *l, const struct compat_20_sys_fhstatfs_args *uap, register_t *retval) { /* { syscallarg(const struct compat_30_fhandle *) fhp; syscallarg(struct statfs12 *) buf; } */ struct statvfs *sbuf; struct compat_30_fhandle fh; struct mount *mp; struct vnode *vp; int error; /* * Must be super user */ if ((error = kauth_authorize_system(l->l_cred, KAUTH_SYSTEM_FILEHANDLE, 0, NULL, NULL, NULL))) return (error); if ((error = copyin(SCARG(uap, fhp), &fh, sizeof(fh))) != 0) return (error); if ((mp = vfs_getvfs(&fh.fh_fsid)) == NULL) return (ESTALE); error = VFS_FHTOVP(mp, (struct fid*)&fh.fh_fid, LK_EXCLUSIVE, &vp); if (error != 0) return (error); mp = vp->v_mount; VOP_UNLOCK(vp); sbuf = STATVFSBUF_GET(); if ((error = VFS_STATVFS(mp, sbuf)) != 0) goto out; error = statvfs_to_statfs12_copy(sbuf, SCARG(uap, buf), 0); out: vrele(vp); STATVFSBUF_PUT(sbuf); return error; } int vfs_syscalls_20_init(void) { return syscall_establish(NULL, vfs_syscalls_20_syscalls); } int vfs_syscalls_20_fini(void) { return syscall_disestablish(NULL, vfs_syscalls_20_syscalls); } |
| 56 1056 119 118 119 1043 1043 1043 1044 1044 1044 1045 1045 1043 225 1055 1054 223 1052 50 1035 681 944 9 1041 736 593 671 896 897 4 897 497 578 578 578 577 542 539 578 576 3 318 566 578 576 3 500 656 656 656 656 656 656 382 633 3 3 3 1050 1050 1048 1050 1050 1050 1049 1050 152 971 659 9 1049 1050 1050 93 119 118 119 1049 959 1060 1060 244 185 1045 1057 1057 137 1045 1044 225 1053 1054 1055 1042 222 223 223 1046 225 225 225 225 1055 1055 1055 1053 1052 53 52 53 53 53 49 4 11 4 47 664 9 657 114 38 38 1 37 132 123 39 39 38 1 1 1 1046 1046 1046 138 138 137 1045 1046 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 197 198 199 200 201 202 203 204 205 206 207 208 209 210 211 212 213 214 215 216 217 218 219 220 221 222 223 224 225 226 227 228 229 230 231 232 233 234 235 236 237 238 239 240 241 242 243 244 245 246 247 248 249 250 251 252 253 254 255 256 257 258 259 260 261 262 263 264 265 266 267 268 269 270 271 272 273 274 275 276 277 278 279 280 281 282 283 284 285 286 287 288 289 290 291 292 293 294 295 296 297 298 299 300 301 302 303 304 305 306 307 308 309 310 311 312 313 314 315 316 317 318 319 320 321 322 323 324 325 326 327 328 329 330 331 332 333 334 335 336 337 338 339 340 341 342 343 344 345 346 347 348 349 350 351 352 353 354 355 356 357 358 359 360 361 362 363 364 365 366 367 368 369 370 371 372 373 374 375 376 377 378 379 380 381 382 383 384 385 386 387 388 389 390 391 392 393 394 395 396 397 398 399 400 401 402 403 404 405 406 407 408 409 410 411 412 413 414 415 416 417 418 419 420 421 422 423 424 425 426 427 428 429 430 431 432 433 434 435 436 437 438 439 440 441 442 443 444 445 446 447 448 449 450 451 452 453 454 455 456 457 458 459 460 461 462 463 464 465 466 467 468 469 470 471 472 473 474 475 476 477 478 479 480 481 482 483 484 485 486 487 488 489 490 491 492 493 494 495 496 497 498 499 500 501 502 503 504 505 506 507 508 509 510 511 512 513 514 515 516 517 518 519 520 521 522 523 524 525 526 527 528 529 530 531 532 533 534 535 536 537 538 539 540 541 542 543 544 545 546 547 548 549 550 551 552 553 554 555 556 557 558 559 560 561 562 563 564 565 566 567 568 569 570 571 572 573 574 575 576 577 578 579 580 581 582 583 584 585 586 587 588 589 590 591 592 593 594 595 596 597 598 599 600 601 602 603 604 605 606 607 608 609 610 611 612 613 614 615 616 617 618 619 620 621 622 623 624 625 626 627 628 629 630 631 632 633 634 635 636 637 638 639 640 641 642 643 644 645 646 647 648 649 650 651 652 653 654 655 656 657 658 659 660 661 662 663 664 665 666 667 668 669 670 671 672 673 674 675 676 677 678 679 680 681 682 683 684 685 686 687 688 689 690 691 692 693 694 695 696 697 698 699 700 701 702 703 704 705 706 707 708 709 710 711 712 713 714 715 716 717 718 719 720 721 722 723 724 725 726 727 728 729 730 731 732 733 734 735 736 737 738 739 740 741 742 743 744 745 746 747 748 749 750 751 752 753 754 755 756 757 758 759 760 761 762 763 764 765 766 767 768 769 770 771 772 773 774 775 776 777 778 779 780 781 782 783 784 785 786 787 788 789 790 791 792 793 794 795 796 797 798 799 800 801 802 803 804 805 806 807 808 809 810 811 812 813 814 815 816 817 818 819 820 821 822 823 824 825 826 827 828 829 830 831 832 833 834 835 836 837 838 839 840 841 842 843 844 845 846 847 848 849 850 851 852 853 854 855 856 857 858 859 860 861 862 863 864 865 866 867 868 869 870 871 872 873 874 875 876 877 878 879 880 881 882 883 884 885 886 887 888 889 890 891 892 893 894 895 896 897 898 899 900 901 902 903 904 905 906 907 908 909 910 911 912 913 914 915 916 917 918 919 920 921 922 923 924 925 926 927 928 929 930 931 932 933 934 935 936 937 938 939 940 941 942 943 944 945 946 947 948 949 950 951 952 953 954 955 956 957 958 959 960 961 962 963 964 965 966 967 968 969 970 971 972 973 974 975 976 977 978 979 980 981 982 983 984 985 986 987 988 989 990 991 992 993 994 995 996 997 998 999 1000 1001 1002 1003 1004 1005 1006 1007 1008 1009 1010 1011 1012 1013 1014 1015 1016 1017 1018 1019 1020 1021 1022 1023 1024 1025 1026 1027 1028 1029 1030 1031 1032 1033 1034 1035 1036 1037 1038 1039 1040 1041 1042 1043 1044 1045 1046 1047 1048 1049 1050 1051 1052 1053 1054 1055 1056 1057 1058 1059 1060 1061 1062 1063 1064 1065 1066 1067 1068 1069 1070 1071 1072 1073 1074 1075 1076 1077 1078 1079 1080 1081 1082 1083 1084 1085 1086 1087 1088 1089 1090 1091 1092 1093 1094 1095 1096 1097 1098 1099 1100 1101 1102 1103 1104 1105 1106 1107 1108 1109 1110 1111 1112 1113 1114 1115 1116 1117 1118 1119 1120 1121 1122 1123 1124 1125 1126 1127 1128 1129 1130 1131 1132 1133 1134 1135 1136 1137 1138 1139 1140 1141 1142 1143 1144 1145 1146 1147 1148 1149 1150 1151 1152 1153 1154 1155 1156 1157 1158 1159 1160 1161 1162 1163 1164 1165 1166 1167 1168 1169 1170 1171 1172 1173 1174 1175 1176 1177 1178 1179 1180 1181 1182 1183 1184 1185 1186 1187 1188 1189 1190 1191 1192 1193 1194 1195 1196 1197 1198 1199 1200 1201 1202 1203 1204 1205 1206 1207 1208 1209 1210 1211 1212 1213 1214 1215 1216 1217 1218 1219 1220 1221 1222 1223 1224 1225 1226 1227 1228 1229 1230 1231 1232 1233 1234 1235 1236 1237 1238 1239 1240 1241 1242 1243 1244 1245 1246 1247 1248 1249 1250 1251 1252 1253 1254 1255 1256 1257 1258 1259 1260 1261 1262 1263 1264 1265 1266 1267 1268 1269 1270 1271 1272 1273 1274 1275 1276 1277 1278 1279 1280 1281 1282 1283 1284 1285 1286 1287 1288 1289 1290 1291 1292 1293 1294 1295 1296 1297 1298 1299 1300 1301 1302 1303 1304 1305 1306 1307 1308 1309 1310 1311 1312 1313 1314 1315 1316 1317 1318 1319 1320 1321 1322 1323 1324 1325 1326 1327 1328 1329 1330 1331 1332 1333 1334 1335 1336 1337 1338 1339 1340 1341 1342 1343 1344 1345 1346 1347 1348 1349 1350 1351 1352 1353 1354 1355 1356 1357 1358 1359 1360 1361 1362 1363 1364 1365 1366 1367 1368 1369 1370 1371 1372 1373 1374 1375 1376 1377 1378 1379 1380 1381 1382 1383 1384 1385 1386 1387 1388 1389 1390 1391 1392 1393 1394 1395 1396 1397 1398 1399 1400 1401 1402 1403 1404 1405 1406 1407 1408 1409 1410 1411 1412 1413 1414 1415 1416 1417 1418 1419 1420 1421 1422 1423 1424 1425 1426 1427 1428 1429 1430 1431 1432 1433 1434 1435 1436 1437 1438 1439 1440 1441 1442 1443 1444 1445 1446 1447 1448 1449 1450 1451 1452 1453 1454 1455 1456 1457 1458 1459 1460 1461 1462 1463 1464 1465 1466 1467 1468 1469 1470 1471 1472 1473 1474 1475 1476 1477 1478 1479 1480 1481 1482 1483 1484 1485 1486 1487 1488 1489 1490 1491 1492 1493 1494 1495 1496 1497 1498 1499 1500 1501 1502 1503 1504 1505 1506 1507 1508 1509 1510 1511 1512 1513 1514 1515 1516 1517 1518 1519 1520 1521 1522 1523 1524 1525 1526 1527 1528 1529 1530 1531 1532 1533 1534 1535 1536 1537 1538 1539 1540 1541 1542 1543 1544 1545 1546 1547 1548 1549 1550 1551 1552 1553 1554 1555 1556 1557 1558 1559 1560 1561 1562 1563 1564 1565 1566 1567 1568 1569 1570 1571 1572 1573 1574 1575 1576 1577 1578 1579 1580 1581 1582 1583 1584 1585 1586 1587 1588 1589 1590 1591 1592 1593 1594 1595 1596 1597 1598 1599 1600 1601 1602 1603 1604 1605 1606 1607 1608 1609 1610 1611 1612 1613 1614 1615 1616 1617 1618 1619 1620 1621 1622 1623 1624 1625 1626 1627 1628 1629 1630 1631 1632 1633 1634 1635 1636 1637 1638 1639 1640 1641 1642 1643 1644 1645 1646 1647 1648 1649 1650 1651 1652 1653 1654 1655 1656 1657 1658 1659 1660 1661 1662 1663 1664 1665 1666 1667 1668 1669 1670 1671 1672 1673 1674 1675 1676 1677 1678 1679 1680 1681 1682 1683 1684 1685 1686 1687 1688 1689 1690 1691 1692 1693 1694 1695 1696 1697 1698 1699 1700 1701 1702 1703 1704 1705 1706 1707 1708 1709 1710 1711 1712 1713 1714 1715 1716 1717 1718 1719 1720 1721 1722 1723 1724 1725 1726 1727 1728 1729 1730 1731 1732 1733 1734 1735 1736 1737 1738 1739 1740 1741 1742 1743 1744 1745 1746 1747 1748 1749 1750 1751 1752 1753 1754 1755 1756 1757 1758 1759 1760 1761 1762 1763 1764 1765 1766 1767 1768 1769 1770 1771 1772 1773 1774 1775 1776 1777 1778 1779 1780 1781 1782 1783 1784 1785 1786 1787 1788 1789 1790 1791 1792 1793 1794 1795 1796 1797 1798 1799 1800 1801 1802 1803 1804 1805 1806 1807 1808 1809 1810 1811 1812 1813 1814 1815 1816 1817 1818 1819 1820 1821 1822 1823 1824 1825 1826 1827 1828 1829 1830 1831 1832 1833 1834 1835 1836 1837 1838 1839 1840 1841 1842 1843 1844 1845 1846 1847 1848 1849 1850 1851 1852 1853 1854 1855 1856 1857 1858 1859 1860 1861 1862 1863 1864 1865 1866 1867 1868 1869 1870 1871 1872 1873 1874 1875 1876 1877 1878 1879 1880 1881 1882 1883 1884 1885 1886 1887 1888 1889 1890 1891 1892 1893 1894 1895 1896 1897 1898 1899 1900 1901 1902 1903 1904 1905 1906 1907 1908 1909 1910 1911 1912 1913 1914 1915 1916 1917 1918 1919 1920 1921 1922 1923 1924 1925 1926 1927 1928 1929 1930 1931 1932 1933 1934 1935 1936 1937 1938 1939 1940 1941 1942 1943 1944 1945 1946 1947 1948 1949 1950 1951 1952 1953 1954 1955 1956 1957 1958 1959 1960 1961 1962 1963 1964 1965 1966 1967 1968 1969 1970 1971 1972 1973 1974 1975 1976 1977 1978 1979 1980 1981 1982 1983 1984 1985 1986 1987 1988 1989 1990 1991 1992 1993 1994 1995 1996 1997 1998 1999 2000 2001 2002 2003 2004 2005 2006 2007 2008 2009 2010 2011 2012 2013 2014 2015 2016 2017 2018 2019 2020 2021 2022 2023 2024 2025 2026 2027 2028 2029 2030 2031 2032 2033 2034 2035 2036 2037 2038 2039 2040 2041 2042 2043 2044 2045 2046 2047 2048 2049 2050 2051 2052 2053 2054 2055 2056 2057 2058 2059 2060 2061 2062 2063 2064 2065 2066 2067 2068 2069 2070 2071 2072 2073 2074 2075 2076 2077 2078 2079 2080 2081 2082 2083 2084 2085 2086 2087 2088 2089 2090 2091 2092 2093 2094 2095 2096 2097 2098 2099 2100 2101 2102 2103 2104 2105 2106 2107 2108 2109 2110 2111 2112 2113 2114 2115 2116 2117 2118 2119 2120 2121 2122 2123 2124 2125 2126 2127 2128 2129 2130 2131 2132 2133 2134 2135 2136 2137 2138 2139 2140 2141 2142 2143 2144 2145 2146 2147 2148 2149 2150 2151 2152 2153 2154 2155 2156 2157 2158 2159 2160 2161 2162 2163 2164 2165 2166 2167 2168 2169 2170 2171 2172 2173 2174 2175 2176 2177 2178 2179 2180 2181 2182 2183 2184 2185 2186 2187 2188 2189 2190 2191 2192 2193 2194 2195 2196 2197 2198 2199 2200 2201 2202 2203 2204 2205 2206 2207 2208 2209 2210 2211 2212 2213 2214 2215 2216 2217 2218 2219 2220 2221 2222 2223 2224 2225 2226 2227 2228 | /* $NetBSD: vfs_bio.c,v 1.303 2022/03/30 14:54:29 riastradh Exp $ */ /*- * Copyright (c) 2007, 2008, 2009, 2019, 2020 The NetBSD Foundation, Inc. * All rights reserved. * * This code is derived from software contributed to The NetBSD Foundation * by Andrew Doran, and by Wasabi Systems, Inc. * * Redistribution and use in source and binary forms, with or without * modification, are permitted provided that the following conditions * are met: * 1. Redistributions of source code must retain the above copyright * notice, this list of conditions and the following disclaimer. * 2. Redistributions in binary form must reproduce the above copyright * notice, this list of conditions and the following disclaimer in the * documentation and/or other materials provided with the distribution. * * THIS SOFTWARE IS PROVIDED BY THE NETBSD FOUNDATION, INC. AND CONTRIBUTORS * ``AS IS'' AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED * TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR * PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE FOUNDATION OR CONTRIBUTORS * BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR * CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF * SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS * INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN * CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) * ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE * POSSIBILITY OF SUCH DAMAGE. */ /*- * Copyright (c) 1982, 1986, 1989, 1993 * The Regents of the University of California. All rights reserved. * (c) UNIX System Laboratories, Inc. * All or some portions of this file are derived from material licensed * to the University of California by American Telephone and Telegraph * Co. or Unix System Laboratories, Inc. and are reproduced herein with * the permission of UNIX System Laboratories, Inc. * * Redistribution and use in source and binary forms, with or without * modification, are permitted provided that the following conditions * are met: * 1. Redistributions of source code must retain the above copyright * notice, this list of conditions and the following disclaimer. * 2. Redistributions in binary form must reproduce the above copyright * notice, this list of conditions and the following disclaimer in the * documentation and/or other materials provided with the distribution. * 3. Neither the name of the University nor the names of its contributors * may be used to endorse or promote products derived from this software * without specific prior written permission. * * THIS SOFTWARE IS PROVIDED BY THE REGENTS AND CONTRIBUTORS ``AS IS'' AND * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE * ARE DISCLAIMED. IN NO EVENT SHALL THE REGENTS OR CONTRIBUTORS BE LIABLE * FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL * DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS * OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) * HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT * LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY * OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF * SUCH DAMAGE. * * @(#)vfs_bio.c 8.6 (Berkeley) 1/11/94 */ /*- * Copyright (c) 1994 Christopher G. Demetriou * * Redistribution and use in source and binary forms, with or without * modification, are permitted provided that the following conditions * are met: * 1. Redistributions of source code must retain the above copyright * notice, this list of conditions and the following disclaimer. * 2. Redistributions in binary form must reproduce the above copyright * notice, this list of conditions and the following disclaimer in the * documentation and/or other materials provided with the distribution. * 3. All advertising materials mentioning features or use of this software * must display the following acknowledgement: * This product includes software developed by the University of * California, Berkeley and its contributors. * 4. Neither the name of the University nor the names of its contributors * may be used to endorse or promote products derived from this software * without specific prior written permission. * * THIS SOFTWARE IS PROVIDED BY THE REGENTS AND CONTRIBUTORS ``AS IS'' AND * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE * ARE DISCLAIMED. IN NO EVENT SHALL THE REGENTS OR CONTRIBUTORS BE LIABLE * FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL * DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS * OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) * HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT * LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY * OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF * SUCH DAMAGE. * * @(#)vfs_bio.c 8.6 (Berkeley) 1/11/94 */ /* * The buffer cache subsystem. * * Some references: * Bach: The Design of the UNIX Operating System (Prentice Hall, 1986) * Leffler, et al.: The Design and Implementation of the 4.3BSD * UNIX Operating System (Addison Welley, 1989) * * Locking * * There are three locks: * - bufcache_lock: protects global buffer cache state. * - BC_BUSY: a long term per-buffer lock. * - buf_t::b_objlock: lock on completion (biowait vs biodone). * * For buffers associated with vnodes (a most common case) b_objlock points * to the vnode_t::v_interlock. Otherwise, it points to generic buffer_lock. * * Lock order: * bufcache_lock -> * buf_t::b_objlock */ #include <sys/cdefs.h> __KERNEL_RCSID(0, "$NetBSD: vfs_bio.c,v 1.303 2022/03/30 14:54:29 riastradh Exp $"); #ifdef _KERNEL_OPT #include "opt_bufcache.h" #include "opt_dtrace.h" #include "opt_biohist.h" #endif #include <sys/param.h> #include <sys/systm.h> #include <sys/kernel.h> #include <sys/proc.h> #include <sys/buf.h> #include <sys/vnode.h> #include <sys/mount.h> #include <sys/resourcevar.h> #include <sys/sysctl.h> #include <sys/conf.h> #include <sys/kauth.h> #include <sys/fstrans.h> #include <sys/intr.h> #include <sys/cpu.h> #include <sys/wapbl.h> #include <sys/bitops.h> #include <sys/cprng.h> #include <sys/sdt.h> #include <uvm/uvm.h> /* extern struct uvm uvm */ #include <miscfs/specfs/specdev.h> SDT_PROVIDER_DEFINE(io); SDT_PROBE_DEFINE4(io, kernel, , bbusy__start, "struct buf *"/*bp*/, "bool"/*intr*/, "int"/*timo*/, "kmutex_t *"/*interlock*/); SDT_PROBE_DEFINE5(io, kernel, , bbusy__done, "struct buf *"/*bp*/, "bool"/*intr*/, "int"/*timo*/, "kmutex_t *"/*interlock*/, "int"/*error*/); SDT_PROBE_DEFINE0(io, kernel, , getnewbuf__start); SDT_PROBE_DEFINE1(io, kernel, , getnewbuf__done, "struct buf *"/*bp*/); SDT_PROBE_DEFINE3(io, kernel, , getblk__start, "struct vnode *"/*vp*/, "daddr_t"/*blkno*/, "int"/*size*/); SDT_PROBE_DEFINE4(io, kernel, , getblk__done, "struct vnode *"/*vp*/, "daddr_t"/*blkno*/, "int"/*size*/, "struct buf *"/*bp*/); SDT_PROBE_DEFINE2(io, kernel, , brelse, "struct buf *"/*bp*/, "int"/*set*/); SDT_PROBE_DEFINE1(io, kernel, , wait__start, "struct buf *"/*bp*/); SDT_PROBE_DEFINE1(io, kernel, , wait__done, "struct buf *"/*bp*/); #ifndef BUFPAGES # define BUFPAGES 0 #endif #ifdef BUFCACHE # if (BUFCACHE < 5) || (BUFCACHE > 95) # error BUFCACHE is not between 5 and 95 # endif #else # define BUFCACHE 15 #endif u_int nbuf; /* desired number of buffer headers */ u_int bufpages = BUFPAGES; /* optional hardwired count */ u_int bufcache = BUFCACHE; /* max % of RAM to use for buffer cache */ /* * Definitions for the buffer free lists. */ #define BQUEUES 3 /* number of free buffer queues */ #define BQ_LOCKED 0 /* super-blocks &c */ #define BQ_LRU 1 /* lru, useful buffers */ #define BQ_AGE 2 /* rubbish */ struct bqueue { TAILQ_HEAD(, buf) bq_queue; uint64_t bq_bytes; buf_t *bq_marker; }; static struct bqueue bufqueues[BQUEUES] __cacheline_aligned; /* Function prototypes */ static void buf_setwm(void); static int buf_trim(void); static void *bufpool_page_alloc(struct pool *, int); static void bufpool_page_free(struct pool *, void *); static buf_t *bio_doread(struct vnode *, daddr_t, int, int); static buf_t *getnewbuf(int, int, int); static int buf_lotsfree(void); static int buf_canrelease(void); static u_long buf_mempoolidx(u_long); static u_long buf_roundsize(u_long); static void *buf_alloc(size_t); static void buf_mrelease(void *, size_t); static void binsheadfree(buf_t *, struct bqueue *); static void binstailfree(buf_t *, struct bqueue *); #ifdef DEBUG static int checkfreelist(buf_t *, struct bqueue *, int); #endif static void biointr(void *); static void biodone2(buf_t *); static void sysctl_kern_buf_setup(void); static void sysctl_vm_buf_setup(void); /* Initialization for biohist */ #include <sys/biohist.h> BIOHIST_DEFINE(biohist); void biohist_init(void) { BIOHIST_INIT(biohist, BIOHIST_SIZE); } /* * Definitions for the buffer hash lists. */ #define BUFHASH(dvp, lbn) \ (&bufhashtbl[(((long)(dvp) >> 8) + (int)(lbn)) & bufhash]) LIST_HEAD(bufhashhdr, buf) *bufhashtbl, invalhash; u_long bufhash; static int bufhash_stats(struct hashstat_sysctl *, bool); static kcondvar_t needbuffer_cv; /* * Buffer queue lock. */ kmutex_t bufcache_lock __cacheline_aligned; kmutex_t buffer_lock __cacheline_aligned; /* Software ISR for completed transfers. */ static void *biodone_sih; /* Buffer pool for I/O buffers. */ static pool_cache_t buf_cache; static pool_cache_t bufio_cache; #define MEMPOOL_INDEX_OFFSET (ilog2(DEV_BSIZE)) /* smallest pool is 512 bytes */ #define NMEMPOOLS (ilog2(MAXBSIZE) - MEMPOOL_INDEX_OFFSET + 1) __CTASSERT((1 << (NMEMPOOLS + MEMPOOL_INDEX_OFFSET - 1)) == MAXBSIZE); /* Buffer memory pools */ static struct pool bmempools[NMEMPOOLS]; static struct vm_map *buf_map; /* * Buffer memory pool allocator. */ static void * bufpool_page_alloc(struct pool *pp, int flags) { return (void *)uvm_km_alloc(buf_map, MAXBSIZE, MAXBSIZE, ((flags & PR_WAITOK) ? 0 : UVM_KMF_NOWAIT|UVM_KMF_TRYLOCK) | UVM_KMF_WIRED); } static void bufpool_page_free(struct pool *pp, void *v) { uvm_km_free(buf_map, (vaddr_t)v, MAXBSIZE, UVM_KMF_WIRED); } static struct pool_allocator bufmempool_allocator = { .pa_alloc = bufpool_page_alloc, .pa_free = bufpool_page_free, .pa_pagesz = MAXBSIZE, }; /* Buffer memory management variables */ u_long bufmem_valimit; u_long bufmem_hiwater; u_long bufmem_lowater; u_long bufmem; /* * MD code can call this to set a hard limit on the amount * of virtual memory used by the buffer cache. */ int buf_setvalimit(vsize_t sz) { /* We need to accommodate at least NMEMPOOLS of MAXBSIZE each */ if (sz < NMEMPOOLS * MAXBSIZE) return EINVAL; bufmem_valimit = sz; return 0; } static void buf_setwm(void) { bufmem_hiwater = buf_memcalc(); /* lowater is approx. 2% of memory (with bufcache = 15) */ #define BUFMEM_WMSHIFT 3 #define BUFMEM_HIWMMIN (64 * 1024 << BUFMEM_WMSHIFT) if (bufmem_hiwater < BUFMEM_HIWMMIN) /* Ensure a reasonable minimum value */ bufmem_hiwater = BUFMEM_HIWMMIN; bufmem_lowater = bufmem_hiwater >> BUFMEM_WMSHIFT; } #ifdef DEBUG int debug_verify_freelist = 0; static int checkfreelist(buf_t *bp, struct bqueue *dp, int ison) { buf_t *b; if (!debug_verify_freelist) return 1; TAILQ_FOREACH(b, &dp->bq_queue, b_freelist) { if (b == bp) return ison ? 1 : 0; } return ison ? 0 : 1; } #endif /* * Insq/Remq for the buffer hash lists. * Call with buffer queue locked. */ static void binsheadfree(buf_t *bp, struct bqueue *dp) { KASSERT(mutex_owned(&bufcache_lock)); KASSERT(bp->b_freelistindex == -1); TAILQ_INSERT_HEAD(&dp->bq_queue, bp, b_freelist); dp->bq_bytes += bp->b_bufsize; bp->b_freelistindex = dp - bufqueues; } static void binstailfree(buf_t *bp, struct bqueue *dp) { KASSERT(mutex_owned(&bufcache_lock)); KASSERTMSG(bp->b_freelistindex == -1, "double free of buffer? " "bp=%p, b_freelistindex=%d\n", bp, bp->b_freelistindex); TAILQ_INSERT_TAIL(&dp->bq_queue, bp, b_freelist); dp->bq_bytes += bp->b_bufsize; bp->b_freelistindex = dp - bufqueues; } void bremfree(buf_t *bp) { struct bqueue *dp; int bqidx = bp->b_freelistindex; KASSERT(mutex_owned(&bufcache_lock)); KASSERT(bqidx != -1); dp = &bufqueues[bqidx]; KDASSERT(checkfreelist(bp, dp, 1)); KASSERT(dp->bq_bytes >= bp->b_bufsize); TAILQ_REMOVE(&dp->bq_queue, bp, b_freelist); dp->bq_bytes -= bp->b_bufsize; /* For the sysctl helper. */ if (bp == dp->bq_marker) dp->bq_marker = NULL; #if defined(DIAGNOSTIC) bp->b_freelistindex = -1; #endif /* defined(DIAGNOSTIC) */ } /* * note that for some ports this is used by pmap bootstrap code to * determine kva size. */ u_long buf_memcalc(void) { u_long n; vsize_t mapsz = 0; /* * Determine the upper bound of memory to use for buffers. * * - If bufpages is specified, use that as the number * pages. * * - Otherwise, use bufcache as the percentage of * physical memory. */ if (bufpages != 0) { n = bufpages; } else { if (bufcache < 5) { printf("forcing bufcache %d -> 5", bufcache); bufcache = 5; } if (bufcache > 95) { printf("forcing bufcache %d -> 95", bufcache); bufcache = 95; } if (buf_map != NULL) mapsz = vm_map_max(buf_map) - vm_map_min(buf_map); n = calc_cache_size(mapsz, bufcache, (buf_map != kernel_map) ? 100 : BUFCACHE_VA_MAXPCT) / PAGE_SIZE; } n <<= PAGE_SHIFT; if (bufmem_valimit != 0 && n > bufmem_valimit) n = bufmem_valimit; return (n); } /* * Initialize buffers and hash links for buffers. */ void bufinit(void) { struct bqueue *dp; int use_std; u_int i; biodone_vfs = biodone; mutex_init(&bufcache_lock, MUTEX_DEFAULT, IPL_NONE); mutex_init(&buffer_lock, MUTEX_DEFAULT, IPL_NONE); cv_init(&needbuffer_cv, "needbuf"); if (bufmem_valimit != 0) { vaddr_t minaddr = 0, maxaddr; buf_map = uvm_km_suballoc(kernel_map, &minaddr, &maxaddr, bufmem_valimit, 0, false, 0); if (buf_map == NULL) panic("bufinit: cannot allocate submap"); } else buf_map = kernel_map; /* * Initialize buffer cache memory parameters. */ bufmem = 0; buf_setwm(); /* On "small" machines use small pool page sizes where possible */ use_std = (physmem < atop(16*1024*1024)); /* * Also use them on systems that can map the pool pages using * a direct-mapped segment. */ #ifdef PMAP_MAP_POOLPAGE use_std = 1; #endif buf_cache = pool_cache_init(sizeof(buf_t), 0, 0, 0, "bufpl", NULL, IPL_SOFTBIO, NULL, NULL, NULL); bufio_cache = pool_cache_init(sizeof(buf_t), 0, 0, 0, "biopl", NULL, IPL_BIO, NULL, NULL, NULL); for (i = 0; i < NMEMPOOLS; i++) { struct pool_allocator *pa; struct pool *pp = &bmempools[i]; u_int size = 1 << (i + MEMPOOL_INDEX_OFFSET); char *name = kmem_alloc(8, KM_SLEEP); /* XXX: never freed */ if (__predict_false(size >= 1048576)) (void)snprintf(name, 8, "buf%um", size / 1048576); else if (__predict_true(size >= 1024)) (void)snprintf(name, 8, "buf%uk", size / 1024); else (void)snprintf(name, 8, "buf%ub", size); pa = (size <= PAGE_SIZE && use_std) ? &pool_allocator_nointr : &bufmempool_allocator; pool_init(pp, size, DEV_BSIZE, 0, 0, name, pa, IPL_NONE); pool_setlowat(pp, 1); pool_sethiwat(pp, 1); } /* Initialize the buffer queues */ for (dp = bufqueues; dp < &bufqueues[BQUEUES]; dp++) { TAILQ_INIT(&dp->bq_queue); dp->bq_bytes = 0; } /* * Estimate hash table size based on the amount of memory we * intend to use for the buffer cache. The average buffer * size is dependent on our clients (i.e. filesystems). * * For now, use an empirical 3K per buffer. */ nbuf = (bufmem_hiwater / 1024) / 3; bufhashtbl = hashinit(nbuf, HASH_LIST, true, &bufhash); sysctl_kern_buf_setup(); sysctl_vm_buf_setup(); hashstat_register("bufhash", bufhash_stats); } void bufinit2(void) { biodone_sih = softint_establish(SOFTINT_BIO | SOFTINT_MPSAFE, biointr, NULL); if (biodone_sih == NULL) panic("bufinit2: can't establish soft interrupt"); } static int buf_lotsfree(void) { u_long guess; /* Always allocate if less than the low water mark. */ if (bufmem < bufmem_lowater) return 1; /* Never allocate if greater than the high water mark. */ if (bufmem > bufmem_hiwater) return 0; /* If there's anything on the AGE list, it should be eaten. */ if (TAILQ_FIRST(&bufqueues[BQ_AGE].bq_queue) != NULL) return 0; /* * The probabily of getting a new allocation is inversely * proportional to the current size of the cache above * the low water mark. Divide the total first to avoid overflows * in the product. */ guess = cprng_fast32() % 16; if ((bufmem_hiwater - bufmem_lowater) / 16 * guess >= (bufmem - bufmem_lowater)) return 1; /* Otherwise don't allocate. */ return 0; } /* * Return estimate of bytes we think need to be * released to help resolve low memory conditions. * * => called with bufcache_lock held. */ static int buf_canrelease(void) { int pagedemand, ninvalid = 0; KASSERT(mutex_owned(&bufcache_lock)); if (bufmem < bufmem_lowater) return 0; if (bufmem > bufmem_hiwater) return bufmem - bufmem_hiwater; ninvalid += bufqueues[BQ_AGE].bq_bytes; pagedemand = uvmexp.freetarg - uvm_availmem(false); if (pagedemand < 0) return ninvalid; return MAX(ninvalid, MIN(2 * MAXBSIZE, MIN((bufmem - bufmem_lowater) / 16, pagedemand * PAGE_SIZE))); } /* * Buffer memory allocation helper functions */ static u_long buf_mempoolidx(u_long size) { u_int n = 0; size -= 1; size >>= MEMPOOL_INDEX_OFFSET; while (size) { size >>= 1; n += 1; } if (n >= NMEMPOOLS) panic("buf mem pool index %d", n); return n; } static u_long buf_roundsize(u_long size) { /* Round up to nearest power of 2 */ return (1 << (buf_mempoolidx(size) + MEMPOOL_INDEX_OFFSET)); } static void * buf_alloc(size_t size) { u_int n = buf_mempoolidx(size); void *addr; while (1) { addr = pool_get(&bmempools[n], PR_NOWAIT); if (addr != NULL) break; /* No memory, see if we can free some. If so, try again */ mutex_enter(&bufcache_lock); if (buf_drain(1) > 0) { mutex_exit(&bufcache_lock); continue; } if (curlwp == uvm.pagedaemon_lwp) { mutex_exit(&bufcache_lock); return NULL; } /* Wait for buffers to arrive on the LRU queue */ cv_timedwait(&needbuffer_cv, &bufcache_lock, hz / 4); mutex_exit(&bufcache_lock); } return addr; } static void buf_mrelease(void *addr, size_t size) { pool_put(&bmempools[buf_mempoolidx(size)], addr); } /* * bread()/breadn() helper. */ static buf_t * bio_doread(struct vnode *vp, daddr_t blkno, int size, int async) { buf_t *bp; struct mount *mp; bp = getblk(vp, blkno, size, 0, 0); /* * getblk() may return NULL if we are the pagedaemon. */ if (bp == NULL) { KASSERT(curlwp == uvm.pagedaemon_lwp); return NULL; } /* * If buffer does not have data valid, start a read. * Note that if buffer is BC_INVAL, getblk() won't return it. * Therefore, it's valid if its I/O has completed or been delayed. */ if (!ISSET(bp->b_oflags, (BO_DONE | BO_DELWRI))) { /* Start I/O for the buffer. */ SET(bp->b_flags, B_READ | async); if (async) BIO_SETPRIO(bp, BPRIO_TIMELIMITED); else BIO_SETPRIO(bp, BPRIO_TIMECRITICAL); VOP_STRATEGY(vp, bp); /* Pay for the read. */ curlwp->l_ru.ru_inblock++; } else if (async) brelse(bp, 0); if (vp->v_type == VBLK) mp = spec_node_getmountedfs(vp); else mp = vp->v_mount; /* * Collect statistics on synchronous and asynchronous reads. * Reads from block devices are charged to their associated * filesystem (if any). */ if (mp != NULL) { if (async == 0) mp->mnt_stat.f_syncreads++; else mp->mnt_stat.f_asyncreads++; } return (bp); } /* * Read a disk block. * This algorithm described in Bach (p.54). */ int bread(struct vnode *vp, daddr_t blkno, int size, int flags, buf_t **bpp) { buf_t *bp; int error; BIOHIST_FUNC(__func__); BIOHIST_CALLED(biohist); /* Get buffer for block. */ bp = *bpp = bio_doread(vp, blkno, size, 0); if (bp == NULL) return ENOMEM; /* Wait for the read to complete, and return result. */ error = biowait(bp); if (error == 0 && (flags & B_MODIFY) != 0) error = fscow_run(bp, true); if (error) { brelse(bp, 0); *bpp = NULL; } return error; } /* * Read-ahead multiple disk blocks. The first is sync, the rest async. * Trivial modification to the breada algorithm presented in Bach (p.55). */ int breadn(struct vnode *vp, daddr_t blkno, int size, daddr_t *rablks, int *rasizes, int nrablks, int flags, buf_t **bpp) { buf_t *bp; int error, i; BIOHIST_FUNC(__func__); BIOHIST_CALLED(biohist); bp = *bpp = bio_doread(vp, blkno, size, 0); if (bp == NULL) return ENOMEM; /* * For each of the read-ahead blocks, start a read, if necessary. */ mutex_enter(&bufcache_lock); for (i = 0; i < nrablks; i++) { /* If it's in the cache, just go on to next one. */ if (incore(vp, rablks[i])) continue; /* Get a buffer for the read-ahead block */ mutex_exit(&bufcache_lock); (void) bio_doread(vp, rablks[i], rasizes[i], B_ASYNC); mutex_enter(&bufcache_lock); } mutex_exit(&bufcache_lock); /* Otherwise, we had to start a read for it; wait until it's valid. */ error = biowait(bp); if (error == 0 && (flags & B_MODIFY) != 0) error = fscow_run(bp, true); if (error) { brelse(bp, 0); *bpp = NULL; } return error; } /* * Block write. Described in Bach (p.56) */ int bwrite(buf_t *bp) { int rv, sync, wasdelayed; struct vnode *vp; struct mount *mp; BIOHIST_FUNC(__func__); BIOHIST_CALLARGS(biohist, "bp=%#jx", (uintptr_t)bp, 0, 0, 0); KASSERT(ISSET(bp->b_cflags, BC_BUSY)); KASSERT(!cv_has_waiters(&bp->b_done)); vp = bp->b_vp; /* * dholland 20160728 AFAICT vp==NULL must be impossible as it * will crash upon reaching VOP_STRATEGY below... see further * analysis on tech-kern. */ KASSERTMSG(vp != NULL, "bwrite given buffer with null vnode"); if (vp != NULL) { KASSERT(bp->b_objlock == vp->v_interlock); if (vp->v_type == VBLK) mp = spec_node_getmountedfs(vp); else mp = vp->v_mount; } else { mp = NULL; } if (mp && mp->mnt_wapbl) { if (bp->b_iodone != mp->mnt_wapbl_op->wo_wapbl_biodone) { bdwrite(bp); return 0; } } /* * Remember buffer type, to switch on it later. If the write was * synchronous, but the file system was mounted with MNT_ASYNC, * convert it to a delayed write. * XXX note that this relies on delayed tape writes being converted * to async, not sync writes (which is safe, but ugly). */ sync = !ISSET(bp->b_flags, B_ASYNC); if (sync && mp != NULL && ISSET(mp->mnt_flag, MNT_ASYNC)) { bdwrite(bp); return (0); } /* * Collect statistics on synchronous and asynchronous writes. * Writes to block devices are charged to their associated * filesystem (if any). */ if (mp != NULL) { if (sync) mp->mnt_stat.f_syncwrites++; else mp->mnt_stat.f_asyncwrites++; } /* * Pay for the I/O operation and make sure the buf is on the correct * vnode queue. */ bp->b_error = 0; wasdelayed = ISSET(bp->b_oflags, BO_DELWRI); CLR(bp->b_flags, B_READ); if (wasdelayed) { mutex_enter(&bufcache_lock); mutex_enter(bp->b_objlock); CLR(bp->b_oflags, BO_DONE | BO_DELWRI); reassignbuf(bp, bp->b_vp); /* Wake anyone trying to busy the buffer via vnode's lists. */ cv_broadcast(&bp->b_busy); mutex_exit(&bufcache_lock); } else { curlwp->l_ru.ru_oublock++; mutex_enter(bp->b_objlock); CLR(bp->b_oflags, BO_DONE | BO_DELWRI); } if (vp != NULL) vp->v_numoutput++; mutex_exit(bp->b_objlock); /* Initiate disk write. */ if (sync) BIO_SETPRIO(bp, BPRIO_TIMECRITICAL); else BIO_SETPRIO(bp, BPRIO_TIMELIMITED); VOP_STRATEGY(vp, bp); if (sync) { /* If I/O was synchronous, wait for it to complete. */ rv = biowait(bp); /* Release the buffer. */ brelse(bp, 0); return (rv); } else { return (0); } } int vn_bwrite(void *v) { struct vop_bwrite_args *ap = v; return (bwrite(ap->a_bp)); } /* * Delayed write. * * The buffer is marked dirty, but is not queued for I/O. * This routine should be used when the buffer is expected * to be modified again soon, typically a small write that * partially fills a buffer. * * NB: magnetic tapes cannot be delayed; they must be * written in the order that the writes are requested. * * Described in Leffler, et al. (pp. 208-213). */ void bdwrite(buf_t *bp) { BIOHIST_FUNC(__func__); BIOHIST_CALLARGS(biohist, "bp=%#jx", (uintptr_t)bp, 0, 0, 0); KASSERT(bp->b_vp == NULL || bp->b_vp->v_tag != VT_UFS || bp->b_vp->v_type == VBLK || ISSET(bp->b_flags, B_COWDONE)); KASSERT(ISSET(bp->b_cflags, BC_BUSY)); KASSERT(!cv_has_waiters(&bp->b_done)); /* If this is a tape block, write the block now. */ if (bdev_type(bp->b_dev) == D_TAPE) { bawrite(bp); return; } if (wapbl_vphaswapbl(bp->b_vp)) { struct mount *mp = wapbl_vptomp(bp->b_vp); if (bp->b_iodone != mp->mnt_wapbl_op->wo_wapbl_biodone) { WAPBL_ADD_BUF(mp, bp); } } /* * If the block hasn't been seen before: * (1) Mark it as having been seen, * (2) Charge for the write, * (3) Make sure it's on its vnode's correct block list. */ KASSERT(bp->b_vp == NULL || bp->b_objlock == bp->b_vp->v_interlock); if (!ISSET(bp->b_oflags, BO_DELWRI)) { mutex_enter(&bufcache_lock); mutex_enter(bp->b_objlock); SET(bp->b_oflags, BO_DELWRI); curlwp->l_ru.ru_oublock++; reassignbuf(bp, bp->b_vp); /* Wake anyone trying to busy the buffer via vnode's lists. */ cv_broadcast(&bp->b_busy); mutex_exit(&bufcache_lock); } else { mutex_enter(bp->b_objlock); } /* Otherwise, the "write" is done, so mark and release the buffer. */ CLR(bp->b_oflags, BO_DONE); mutex_exit(bp->b_objlock); brelse(bp, 0); } /* * Asynchronous block write; just an asynchronous bwrite(). */ void bawrite(buf_t *bp) { KASSERT(ISSET(bp->b_cflags, BC_BUSY)); KASSERT(bp->b_vp != NULL); SET(bp->b_flags, B_ASYNC); VOP_BWRITE(bp->b_vp, bp); } /* * Release a buffer on to the free lists. * Described in Bach (p. 46). */ void brelsel(buf_t *bp, int set) { struct bqueue *bufq; struct vnode *vp; SDT_PROBE2(io, kernel, , brelse, bp, set); KASSERT(bp != NULL); KASSERT(mutex_owned(&bufcache_lock)); KASSERT(!cv_has_waiters(&bp->b_done)); SET(bp->b_cflags, set); KASSERT(ISSET(bp->b_cflags, BC_BUSY)); KASSERT(bp->b_iodone == NULL); /* Wake up any processes waiting for any buffer to become free. */ cv_signal(&needbuffer_cv); /* Wake up any proceeses waiting for _this_ buffer to become free */ if (ISSET(bp->b_cflags, BC_WANTED)) CLR(bp->b_cflags, BC_WANTED|BC_AGE); /* If it's clean clear the copy-on-write flag. */ if (ISSET(bp->b_flags, B_COWDONE)) { mutex_enter(bp->b_objlock); if (!ISSET(bp->b_oflags, BO_DELWRI)) CLR(bp->b_flags, B_COWDONE); mutex_exit(bp->b_objlock); } /* * Determine which queue the buffer should be on, then put it there. */ /* If it's locked, don't report an error; try again later. */ if (ISSET(bp->b_flags, B_LOCKED)) bp->b_error = 0; /* If it's not cacheable, or an error, mark it invalid. */ if (ISSET(bp->b_cflags, BC_NOCACHE) || bp->b_error != 0) SET(bp->b_cflags, BC_INVAL); if (ISSET(bp->b_cflags, BC_VFLUSH)) { /* * This is a delayed write buffer that was just flushed to * disk. It is still on the LRU queue. If it's become * invalid, then we need to move it to a different queue; * otherwise leave it in its current position. */ CLR(bp->b_cflags, BC_VFLUSH); if (!ISSET(bp->b_cflags, BC_INVAL|BC_AGE) && !ISSET(bp->b_flags, B_LOCKED) && bp->b_error == 0) { KDASSERT(checkfreelist(bp, &bufqueues[BQ_LRU], 1)); goto already_queued; } else { bremfree(bp); } } KDASSERT(checkfreelist(bp, &bufqueues[BQ_AGE], 0)); KDASSERT(checkfreelist(bp, &bufqueues[BQ_LRU], 0)); KDASSERT(checkfreelist(bp, &bufqueues[BQ_LOCKED], 0)); if ((bp->b_bufsize <= 0) || ISSET(bp->b_cflags, BC_INVAL)) { /* * If it's invalid or empty, dissociate it from its vnode * and put on the head of the appropriate queue. */ if (ISSET(bp->b_flags, B_LOCKED)) { if (wapbl_vphaswapbl(vp = bp->b_vp)) { struct mount *mp = wapbl_vptomp(vp); KASSERT(bp->b_iodone != mp->mnt_wapbl_op->wo_wapbl_biodone); WAPBL_REMOVE_BUF(mp, bp); } } mutex_enter(bp->b_objlock); CLR(bp->b_oflags, BO_DONE|BO_DELWRI); if ((vp = bp->b_vp) != NULL) { KASSERT(bp->b_objlock == vp->v_interlock); reassignbuf(bp, bp->b_vp); brelvp(bp); mutex_exit(vp->v_interlock); } else { KASSERT(bp->b_objlock == &buffer_lock); mutex_exit(bp->b_objlock); } /* We want to dispose of the buffer, so wake everybody. */ cv_broadcast(&bp->b_busy); if (bp->b_bufsize <= 0) /* no data */ goto already_queued; else /* invalid data */ bufq = &bufqueues[BQ_AGE]; binsheadfree(bp, bufq); } else { /* * It has valid data. Put it on the end of the appropriate * queue, so that it'll stick around for as long as possible. * If buf is AGE, but has dependencies, must put it on last * bufqueue to be scanned, ie LRU. This protects against the * livelock where BQ_AGE only has buffers with dependencies, * and we thus never get to the dependent buffers in BQ_LRU. */ if (ISSET(bp->b_flags, B_LOCKED)) { /* locked in core */ bufq = &bufqueues[BQ_LOCKED]; } else if (!ISSET(bp->b_cflags, BC_AGE)) { /* valid data */ bufq = &bufqueues[BQ_LRU]; } else { /* stale but valid data */ bufq = &bufqueues[BQ_AGE]; } binstailfree(bp, bufq); } already_queued: /* Unlock the buffer. */ CLR(bp->b_cflags, BC_AGE|BC_BUSY|BC_NOCACHE); CLR(bp->b_flags, B_ASYNC); /* * Wake only the highest priority waiter on the lock, in order to * prevent a thundering herd: many LWPs simultaneously awakening and * competing for the buffer's lock. Testing in 2019 revealed this * to reduce contention on bufcache_lock tenfold during a kernel * compile. Here and elsewhere, when the buffer is changing * identity, being disposed of, or moving from one list to another, * we wake all lock requestors. */ if (bp->b_bufsize <= 0) { cv_broadcast(&bp->b_busy); buf_destroy(bp); #ifdef DEBUG memset((char *)bp, 0, sizeof(*bp)); #endif pool_cache_put(buf_cache, bp); } else cv_signal(&bp->b_busy); } void brelse(buf_t *bp, int set) { mutex_enter(&bufcache_lock); brelsel(bp, set); mutex_exit(&bufcache_lock); } /* * Determine if a block is in the cache. * Just look on what would be its hash chain. If it's there, return * a pointer to it, unless it's marked invalid. If it's marked invalid, * we normally don't return the buffer, unless the caller explicitly * wants us to. */ buf_t * incore(struct vnode *vp, daddr_t blkno) { buf_t *bp; KASSERT(mutex_owned(&bufcache_lock)); /* Search hash chain */ LIST_FOREACH(bp, BUFHASH(vp, blkno), b_hash) { if (bp->b_lblkno == blkno && bp->b_vp == vp && !ISSET(bp->b_cflags, BC_INVAL)) { KASSERT(bp->b_objlock == vp->v_interlock); return (bp); } } return (NULL); } /* * Get a block of requested size that is associated with * a given vnode and block offset. If it is found in the * block cache, mark it as having been found, make it busy * and return it. Otherwise, return an empty block of the * correct size. It is up to the caller to insure that the * cached blocks be of the correct size. */ buf_t * getblk(struct vnode *vp, daddr_t blkno, int size, int slpflag, int slptimeo) { int err, preserve; buf_t *bp; mutex_enter(&bufcache_lock); SDT_PROBE3(io, kernel, , getblk__start, vp, blkno, size); loop: bp = incore(vp, blkno); if (bp != NULL) { err = bbusy(bp, ((slpflag & PCATCH) != 0), slptimeo, NULL); if (err != 0) { if (err == EPASSTHROUGH) goto loop; mutex_exit(&bufcache_lock); SDT_PROBE4(io, kernel, , getblk__done, vp, blkno, size, NULL); return (NULL); } KASSERT(!cv_has_waiters(&bp->b_done)); #ifdef DIAGNOSTIC if (ISSET(bp->b_oflags, BO_DONE|BO_DELWRI) && bp->b_bcount < size && vp->v_type != VBLK) panic("getblk: block size invariant failed"); #endif bremfree(bp); preserve = 1; } else { if ((bp = getnewbuf(slpflag, slptimeo, 0)) == NULL) goto loop; if (incore(vp, blkno) != NULL) { /* The block has come into memory in the meantime. */ brelsel(bp, 0); goto loop; } LIST_INSERT_HEAD(BUFHASH(vp, blkno), bp, b_hash); bp->b_blkno = bp->b_lblkno = bp->b_rawblkno = blkno; mutex_enter(vp->v_interlock); bgetvp(vp, bp); mutex_exit(vp->v_interlock); preserve = 0; } mutex_exit(&bufcache_lock); /* * LFS can't track total size of B_LOCKED buffer (locked_queue_bytes) * if we re-size buffers here. */ if (ISSET(bp->b_flags, B_LOCKED)) { KASSERT(bp->b_bufsize >= size); } else { if (allocbuf(bp, size, preserve)) { mutex_enter(&bufcache_lock); LIST_REMOVE(bp, b_hash); brelsel(bp, BC_INVAL); mutex_exit(&bufcache_lock); SDT_PROBE4(io, kernel, , getblk__done, vp, blkno, size, NULL); return NULL; } } BIO_SETPRIO(bp, BPRIO_DEFAULT); SDT_PROBE4(io, kernel, , getblk__done, vp, blkno, size, bp); return (bp); } /* * Get an empty, disassociated buffer of given size. */ buf_t * geteblk(int size) { buf_t *bp; int error __diagused; mutex_enter(&bufcache_lock); while ((bp = getnewbuf(0, 0, 0)) == NULL) ; SET(bp->b_cflags, BC_INVAL); LIST_INSERT_HEAD(&invalhash, bp, b_hash); mutex_exit(&bufcache_lock); BIO_SETPRIO(bp, BPRIO_DEFAULT); error = allocbuf(bp, size, 0); KASSERT(error == 0); return (bp); } /* * Expand or contract the actual memory allocated to a buffer. * * If the buffer shrinks, data is lost, so it's up to the * caller to have written it out *first*; this routine will not * start a write. If the buffer grows, it's the callers * responsibility to fill out the buffer's additional contents. */ int allocbuf(buf_t *bp, int size, int preserve) { void *addr; vsize_t oldsize, desired_size; int oldcount; int delta; desired_size = buf_roundsize(size); if (desired_size > MAXBSIZE) printf("allocbuf: buffer larger than MAXBSIZE requested"); oldcount = bp->b_bcount; bp->b_bcount = size; oldsize = bp->b_bufsize; if (oldsize == desired_size) { /* * Do not short cut the WAPBL resize, as the buffer length * could still have changed and this would corrupt the * tracking of the transaction length. */ goto out; } /* * If we want a buffer of a different size, re-allocate the * buffer's memory; copy old content only if needed. */ addr = buf_alloc(desired_size); if (addr == NULL) return ENOMEM; if (preserve) memcpy(addr, bp->b_data, MIN(oldsize,desired_size)); if (bp->b_data != NULL) buf_mrelease(bp->b_data, oldsize); bp->b_data = addr; bp->b_bufsize = desired_size; /* * Update overall buffer memory counter (protected by bufcache_lock) */ delta = (long)desired_size - (long)oldsize; mutex_enter(&bufcache_lock); if ((bufmem += delta) > bufmem_hiwater) { /* * Need to trim overall memory usage. */ while (buf_canrelease()) { if (preempt_needed()) { mutex_exit(&bufcache_lock); preempt(); mutex_enter(&bufcache_lock); } if (buf_trim() == 0) break; } } mutex_exit(&bufcache_lock); out: if (wapbl_vphaswapbl(bp->b_vp)) WAPBL_RESIZE_BUF(wapbl_vptomp(bp->b_vp), bp, oldsize, oldcount); return 0; } /* * Find a buffer which is available for use. * Select something from a free list. * Preference is to AGE list, then LRU list. * * Called with the buffer queues locked. * Return buffer locked. */ static buf_t * getnewbuf(int slpflag, int slptimeo, int from_bufq) { buf_t *bp; struct vnode *vp; struct mount *transmp = NULL; SDT_PROBE0(io, kernel, , getnewbuf__start); start: KASSERT(mutex_owned(&bufcache_lock)); /* * Get a new buffer from the pool. */ if (!from_bufq && buf_lotsfree()) { mutex_exit(&bufcache_lock); bp = pool_cache_get(buf_cache, PR_NOWAIT); if (bp != NULL) { memset((char *)bp, 0, sizeof(*bp)); buf_init(bp); SET(bp->b_cflags, BC_BUSY); /* mark buffer busy */ mutex_enter(&bufcache_lock); #if defined(DIAGNOSTIC) bp->b_freelistindex = -1; #endif /* defined(DIAGNOSTIC) */ SDT_PROBE1(io, kernel, , getnewbuf__done, bp); return (bp); } mutex_enter(&bufcache_lock); } KASSERT(mutex_owned(&bufcache_lock)); if ((bp = TAILQ_FIRST(&bufqueues[BQ_AGE].bq_queue)) != NULL) { KASSERT(!ISSET(bp->b_oflags, BO_DELWRI)); } else { TAILQ_FOREACH(bp, &bufqueues[BQ_LRU].bq_queue, b_freelist) { if (ISSET(bp->b_cflags, BC_VFLUSH) || !ISSET(bp->b_oflags, BO_DELWRI)) break; if (fstrans_start_nowait(bp->b_vp->v_mount) == 0) { KASSERT(transmp == NULL); transmp = bp->b_vp->v_mount; break; } } } if (bp != NULL) { KASSERT(!ISSET(bp->b_cflags, BC_BUSY) || ISSET(bp->b_cflags, BC_VFLUSH)); bremfree(bp); /* Buffer is no longer on free lists. */ SET(bp->b_cflags, BC_BUSY); /* Wake anyone trying to lock the old identity. */ cv_broadcast(&bp->b_busy); } else { /* * XXX: !from_bufq should be removed. */ if (!from_bufq || curlwp != uvm.pagedaemon_lwp) { /* wait for a free buffer of any kind */ if ((slpflag & PCATCH) != 0) (void)cv_timedwait_sig(&needbuffer_cv, &bufcache_lock, slptimeo); else (void)cv_timedwait(&needbuffer_cv, &bufcache_lock, slptimeo); } SDT_PROBE1(io, kernel, , getnewbuf__done, NULL); return (NULL); } #ifdef DIAGNOSTIC if (bp->b_bufsize <= 0) panic("buffer %p: on queue but empty", bp); #endif if (ISSET(bp->b_cflags, BC_VFLUSH)) { /* * This is a delayed write buffer being flushed to disk. Make * sure it gets aged out of the queue when it's finished, and * leave it off the LRU queue. */ CLR(bp->b_cflags, BC_VFLUSH); SET(bp->b_cflags, BC_AGE); goto start; } KASSERT(ISSET(bp->b_cflags, BC_BUSY)); KASSERT(!cv_has_waiters(&bp->b_done)); /* * If buffer was a delayed write, start it and return NULL * (since we might sleep while starting the write). */ if (ISSET(bp->b_oflags, BO_DELWRI)) { /* * This buffer has gone through the LRU, so make sure it gets * reused ASAP. */ SET(bp->b_cflags, BC_AGE); mutex_exit(&bufcache_lock); bawrite(bp); KASSERT(transmp != NULL); fstrans_done(transmp); mutex_enter(&bufcache_lock); SDT_PROBE1(io, kernel, , getnewbuf__done, NULL); return (NULL); } KASSERT(transmp == NULL); vp = bp->b_vp; /* clear out various other fields */ bp->b_cflags = BC_BUSY; bp->b_oflags = 0; bp->b_flags = 0; bp->b_dev = NODEV; bp->b_blkno = 0; bp->b_lblkno = 0; bp->b_rawblkno = 0; bp->b_iodone = 0; bp->b_error = 0; bp->b_resid = 0; bp->b_bcount = 0; LIST_REMOVE(bp, b_hash); /* Disassociate us from our vnode, if we had one... */ if (vp != NULL) { mutex_enter(vp->v_interlock); brelvp(bp); mutex_exit(vp->v_interlock); } SDT_PROBE1(io, kernel, , getnewbuf__done, bp); return (bp); } /* * Invalidate the specified buffer if it exists. */ void binvalbuf(struct vnode *vp, daddr_t blkno) { buf_t *bp; int err; mutex_enter(&bufcache_lock); loop: bp = incore(vp, blkno); if (bp != NULL) { err = bbusy(bp, 0, 0, NULL); if (err == EPASSTHROUGH) goto loop; bremfree(bp); if (ISSET(bp->b_oflags, BO_DELWRI)) { SET(bp->b_cflags, BC_NOCACHE); mutex_exit(&bufcache_lock); bwrite(bp); } else { brelsel(bp, BC_INVAL); mutex_exit(&bufcache_lock); } } else mutex_exit(&bufcache_lock); } /* * Attempt to free an aged buffer off the queues. * Called with queue lock held. * Returns the amount of buffer memory freed. */ static int buf_trim(void) { buf_t *bp; long size; KASSERT(mutex_owned(&bufcache_lock)); /* Instruct getnewbuf() to get buffers off the queues */ if ((bp = getnewbuf(PCATCH, 1, 1)) == NULL) return 0; KASSERT((bp->b_cflags & BC_WANTED) == 0); size = bp->b_bufsize; bufmem -= size; if (size > 0) { buf_mrelease(bp->b_data, size); bp->b_bcount = bp->b_bufsize = 0; } /* brelse() will return the buffer to the global buffer pool */ brelsel(bp, 0); return size; } int buf_drain(int n) { int size = 0, sz; KASSERT(mutex_owned(&bufcache_lock)); while (size < n && bufmem > bufmem_lowater) { sz = buf_trim(); if (sz <= 0) break; size += sz; } return size; } /* * Wait for operations on the buffer to complete. * When they do, extract and return the I/O's error value. */ int biowait(buf_t *bp) { BIOHIST_FUNC(__func__); KASSERT(ISSET(bp->b_cflags, BC_BUSY)); SDT_PROBE1(io, kernel, , wait__start, bp); mutex_enter(bp->b_objlock); BIOHIST_CALLARGS(biohist, "bp=%#jx, oflags=0x%jx, ret_addr=%#jx", (uintptr_t)bp, bp->b_oflags, (uintptr_t)__builtin_return_address(0), 0); while (!ISSET(bp->b_oflags, BO_DONE | BO_DELWRI)) { BIOHIST_LOG(biohist, "waiting bp=%#jx", (uintptr_t)bp, 0, 0, 0); cv_wait(&bp->b_done, bp->b_objlock); } mutex_exit(bp->b_objlock); SDT_PROBE1(io, kernel, , wait__done, bp); BIOHIST_LOG(biohist, "return %jd", bp->b_error, 0, 0, 0); return bp->b_error; } /* * Mark I/O complete on a buffer. * * If a callback has been requested, e.g. the pageout * daemon, do so. Otherwise, awaken waiting processes. * * [ Leffler, et al., says on p.247: * "This routine wakes up the blocked process, frees the buffer * for an asynchronous write, or, for a request by the pagedaemon * process, invokes a procedure specified in the buffer structure" ] * * In real life, the pagedaemon (or other system processes) wants * to do async stuff too, and doesn't want the buffer brelse()'d. * (for swap pager, that puts swap buffers on the free lists (!!!), * for the vn device, that puts allocated buffers on the free lists!) */ void biodone(buf_t *bp) { int s; BIOHIST_FUNC(__func__); KASSERT(!ISSET(bp->b_oflags, BO_DONE)); if (cpu_intr_p()) { /* From interrupt mode: defer to a soft interrupt. */ s = splvm(); TAILQ_INSERT_TAIL(&curcpu()->ci_data.cpu_biodone, bp, b_actq); BIOHIST_CALLARGS(biohist, "bp=%#jx, softint scheduled", (uintptr_t)bp, 0, 0, 0); softint_schedule(biodone_sih); splx(s); } else { /* Process now - the buffer may be freed soon. */ biodone2(bp); } } SDT_PROBE_DEFINE1(io, kernel, , done, "struct buf *"/*bp*/); static void biodone2(buf_t *bp) { void (*callout)(buf_t *); SDT_PROBE1(io, kernel, ,done, bp); BIOHIST_FUNC(__func__); BIOHIST_CALLARGS(biohist, "bp=%#jx", (uintptr_t)bp, 0, 0, 0); mutex_enter(bp->b_objlock); /* Note that the transfer is done. */ if (ISSET(bp->b_oflags, BO_DONE)) panic("biodone2 already"); CLR(bp->b_flags, B_COWDONE); SET(bp->b_oflags, BO_DONE); BIO_SETPRIO(bp, BPRIO_DEFAULT); /* Wake up waiting writers. */ if (!ISSET(bp->b_flags, B_READ)) vwakeup(bp); if ((callout = bp->b_iodone) != NULL) { BIOHIST_LOG(biohist, "callout %#jx", (uintptr_t)callout, 0, 0, 0); /* Note callout done, then call out. */ KASSERT(!cv_has_waiters(&bp->b_done)); bp->b_iodone = NULL; mutex_exit(bp->b_objlock); (*callout)(bp); } else if (ISSET(bp->b_flags, B_ASYNC)) { /* If async, release. */ BIOHIST_LOG(biohist, "async", 0, 0, 0, 0); KASSERT(!cv_has_waiters(&bp->b_done)); mutex_exit(bp->b_objlock); brelse(bp, 0); } else { /* Otherwise just wake up waiters in biowait(). */ BIOHIST_LOG(biohist, "wake-up", 0, 0, 0, 0); cv_broadcast(&bp->b_done); mutex_exit(bp->b_objlock); } } static void biointr(void *cookie) { struct cpu_info *ci; buf_t *bp; int s; BIOHIST_FUNC(__func__); BIOHIST_CALLED(biohist); ci = curcpu(); s = splvm(); while (!TAILQ_EMPTY(&ci->ci_data.cpu_biodone)) { KASSERT(curcpu() == ci); bp = TAILQ_FIRST(&ci->ci_data.cpu_biodone); TAILQ_REMOVE(&ci->ci_data.cpu_biodone, bp, b_actq); splx(s); BIOHIST_LOG(biohist, "bp=%#jx", (uintptr_t)bp, 0, 0, 0); biodone2(bp); s = splvm(); } splx(s); } static void sysctl_fillbuf(const buf_t *i, struct buf_sysctl *o) { const bool allowaddr = get_expose_address(curproc); memset(o, 0, sizeof(*o)); o->b_flags = i->b_flags | i->b_cflags | i->b_oflags; o->b_error = i->b_error; o->b_prio = i->b_prio; o->b_dev = i->b_dev; o->b_bufsize = i->b_bufsize; o->b_bcount = i->b_bcount; o->b_resid = i->b_resid; COND_SET_VALUE(o->b_addr, PTRTOUINT64(i->b_data), allowaddr); o->b_blkno = i->b_blkno; o->b_rawblkno = i->b_rawblkno; COND_SET_VALUE(o->b_iodone, PTRTOUINT64(i->b_iodone), allowaddr); COND_SET_VALUE(o->b_proc, PTRTOUINT64(i->b_proc), allowaddr); COND_SET_VALUE(o->b_vp, PTRTOUINT64(i->b_vp), allowaddr); COND_SET_VALUE(o->b_saveaddr, PTRTOUINT64(i->b_saveaddr), allowaddr); o->b_lblkno = i->b_lblkno; } static int sysctl_dobuf(SYSCTLFN_ARGS) { buf_t *bp; struct buf_sysctl bs; struct bqueue *bq; char *dp; u_int i, op, arg; size_t len, needed, elem_size, out_size; int error, elem_count, retries; if (namelen == 1 && name[0] == CTL_QUERY) return (sysctl_query(SYSCTLFN_CALL(rnode))); if (namelen != 4) return (EINVAL); retries = 100; retry: dp = oldp; len = (oldp != NULL) ? *oldlenp : 0; op = name[0]; arg = name[1]; elem_size = name[2]; elem_count = name[3]; out_size = MIN(sizeof(bs), elem_size); /* * at the moment, these are just "placeholders" to make the * API for retrieving kern.buf data more extensible in the * future. * * XXX kern.buf currently has "netbsd32" issues. hopefully * these will be resolved at a later point. */ if (op != KERN_BUF_ALL || arg != KERN_BUF_ALL || elem_size < 1 || elem_count < 0) return (EINVAL); if (oldp == NULL) { /* count only, don't run through the buffer queues */ needed = pool_cache_nget(buf_cache) - pool_cache_nput(buf_cache); *oldlenp = (needed + KERN_BUFSLOP) * elem_size; return 0; } error = 0; needed = 0; sysctl_unlock(); mutex_enter(&bufcache_lock); for (i = 0; i < BQUEUES; i++) { bq = &bufqueues[i]; TAILQ_FOREACH(bp, &bq->bq_queue, b_freelist) { bq->bq_marker = bp; if (len >= elem_size && elem_count > 0) { sysctl_fillbuf(bp, &bs); mutex_exit(&bufcache_lock); error = copyout(&bs, dp, out_size); mutex_enter(&bufcache_lock); if (error) break; if (bq->bq_marker != bp) { /* * This sysctl node is only for * statistics. Retry; if the * queue keeps changing, then * bail out. */ if (retries-- == 0) { error = EAGAIN; break; } mutex_exit(&bufcache_lock); sysctl_relock(); goto retry; } dp += elem_size; len -= elem_size; } needed += elem_size; if (elem_count > 0 && elem_count != INT_MAX) elem_count--; } if (error != 0) break; } mutex_exit(&bufcache_lock); sysctl_relock(); *oldlenp = needed; return (error); } static int sysctl_bufvm_update(SYSCTLFN_ARGS) { int error, rv; struct sysctlnode node; unsigned int temp_bufcache; unsigned long temp_water; /* Take a copy of the supplied node and its data */ node = *rnode; if (node.sysctl_data == &bufcache) { node.sysctl_data = &temp_bufcache; temp_bufcache = *(unsigned int *)rnode->sysctl_data; } else { node.sysctl_data = &temp_water; temp_water = *(unsigned long *)rnode->sysctl_data; } /* Update the copy */ error = sysctl_lookup(SYSCTLFN_CALL(&node)); if (error || newp == NULL) return (error); if (rnode->sysctl_data == &bufcache) { if (temp_bufcache > 100) return (EINVAL); bufcache = temp_bufcache; buf_setwm(); } else if (rnode->sysctl_data == &bufmem_lowater) { if (bufmem_hiwater - temp_water < 16) return (EINVAL); bufmem_lowater = temp_water; } else if (rnode->sysctl_data == &bufmem_hiwater) { if (temp_water - bufmem_lowater < 16) return (EINVAL); bufmem_hiwater = temp_water; } else return (EINVAL); /* Drain until below new high water mark */ sysctl_unlock(); mutex_enter(&bufcache_lock); while (bufmem > bufmem_hiwater) { rv = buf_drain((bufmem - bufmem_hiwater) / (2 * 1024)); if (rv <= 0) break; } mutex_exit(&bufcache_lock); sysctl_relock(); return 0; } static struct sysctllog *vfsbio_sysctllog; static void sysctl_kern_buf_setup(void) { sysctl_createv(&vfsbio_sysctllog, 0, NULL, NULL, CTLFLAG_PERMANENT, CTLTYPE_NODE, "buf", SYSCTL_DESCR("Kernel buffer cache information"), sysctl_dobuf, 0, NULL, 0, CTL_KERN, KERN_BUF, CTL_EOL); } static void sysctl_vm_buf_setup(void) { sysctl_createv(&vfsbio_sysctllog, 0, NULL, NULL, CTLFLAG_PERMANENT|CTLFLAG_READWRITE, CTLTYPE_INT, "bufcache", SYSCTL_DESCR("Percentage of physical memory to use for " "buffer cache"), sysctl_bufvm_update, 0, &bufcache, 0, CTL_VM, CTL_CREATE, CTL_EOL); sysctl_createv(&vfsbio_sysctllog, 0, NULL, NULL, CTLFLAG_PERMANENT|CTLFLAG_READONLY, CTLTYPE_LONG, "bufmem", SYSCTL_DESCR("Amount of kernel memory used by buffer " "cache"), NULL, 0, &bufmem, 0, CTL_VM, CTL_CREATE, CTL_EOL); sysctl_createv(&vfsbio_sysctllog, 0, NULL, NULL, CTLFLAG_PERMANENT|CTLFLAG_READWRITE, CTLTYPE_LONG, "bufmem_lowater", SYSCTL_DESCR("Minimum amount of kernel memory to " "reserve for buffer cache"), sysctl_bufvm_update, 0, &bufmem_lowater, 0, CTL_VM, CTL_CREATE, CTL_EOL); sysctl_createv(&vfsbio_sysctllog, 0, NULL, NULL, CTLFLAG_PERMANENT|CTLFLAG_READWRITE, CTLTYPE_LONG, "bufmem_hiwater", SYSCTL_DESCR("Maximum amount of kernel memory to use " "for buffer cache"), sysctl_bufvm_update, 0, &bufmem_hiwater, 0, CTL_VM, CTL_CREATE, CTL_EOL); } static int bufhash_stats(struct hashstat_sysctl *hs, bool fill) { buf_t *bp; uint64_t chain; strlcpy(hs->hash_name, "bufhash", sizeof(hs->hash_name)); strlcpy(hs->hash_desc, "buffer hash", sizeof(hs->hash_desc)); if (!fill) return 0; hs->hash_size = bufhash + 1; for (size_t i = 0; i < hs->hash_size; i++) { chain = 0; mutex_enter(&bufcache_lock); LIST_FOREACH(bp, &bufhashtbl[i], b_hash) { chain++; } mutex_exit(&bufcache_lock); if (chain > 0) { hs->hash_used++; hs->hash_items += chain; if (chain > hs->hash_maxchain) hs->hash_maxchain = chain; } preempt_point(); } return 0; } #ifdef DEBUG /* * Print out statistics on the current allocation of the buffer pool. * Can be enabled to print out on every ``sync'' by setting "syncprt" * in vfs_syscalls.c using sysctl. */ void vfs_bufstats(void) { int i, j, count; buf_t *bp; struct bqueue *dp; int counts[MAXBSIZE / MIN_PAGE_SIZE + 1]; static const char *bname[BQUEUES] = { "LOCKED", "LRU", "AGE" }; for (dp = bufqueues, i = 0; dp < &bufqueues[BQUEUES]; dp++, i++) { count = 0; memset(counts, 0, sizeof(counts)); TAILQ_FOREACH(bp, &dp->bq_queue, b_freelist) { counts[bp->b_bufsize / PAGE_SIZE]++; count++; } printf("%s: total-%d", bname[i], count); for (j = 0; j <= MAXBSIZE / PAGE_SIZE; j++) if (counts[j] != 0) printf(", %d-%d", j * PAGE_SIZE, counts[j]); printf("\n"); } } #endif /* DEBUG */ /* ------------------------------ */ buf_t * getiobuf(struct vnode *vp, bool waitok) { buf_t *bp; bp = pool_cache_get(bufio_cache, (waitok ? PR_WAITOK : PR_NOWAIT)); if (bp == NULL) return bp; buf_init(bp); if ((bp->b_vp = vp) != NULL) { bp->b_objlock = vp->v_interlock; } else { KASSERT(bp->b_objlock == &buffer_lock); } return bp; } void putiobuf(buf_t *bp) { buf_destroy(bp); pool_cache_put(bufio_cache, bp); } /* * nestiobuf_iodone: b_iodone callback for nested buffers. */ void nestiobuf_iodone(buf_t *bp) { buf_t *mbp = bp->b_private; int error; int donebytes; KASSERT(bp->b_bcount <= bp->b_bufsize); KASSERT(mbp != bp); error = bp->b_error; if (bp->b_error == 0 && (bp->b_bcount < bp->b_bufsize || bp->b_resid > 0)) { /* * Not all got transferred, raise an error. We have no way to * propagate these conditions to mbp. */ error = EIO; } donebytes = bp->b_bufsize; putiobuf(bp); nestiobuf_done(mbp, donebytes, error); } /* * nestiobuf_setup: setup a "nested" buffer. * * => 'mbp' is a "master" buffer which is being divided into sub pieces. * => 'bp' should be a buffer allocated by getiobuf. * => 'offset' is a byte offset in the master buffer. * => 'size' is a size in bytes of this nested buffer. */ void nestiobuf_setup(buf_t *mbp, buf_t *bp, int offset, size_t size) { const int b_pass = mbp->b_flags & (B_READ|B_PHYS|B_RAW|B_MEDIA_FLAGS); struct vnode *vp = mbp->b_vp; KASSERT(mbp->b_bcount >= offset + size); bp->b_vp = vp; bp->b_dev = mbp->b_dev; bp->b_objlock = mbp->b_objlock; bp->b_cflags = BC_BUSY; bp->b_flags = B_ASYNC | b_pass; bp->b_iodone = nestiobuf_iodone; bp->b_data = (char *)mbp->b_data + offset; bp->b_resid = bp->b_bcount = size; bp->b_bufsize = bp->b_bcount; bp->b_private = mbp; BIO_COPYPRIO(bp, mbp); if (BUF_ISWRITE(bp) && vp != NULL) { mutex_enter(vp->v_interlock); vp->v_numoutput++; mutex_exit(vp->v_interlock); } } /* * nestiobuf_done: propagate completion to the master buffer. * * => 'donebytes' specifies how many bytes in the 'mbp' is completed. * => 'error' is an errno(2) that 'donebytes' has been completed with. */ void nestiobuf_done(buf_t *mbp, int donebytes, int error) { if (donebytes == 0) { return; } mutex_enter(mbp->b_objlock); KASSERT(mbp->b_resid >= donebytes); mbp->b_resid -= donebytes; if (error) mbp->b_error = error; if (mbp->b_resid == 0) { if (mbp->b_error) mbp->b_resid = mbp->b_bcount; mutex_exit(mbp->b_objlock); biodone(mbp); } else mutex_exit(mbp->b_objlock); } void buf_init(buf_t *bp) { cv_init(&bp->b_busy, "biolock"); cv_init(&bp->b_done, "biowait"); bp->b_dev = NODEV; bp->b_error = 0; bp->b_flags = 0; bp->b_cflags = 0; bp->b_oflags = 0; bp->b_objlock = &buffer_lock; bp->b_iodone = NULL; bp->b_dev = NODEV; bp->b_vnbufs.le_next = NOLIST; BIO_SETPRIO(bp, BPRIO_DEFAULT); } void buf_destroy(buf_t *bp) { cv_destroy(&bp->b_done); cv_destroy(&bp->b_busy); } int bbusy(buf_t *bp, bool intr, int timo, kmutex_t *interlock) { int error; KASSERT(mutex_owned(&bufcache_lock)); SDT_PROBE4(io, kernel, , bbusy__start, bp, intr, timo, interlock); if ((bp->b_cflags & BC_BUSY) != 0) { if (curlwp == uvm.pagedaemon_lwp) { error = EDEADLK; goto out; } bp->b_cflags |= BC_WANTED; if (interlock != NULL) mutex_exit(interlock); if (intr) { error = cv_timedwait_sig(&bp->b_busy, &bufcache_lock, timo); } else { error = cv_timedwait(&bp->b_busy, &bufcache_lock, timo); } /* * At this point the buffer may be gone: don't touch it * again. The caller needs to find it again and retry. */ if (interlock != NULL) mutex_enter(interlock); if (error == 0) error = EPASSTHROUGH; } else { bp->b_cflags |= BC_BUSY; error = 0; } out: SDT_PROBE5(io, kernel, , bbusy__done, bp, intr, timo, interlock, error); return error; } /* * Nothing outside this file should really need to know about nbuf, * but a few things still want to read it, so give them a way to do that. */ u_int buf_nbuf(void) { return nbuf; } |
| 4 4 150 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 197 198 199 200 201 202 203 204 205 206 207 208 209 210 211 212 213 214 215 216 217 218 219 220 221 222 223 224 225 226 227 228 229 230 231 232 233 234 235 236 237 238 239 240 241 242 243 244 245 246 247 248 249 250 251 252 253 254 255 256 257 258 259 260 261 262 263 264 265 266 267 268 269 270 271 272 273 274 275 276 277 278 279 280 281 282 283 284 285 286 287 288 289 290 291 292 293 294 295 296 297 298 299 300 301 302 303 304 305 306 307 308 309 310 311 312 313 314 315 316 317 318 319 320 321 322 323 324 325 | /* $NetBSD: signalvar.h,v 1.104 2021/11/01 05:07:17 thorpej Exp $ */ /* * Copyright (c) 1991, 1993 * The Regents of the University of California. All rights reserved. * * Redistribution and use in source and binary forms, with or without * modification, are permitted provided that the following conditions * are met: * 1. Redistributions of source code must retain the above copyright * notice, this list of conditions and the following disclaimer. * 2. Redistributions in binary form must reproduce the above copyright * notice, this list of conditions and the following disclaimer in the * documentation and/or other materials provided with the distribution. * 3. Neither the name of the University nor the names of its contributors * may be used to endorse or promote products derived from this software * without specific prior written permission. * * THIS SOFTWARE IS PROVIDED BY THE REGENTS AND CONTRIBUTORS ``AS IS'' AND * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE * ARE DISCLAIMED. IN NO EVENT SHALL THE REGENTS OR CONTRIBUTORS BE LIABLE * FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL * DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS * OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) * HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT * LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY * OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF * SUCH DAMAGE. * * @(#)signalvar.h 8.6 (Berkeley) 2/19/95 */ #ifndef _SYS_SIGNALVAR_H_ #define _SYS_SIGNALVAR_H_ #include <sys/siginfo.h> #include <sys/queue.h> #include <sys/mutex.h> #include <sys/stdbool.h> #ifndef _KERNEL #include <string.h> /* Required for memset(3) and memcpy(3) prototypes */ #endif /* _KERNEL */ /* * Kernel signal definitions and data structures, * not exported to user programs. */ /* * Queue of signals. */ typedef TAILQ_HEAD(ksiginfoq, ksiginfo) ksiginfoq_t; /* * Process signal actions, possibly shared between processes. */ struct sigacts { struct sigact_sigdesc { struct sigaction sd_sigact; const void *sd_tramp; int sd_vers; } sa_sigdesc[NSIG]; /* disposition of signals */ int sa_refcnt; /* reference count */ kmutex_t sa_mutex; /* lock on sa_refcnt */ }; /* * Pending signals, per LWP and per process. */ typedef struct sigpend { ksiginfoq_t sp_info; sigset_t sp_set; } sigpend_t; /* * Process signal state. */ struct sigctx { struct _ksiginfo ps_info; /* for core dump/debugger XXX */ int ps_lwp; /* for core dump/debugger XXX */ bool ps_faked; /* for core dump/debugger XXX */ void *ps_sigcode; /* address of signal trampoline */ sigset_t ps_sigignore; /* Signals being ignored. */ sigset_t ps_sigcatch; /* Signals being caught by user. */ sigset_t ps_sigpass; /* Signals evading the debugger. */ }; /* additional signal action values, used only temporarily/internally */ #define SIG_CATCH (void (*)(int))2 /* * get signal action for process and signal; currently only for current process */ #define SIGACTION(p, sig) (p->p_sigacts->sa_sigdesc[(sig)].sd_sigact) #define SIGACTION_PS(ps, sig) (ps->sa_sigdesc[(sig)].sd_sigact) /* * Copy a sigaction structure without padding. */ static __inline void sigaction_copy(struct sigaction *dst, const struct sigaction *src) { memset(dst, 0, sizeof(*dst)); dst->_sa_u._sa_handler = src->_sa_u._sa_handler; memcpy(&dst->sa_mask, &src->sa_mask, sizeof(dst->sa_mask)); dst->sa_flags = src->sa_flags; } /* * Signal properties and actions. * The array below categorizes the signals and their default actions * according to the following properties: */ #define SA_KILL 0x0001 /* terminates process by default */ #define SA_CORE 0x0002 /* ditto and coredumps */ #define SA_STOP 0x0004 /* suspend process */ #define SA_TTYSTOP 0x0008 /* ditto, from tty */ #define SA_IGNORE 0x0010 /* ignore by default */ #define SA_CONT 0x0020 /* continue if suspended */ #define SA_CANTMASK 0x0040 /* non-maskable, catchable */ #define SA_NORESET 0x0080 /* not reset when caught */ #define SA_TOLWP 0x0100 /* to LWP that generated, if local */ #define SA_TOALL 0x0200 /* always to all LWPs */ #ifdef _KERNEL #include <sys/systm.h> /* for copyin_t/copyout_t */ extern sigset_t contsigmask, stopsigmask, sigcantmask; struct vnode; struct coredump_iostate; /* * Machine-independent functions: */ int coredump_netbsd(struct lwp *, struct coredump_iostate *); int coredump_netbsd32(struct lwp *, struct coredump_iostate *); int real_coredump_netbsd(struct lwp *, struct coredump_iostate *); void execsigs(struct proc *); int issignal(struct lwp *); void pgsignal(struct pgrp *, int, int); void kpgsignal(struct pgrp *, struct ksiginfo *, void *, int); void postsig(int); void psignal(struct proc *, int); void kpsignal(struct proc *, struct ksiginfo *, void *); void child_psignal(struct proc *, int); void siginit(struct proc *); void trapsignal(struct lwp *, struct ksiginfo *); void sigexit(struct lwp *, int) __dead; void killproc(struct proc *, const char *); void setsigvec(struct proc *, int, struct sigaction *); int killpg1(struct lwp *, struct ksiginfo *, int, int); void proc_unstop(struct proc *p); void eventswitch(int, int, int); void eventswitchchild(struct proc *, int, int); int sigaction1(struct lwp *, int, const struct sigaction *, struct sigaction *, const void *, int); int sigprocmask1(struct lwp *, int, const sigset_t *, sigset_t *); void sigpending1(struct lwp *, sigset_t *); void sigsuspendsetup(struct lwp *, const sigset_t *); void sigsuspendteardown(struct lwp *); int sigsuspend1(struct lwp *, const sigset_t *); int sigaltstack1(struct lwp *, const stack_t *, stack_t *); int sigismasked(struct lwp *, int); int sigget(sigpend_t *, ksiginfo_t *, int, const sigset_t *); void sigclear(sigpend_t *, const sigset_t *, ksiginfoq_t *); void sigclearall(struct proc *, const sigset_t *, ksiginfoq_t *); int kpsignal2(struct proc *, ksiginfo_t *); void signal_init(void); struct sigacts *sigactsinit(struct proc *, int); void sigactsunshare(struct proc *); void sigactsfree(struct sigacts *); void kpsendsig(struct lwp *, const struct ksiginfo *, const sigset_t *); void sendsig_reset(struct lwp *, int); void sendsig(const struct ksiginfo *, const sigset_t *); ksiginfo_t *ksiginfo_alloc(struct proc *, ksiginfo_t *, int); void ksiginfo_free(ksiginfo_t *); void ksiginfo_queue_drain0(ksiginfoq_t *); struct sys_____sigtimedwait50_args; int sigtimedwait1(struct lwp *, const struct sys_____sigtimedwait50_args *, register_t *, copyin_t, copyout_t, copyin_t, copyout_t); void signotify(struct lwp *); int sigispending(struct lwp *, int); /* * Machine-dependent functions: */ void sendsig_sigcontext(const struct ksiginfo *, const sigset_t *); void sendsig_siginfo(const struct ksiginfo *, const sigset_t *); extern struct pool ksiginfo_pool; /* * firstsig: * * Return the first signal in a signal set. */ static __inline int firstsig(const sigset_t *ss) { int sig; sig = ffs(ss->__bits[0]); if (sig != 0) return (sig); #if NSIG > 33 sig = ffs(ss->__bits[1]); if (sig != 0) return (sig + 32); #endif #if NSIG > 65 sig = ffs(ss->__bits[2]); if (sig != 0) return (sig + 64); #endif #if NSIG > 97 sig = ffs(ss->__bits[3]); if (sig != 0) return (sig + 96); #endif return (0); } static __inline void ksiginfo_queue_init(ksiginfoq_t *kq) { TAILQ_INIT(kq); } static __inline void ksiginfo_queue_drain(ksiginfoq_t *kq) { if (!TAILQ_EMPTY(kq)) ksiginfo_queue_drain0(kq); } #endif /* _KERNEL */ #ifdef _KERNEL #ifdef SIGPROP const int sigprop[NSIG] = { 0, /* 0 unused */ SA_KILL, /* 1 SIGHUP */ SA_KILL, /* 2 SIGINT */ SA_KILL|SA_CORE, /* 3 SIGQUIT */ SA_KILL|SA_CORE|SA_NORESET|SA_TOLWP, /* 4 SIGILL */ SA_KILL|SA_CORE|SA_NORESET|SA_TOLWP, /* 5 SIGTRAP */ SA_KILL|SA_CORE, /* 6 SIGABRT */ SA_KILL|SA_CORE|SA_TOLWP, /* 7 SIGEMT */ SA_KILL|SA_CORE|SA_TOLWP, /* 8 SIGFPE */ SA_KILL|SA_CANTMASK|SA_TOALL, /* 9 SIGKILL */ SA_KILL|SA_CORE|SA_TOLWP, /* 10 SIGBUS */ SA_KILL|SA_CORE|SA_TOLWP, /* 11 SIGSEGV */ SA_KILL|SA_CORE|SA_TOLWP, /* 12 SIGSYS */ SA_KILL, /* 13 SIGPIPE */ SA_KILL, /* 14 SIGALRM */ SA_KILL, /* 15 SIGTERM */ SA_IGNORE, /* 16 SIGURG */ SA_STOP|SA_CANTMASK|SA_TOALL, /* 17 SIGSTOP */ SA_STOP|SA_TTYSTOP|SA_TOALL, /* 18 SIGTSTP */ SA_IGNORE|SA_CONT|SA_TOALL, /* 19 SIGCONT */ SA_IGNORE, /* 20 SIGCHLD */ SA_STOP|SA_TTYSTOP|SA_TOALL, /* 21 SIGTTIN */ SA_STOP|SA_TTYSTOP|SA_TOALL, /* 22 SIGTTOU */ SA_IGNORE, /* 23 SIGIO */ SA_KILL, /* 24 SIGXCPU */ SA_KILL, /* 25 SIGXFSZ */ SA_KILL, /* 26 SIGVTALRM */ SA_KILL, /* 27 SIGPROF */ SA_IGNORE, /* 28 SIGWINCH */ SA_IGNORE, /* 29 SIGINFO */ SA_KILL, /* 30 SIGUSR1 */ SA_KILL, /* 31 SIGUSR2 */ SA_IGNORE|SA_NORESET, /* 32 SIGPWR */ SA_KILL, /* 33 SIGRTMIN + 0 */ SA_KILL, /* 34 SIGRTMIN + 1 */ SA_KILL, /* 35 SIGRTMIN + 2 */ SA_KILL, /* 36 SIGRTMIN + 3 */ SA_KILL, /* 37 SIGRTMIN + 4 */ SA_KILL, /* 38 SIGRTMIN + 5 */ SA_KILL, /* 39 SIGRTMIN + 6 */ SA_KILL, /* 40 SIGRTMIN + 7 */ SA_KILL, /* 41 SIGRTMIN + 8 */ SA_KILL, /* 42 SIGRTMIN + 9 */ SA_KILL, /* 43 SIGRTMIN + 10 */ SA_KILL, /* 44 SIGRTMIN + 11 */ SA_KILL, /* 45 SIGRTMIN + 12 */ SA_KILL, /* 46 SIGRTMIN + 13 */ SA_KILL, /* 47 SIGRTMIN + 14 */ SA_KILL, /* 48 SIGRTMIN + 15 */ SA_KILL, /* 49 SIGRTMIN + 16 */ SA_KILL, /* 50 SIGRTMIN + 17 */ SA_KILL, /* 51 SIGRTMIN + 18 */ SA_KILL, /* 52 SIGRTMIN + 19 */ SA_KILL, /* 53 SIGRTMIN + 20 */ SA_KILL, /* 54 SIGRTMIN + 21 */ SA_KILL, /* 55 SIGRTMIN + 22 */ SA_KILL, /* 56 SIGRTMIN + 23 */ SA_KILL, /* 57 SIGRTMIN + 24 */ SA_KILL, /* 58 SIGRTMIN + 25 */ SA_KILL, /* 59 SIGRTMIN + 26 */ SA_KILL, /* 60 SIGRTMIN + 27 */ SA_KILL, /* 61 SIGRTMIN + 28 */ SA_KILL, /* 62 SIGRTMIN + 29 */ SA_KILL, /* 63 SIGRTMIN + 30 */ }; #undef SIGPROP #else extern const int sigprop[NSIG]; #endif /* SIGPROP */ #endif /* _KERNEL */ #endif /* !_SYS_SIGNALVAR_H_ */ |
| 1 1 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 197 198 199 200 201 202 203 204 205 206 207 208 209 210 211 212 213 214 215 216 217 218 219 220 221 222 223 224 225 226 227 228 229 230 231 232 233 234 235 236 237 238 239 240 241 242 243 244 245 246 247 248 249 250 251 252 253 254 255 256 257 258 259 260 261 262 263 264 265 266 267 268 269 270 271 272 273 274 275 276 277 278 279 280 281 282 283 284 285 286 287 288 289 290 291 292 293 294 295 296 297 298 299 300 301 302 303 304 305 306 307 308 309 310 311 312 313 314 315 316 317 318 319 320 321 322 323 324 325 326 327 328 329 330 331 332 333 334 335 336 337 338 339 340 341 342 343 344 345 346 347 348 349 350 351 352 353 354 355 356 357 358 359 360 361 362 363 364 365 366 367 368 369 370 371 372 373 374 375 376 377 378 379 380 381 382 383 384 385 386 387 388 389 390 391 392 393 394 395 396 397 398 399 400 401 402 403 404 405 406 407 408 409 410 411 412 413 414 415 416 417 418 419 420 421 422 423 424 425 426 427 428 429 430 431 432 433 434 435 436 437 438 439 440 441 442 443 444 445 446 447 448 449 450 451 452 453 454 455 456 457 458 459 460 461 462 463 464 465 466 467 468 469 470 471 472 473 474 475 476 477 478 479 480 481 482 483 484 485 486 487 488 489 490 491 492 493 494 495 496 497 498 499 500 501 502 503 504 505 506 507 508 509 510 511 512 513 514 515 516 517 518 519 520 521 522 523 524 525 526 527 528 529 530 531 532 533 534 535 536 537 538 539 540 541 542 543 544 545 546 547 548 549 550 551 552 553 554 555 556 557 558 559 560 561 562 563 564 565 566 567 568 569 570 571 572 573 574 575 576 577 578 579 580 581 582 583 584 585 586 587 588 589 590 591 592 593 594 595 596 597 598 599 600 601 602 603 604 605 606 607 608 609 610 611 612 613 614 615 616 617 618 619 620 621 622 623 624 625 626 627 628 629 630 631 632 633 634 635 636 637 638 639 640 641 642 643 644 645 646 647 648 649 650 651 652 653 654 655 656 657 658 659 660 661 662 663 664 665 666 667 668 669 670 671 672 673 674 675 676 677 678 679 680 681 682 683 684 685 686 687 688 689 690 691 692 693 694 695 696 697 698 699 700 701 702 703 704 705 706 707 708 709 710 711 712 713 714 715 716 717 718 719 720 721 722 723 724 725 726 727 728 729 730 731 732 733 734 735 736 737 738 739 740 741 742 743 744 745 746 747 748 749 750 751 752 753 754 755 756 757 758 759 760 761 762 763 764 765 766 767 768 769 770 771 772 773 774 775 776 777 778 779 780 781 782 783 784 785 786 787 788 789 790 791 792 793 794 795 796 797 798 799 800 801 802 803 804 805 806 807 808 809 810 811 812 813 814 815 816 817 818 819 820 821 822 823 824 825 826 827 828 829 830 831 832 833 834 835 836 837 838 839 840 841 842 843 844 845 846 847 848 849 850 851 852 853 854 855 856 857 858 859 860 861 862 863 864 865 866 867 868 869 870 871 872 873 874 875 876 877 878 879 880 881 882 883 884 885 886 887 888 889 890 891 892 893 894 895 896 897 898 899 900 901 902 903 904 905 906 907 908 909 910 911 912 913 914 915 916 917 918 919 920 921 922 923 924 925 926 927 928 929 930 931 932 933 934 935 936 937 938 939 940 941 942 943 944 945 946 947 948 949 950 951 952 953 954 955 956 957 958 959 960 961 962 963 964 965 966 967 968 969 970 971 972 973 974 975 976 977 978 979 980 981 982 983 984 985 986 987 988 989 990 991 992 993 994 995 996 997 998 999 1000 1001 1002 1003 1004 1005 1006 1007 1008 1009 1010 1011 1012 1013 1014 1015 1016 1017 1018 1019 1020 1021 1022 1023 1024 1025 1026 1027 1028 1029 1030 1031 1032 1033 1034 1035 1036 1037 1038 1039 1040 1041 1042 1043 1044 1045 1046 1047 1048 1049 1050 1051 1052 1053 1054 1055 1056 1057 1058 1059 1060 1061 1062 1063 1064 1065 1066 1067 1068 1069 1070 1071 1072 1073 1074 1075 1076 1077 1078 1079 1080 1081 1082 1083 1084 1085 1086 1087 1088 1089 1090 1091 1092 1093 1094 1095 1096 1097 1098 1099 1100 1101 1102 1103 1104 1105 1106 1107 1108 1109 1110 1111 1112 1113 1114 1115 1116 1117 1118 1119 1120 1121 1122 1123 1124 1125 1126 1127 1128 1129 1130 1131 1132 1133 1134 1135 1136 1137 1138 1139 1140 1141 1142 1143 1144 1145 1146 1147 1148 1149 1150 1151 1152 1153 1154 1155 1156 1157 1158 1159 1160 1161 1162 1163 1164 1165 1166 1167 1168 1169 1170 1171 1172 1173 1174 1175 1176 1177 1178 1179 1180 1181 1182 1183 1184 1185 1186 1187 1188 1189 1190 1191 1192 1193 1194 1195 1196 1197 1198 1199 1200 1201 1202 1203 1204 1205 1206 1207 1208 1209 1210 1211 1212 1213 1214 1215 1216 1217 1218 1219 1220 1221 1222 1223 1224 1225 1226 1227 1228 1229 1230 1231 1232 1233 1234 1235 1236 1237 1238 1239 1240 1241 1242 1243 1244 1245 1246 1247 1248 1249 1250 1251 1252 1253 1254 1255 1256 1257 1258 1259 1260 1261 1262 1263 1264 1265 1266 1267 1268 1269 1270 1271 1272 1273 1274 1275 1276 1277 1278 1279 1280 1281 1282 1283 1284 1285 1286 1287 1288 1289 1290 1291 1292 1293 1294 1295 1296 1297 1298 1299 1300 1301 1302 1303 1304 1305 1306 1307 1308 1309 1310 1311 1312 1313 1314 1315 1316 1317 1318 1319 1320 1321 1322 1323 1324 1325 1326 1327 1328 1329 1330 1331 1332 1333 1334 1335 1336 1337 1338 1339 1340 1341 1342 1343 1344 1345 1346 1347 1348 1349 1350 1351 1352 1353 1354 1355 1356 1357 1358 1359 1360 1361 1362 1363 1364 1365 1366 1367 1368 1369 1370 1371 1372 1373 1374 1375 1376 1377 1378 1379 1380 1381 1382 1383 1384 1385 1386 1387 1388 1389 1390 1391 1392 1393 1394 1395 1396 1397 1398 1399 1400 1401 1402 1403 1404 1405 1406 1407 1408 1409 1410 1411 1412 1413 1414 1415 1416 1417 1418 1419 1420 1421 1422 1423 1424 1425 1426 1427 1428 1429 1430 1431 1432 1433 1434 1435 1436 1437 1438 1439 1440 1441 1442 1443 1444 1445 1446 1447 1448 1449 1450 1451 1452 1453 1454 1455 1456 1457 1458 1459 1460 1461 1462 1463 1464 1465 1466 1467 1468 1469 1470 1471 1472 1473 1474 1475 1476 1477 1478 1479 1480 1481 1482 1483 1484 1485 1486 1487 1488 1489 1490 1491 1492 1493 1494 1495 1496 1497 1498 1499 1500 1501 1502 1503 1504 1505 1506 1507 1508 1509 1510 1511 1512 1513 1514 1515 1516 1517 1518 1519 1520 1521 1522 1523 1524 1525 1526 1527 1528 1529 1530 1531 1532 1533 1534 1535 1536 1537 1538 1539 1540 1541 1542 1543 1544 1545 1546 1547 1548 1549 1550 1551 1552 1553 1554 1555 1556 1557 1558 1559 1560 1561 1562 1563 1564 1565 1566 1567 1568 1569 1570 1571 1572 1573 1574 1575 1576 1577 1578 1579 1580 1581 1582 1583 1584 1585 1586 1587 1588 1589 1590 1591 1592 1593 1594 1595 1596 1597 1598 1599 1600 1601 1602 1603 1604 1605 1606 1607 1608 1609 1610 1611 1612 1613 1614 1615 1616 1617 1618 1619 1620 1621 1622 1623 1624 1625 1626 1627 1628 1629 1630 1631 1632 1633 1634 1635 1636 1637 1638 1639 1640 1641 1642 1643 1644 1645 1646 1647 1648 1649 1650 1651 1652 1653 1654 1655 1656 1657 1658 1659 1660 1661 1662 1663 1664 1665 1666 1667 1668 1669 1670 1671 1672 1673 1674 1675 1676 1677 1678 1679 1680 1681 1682 1683 1684 1685 1686 1687 1688 1689 1690 1691 1692 1693 1694 1695 1696 1697 1698 1699 1700 1701 1702 1703 1704 1705 1706 1707 1708 1709 1710 1711 1712 1713 1714 1715 1716 1717 1718 1719 1720 1721 1722 1723 1724 1725 1726 1727 1728 1729 1730 1731 1732 1733 1734 1735 1736 1737 1738 1739 1740 1741 1742 1743 1744 1745 1746 1747 1748 1749 1750 1751 1752 1753 1754 1755 1756 1757 1758 1759 1760 1761 1762 1763 1764 1765 1766 1767 1768 1769 1770 1771 1772 1773 1774 1775 1776 1777 1778 1779 1780 1781 1782 1783 1784 1785 1786 1787 1788 1789 1790 1791 1792 1793 1794 1795 1796 1797 1798 1799 1800 1801 1802 1803 1804 1805 1806 1807 1808 1809 1810 1811 1812 1813 1814 1815 1816 1817 1818 1819 1820 1821 1822 1823 1824 1825 1826 1827 1828 1829 1830 1831 1832 1833 1834 1835 1836 1837 1838 1839 1840 1841 1842 1843 1844 1845 1846 1847 1848 1849 1850 1851 1852 1853 1854 1855 1856 1857 1858 1859 1860 1861 1862 1863 1864 1865 1866 1867 1868 1869 1870 1871 1872 1873 1874 1875 1876 1877 1878 1879 1880 1881 1882 1883 1884 1885 1886 1887 1888 1889 1890 1891 1892 1893 1894 1895 1896 1897 1898 1899 1900 1901 1902 1903 1904 1905 1906 1907 1908 1909 1910 1911 1912 1913 1914 1915 1916 1917 1918 1919 1920 1921 1922 1923 1924 1925 1926 1927 1928 1929 1930 1931 1932 1933 1934 1935 1936 1937 1938 1939 1940 1941 1942 1943 1944 1945 1946 1947 1948 1949 1950 1951 1952 1953 1954 1955 1956 1957 1958 1959 1960 1961 1962 1963 1964 1965 1966 1967 1968 1969 1970 1971 1972 1973 1974 1975 1976 1977 1978 1979 1980 1981 1982 1983 1984 1985 1986 1987 1988 1989 1990 1991 1992 1993 1994 1995 1996 1997 1998 1999 2000 2001 2002 2003 2004 2005 2006 2007 2008 2009 2010 2011 2012 2013 2014 2015 2016 2017 2018 2019 2020 2021 2022 2023 2024 2025 2026 2027 2028 2029 2030 2031 2032 2033 2034 2035 2036 2037 2038 2039 2040 2041 2042 2043 2044 2045 2046 2047 2048 2049 2050 2051 2052 2053 2054 2055 2056 2057 2058 2059 2060 2061 2062 2063 2064 2065 2066 2067 2068 2069 2070 2071 2072 2073 2074 2075 2076 2077 2078 2079 2080 2081 2082 2083 2084 2085 2086 2087 2088 2089 2090 2091 2092 2093 2094 2095 2096 2097 2098 2099 2100 2101 2102 2103 2104 2105 2106 2107 2108 2109 2110 2111 2112 2113 2114 2115 2116 2117 2118 2119 2120 2121 2122 2123 2124 2125 2126 2127 2128 2129 2130 2131 2132 2133 2134 2135 2136 2137 2138 2139 2140 2141 2142 2143 2144 2145 2146 2147 2148 2149 2150 2151 2152 2153 2154 2155 2156 2157 2158 2159 2160 2161 2162 2163 2164 2165 2166 2167 2168 2169 2170 2171 2172 2173 2174 2175 2176 2177 2178 2179 2180 2181 2182 2183 2184 2185 2186 2187 2188 2189 2190 2191 2192 2193 2194 2195 2196 2197 2198 2199 2200 2201 2202 2203 2204 2205 2206 2207 2208 2209 2210 2211 2212 2213 2214 2215 2216 2217 2218 2219 2220 2221 2222 2223 2224 2225 2226 2227 2228 2229 2230 2231 2232 2233 2234 2235 2236 2237 2238 2239 2240 2241 2242 2243 2244 2245 2246 2247 2248 2249 2250 2251 2252 2253 2254 2255 2256 2257 2258 2259 2260 2261 2262 2263 2264 2265 2266 2267 2268 2269 2270 2271 2272 2273 2274 2275 2276 2277 2278 2279 2280 2281 2282 2283 2284 2285 2286 2287 2288 2289 2290 2291 2292 2293 2294 2295 2296 | /* $NetBSD: intr.c,v 1.160 2022/03/12 15:50:45 riastradh Exp $ */ /* * Copyright (c) 2007, 2008, 2009, 2019 The NetBSD Foundation, Inc. * All rights reserved. * * This code is derived from software contributed to The NetBSD Foundation * by Andrew Doran, and by Jason R. Thorpe. * * Redistribution and use in source and binary forms, with or without * modification, are permitted provided that the following conditions * are met: * 1. Redistributions of source code must retain the above copyright * notice, this list of conditions and the following disclaimer. * 2. Redistributions in binary form must reproduce the above copyright * notice, this list of conditions and the following disclaimer in the * documentation and/or other materials provided with the distribution. * * THIS SOFTWARE IS PROVIDED BY THE NETBSD FOUNDATION, INC. AND CONTRIBUTORS * ``AS IS'' AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED * TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR * PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE FOUNDATION OR CONTRIBUTORS * BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR * CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF * SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS * INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN * CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) * ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE * POSSIBILITY OF SUCH DAMAGE. */ /* * Copyright 2002 (c) Wasabi Systems, Inc. * All rights reserved. * * Written by Frank van der Linden for Wasabi Systems, Inc. * * Redistribution and use in source and binary forms, with or without * modification, are permitted provided that the following conditions * are met: * 1. Redistributions of source code must retain the above copyright * notice, this list of conditions and the following disclaimer. * 2. Redistributions in binary form must reproduce the above copyright * notice, this list of conditions and the following disclaimer in the * documentation and/or other materials provided with the distribution. * 3. All advertising materials mentioning features or use of this software * must display the following acknowledgement: * This product includes software developed for the NetBSD Project by * Wasabi Systems, Inc. * 4. The name of Wasabi Systems, Inc. may not be used to endorse * or promote products derived from this software without specific prior * written permission. * * THIS SOFTWARE IS PROVIDED BY WASABI SYSTEMS, INC. ``AS IS'' AND * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED * TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR * PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL WASABI SYSTEMS, INC * BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR * CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF * SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS * INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN * CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) * ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE * POSSIBILITY OF SUCH DAMAGE. */ /*- * Copyright (c) 1991 The Regents of the University of California. * All rights reserved. * * This code is derived from software contributed to Berkeley by * William Jolitz. * * Redistribution and use in source and binary forms, with or without * modification, are permitted provided that the following conditions * are met: * 1. Redistributions of source code must retain the above copyright * notice, this list of conditions and the following disclaimer. * 2. Redistributions in binary form must reproduce the above copyright * notice, this list of conditions and the following disclaimer in the * documentation and/or other materials provided with the distribution. * 3. Neither the name of the University nor the names of its contributors * may be used to endorse or promote products derived from this software * without specific prior written permission. * * THIS SOFTWARE IS PROVIDED BY THE REGENTS AND CONTRIBUTORS ``AS IS'' AND * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE * ARE DISCLAIMED. IN NO EVENT SHALL THE REGENTS OR CONTRIBUTORS BE LIABLE * FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL * DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS * OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) * HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT * LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY * OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF * SUCH DAMAGE. * * @(#)isa.c 7.2 (Berkeley) 5/13/91 */ /*- * Copyright (c) 1993, 1994 Charles Hannum. * * Redistribution and use in source and binary forms, with or without * modification, are permitted provided that the following conditions * are met: * 1. Redistributions of source code must retain the above copyright * notice, this list of conditions and the following disclaimer. * 2. Redistributions in binary form must reproduce the above copyright * notice, this list of conditions and the following disclaimer in the * documentation and/or other materials provided with the distribution. * 3. All advertising materials mentioning features or use of this software * must display the following acknowledgement: * This product includes software developed by the University of * California, Berkeley and its contributors. * 4. Neither the name of the University nor the names of its contributors * may be used to endorse or promote products derived from this software * without specific prior written permission. * * THIS SOFTWARE IS PROVIDED BY THE REGENTS AND CONTRIBUTORS ``AS IS'' AND * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE * ARE DISCLAIMED. IN NO EVENT SHALL THE REGENTS OR CONTRIBUTORS BE LIABLE * FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL * DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS * OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) * HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT * LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY * OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF * SUCH DAMAGE. * * @(#)isa.c 7.2 (Berkeley) 5/13/91 */ #include <sys/cdefs.h> __KERNEL_RCSID(0, "$NetBSD: intr.c,v 1.160 2022/03/12 15:50:45 riastradh Exp $"); #include "opt_intrdebug.h" #include "opt_multiprocessor.h" #include "opt_acpi.h" #include <sys/param.h> #include <sys/systm.h> #include <sys/kernel.h> #include <sys/syslog.h> #include <sys/device.h> #include <sys/kmem.h> #include <sys/proc.h> #include <sys/errno.h> #include <sys/intr.h> #include <sys/cpu.h> #include <sys/atomic.h> #include <sys/xcall.h> #include <sys/interrupt.h> #include <sys/reboot.h> /* for AB_VERBOSE */ #include <sys/kauth.h> #include <sys/conf.h> #include <uvm/uvm_extern.h> #include <machine/i8259.h> #include <machine/pio.h> #include "ioapic.h" #include "lapic.h" #include "pci.h" #include "acpica.h" #ifndef XENPV #include "hyperv.h" #if NHYPERV > 0 #include <dev/hyperv/hypervvar.h> extern void Xresume_hyperv_hypercall(void); extern void Xrecurse_hyperv_hypercall(void); #endif #endif #if NIOAPIC > 0 || NACPICA > 0 #include <machine/i82093var.h> #include <machine/mpbiosvar.h> #include <machine/mpacpi.h> #endif #if NLAPIC > 0 #include <machine/i82489var.h> #endif #if NPCI > 0 #include <dev/pci/ppbreg.h> #endif #include <x86/pci/msipic.h> #include <x86/pci/pci_msi_machdep.h> #if NPCI == 0 || !defined(__HAVE_PCI_MSI_MSIX) #define msipic_is_msi_pic(PIC) (false) #endif #ifdef DDB #include <ddb/db_output.h> #endif #ifdef INTRDEBUG #define DPRINTF(msg) printf msg #else #define DPRINTF(msg) #endif static SIMPLEQ_HEAD(, intrsource) io_interrupt_sources = SIMPLEQ_HEAD_INITIALIZER(io_interrupt_sources); static kmutex_t intr_distribute_lock; static int intr_allocate_slot_cpu(struct cpu_info *, struct pic *, int, int *, struct intrsource *); static int __noinline intr_allocate_slot(struct pic *, int, int, struct cpu_info **, int *, int *, struct intrsource *); static void intr_source_free(struct cpu_info *, int, struct pic *, int); static void intr_establish_xcall(void *, void *); static void intr_disestablish_xcall(void *, void *); static const char *legacy_intr_string(int, char *, size_t, struct pic *); static const char *xen_intr_string(int, char *, size_t, struct pic *); #if defined(INTRSTACKSIZE) static inline bool redzone_const_or_false(bool); static inline int redzone_const_or_zero(int); #endif static void intr_redistribute_xc_t(void *, void *); static void intr_redistribute_xc_s1(void *, void *); static void intr_redistribute_xc_s2(void *, void *); static bool intr_redistribute(struct cpu_info *); static struct intrsource *intr_get_io_intrsource(const char *); static void intr_free_io_intrsource_direct(struct intrsource *); static int intr_num_handlers(struct intrsource *); static int intr_find_unused_slot(struct cpu_info *, int *); static void intr_activate_xcall(void *, void *); static void intr_deactivate_xcall(void *, void *); static void intr_get_affinity(struct intrsource *, kcpuset_t *); static int intr_set_affinity(struct intrsource *, const kcpuset_t *); /* * Fill in default interrupt table (in case of spurious interrupt * during configuration of kernel), setup interrupt control unit */ void intr_default_setup(void) { struct idt_vec *iv = &(cpu_info_primary.ci_idtvec); int i; /* icu vectors */ for (i = 0; i < NUM_LEGACY_IRQS; i++) { idt_vec_reserve(iv, ICU_OFFSET + i); idt_vec_set(iv, ICU_OFFSET + i, legacy_stubs[i].ist_entry); } /* * Eventually might want to check if it's actually there. */ i8259_default_setup(); mutex_init(&intr_distribute_lock, MUTEX_DEFAULT, IPL_NONE); } /* * Handle a NMI, possibly a machine check. * return true to panic system, false to ignore. */ void x86_nmi(void) { log(LOG_CRIT, "NMI port 61 %x, port 70 %x\n", inb(0x61), inb(0x70)); } /* * Create an interrupt id such as "ioapic0 pin 9". This interrupt id is used * by MI code and intrctl(8). */ const char * intr_create_intrid(int legacy_irq, struct pic *pic, int pin, char *buf, size_t len) { int ih = 0; #if NPCI > 0 #if defined(__HAVE_PCI_MSI_MSIX) if ((pic->pic_type == PIC_MSI) || (pic->pic_type == PIC_MSIX)) { uint64_t pih; int dev, vec; dev = msipic_get_devid(pic); vec = pin; pih = __SHIFTIN((uint64_t)dev, MSI_INT_DEV_MASK) | __SHIFTIN((uint64_t)vec, MSI_INT_VEC_MASK) | APIC_INT_VIA_MSI; if (pic->pic_type == PIC_MSI) MSI_INT_MAKE_MSI(pih); else if (pic->pic_type == PIC_MSIX) MSI_INT_MAKE_MSIX(pih); return x86_pci_msi_string(NULL, pih, buf, len); } #endif /* __HAVE_PCI_MSI_MSIX */ #endif if (pic->pic_type == PIC_XEN) { ih = pin; /* Port == pin */ return xen_intr_string(pin, buf, len, pic); } /* * If the device is pci, "legacy_irq" is always -1. Least 8 bit of "ih" * is only used in intr_string() to show the irq number. * If the device is "legacy"(such as floppy), it should not use * intr_string(). */ if (pic->pic_type == PIC_I8259) { ih = legacy_irq; return legacy_intr_string(ih, buf, len, pic); } #if NIOAPIC > 0 || NACPICA > 0 ih = ((pic->pic_apicid << APIC_INT_APIC_SHIFT) & APIC_INT_APIC_MASK) | ((pin << APIC_INT_PIN_SHIFT) & APIC_INT_PIN_MASK); if (pic->pic_type == PIC_IOAPIC) { ih |= APIC_INT_VIA_APIC; } ih |= pin; return intr_string(ih, buf, len); #endif return NULL; /* No pic found! */ } /* * Find intrsource from io_interrupt_sources list. */ static struct intrsource * intr_get_io_intrsource(const char *intrid) { struct intrsource *isp; KASSERT(mutex_owned(&cpu_lock)); SIMPLEQ_FOREACH(isp, &io_interrupt_sources, is_list) { KASSERT(isp->is_intrid != NULL); if (strncmp(intrid, isp->is_intrid, INTRIDBUF - 1) == 0) return isp; } return NULL; } /* * Allocate intrsource and add to io_interrupt_sources list. */ struct intrsource * intr_allocate_io_intrsource(const char *intrid) { CPU_INFO_ITERATOR cii; struct cpu_info *ci; struct intrsource *isp; struct percpu_evcnt *pep; KASSERT(mutex_owned(&cpu_lock)); if (intrid == NULL) return NULL; isp = kmem_zalloc(sizeof(*isp), KM_SLEEP); pep = kmem_zalloc(sizeof(*pep) * ncpu, KM_SLEEP); isp->is_saved_evcnt = pep; for (CPU_INFO_FOREACH(cii, ci)) { pep->cpuid = ci->ci_cpuid; pep++; } strlcpy(isp->is_intrid, intrid, sizeof(isp->is_intrid)); SIMPLEQ_INSERT_TAIL(&io_interrupt_sources, isp, is_list); return isp; } /* * Remove from io_interrupt_sources list and free by the intrsource pointer. */ static void intr_free_io_intrsource_direct(struct intrsource *isp) { KASSERT(mutex_owned(&cpu_lock)); SIMPLEQ_REMOVE(&io_interrupt_sources, isp, intrsource, is_list); /* Is this interrupt established? */ if (isp->is_evname[0] != '\0') { evcnt_detach(&isp->is_evcnt); isp->is_evname[0] = '\0'; } kmem_free(isp->is_saved_evcnt, sizeof(*(isp->is_saved_evcnt)) * ncpu); kmem_free(isp, sizeof(*isp)); } /* * Remove from io_interrupt_sources list and free by the interrupt id. * This function can be used by MI code. */ void intr_free_io_intrsource(const char *intrid) { struct intrsource *isp; KASSERT(mutex_owned(&cpu_lock)); if (intrid == NULL) return; if ((isp = intr_get_io_intrsource(intrid)) == NULL) { return; } /* If the interrupt uses shared IRQ, don't free yet. */ if (isp->is_handlers != NULL) { return; } intr_free_io_intrsource_direct(isp); } static int intr_allocate_slot_cpu(struct cpu_info *ci, struct pic *pic, int pin, int *index, struct intrsource *chained) { int slot, i; struct intrsource *isp; KASSERT(mutex_owned(&cpu_lock)); if (pic == &i8259_pic) { KASSERT(CPU_IS_PRIMARY(ci)); slot = pin; } else { int start = 0; int max = MAX_INTR_SOURCES; slot = -1; /* avoid reserved slots for legacy interrupts. */ if (CPU_IS_PRIMARY(ci) && msipic_is_msi_pic(pic)) start = NUM_LEGACY_IRQS; /* don't step over Xen's slots */ if (vm_guest == VM_GUEST_XENPVH) max = SIR_XENIPL_VM; /* * intr_allocate_slot has checked for an existing mapping. * Now look for a free slot. */ for (i = start; i < max ; i++) { if (ci->ci_isources[i] == NULL) { slot = i; break; } } if (slot == -1) { return EBUSY; } } isp = ci->ci_isources[slot]; if (isp == NULL) { const char *via; isp = chained; KASSERT(isp != NULL); if (pic->pic_type == PIC_MSI || pic->pic_type == PIC_MSIX) via = "vec"; else via = "pin"; snprintf(isp->is_evname, sizeof (isp->is_evname), "%s %d", via, pin); evcnt_attach_dynamic(&isp->is_evcnt, EVCNT_TYPE_INTR, NULL, pic->pic_name, isp->is_evname); isp->is_active_cpu = ci->ci_cpuid; ci->ci_isources[slot] = isp; } *index = slot; return 0; } /* * A simple round-robin allocator to assign interrupts to CPUs. */ static int __noinline intr_allocate_slot(struct pic *pic, int pin, int level, struct cpu_info **cip, int *index, int *idt_slot, struct intrsource *chained) { CPU_INFO_ITERATOR cii; struct cpu_info *ci, *lci; struct intrsource *isp; int slot = 0, idtvec, error; KASSERT(mutex_owned(&cpu_lock)); /* First check if this pin is already used by an interrupt vector. */ for (CPU_INFO_FOREACH(cii, ci)) { for (slot = 0 ; slot < MAX_INTR_SOURCES ; slot++) { if ((isp = ci->ci_isources[slot]) == NULL) { continue; } if (isp->is_pic == pic && pin != -1 && isp->is_pin == pin) { *idt_slot = isp->is_idtvec; *index = slot; *cip = ci; return 0; } } } /* * The pic/pin combination doesn't have an existing mapping. * Find a slot for a new interrupt source. For the i8259 case, * we always use reserved slots of the primary CPU. Otherwise, * we make an attempt to balance the interrupt load. * * PIC and APIC usage are essentially exclusive, so the reservation * of the ISA slots is ignored when assigning IOAPIC slots. */ if (pic == &i8259_pic) { /* * Must be directed to BP. */ ci = &cpu_info_primary; error = intr_allocate_slot_cpu(ci, pic, pin, &slot, chained); } else { /* * Find least loaded AP/BP and try to allocate there. */ ci = NULL; for (CPU_INFO_FOREACH(cii, lci)) { if ((lci->ci_schedstate.spc_flags & SPCF_NOINTR) != 0) { continue; } #if 0 if (ci == NULL || ci->ci_nintrhand > lci->ci_nintrhand) { ci = lci; } #else ci = &cpu_info_primary; #endif } KASSERT(ci != NULL); error = intr_allocate_slot_cpu(ci, pic, pin, &slot, chained); /* * If that did not work, allocate anywhere. */ if (error != 0) { for (CPU_INFO_FOREACH(cii, ci)) { if ((ci->ci_schedstate.spc_flags & SPCF_NOINTR) != 0) { continue; } error = intr_allocate_slot_cpu(ci, pic, pin, &slot, chained); if (error == 0) { break; } } } } if (error != 0) { return error; } KASSERT(ci != NULL); /* * Now allocate an IDT vector. * For the 8259 these are reserved up front. */ if (pic == &i8259_pic) { idtvec = ICU_OFFSET + pin; } else { /* * TODO to support MSI (not MSI-X) multiple vectors * * PCI Local Bus Specification Revision 3.0 says the devices * which use MSI multiple vectors increment the low order bits * of MSI message data. * On the other hand, Intel SDM "10.11.2 Message Data Register * Format" says the 7:0 bits of MSI message data mean Interrupt * Descriptor Table(IDT) vector. * As the result of these two documents, the IDT vectors which * are used by a device using MSI multiple vectors must be * continuous. */ struct idt_vec *iv; iv = idt_vec_ref(&ci->ci_idtvec); idtvec = idt_vec_alloc(iv, APIC_LEVEL(level), IDT_INTR_HIGH); } if (idtvec < 0) { evcnt_detach(&ci->ci_isources[slot]->is_evcnt); ci->ci_isources[slot]->is_evname[0] = '\0'; ci->ci_isources[slot] = NULL; return EBUSY; } ci->ci_isources[slot]->is_idtvec = idtvec; *idt_slot = idtvec; *index = slot; *cip = ci; return 0; } static void intr_source_free(struct cpu_info *ci, int slot, struct pic *pic, int idtvec) { struct intrsource *isp; struct idt_vec *iv; isp = ci->ci_isources[slot]; iv = idt_vec_ref(&ci->ci_idtvec); if (isp->is_handlers != NULL) return; ci->ci_isources[slot] = NULL; if (pic != &i8259_pic) idt_vec_free(iv, idtvec); isp->is_recurse = NULL; isp->is_resume = NULL; } #ifdef MULTIPROCESSOR static int intr_biglock_wrapper(void *); /* * intr_biglock_wrapper: grab biglock and call a real interrupt handler. */ static int intr_biglock_wrapper(void *vp) { struct intrhand *ih = vp; int locks; int ret; KERNEL_LOCK(1, NULL); locks = curcpu()->ci_biglock_count; ret = (*ih->ih_realfun)(ih->ih_realarg); KASSERTMSG(locks == curcpu()->ci_biglock_count, "%s @ %p slipped locks %d -> %d", ih->ih_xname, ih->ih_realfun, locks, curcpu()->ci_biglock_count); KERNEL_UNLOCK_ONE(NULL); return ret; } #endif /* MULTIPROCESSOR */ /* * Append device name to intrsource. If device A and device B share IRQ number, * the device name of the interrupt id is "device A, device B". */ static void intr_append_intrsource_xname(struct intrsource *isp, const char *xname) { if (isp->is_xname[0] != '\0') strlcat(isp->is_xname, ", ", sizeof(isp->is_xname)); strlcat(isp->is_xname, xname, sizeof(isp->is_xname)); } /* * Called on bound CPU to handle calling pic_hwunmask from contexts * that are not already running on the bound CPU. * * => caller (on initiating CPU) holds cpu_lock on our behalf * => arg1: struct intrhand *ih */ static void intr_hwunmask_xcall(void *arg1, void *arg2) { struct intrhand * const ih = arg1; struct cpu_info * const ci = ih->ih_cpu; KASSERT(ci == curcpu() || !mp_online); const u_long psl = x86_read_psl(); x86_disable_intr(); struct intrsource * const source = ci->ci_isources[ih->ih_slot]; struct pic * const pic = source->is_pic; if (source->is_mask_count == 0) { (*pic->pic_hwunmask)(pic, ih->ih_pin); } x86_write_psl(psl); } /* * Handle per-CPU component of interrupt establish. * * => caller (on initiating CPU) holds cpu_lock on our behalf * => arg1: struct intrhand *ih * => arg2: int idt_vec */ static void intr_establish_xcall(void *arg1, void *arg2) { struct idt_vec *iv; struct intrsource *source; struct intrstub *stubp; struct intrhand *ih; struct cpu_info *ci; int idt_vec; u_long psl; ih = arg1; KASSERT(ih->ih_cpu == curcpu() || !mp_online); ci = ih->ih_cpu; source = ci->ci_isources[ih->ih_slot]; idt_vec = (int)(intptr_t)arg2; iv = idt_vec_ref(&ci->ci_idtvec); /* Disable interrupts locally. */ psl = x86_read_psl(); x86_disable_intr(); /* Link in the handler and re-calculate masks. */ *(ih->ih_prevp) = ih; x86_intr_calculatemasks(ci); /* Hook in new IDT vector and SPL state. */ if (source->is_resume == NULL || source->is_idtvec != idt_vec) { if (source->is_idtvec != 0 && source->is_idtvec != idt_vec) idt_vec_free(iv, source->is_idtvec); source->is_idtvec = idt_vec; if (source->is_type == IST_LEVEL) { stubp = &source->is_pic->pic_level_stubs[ih->ih_slot]; } else { stubp = &source->is_pic->pic_edge_stubs[ih->ih_slot]; } source->is_resume = stubp->ist_resume; source->is_recurse = stubp->ist_recurse; idt_vec_set(iv, idt_vec, stubp->ist_entry); } /* Re-enable interrupts locally. */ x86_write_psl(psl); } void * intr_establish_xname(int legacy_irq, struct pic *pic, int pin, int type, int level, int (*handler)(void *), void *arg, bool known_mpsafe, const char *xname) { struct intrhand **p, *q, *ih; struct cpu_info *ci; int slot, error, idt_vec; struct intrsource *chained, *source; #ifdef MULTIPROCESSOR bool mpsafe = (known_mpsafe || level != IPL_VM); #endif /* MULTIPROCESSOR */ uint64_t where; const char *intrstr; char intrstr_buf[INTRIDBUF]; KASSERTMSG((legacy_irq == -1 || (0 <= legacy_irq && legacy_irq < 16)), "bad legacy IRQ value: %d", legacy_irq); KASSERTMSG((legacy_irq != -1 || pic != &i8259_pic), "non-legacy IRQ on i8259"); ih = kmem_alloc(sizeof(*ih), KM_SLEEP); intrstr = intr_create_intrid(legacy_irq, pic, pin, intrstr_buf, sizeof(intrstr_buf)); KASSERT(intrstr != NULL); mutex_enter(&cpu_lock); /* allocate intrsource pool, if not yet. */ chained = intr_get_io_intrsource(intrstr); if (chained == NULL) { if (msipic_is_msi_pic(pic)) { mutex_exit(&cpu_lock); kmem_free(ih, sizeof(*ih)); printf("%s: %s has no intrsource\n", __func__, intrstr); return NULL; } chained = intr_allocate_io_intrsource(intrstr); if (chained == NULL) { mutex_exit(&cpu_lock); kmem_free(ih, sizeof(*ih)); printf("%s: can't allocate io_intersource\n", __func__); return NULL; } } error = intr_allocate_slot(pic, pin, level, &ci, &slot, &idt_vec, chained); if (error != 0) { intr_free_io_intrsource_direct(chained); mutex_exit(&cpu_lock); kmem_free(ih, sizeof(*ih)); printf("failed to allocate interrupt slot for PIC %s pin %d\n", pic->pic_name, pin); return NULL; } source = ci->ci_isources[slot]; if (source->is_handlers != NULL && source->is_pic->pic_type != pic->pic_type) { intr_free_io_intrsource_direct(chained); mutex_exit(&cpu_lock); kmem_free(ih, sizeof(*ih)); printf("%s: can't share intr source between " "different PIC types (legacy_irq %d pin %d slot %d)\n", __func__, legacy_irq, pin, slot); return NULL; } source->is_pin = pin; source->is_pic = pic; intr_append_intrsource_xname(source, xname); switch (source->is_type) { case IST_NONE: source->is_type = type; break; case IST_EDGE: case IST_LEVEL: if (source->is_type == type) break; /* FALLTHROUGH */ case IST_PULSE: if (type != IST_NONE) { intr_source_free(ci, slot, pic, idt_vec); intr_free_io_intrsource_direct(chained); mutex_exit(&cpu_lock); kmem_free(ih, sizeof(*ih)); printf("%s: pic %s pin %d: can't share " "type %d with %d\n", __func__, pic->pic_name, pin, source->is_type, type); return NULL; } break; default: panic("%s: bad intr type %d for pic %s pin %d\n", __func__, source->is_type, pic->pic_name, pin); /* NOTREACHED */ } /* * If the establishing interrupt uses shared IRQ, the interrupt uses * "ci->ci_isources[slot]" instead of allocated by the establishing * device's pci_intr_alloc() or this function. */ if (source->is_handlers != NULL) { struct intrsource *isp, *nisp; SIMPLEQ_FOREACH_SAFE(isp, &io_interrupt_sources, is_list, nisp) { if (strncmp(intrstr, isp->is_intrid, INTRIDBUF - 1) == 0 && isp->is_handlers == NULL) intr_free_io_intrsource_direct(isp); } } /* * We're now committed. Mask the interrupt in hardware and * count it for load distribution. */ (*pic->pic_hwmask)(pic, pin); (ci->ci_nintrhand)++; /* * Figure out where to put the handler. * This is O(N^2), but we want to preserve the order, and N is * generally small. */ for (p = &ci->ci_isources[slot]->is_handlers; (q = *p) != NULL && q->ih_level > level; p = &q->ih_next) { /* nothing */; } ih->ih_pic = pic; ih->ih_fun = ih->ih_realfun = handler; ih->ih_arg = ih->ih_realarg = arg; ih->ih_prevp = p; ih->ih_next = *p; ih->ih_level = level; ih->ih_pin = pin; ih->ih_cpu = ci; ih->ih_slot = slot; strlcpy(ih->ih_xname, xname, sizeof(ih->ih_xname)); #ifdef MULTIPROCESSOR if (!mpsafe) { ih->ih_fun = intr_biglock_wrapper; ih->ih_arg = ih; } #endif /* MULTIPROCESSOR */ /* * Call out to the remote CPU to update its interrupt state. * Only make RPCs if the APs are up and running. */ if (ci == curcpu() || !mp_online) { intr_establish_xcall(ih, (void *)(intptr_t)idt_vec); } else { where = xc_unicast(0, intr_establish_xcall, ih, (void *)(intptr_t)idt_vec, ci); xc_wait(where); } /* All set up, so add a route for the interrupt and unmask it. */ (*pic->pic_addroute)(pic, ci, pin, idt_vec, type); if (ci == curcpu() || !mp_online) { intr_hwunmask_xcall(ih, NULL); } else { where = xc_unicast(0, intr_hwunmask_xcall, ih, NULL, ci); xc_wait(where); } mutex_exit(&cpu_lock); if (bootverbose || cpu_index(ci) != 0) aprint_verbose("allocated pic %s type %s pin %d level %d to " "%s slot %d idt entry %d\n", pic->pic_name, type == IST_EDGE ? "edge" : "level", pin, level, device_xname(ci->ci_dev), slot, idt_vec); return ih; } void * intr_establish(int legacy_irq, struct pic *pic, int pin, int type, int level, int (*handler)(void *), void *arg, bool known_mpsafe) { return intr_establish_xname(legacy_irq, pic, pin, type, level, handler, arg, known_mpsafe, "unknown"); } /* * Called on bound CPU to handle intr_mask() / intr_unmask(). * * => caller (on initiating CPU) holds cpu_lock on our behalf * => arg1: struct intrhand *ih * => arg2: true -> mask, false -> unmask. */ static void intr_mask_xcall(void *arg1, void *arg2) { struct intrhand * const ih = arg1; const uintptr_t mask = (uintptr_t)arg2; struct cpu_info * const ci = ih->ih_cpu; bool force_pending = false; KASSERT(ci == curcpu() || !mp_online); /* * We need to disable interrupts to hold off the interrupt * vectors. */ const u_long psl = x86_read_psl(); x86_disable_intr(); struct intrsource * const source = ci->ci_isources[ih->ih_slot]; struct pic * const pic = source->is_pic; if (mask) { source->is_mask_count++; KASSERT(source->is_mask_count != 0); if (source->is_mask_count == 1) { (*pic->pic_hwmask)(pic, ih->ih_pin); } } else { KASSERT(source->is_mask_count != 0); if (--source->is_mask_count == 0) { /* * If this interrupt source is being moved, don't * unmask it at the hw. */ if (! source->is_distribute_pending) { (*pic->pic_hwunmask)(pic, ih->ih_pin); } /* * For level-sensitive interrupts, the hardware * will let us know. For everything else, we * need to explicitly handle interrupts that * happened when when the source was masked. */ const uint32_t bit = (1U << ih->ih_slot); if (ci->ci_imasked & bit) { ci->ci_imasked &= ~bit; if (source->is_type != IST_LEVEL) { ci->ci_ipending |= bit; force_pending = true; } } } } /* Re-enable interrupts. */ x86_write_psl(psl); if (force_pending) { /* Force processing of any pending interrupts. */ splx(splhigh()); } } static void intr_mask_internal(struct intrhand * const ih, const bool mask) { /* * Call out to the remote CPU to update its interrupt state. * Only make RPCs if the APs are up and running. */ mutex_enter(&cpu_lock); struct cpu_info * const ci = ih->ih_cpu; void * const mask_arg = (void *)(uintptr_t)mask; if (ci == curcpu() || !mp_online) { intr_mask_xcall(ih, mask_arg); } else { const uint64_t where = xc_unicast(0, intr_mask_xcall, ih, mask_arg, ci); xc_wait(where); } mutex_exit(&cpu_lock); } void intr_mask(struct intrhand *ih) { if (cpu_intr_p()) { /* * Special case of calling intr_mask() from an interrupt * handler: we MUST be called from the bound CPU for this * interrupt (presumably from a handler we're about to * mask). * * We can't take the cpu_lock in this case, and we must * therefore be extra careful. */ KASSERT(ih->ih_cpu == curcpu() || !mp_online); intr_mask_xcall(ih, (void *)(uintptr_t)true); return; } intr_mask_internal(ih, true); } void intr_unmask(struct intrhand *ih) { /* * This is not safe to call from an interrupt context because * we don't want to accidentally unmask an interrupt source * that's masked because it's being serviced. */ KASSERT(!cpu_intr_p()); intr_mask_internal(ih, false); } /* * Called on bound CPU to handle intr_disestablish(). * * => caller (on initiating CPU) holds cpu_lock on our behalf * => arg1: struct intrhand *ih * => arg2: unused */ static void intr_disestablish_xcall(void *arg1, void *arg2) { struct intrhand **p, *q; struct cpu_info *ci; struct pic *pic; struct intrsource *source; struct intrhand *ih; u_long psl; int idtvec; ih = arg1; ci = ih->ih_cpu; KASSERT(ci == curcpu() || !mp_online); /* Disable interrupts locally. */ psl = x86_read_psl(); x86_disable_intr(); pic = ci->ci_isources[ih->ih_slot]->is_pic; source = ci->ci_isources[ih->ih_slot]; idtvec = source->is_idtvec; (*pic->pic_hwmask)(pic, ih->ih_pin); atomic_and_32(&ci->ci_ipending, ~(1 << ih->ih_slot)); /* * Remove the handler from the chain. */ for (p = &source->is_handlers; (q = *p) != NULL && q != ih; p = &q->ih_next) ; if (q == NULL) { x86_write_psl(psl); panic("%s: handler not registered", __func__); /* NOTREACHED */ } *p = q->ih_next; x86_intr_calculatemasks(ci); /* * If there is no any handler, 1) do delroute because it has no * any source and 2) dont' hwunmask to prevent spurious interrupt. * * If there is any handler, 1) don't delroute because it has source * and 2) do hwunmask to be able to get interrupt again. * */ if (source->is_handlers == NULL) (*pic->pic_delroute)(pic, ci, ih->ih_pin, idtvec, source->is_type); else if (source->is_mask_count == 0) (*pic->pic_hwunmask)(pic, ih->ih_pin); /* If the source is free we can drop it now. */ intr_source_free(ci, ih->ih_slot, pic, idtvec); /* Re-enable interrupts. */ x86_write_psl(psl); DPRINTF(("%s: remove slot %d (pic %s pin %d vec %d)\n", device_xname(ci->ci_dev), ih->ih_slot, pic->pic_name, ih->ih_pin, idtvec)); } static int intr_num_handlers(struct intrsource *isp) { struct intrhand *ih; int num; num = 0; for (ih = isp->is_handlers; ih != NULL; ih = ih->ih_next) num++; return num; } /* * Deregister an interrupt handler. */ void intr_disestablish(struct intrhand *ih) { struct cpu_info *ci; struct intrsource *isp; uint64_t where; /* * Count the removal for load balancing. * Call out to the remote CPU to update its interrupt state. * Only make RPCs if the APs are up and running. */ mutex_enter(&cpu_lock); ci = ih->ih_cpu; (ci->ci_nintrhand)--; KASSERT(ci->ci_nintrhand >= 0); isp = ci->ci_isources[ih->ih_slot]; if (ci == curcpu() || !mp_online) { intr_disestablish_xcall(ih, NULL); } else { where = xc_unicast(0, intr_disestablish_xcall, ih, NULL, ci); xc_wait(where); } if (!msipic_is_msi_pic(isp->is_pic) && intr_num_handlers(isp) < 1) { intr_free_io_intrsource_direct(isp); } mutex_exit(&cpu_lock); kmem_free(ih, sizeof(*ih)); } static const char * xen_intr_string(int port, char *buf, size_t len, struct pic *pic) { KASSERT(pic->pic_type == PIC_XEN); KASSERT(port >= 0); snprintf(buf, len, "%s chan %d", pic->pic_name, port); return buf; } static const char * legacy_intr_string(int ih, char *buf, size_t len, struct pic *pic) { int legacy_irq; KASSERT(pic->pic_type == PIC_I8259); #if NLAPIC > 0 KASSERT(APIC_IRQ_ISLEGACY(ih)); legacy_irq = APIC_IRQ_LEGACY_IRQ(ih); #else legacy_irq = ih; #endif KASSERT(legacy_irq >= 0 && legacy_irq < 16); snprintf(buf, len, "%s pin %d", pic->pic_name, legacy_irq); return buf; } const char * intr_string(intr_handle_t ih, char *buf, size_t len) { #if NIOAPIC > 0 struct ioapic_softc *pic; #endif if (ih == 0) panic("%s: bogus handle 0x%" PRIx64, __func__, ih); #if NIOAPIC > 0 if (ih & APIC_INT_VIA_APIC) { pic = ioapic_find(APIC_IRQ_APIC(ih)); if (pic != NULL) { snprintf(buf, len, "%s pin %d", device_xname(pic->sc_dev), APIC_IRQ_PIN(ih)); } else { snprintf(buf, len, "apic %d int %d (irq %d)", APIC_IRQ_APIC(ih), APIC_IRQ_PIN(ih), APIC_IRQ_LEGACY_IRQ(ih)); } } else snprintf(buf, len, "irq %d", APIC_IRQ_LEGACY_IRQ(ih)); #elif NLAPIC > 0 snprintf(buf, len, "irq %d", APIC_IRQ_LEGACY_IRQ(ih)); #else snprintf(buf, len, "irq %d", (int) ih); #endif return buf; } /* * Fake interrupt handler structures for the benefit of symmetry with * other interrupt sources, and the benefit of x86_intr_calculatemasks() */ struct intrhand fake_timer_intrhand; struct intrhand fake_ipi_intrhand; #if NHYPERV > 0 struct intrhand fake_hyperv_intrhand; #endif #if NLAPIC > 0 && defined(MULTIPROCESSOR) static const char *x86_ipi_names[X86_NIPI] = X86_IPI_NAMES; #endif #if defined(INTRSTACKSIZE) static inline bool redzone_const_or_false(bool x) { #ifdef DIAGNOSTIC return x; #else return false; #endif /* !DIAGNOSTIC */ } static inline int redzone_const_or_zero(int x) { return redzone_const_or_false(true) ? x : 0; } #endif /* * Initialize all handlers that aren't dynamically allocated, and exist * for each CPU. */ void cpu_intr_init(struct cpu_info *ci) { #if (NLAPIC > 0) || defined(MULTIPROCESSOR) || \ (NHYPERV > 0) struct intrsource *isp; #endif #if NLAPIC > 0 static int first = 1; #if defined(MULTIPROCESSOR) int i; #endif #endif #if NLAPIC > 0 isp = kmem_zalloc(sizeof(*isp), KM_SLEEP); isp->is_recurse = Xrecurse_lapic_ltimer; isp->is_resume = Xresume_lapic_ltimer; fake_timer_intrhand.ih_pic = &local_pic; fake_timer_intrhand.ih_level = IPL_CLOCK; isp->is_handlers = &fake_timer_intrhand; isp->is_pic = &local_pic; ci->ci_isources[LIR_TIMER] = isp; evcnt_attach_dynamic(&isp->is_evcnt, first ? EVCNT_TYPE_INTR : EVCNT_TYPE_MISC, NULL, device_xname(ci->ci_dev), "timer"); first = 0; #ifdef MULTIPROCESSOR isp = kmem_zalloc(sizeof(*isp), KM_SLEEP); isp->is_recurse = Xrecurse_lapic_ipi; isp->is_resume = Xresume_lapic_ipi; fake_ipi_intrhand.ih_pic = &local_pic; fake_ipi_intrhand.ih_level = IPL_HIGH; isp->is_handlers = &fake_ipi_intrhand; isp->is_pic = &local_pic; ci->ci_isources[LIR_IPI] = isp; for (i = 0; i < X86_NIPI; i++) evcnt_attach_dynamic(&ci->ci_ipi_events[i], EVCNT_TYPE_MISC, NULL, device_xname(ci->ci_dev), x86_ipi_names[i]); #endif #if NHYPERV > 0 if (hyperv_hypercall_enabled()) { isp = kmem_zalloc(sizeof(*isp), KM_SLEEP); isp->is_recurse = Xrecurse_hyperv_hypercall; isp->is_resume = Xresume_hyperv_hypercall; fake_hyperv_intrhand.ih_level = IPL_NET; isp->is_handlers = &fake_hyperv_intrhand; isp->is_pic = &local_pic; ci->ci_isources[LIR_HV] = isp; evcnt_attach_dynamic(&isp->is_evcnt, EVCNT_TYPE_INTR, NULL, device_xname(ci->ci_dev), "Hyper-V hypercall"); } #endif #endif #if defined(__HAVE_PREEMPTION) x86_init_preempt(ci); #endif x86_intr_calculatemasks(ci); #if defined(INTRSTACKSIZE) vaddr_t istack; /* * If the red zone is activated, protect both the top and * the bottom of the stack with an unmapped page. */ istack = uvm_km_alloc(kernel_map, INTRSTACKSIZE + redzone_const_or_zero(2 * PAGE_SIZE), 0, UVM_KMF_WIRED | UVM_KMF_ZERO); if (redzone_const_or_false(true)) { pmap_kremove(istack, PAGE_SIZE); pmap_kremove(istack + INTRSTACKSIZE + PAGE_SIZE, PAGE_SIZE); pmap_update(pmap_kernel()); } /* * 33 used to be 1. Arbitrarily reserve 32 more register_t's * of space for ddb(4) to examine some subroutine arguments * and to hunt for the next stack frame. */ ci->ci_intrstack = (char *)istack + redzone_const_or_zero(PAGE_SIZE) + INTRSTACKSIZE - 33 * sizeof(register_t); #endif ci->ci_idepth = -1; } #if defined(INTRDEBUG) || defined(DDB) void intr_printconfig(void) { int i; struct intrhand *ih; struct intrsource *isp; struct cpu_info *ci; CPU_INFO_ITERATOR cii; void (*pr)(const char *, ...); pr = printf; #ifdef DDB extern int db_active; if (db_active) { pr = db_printf; } #endif for (CPU_INFO_FOREACH(cii, ci)) { (*pr)("%s: interrupt masks:\n", device_xname(ci->ci_dev)); for (i = 0; i < NIPL; i++) (*pr)("IPL %d mask %08lx unmask %08lx\n", i, (u_long)ci->ci_imask[i], (u_long)ci->ci_iunmask[i]); for (i = 0; i < MAX_INTR_SOURCES; i++) { isp = ci->ci_isources[i]; if (isp == NULL) continue; (*pr)("%s source %d is pin %d from pic %s type %d " "maxlevel %d\n", device_xname(ci->ci_dev), i, isp->is_pin, isp->is_pic->pic_name, isp->is_type, isp->is_maxlevel); for (ih = isp->is_handlers; ih != NULL; ih = ih->ih_next) (*pr)("\thandler %p level %d\n", ih->ih_fun, ih->ih_level); #if NIOAPIC > 0 if (isp->is_pic->pic_type == PIC_IOAPIC) { struct ioapic_softc *sc; sc = isp->is_pic->pic_ioapic; (*pr)("\tioapic redir 0x%x\n", sc->sc_pins[isp->is_pin].ip_map->redir); } #endif } } } #endif /* * Save current affinitied cpu's interrupt count. */ static void intr_save_evcnt(struct intrsource *source, cpuid_t cpuid) { struct percpu_evcnt *pep; uint64_t curcnt; int i; curcnt = source->is_evcnt.ev_count; pep = source->is_saved_evcnt; for (i = 0; i < ncpu; i++) { if (pep[i].cpuid == cpuid) { pep[i].count = curcnt; break; } } } /* * Restore current affinitied cpu's interrupt count. */ static void intr_restore_evcnt(struct intrsource *source, cpuid_t cpuid) { struct percpu_evcnt *pep; int i; pep = source->is_saved_evcnt; for (i = 0; i < ncpu; i++) { if (pep[i].cpuid == cpuid) { source->is_evcnt.ev_count = pep[i].count; break; } } } static void intr_redistribute_xc_t(void *arg1, void *arg2) { struct cpu_info *ci; struct intrsource *isp; int slot; u_long psl; ci = curcpu(); isp = arg1; slot = (int)(intptr_t)arg2; /* Disable interrupts locally. */ psl = x86_read_psl(); x86_disable_intr(); /* Hook it in and re-calculate masks. */ ci->ci_isources[slot] = isp; x86_intr_calculatemasks(curcpu()); /* Re-enable interrupts locally. */ x86_write_psl(psl); } static void intr_redistribute_xc_s1(void *arg1, void *arg2) { struct pic *pic; struct intrsource *isp; struct cpu_info *nci; u_long psl; isp = arg1; nci = arg2; /* * Disable interrupts on-chip and mask the pin. Back out * and let the interrupt be processed if one is pending. */ pic = isp->is_pic; for (;;) { psl = x86_read_psl(); x86_disable_intr(); if ((*pic->pic_trymask)(pic, isp->is_pin)) { break; } x86_write_psl(psl); DELAY(1000); } /* pic_addroute will unmask the interrupt. */ (*pic->pic_addroute)(pic, nci, isp->is_pin, isp->is_idtvec, isp->is_type); x86_write_psl(psl); } static void intr_redistribute_xc_s2(void *arg1, void *arg2) { struct cpu_info *ci; u_long psl; int slot; ci = curcpu(); slot = (int)(uintptr_t)arg1; /* Disable interrupts locally. */ psl = x86_read_psl(); x86_disable_intr(); /* Patch out the source and re-calculate masks. */ ci->ci_isources[slot] = NULL; x86_intr_calculatemasks(ci); /* Re-enable interrupts locally. */ x86_write_psl(psl); } static bool intr_redistribute(struct cpu_info *oci) { struct intrsource *isp; struct intrhand *ih; CPU_INFO_ITERATOR cii; struct cpu_info *nci, *ici; int oslot, nslot; uint64_t where; KASSERT(mutex_owned(&cpu_lock)); /* Look for an interrupt source that we can migrate. */ for (oslot = 0; oslot < MAX_INTR_SOURCES; oslot++) { if ((isp = oci->ci_isources[oslot]) == NULL) { continue; } if (isp->is_pic->pic_type == PIC_IOAPIC) { break; } } if (oslot == MAX_INTR_SOURCES) { return false; } /* Find least loaded CPU and try to move there. */ nci = NULL; for (CPU_INFO_FOREACH(cii, ici)) { if ((ici->ci_schedstate.spc_flags & SPCF_NOINTR) != 0) { continue; } KASSERT(ici != oci); if (nci == NULL || nci->ci_nintrhand > ici->ci_nintrhand) { nci = ici; } } if (nci == NULL) { return false; } for (nslot = 0; nslot < MAX_INTR_SOURCES; nslot++) { if (nci->ci_isources[nslot] == NULL) { break; } } /* If that did not work, allocate anywhere. */ if (nslot == MAX_INTR_SOURCES) { for (CPU_INFO_FOREACH(cii, nci)) { if ((nci->ci_schedstate.spc_flags & SPCF_NOINTR) != 0) { continue; } KASSERT(nci != oci); for (nslot = 0; nslot < MAX_INTR_SOURCES; nslot++) { if (nci->ci_isources[nslot] == NULL) { break; } } if (nslot != MAX_INTR_SOURCES) { break; } } } if (nslot == MAX_INTR_SOURCES) { return false; } /* * Now we have new CPU and new slot. Run a cross-call to set up * the new vector on the target CPU. */ where = xc_unicast(0, intr_redistribute_xc_t, isp, (void *)(intptr_t)nslot, nci); xc_wait(where); /* * We're ready to go on the target CPU. Run a cross call to * reroute the interrupt away from the source CPU. */ where = xc_unicast(0, intr_redistribute_xc_s1, isp, nci, oci); xc_wait(where); /* Sleep for (at least) 10ms to allow the change to take hold. */ (void)kpause("intrdist", false, mstohz(10), NULL); /* Complete removal from the source CPU. */ where = xc_unicast(0, intr_redistribute_xc_s2, (void *)(uintptr_t)oslot, NULL, oci); xc_wait(where); /* Finally, take care of book-keeping. */ for (ih = isp->is_handlers; ih != NULL; ih = ih->ih_next) { oci->ci_nintrhand--; nci->ci_nintrhand++; ih->ih_cpu = nci; } intr_save_evcnt(isp, oci->ci_cpuid); intr_restore_evcnt(isp, nci->ci_cpuid); isp->is_active_cpu = nci->ci_cpuid; return true; } void cpu_intr_redistribute(void) { CPU_INFO_ITERATOR cii; struct cpu_info *ci; KASSERT(mutex_owned(&cpu_lock)); KASSERT(mp_online); /* Direct interrupts away from shielded CPUs. */ for (CPU_INFO_FOREACH(cii, ci)) { if ((ci->ci_schedstate.spc_flags & SPCF_NOINTR) == 0) { continue; } while (intr_redistribute(ci)) { /* nothing */ } } /* XXX should now re-balance */ } u_int cpu_intr_count(struct cpu_info *ci) { KASSERT(ci->ci_nintrhand >= 0); return ci->ci_nintrhand; } static int intr_find_unused_slot(struct cpu_info *ci, int *index) { int slot, i; KASSERT(mutex_owned(&cpu_lock)); slot = -1; for (i = 0; i < MAX_INTR_SOURCES ; i++) { if (ci->ci_isources[i] == NULL) { slot = i; break; } } if (slot == -1) { DPRINTF(("cannot allocate ci_isources\n")); return EBUSY; } *index = slot; return 0; } /* * Let cpu_info ready to accept the interrupt. */ static void intr_activate_xcall(void *arg1, void *arg2) { struct cpu_info *ci; struct intrsource *source; struct intrstub *stubp; struct intrhand *ih; struct idt_vec *iv; u_long psl; int idt_vec; int slot; ih = arg1; kpreempt_disable(); KASSERT(ih->ih_cpu == curcpu() || !mp_online); ci = ih->ih_cpu; slot = ih->ih_slot; source = ci->ci_isources[slot]; idt_vec = source->is_idtvec; iv = idt_vec_ref(&ci->ci_idtvec); psl = x86_read_psl(); x86_disable_intr(); x86_intr_calculatemasks(ci); if (source->is_type == IST_LEVEL) { stubp = &source->is_pic->pic_level_stubs[slot]; } else { stubp = &source->is_pic->pic_edge_stubs[slot]; } source->is_resume = stubp->ist_resume; source->is_recurse = stubp->ist_recurse; idt_vec_set(iv, idt_vec, stubp->ist_entry); x86_write_psl(psl); kpreempt_enable(); } /* * Let cpu_info not accept the interrupt. */ static void intr_deactivate_xcall(void *arg1, void *arg2) { struct cpu_info *ci; struct intrhand *ih, *lih; struct intrsource *isp; u_long psl; int idt_vec; int slot; ih = arg1; kpreempt_disable(); KASSERT(ih->ih_cpu == curcpu() || !mp_online); ci = ih->ih_cpu; slot = ih->ih_slot; isp = ci->ci_isources[slot]; idt_vec = isp->is_idtvec; psl = x86_read_psl(); x86_disable_intr(); /* Move all devices sharing IRQ number. */ ci->ci_isources[slot] = NULL; for (lih = ih; lih != NULL; lih = lih->ih_next) { ci->ci_nintrhand--; } x86_intr_calculatemasks(ci); if (idt_vec_is_pcpu()) { idt_vec_free(&ci->ci_idtvec, idt_vec); } else { /* * Skip unsetgate(), because the same idt[] entry is * overwritten in intr_activate_xcall(). */ } x86_write_psl(psl); kpreempt_enable(); } static void intr_get_affinity(struct intrsource *isp, kcpuset_t *cpuset) { struct cpu_info *ci; KASSERT(mutex_owned(&cpu_lock)); if (isp == NULL) { kcpuset_zero(cpuset); return; } KASSERTMSG(isp->is_handlers != NULL, "Don't get affinity for the device which is not established."); ci = isp->is_handlers->ih_cpu; if (ci == NULL) { kcpuset_zero(cpuset); return; } kcpuset_set(cpuset, cpu_index(ci)); return; } static int intr_set_affinity(struct intrsource *isp, const kcpuset_t *cpuset) { struct cpu_info *oldci, *newci; struct intrhand *ih, *lih; struct pic *pic; u_int cpu_idx; int old_idtvec, new_idtvec; int oldslot, newslot; int err; int pin; KASSERT(mutex_owned(&intr_distribute_lock)); KASSERT(mutex_owned(&cpu_lock)); /* XXX * logical destination mode is not supported, use lowest index cpu. */ cpu_idx = kcpuset_ffs(cpuset) - 1; newci = cpu_lookup(cpu_idx); if (newci == NULL) { DPRINTF(("invalid cpu index: %u\n", cpu_idx)); return EINVAL; } if ((newci->ci_schedstate.spc_flags & SPCF_NOINTR) != 0) { DPRINTF(("the cpu is set nointr shield. index:%u\n", cpu_idx)); return EINVAL; } if (isp == NULL) { DPRINTF(("invalid intrctl handler\n")); return EINVAL; } /* i8259_pic supports only primary cpu, see i8259.c. */ pic = isp->is_pic; if (pic == &i8259_pic) { DPRINTF(("i8259 pic does not support set_affinity\n")); return ENOTSUP; } ih = isp->is_handlers; KASSERTMSG(ih != NULL, "Don't set affinity for the device which is not established."); oldci = ih->ih_cpu; if (newci == oldci) /* nothing to do */ return 0; oldslot = ih->ih_slot; err = intr_find_unused_slot(newci, &newslot); if (err) { DPRINTF(("failed to allocate interrupt slot for PIC %s intrid " "%s\n", isp->is_pic->pic_name, isp->is_intrid)); return err; } old_idtvec = isp->is_idtvec; if (idt_vec_is_pcpu()) { new_idtvec = idt_vec_alloc(&newci->ci_idtvec, APIC_LEVEL(ih->ih_level), IDT_INTR_HIGH); if (new_idtvec == 0) return EBUSY; DPRINTF(("interrupt from cpu%d vec %d to cpu%d vec %d\n", cpu_index(oldci), old_idtvec, cpu_index(newci), new_idtvec)); } else { new_idtvec = isp->is_idtvec; } /* Prevent intr_unmask() from reenabling the source at the hw. */ isp->is_distribute_pending = true; pin = isp->is_pin; (*pic->pic_hwmask)(pic, pin); /* for ci_ipending check */ while (oldci->ci_ipending & (1 << oldslot)) { (void)kpause("intrdist", false, 1, &cpu_lock); } kpreempt_disable(); /* deactivate old interrupt setting */ if (oldci == curcpu() || !mp_online) { intr_deactivate_xcall(ih, NULL); } else { uint64_t where; where = xc_unicast(0, intr_deactivate_xcall, ih, NULL, oldci); xc_wait(where); } intr_save_evcnt(isp, oldci->ci_cpuid); (*pic->pic_delroute)(pic, oldci, pin, old_idtvec, isp->is_type); /* activate new interrupt setting */ isp->is_idtvec = new_idtvec; newci->ci_isources[newslot] = isp; for (lih = ih; lih != NULL; lih = lih->ih_next) { newci->ci_nintrhand++; lih->ih_cpu = newci; lih->ih_slot = newslot; } if (newci == curcpu() || !mp_online) { intr_activate_xcall(ih, NULL); } else { uint64_t where; where = xc_unicast(0, intr_activate_xcall, ih, NULL, newci); xc_wait(where); } intr_restore_evcnt(isp, newci->ci_cpuid); isp->is_active_cpu = newci->ci_cpuid; (*pic->pic_addroute)(pic, newci, pin, new_idtvec, isp->is_type); isp->is_distribute_pending = false; if (newci == curcpu() || !mp_online) { intr_hwunmask_xcall(ih, NULL); } else { uint64_t where; where = xc_unicast(0, intr_hwunmask_xcall, ih, NULL, newci); xc_wait(where); } kpreempt_enable(); return err; } static bool intr_is_affinity_intrsource(struct intrsource *isp, const kcpuset_t *cpuset) { struct cpu_info *ci; KASSERT(mutex_owned(&cpu_lock)); /* * The device is already pci_intr_alloc'ed, however it is not * established yet. */ if (isp->is_handlers == NULL) return false; ci = isp->is_handlers->ih_cpu; KASSERT(ci != NULL); return kcpuset_isset(cpuset, cpu_index(ci)); } static struct intrhand * intr_get_handler(const char *intrid) { struct intrsource *isp; KASSERT(mutex_owned(&cpu_lock)); isp = intr_get_io_intrsource(intrid); if (isp == NULL) return NULL; return isp->is_handlers; } uint64_t x86_intr_get_count(const char *intrid, u_int cpu_idx) { struct cpu_info *ci; struct intrsource *isp; struct intrhand *ih; struct percpu_evcnt pep; cpuid_t cpuid; int i, slot; uint64_t count = 0; KASSERT(mutex_owned(&cpu_lock)); ci = cpu_lookup(cpu_idx); cpuid = ci->ci_cpuid; ih = intr_get_handler(intrid); if (ih == NULL) { count = 0; goto out; } slot = ih->ih_slot; isp = ih->ih_cpu->ci_isources[slot]; for (i = 0; i < ncpu; i++) { pep = isp->is_saved_evcnt[i]; if (cpuid == pep.cpuid) { if (isp->is_active_cpu == pep.cpuid) { count = isp->is_evcnt.ev_count; goto out; } else { count = pep.count; goto out; } } } out: return count; } void x86_intr_get_assigned(const char *intrid, kcpuset_t *cpuset) { struct cpu_info *ci; struct intrhand *ih; KASSERT(mutex_owned(&cpu_lock)); kcpuset_zero(cpuset); ih = intr_get_handler(intrid); if (ih == NULL) return; ci = ih->ih_cpu; kcpuset_set(cpuset, cpu_index(ci)); } void x86_intr_get_devname(const char *intrid, char *buf, size_t len) { struct intrsource *isp; struct intrhand *ih; int slot; KASSERT(mutex_owned(&cpu_lock)); ih = intr_get_handler(intrid); if (ih == NULL) { buf[0] = '\0'; return; } slot = ih->ih_slot; isp = ih->ih_cpu->ci_isources[slot]; strlcpy(buf, isp->is_xname, len); } /* * MI interface for subr_interrupt.c */ uint64_t interrupt_get_count(const char *intrid, u_int cpu_idx) { struct intrsource *isp; uint64_t count = 0; mutex_enter(&cpu_lock); isp = intr_get_io_intrsource(intrid); if (isp != NULL) count = isp->is_pic->pic_intr_get_count(intrid, cpu_idx); mutex_exit(&cpu_lock); return count; } /* * MI interface for subr_interrupt.c */ void interrupt_get_assigned(const char *intrid, kcpuset_t *cpuset) { struct intrsource *isp; mutex_enter(&cpu_lock); isp = intr_get_io_intrsource(intrid); if (isp != NULL) isp->is_pic->pic_intr_get_assigned(intrid, cpuset); mutex_exit(&cpu_lock); } /* * MI interface for subr_interrupt.c */ void interrupt_get_available(kcpuset_t *cpuset) { CPU_INFO_ITERATOR cii; struct cpu_info *ci; kcpuset_zero(cpuset); mutex_enter(&cpu_lock); for (CPU_INFO_FOREACH(cii, ci)) { if ((ci->ci_schedstate.spc_flags & SPCF_NOINTR) == 0) { kcpuset_set(cpuset, cpu_index(ci)); } } mutex_exit(&cpu_lock); } /* * MI interface for subr_interrupt.c */ void interrupt_get_devname(const char *intrid, char *buf, size_t len) { struct intrsource *isp; mutex_enter(&cpu_lock); isp = intr_get_io_intrsource(intrid); if (isp != NULL) { if (isp->is_pic->pic_intr_get_devname == NULL) { printf("NULL get_devname intrid %s pic %s\n", intrid, isp->is_pic->pic_name); } else { isp->is_pic->pic_intr_get_devname(intrid, buf, len); } } mutex_exit(&cpu_lock); } static int intr_distribute_locked(struct intrhand *ih, const kcpuset_t *newset, kcpuset_t *oldset) { struct intrsource *isp; int slot; KASSERT(mutex_owned(&intr_distribute_lock)); KASSERT(mutex_owned(&cpu_lock)); if (ih == NULL) return EINVAL; slot = ih->ih_slot; isp = ih->ih_cpu->ci_isources[slot]; KASSERT(isp != NULL); if (oldset != NULL) intr_get_affinity(isp, oldset); return intr_set_affinity(isp, newset); } /* * MI interface for subr_interrupt.c */ int interrupt_distribute(void *cookie, const kcpuset_t *newset, kcpuset_t *oldset) { int error; struct intrhand *ih = cookie; mutex_enter(&intr_distribute_lock); mutex_enter(&cpu_lock); error = intr_distribute_locked(ih, newset, oldset); mutex_exit(&cpu_lock); mutex_exit(&intr_distribute_lock); return error; } /* * MI interface for subr_interrupt.c */ int interrupt_distribute_handler(const char *intrid, const kcpuset_t *newset, kcpuset_t *oldset) { int error; struct intrhand *ih; mutex_enter(&intr_distribute_lock); mutex_enter(&cpu_lock); ih = intr_get_handler(intrid); if (ih == NULL) { error = ENOENT; goto out; } error = intr_distribute_locked(ih, newset, oldset); out: mutex_exit(&cpu_lock); mutex_exit(&intr_distribute_lock); return error; } /* * MI interface for subr_interrupt.c */ struct intrids_handler * interrupt_construct_intrids(const kcpuset_t *cpuset) { struct intrsource *isp; struct intrids_handler *ii_handler; intrid_t *ids; int i, count; if (kcpuset_iszero(cpuset)) return 0; /* * Count the number of interrupts which affinity to any cpu of * "cpuset". */ count = 0; mutex_enter(&cpu_lock); SIMPLEQ_FOREACH(isp, &io_interrupt_sources, is_list) { if (intr_is_affinity_intrsource(isp, cpuset)) count++; } mutex_exit(&cpu_lock); ii_handler = kmem_zalloc(sizeof(int) + sizeof(intrid_t) * count, KM_SLEEP); if (ii_handler == NULL) return NULL; ii_handler->iih_nids = count; if (count == 0) return ii_handler; ids = ii_handler->iih_intrids; i = 0; mutex_enter(&cpu_lock); SIMPLEQ_FOREACH(isp, &io_interrupt_sources, is_list) { /* Ignore devices attached after counting "count". */ if (i >= count) { DPRINTF(("New devices are attached after counting.\n")); break; } if (!intr_is_affinity_intrsource(isp, cpuset)) continue; strncpy(ids[i], isp->is_intrid, sizeof(intrid_t)); i++; } mutex_exit(&cpu_lock); return ii_handler; } /* * MI interface for subr_interrupt.c */ void interrupt_destruct_intrids(struct intrids_handler *ii_handler) { size_t iih_size; if (ii_handler == NULL) return; iih_size = sizeof(int) + sizeof(intrid_t) * ii_handler->iih_nids; kmem_free(ii_handler, iih_size); } |
| 58 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 1 1 2 2 1 15 15 1 1 1 12 12 10 5 6 23 7 15 23 10 15 60 60 60 60 59 60 60 59 56 59 59 59 59 59 60 60 57 1 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 197 198 199 200 201 202 203 204 205 206 207 208 209 210 211 212 213 214 215 216 217 218 219 220 221 222 223 224 225 226 227 228 229 230 231 232 233 234 235 236 237 238 239 240 241 242 243 244 245 246 247 248 249 250 251 252 253 254 255 256 257 258 259 260 261 262 263 264 265 266 267 268 269 270 271 272 273 274 275 276 277 278 279 280 281 282 283 284 285 286 287 288 289 290 291 292 293 294 295 296 297 298 299 300 301 302 303 304 305 306 307 308 309 310 311 312 313 314 315 316 317 318 319 320 321 322 323 324 325 326 327 328 329 330 331 332 333 334 335 336 337 338 339 340 341 342 343 344 345 346 347 348 349 350 351 352 353 354 355 356 357 358 359 360 361 362 363 364 365 366 367 368 369 370 371 372 373 374 375 376 377 378 379 380 381 382 383 384 385 386 387 388 389 390 391 392 393 394 395 396 397 398 399 400 401 402 403 404 405 406 407 408 409 410 411 412 413 414 415 416 417 418 419 420 421 422 423 424 425 426 427 428 429 430 431 432 433 434 435 436 437 438 439 440 441 442 443 444 445 446 447 448 449 450 451 452 453 454 455 456 457 458 459 460 461 462 463 464 465 466 467 468 469 470 471 472 473 474 475 476 477 478 479 480 481 482 483 484 485 486 487 488 489 490 491 492 493 494 495 496 497 498 499 500 501 502 503 504 505 506 507 508 509 510 511 512 513 514 515 516 517 518 519 520 521 522 523 524 525 526 527 528 529 530 531 532 533 534 535 536 537 538 539 540 541 542 543 544 545 546 547 548 549 550 551 552 553 554 555 556 557 558 559 560 561 562 563 564 565 566 567 568 569 570 571 572 573 574 575 576 577 578 579 580 581 582 583 584 585 586 587 588 589 590 591 592 593 594 595 596 597 598 599 600 601 602 603 604 605 606 607 608 609 610 611 612 613 614 615 616 617 618 619 620 621 622 623 624 625 626 627 628 629 630 631 632 633 634 635 636 637 638 639 640 641 642 643 644 645 646 647 648 649 650 651 652 653 654 655 656 657 658 659 660 661 662 663 664 665 666 667 668 669 670 671 672 673 674 675 676 677 678 679 680 681 682 683 684 685 686 687 688 689 690 691 692 693 694 695 696 697 698 699 700 701 702 703 704 705 706 707 708 709 710 711 712 713 714 715 716 717 718 719 720 721 722 723 724 725 726 727 728 729 730 731 732 733 734 735 736 737 738 739 740 741 742 743 744 745 746 747 748 749 750 751 752 753 754 755 756 757 758 759 760 761 762 763 764 765 766 767 768 769 770 771 772 773 774 775 776 777 778 779 780 781 782 783 784 785 786 787 788 789 790 791 792 793 794 795 796 797 798 799 800 801 802 803 804 805 806 807 808 809 810 811 812 813 814 815 816 817 818 819 820 821 822 823 824 825 826 827 828 829 830 831 832 833 834 835 836 837 838 839 840 841 842 843 844 845 846 847 848 849 850 851 852 853 854 855 856 857 858 859 860 861 862 863 864 865 866 867 868 869 870 871 872 873 874 875 876 877 878 879 880 881 882 883 884 885 886 887 888 889 890 891 892 893 894 895 896 897 898 899 900 901 902 903 904 905 906 907 908 909 910 911 912 913 914 915 916 917 918 919 920 921 922 923 924 925 926 927 928 929 930 931 932 933 934 935 936 937 938 939 940 941 942 943 944 945 946 947 948 949 950 951 952 953 954 955 956 957 958 959 960 961 962 963 964 965 966 967 968 969 970 971 972 973 974 975 976 977 978 979 980 981 982 983 984 985 986 987 988 989 990 991 992 993 994 995 996 997 998 999 1000 1001 1002 1003 1004 1005 1006 1007 1008 1009 1010 1011 1012 1013 1014 1015 1016 1017 1018 1019 1020 1021 1022 1023 1024 1025 1026 1027 1028 1029 1030 1031 1032 1033 1034 1035 1036 1037 1038 1039 1040 1041 1042 1043 1044 1045 1046 1047 1048 1049 1050 1051 1052 1053 1054 1055 1056 1057 1058 1059 1060 1061 1062 1063 1064 1065 1066 1067 1068 1069 1070 1071 1072 1073 1074 1075 1076 1077 1078 1079 1080 1081 1082 1083 1084 1085 1086 1087 1088 1089 1090 1091 1092 1093 1094 1095 1096 1097 1098 1099 1100 1101 1102 1103 1104 1105 1106 1107 1108 1109 1110 1111 1112 1113 1114 1115 1116 1117 1118 1119 1120 1121 1122 1123 1124 1125 1126 1127 1128 1129 1130 1131 1132 1133 1134 1135 1136 1137 1138 1139 1140 1141 1142 1143 1144 1145 1146 1147 1148 1149 1150 1151 1152 1153 1154 1155 1156 1157 1158 1159 1160 1161 1162 1163 1164 1165 1166 1167 1168 1169 1170 1171 1172 1173 1174 1175 1176 1177 1178 1179 1180 1181 1182 1183 1184 1185 1186 1187 1188 1189 1190 1191 1192 1193 1194 1195 1196 1197 1198 1199 1200 1201 1202 1203 1204 1205 1206 1207 1208 1209 1210 1211 1212 1213 1214 1215 1216 1217 1218 1219 1220 1221 1222 1223 1224 1225 1226 1227 1228 1229 1230 1231 1232 1233 | /* $NetBSD: prop_object.c,v 1.35 2022/08/07 23:49:46 riastradh Exp $ */ /*- * Copyright (c) 2006, 2007 The NetBSD Foundation, Inc. * All rights reserved. * * This code is derived from software contributed to The NetBSD Foundation * by Jason R. Thorpe. * * Redistribution and use in source and binary forms, with or without * modification, are permitted provided that the following conditions * are met: * 1. Redistributions of source code must retain the above copyright * notice, this list of conditions and the following disclaimer. * 2. Redistributions in binary form must reproduce the above copyright * notice, this list of conditions and the following disclaimer in the * documentation and/or other materials provided with the distribution. * * THIS SOFTWARE IS PROVIDED BY THE NETBSD FOUNDATION, INC. AND CONTRIBUTORS * ``AS IS'' AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED * TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR * PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE FOUNDATION OR CONTRIBUTORS * BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR * CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF * SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS * INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN * CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) * ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE * POSSIBILITY OF SUCH DAMAGE. */ #include "prop_object_impl.h" #include <prop/prop_object.h> #ifdef _PROP_NEED_REFCNT_MTX static pthread_mutex_t _prop_refcnt_mtx = PTHREAD_MUTEX_INITIALIZER; #endif /* _PROP_NEED_REFCNT_MTX */ #if !defined(_KERNEL) && !defined(_STANDALONE) #include <sys/mman.h> #include <sys/stat.h> #include <errno.h> #include <fcntl.h> #include <limits.h> #include <unistd.h> #endif #ifdef _STANDALONE void * _prop_standalone_calloc(size_t size) { void *rv; rv = alloc(size); if (rv != NULL) memset(rv, 0, size); return (rv); } void * _prop_standalone_realloc(void *v, size_t size) { void *rv; rv = alloc(size); if (rv != NULL) { memcpy(rv, v, size); /* XXX */ dealloc(v, 0); /* XXX */ } return (rv); } #endif /* _STANDALONE */ /* * _prop_object_init -- * Initialize an object. Called when sub-classes create * an instance. */ void _prop_object_init(struct _prop_object *po, const struct _prop_object_type *pot) { po->po_type = pot; po->po_refcnt = 1; } /* * _prop_object_fini -- * Finalize an object. Called when sub-classes destroy * an instance. */ /*ARGSUSED*/ void _prop_object_fini(struct _prop_object *po _PROP_ARG_UNUSED) { /* Nothing to do, currently. */ } /* * _prop_object_externalize_start_tag -- * Append an XML-style start tag to the externalize buffer. */ bool _prop_object_externalize_start_tag( struct _prop_object_externalize_context *ctx, const char *tag) { unsigned int i; for (i = 0; i < ctx->poec_depth; i++) { if (_prop_object_externalize_append_char(ctx, '\t') == false) return (false); } if (_prop_object_externalize_append_char(ctx, '<') == false || _prop_object_externalize_append_cstring(ctx, tag) == false || _prop_object_externalize_append_char(ctx, '>') == false) return (false); return (true); } /* * _prop_object_externalize_end_tag -- * Append an XML-style end tag to the externalize buffer. */ bool _prop_object_externalize_end_tag( struct _prop_object_externalize_context *ctx, const char *tag) { if (_prop_object_externalize_append_char(ctx, '<') == false || _prop_object_externalize_append_char(ctx, '/') == false || _prop_object_externalize_append_cstring(ctx, tag) == false || _prop_object_externalize_append_char(ctx, '>') == false || _prop_object_externalize_append_char(ctx, '\n') == false) return (false); return (true); } /* * _prop_object_externalize_empty_tag -- * Append an XML-style empty tag to the externalize buffer. */ bool _prop_object_externalize_empty_tag( struct _prop_object_externalize_context *ctx, const char *tag) { unsigned int i; for (i = 0; i < ctx->poec_depth; i++) { if (_prop_object_externalize_append_char(ctx, '\t') == false) return (false); } if (_prop_object_externalize_append_char(ctx, '<') == false || _prop_object_externalize_append_cstring(ctx, tag) == false || _prop_object_externalize_append_char(ctx, '/') == false || _prop_object_externalize_append_char(ctx, '>') == false || _prop_object_externalize_append_char(ctx, '\n') == false) return (false); return (true); } /* * _prop_object_externalize_append_cstring -- * Append a C string to the externalize buffer. */ bool _prop_object_externalize_append_cstring( struct _prop_object_externalize_context *ctx, const char *cp) { while (*cp != '\0') { if (_prop_object_externalize_append_char(ctx, (unsigned char) *cp) == false) return (false); cp++; } return (true); } /* * _prop_object_externalize_append_encoded_cstring -- * Append an encoded C string to the externalize buffer. */ bool _prop_object_externalize_append_encoded_cstring( struct _prop_object_externalize_context *ctx, const char *cp) { while (*cp != '\0') { switch (*cp) { case '<': if (_prop_object_externalize_append_cstring(ctx, "<") == false) return (false); break; case '>': if (_prop_object_externalize_append_cstring(ctx, ">") == false) return (false); break; case '&': if (_prop_object_externalize_append_cstring(ctx, "&") == false) return (false); break; default: if (_prop_object_externalize_append_char(ctx, (unsigned char) *cp) == false) return (false); break; } cp++; } return (true); } #define BUF_EXPAND 256 /* * _prop_object_externalize_append_char -- * Append a single character to the externalize buffer. */ bool _prop_object_externalize_append_char( struct _prop_object_externalize_context *ctx, unsigned char c) { _PROP_ASSERT(ctx->poec_capacity != 0); _PROP_ASSERT(ctx->poec_buf != NULL); _PROP_ASSERT(ctx->poec_len <= ctx->poec_capacity); if (ctx->poec_len == ctx->poec_capacity) { char *cp = _PROP_REALLOC(ctx->poec_buf, ctx->poec_capacity + BUF_EXPAND, M_TEMP); if (cp == NULL) return (false); ctx->poec_capacity = ctx->poec_capacity + BUF_EXPAND; ctx->poec_buf = cp; } ctx->poec_buf[ctx->poec_len++] = c; return (true); } /* * _prop_object_externalize_header -- * Append the standard XML header to the externalize buffer. */ bool _prop_object_externalize_header(struct _prop_object_externalize_context *ctx) { static const char _plist_xml_header[] = "<?xml version=\"1.0\" encoding=\"UTF-8\"?>\n" "<!DOCTYPE plist PUBLIC \"-//Apple Computer//DTD PLIST 1.0//EN\" \"http://www.apple.com/DTDs/PropertyList-1.0.dtd\">\n"; if (_prop_object_externalize_append_cstring(ctx, _plist_xml_header) == false || _prop_object_externalize_start_tag(ctx, "plist version=\"1.0\"") == false || _prop_object_externalize_append_char(ctx, '\n') == false) return (false); return (true); } /* * _prop_object_externalize_footer -- * Append the standard XML footer to the externalize buffer. This * also NUL-terminates the buffer. */ bool _prop_object_externalize_footer(struct _prop_object_externalize_context *ctx) { if (_prop_object_externalize_end_tag(ctx, "plist") == false || _prop_object_externalize_append_char(ctx, '\0') == false) return (false); return (true); } /* * _prop_object_externalize_context_alloc -- * Allocate an externalize context. */ struct _prop_object_externalize_context * _prop_object_externalize_context_alloc(void) { struct _prop_object_externalize_context *ctx; ctx = _PROP_MALLOC(sizeof(*ctx), M_TEMP); if (ctx != NULL) { ctx->poec_buf = _PROP_MALLOC(BUF_EXPAND, M_TEMP); if (ctx->poec_buf == NULL) { _PROP_FREE(ctx, M_TEMP); return (NULL); } ctx->poec_len = 0; ctx->poec_capacity = BUF_EXPAND; ctx->poec_depth = 0; } return (ctx); } /* * _prop_object_externalize_context_free -- * Free an externalize context. */ void _prop_object_externalize_context_free( struct _prop_object_externalize_context *ctx) { /* Buffer is always freed by the caller. */ _PROP_FREE(ctx, M_TEMP); } /* * _prop_object_internalize_skip_comment -- * Skip the body and end tag of a comment. */ static bool _prop_object_internalize_skip_comment( struct _prop_object_internalize_context *ctx) { const char *cp = ctx->poic_cp; while (!_PROP_EOF(*cp)) { if (cp[0] == '-' && cp[1] == '-' && cp[2] == '>') { ctx->poic_cp = cp + 3; return (true); } cp++; } return (false); /* ran out of buffer */ } /* * _prop_object_internalize_find_tag -- * Find the next tag in an XML stream. Optionally compare the found * tag to an expected tag name. State of the context is undefined * if this routine returns false. Upon success, the context points * to the first octet after the tag. */ bool _prop_object_internalize_find_tag(struct _prop_object_internalize_context *ctx, const char *tag, _prop_tag_type_t type) { const char *cp; size_t taglen; if (tag != NULL) taglen = strlen(tag); else taglen = 0; start_over: cp = ctx->poic_cp; /* * Find the start of the tag. */ while (_PROP_ISSPACE(*cp)) cp++; if (_PROP_EOF(*cp)) return (false); if (*cp != '<') return (false); ctx->poic_tag_start = cp++; if (_PROP_EOF(*cp)) return (false); if (*cp == '!') { if (cp[1] != '-' || cp[2] != '-') return (false); /* * Comment block -- only allowed if we are allowed to * return a start tag. */ if (type == _PROP_TAG_TYPE_END) return (false); ctx->poic_cp = cp + 3; if (_prop_object_internalize_skip_comment(ctx) == false) return (false); goto start_over; } if (*cp == '/') { if (type != _PROP_TAG_TYPE_END && type != _PROP_TAG_TYPE_EITHER) return (false); cp++; if (_PROP_EOF(*cp)) return (false); ctx->poic_tag_type = _PROP_TAG_TYPE_END; } else { if (type != _PROP_TAG_TYPE_START && type != _PROP_TAG_TYPE_EITHER) return (false); ctx->poic_tag_type = _PROP_TAG_TYPE_START; } ctx->poic_tagname = cp; while (!_PROP_ISSPACE(*cp) && *cp != '/' && *cp != '>') { if (_PROP_EOF(*cp)) return (false); cp++; } ctx->poic_tagname_len = cp - ctx->poic_tagname; /* Make sure this is the tag we're looking for. */ if (tag != NULL && (taglen != ctx->poic_tagname_len || memcmp(tag, ctx->poic_tagname, taglen) != 0)) return (false); /* Check for empty tag. */ if (*cp == '/') { if (ctx->poic_tag_type != _PROP_TAG_TYPE_START) return(false); /* only valid on start tags */ ctx->poic_is_empty_element = true; cp++; if (_PROP_EOF(*cp) || *cp != '>') return (false); } else ctx->poic_is_empty_element = false; /* Easy case of no arguments. */ if (*cp == '>') { ctx->poic_tagattr = NULL; ctx->poic_tagattr_len = 0; ctx->poic_tagattrval = NULL; ctx->poic_tagattrval_len = 0; ctx->poic_cp = cp + 1; return (true); } _PROP_ASSERT(!_PROP_EOF(*cp)); cp++; if (_PROP_EOF(*cp)) return (false); while (_PROP_ISSPACE(*cp)) cp++; if (_PROP_EOF(*cp)) return (false); ctx->poic_tagattr = cp; while (!_PROP_ISSPACE(*cp) && *cp != '=') { if (_PROP_EOF(*cp)) return (false); cp++; } ctx->poic_tagattr_len = cp - ctx->poic_tagattr; cp++; if (*cp != '\"') return (false); cp++; if (_PROP_EOF(*cp)) return (false); ctx->poic_tagattrval = cp; while (*cp != '\"') { if (_PROP_EOF(*cp)) return (false); cp++; } ctx->poic_tagattrval_len = cp - ctx->poic_tagattrval; cp++; if (*cp != '>') return (false); ctx->poic_cp = cp + 1; return (true); } /* * _prop_object_internalize_decode_string -- * Decode an encoded string. */ bool _prop_object_internalize_decode_string( struct _prop_object_internalize_context *ctx, char *target, size_t targsize, size_t *sizep, const char **cpp) { const char *src; size_t tarindex; char c; tarindex = 0; src = ctx->poic_cp; for (;;) { if (_PROP_EOF(*src)) return (false); if (*src == '<') { break; } if ((c = *src) == '&') { if (src[1] == 'a' && src[2] == 'm' && src[3] == 'p' && src[4] == ';') { c = '&'; src += 5; } else if (src[1] == 'l' && src[2] == 't' && src[3] == ';') { c = '<'; src += 4; } else if (src[1] == 'g' && src[2] == 't' && src[3] == ';') { c = '>'; src += 4; } else if (src[1] == 'a' && src[2] == 'p' && src[3] == 'o' && src[4] == 's' && src[5] == ';') { c = '\''; src += 6; } else if (src[1] == 'q' && src[2] == 'u' && src[3] == 'o' && src[4] == 't' && src[5] == ';') { c = '\"'; src += 6; } else return (false); } else src++; if (target) { if (tarindex >= targsize) return (false); target[tarindex] = c; } tarindex++; } _PROP_ASSERT(*src == '<'); if (sizep != NULL) *sizep = tarindex; if (cpp != NULL) *cpp = src; return (true); } /* * _prop_object_internalize_match -- * Returns true if the two character streams match. */ bool _prop_object_internalize_match(const char *str1, size_t len1, const char *str2, size_t len2) { return (len1 == len2 && memcmp(str1, str2, len1) == 0); } #define INTERNALIZER(t, f) \ { t, sizeof(t) - 1, f } static const struct _prop_object_internalizer { const char *poi_tag; size_t poi_taglen; prop_object_internalizer_t poi_intern; } _prop_object_internalizer_table[] = { INTERNALIZER("array", _prop_array_internalize), INTERNALIZER("true", _prop_bool_internalize), INTERNALIZER("false", _prop_bool_internalize), INTERNALIZER("data", _prop_data_internalize), INTERNALIZER("dict", _prop_dictionary_internalize), INTERNALIZER("integer", _prop_number_internalize), INTERNALIZER("string", _prop_string_internalize), { 0, 0, NULL } }; #undef INTERNALIZER /* * _prop_object_internalize_by_tag -- * Determine the object type from the tag in the context and * internalize it. */ prop_object_t _prop_object_internalize_by_tag(struct _prop_object_internalize_context *ctx) { const struct _prop_object_internalizer *poi; prop_object_t obj, parent_obj; void *data, *iter; prop_object_internalizer_continue_t iter_func; struct _prop_stack stack; _prop_stack_init(&stack); match_start: for (poi = _prop_object_internalizer_table; poi->poi_tag != NULL; poi++) { if (_prop_object_internalize_match(ctx->poic_tagname, ctx->poic_tagname_len, poi->poi_tag, poi->poi_taglen)) break; } if ((poi == NULL) || (poi->poi_tag == NULL)) { while (_prop_stack_pop(&stack, &obj, &iter, &data, NULL)) { iter_func = (prop_object_internalizer_continue_t)iter; (*iter_func)(&stack, &obj, ctx, data, NULL); } return (NULL); } obj = NULL; if (!(*poi->poi_intern)(&stack, &obj, ctx)) goto match_start; parent_obj = obj; while (_prop_stack_pop(&stack, &parent_obj, &iter, &data, NULL)) { iter_func = (prop_object_internalizer_continue_t)iter; if (!(*iter_func)(&stack, &parent_obj, ctx, data, obj)) goto match_start; obj = parent_obj; } return (parent_obj); } prop_object_t _prop_generic_internalize(const char *xml, const char *master_tag) { prop_object_t obj = NULL; struct _prop_object_internalize_context *ctx; ctx = _prop_object_internalize_context_alloc(xml); if (ctx == NULL) return (NULL); /* We start with a <plist> tag. */ if (_prop_object_internalize_find_tag(ctx, "plist", _PROP_TAG_TYPE_START) == false) goto out; /* Plist elements cannot be empty. */ if (ctx->poic_is_empty_element) goto out; /* * We don't understand any plist attributes, but Apple XML * property lists often have a "version" attribute. If we * see that one, we simply ignore it. */ if (ctx->poic_tagattr != NULL && !_PROP_TAGATTR_MATCH(ctx, "version")) goto out; /* Next we expect to see opening master_tag. */ if (_prop_object_internalize_find_tag(ctx, master_tag, _PROP_TAG_TYPE_START) == false) goto out; obj = _prop_object_internalize_by_tag(ctx); if (obj == NULL) goto out; /* * We've advanced past the closing master_tag. * Now we want </plist>. */ if (_prop_object_internalize_find_tag(ctx, "plist", _PROP_TAG_TYPE_END) == false) { prop_object_release(obj); obj = NULL; } out: _prop_object_internalize_context_free(ctx); return (obj); } /* * _prop_object_internalize_context_alloc -- * Allocate an internalize context. */ struct _prop_object_internalize_context * _prop_object_internalize_context_alloc(const char *xml) { struct _prop_object_internalize_context *ctx; ctx = _PROP_MALLOC(sizeof(*ctx), M_TEMP); if (ctx == NULL) return (NULL); ctx->poic_xml = ctx->poic_cp = xml; /* * Skip any whitespace and XML preamble stuff that we don't * know about / care about. */ for (;;) { while (_PROP_ISSPACE(*xml)) xml++; if (_PROP_EOF(*xml) || *xml != '<') goto bad; #define MATCH(str) (strncmp(&xml[1], str, strlen(str)) == 0) /* * Skip over the XML preamble that Apple XML property * lists usually include at the top of the file. */ if (MATCH("?xml ") || MATCH("!DOCTYPE plist")) { while (*xml != '>' && !_PROP_EOF(*xml)) xml++; if (_PROP_EOF(*xml)) goto bad; xml++; /* advance past the '>' */ continue; } if (MATCH("<!--")) { ctx->poic_cp = xml + 4; if (_prop_object_internalize_skip_comment(ctx) == false) goto bad; xml = ctx->poic_cp; continue; } #undef MATCH /* * We don't think we should skip it, so let's hope we can * parse it. */ break; } ctx->poic_cp = xml; return (ctx); bad: _PROP_FREE(ctx, M_TEMP); return (NULL); } /* * _prop_object_internalize_context_free -- * Free an internalize context. */ void _prop_object_internalize_context_free( struct _prop_object_internalize_context *ctx) { _PROP_FREE(ctx, M_TEMP); } #if !defined(_KERNEL) && !defined(_STANDALONE) /* * _prop_object_externalize_file_dirname -- * dirname(3), basically. We have to roll our own because the * system dirname(3) isn't reentrant. */ static void _prop_object_externalize_file_dirname(const char *path, char *result) { const char *lastp; size_t len; /* * If `path' is a NULL pointer or points to an empty string, * return ".". */ if (path == NULL || *path == '\0') goto singledot; /* String trailing slashes, if any. */ lastp = path + strlen(path) - 1; while (lastp != path && *lastp == '/') lastp--; /* Terminate path at the last occurrence of '/'. */ do { if (*lastp == '/') { /* Strip trailing slashes, if any. */ while (lastp != path && *lastp == '/') lastp--; /* ...and copy the result into the result buffer. */ len = (lastp - path) + 1 /* last char */; if (len > (PATH_MAX - 1)) len = PATH_MAX - 1; memcpy(result, path, len); result[len] = '\0'; return; } } while (--lastp >= path); /* No /'s found, return ".". */ singledot: strcpy(result, "."); } /* * _prop_object_externalize_write_file -- * Write an externalized dictionary to the specified file. * The file is written atomically from the caller's perspective, * and the mode set to 0666 modified by the caller's umask. */ bool _prop_object_externalize_write_file(const char *fname, const char *xml, size_t len) { char tname[PATH_MAX]; int fd; int save_errno; mode_t myumask; if (len > SSIZE_MAX) { errno = EFBIG; return (false); } /* * Get the directory name where the file is to be written * and create the temporary file. */ _prop_object_externalize_file_dirname(fname, tname); #define PLISTTMP "/.plistXXXXXX" if (strlen(tname) + strlen(PLISTTMP) >= sizeof(tname)) { errno = ENAMETOOLONG; return (false); } strcat(tname, PLISTTMP); #undef PLISTTMP if ((fd = mkstemp(tname)) == -1) return (false); if (write(fd, xml, len) != (ssize_t)len) goto bad; if (fsync(fd) == -1) goto bad; myumask = umask(0); (void)umask(myumask); if (fchmod(fd, 0666 & ~myumask) == -1) goto bad; (void) close(fd); fd = -1; if (rename(tname, fname) == -1) goto bad; return (true); bad: save_errno = errno; if (fd != -1) (void) close(fd); (void) unlink(tname); errno = save_errno; return (false); } /* * _prop_object_internalize_map_file -- * Map a file for the purpose of internalizing it. */ struct _prop_object_internalize_mapped_file * _prop_object_internalize_map_file(const char *fname) { struct stat sb; struct _prop_object_internalize_mapped_file *mf; size_t pgsize = (size_t)sysconf(_SC_PAGESIZE); size_t pgmask = pgsize - 1; bool need_guard = false; int fd; mf = _PROP_MALLOC(sizeof(*mf), M_TEMP); if (mf == NULL) return (NULL); fd = open(fname, O_RDONLY, 0400); if (fd == -1) { _PROP_FREE(mf, M_TEMP); return (NULL); } if (fstat(fd, &sb) == -1) { (void) close(fd); _PROP_FREE(mf, M_TEMP); return (NULL); } mf->poimf_mapsize = ((size_t)sb.st_size + pgmask) & ~pgmask; if (mf->poimf_mapsize < (size_t)sb.st_size) { (void) close(fd); _PROP_FREE(mf, M_TEMP); return (NULL); } /* * If the file length is an integral number of pages, then we * need to map a guard page at the end in order to provide the * necessary NUL-termination of the buffer. */ if ((sb.st_size & pgmask) == 0) need_guard = true; mf->poimf_xml = mmap(NULL, need_guard ? mf->poimf_mapsize + pgsize : mf->poimf_mapsize, PROT_READ, MAP_FILE|MAP_SHARED, fd, (off_t)0); (void) close(fd); if (mf->poimf_xml == MAP_FAILED) { _PROP_FREE(mf, M_TEMP); return (NULL); } #ifdef POSIX_MADV_SEQUENTIAL (void) posix_madvise(mf->poimf_xml, mf->poimf_mapsize, POSIX_MADV_SEQUENTIAL); #endif if (need_guard) { if (mmap(mf->poimf_xml + mf->poimf_mapsize, pgsize, PROT_READ, MAP_ANON|MAP_PRIVATE|MAP_FIXED, -1, (off_t)0) == MAP_FAILED) { (void) munmap(mf->poimf_xml, mf->poimf_mapsize); _PROP_FREE(mf, M_TEMP); return (NULL); } mf->poimf_mapsize += pgsize; } return (mf); } /* * _prop_object_internalize_unmap_file -- * Unmap a file previously mapped for internalizing. */ void _prop_object_internalize_unmap_file( struct _prop_object_internalize_mapped_file *mf) { #ifdef POSIX_MADV_DONTNEED (void) posix_madvise(mf->poimf_xml, mf->poimf_mapsize, POSIX_MADV_DONTNEED); #endif (void) munmap(mf->poimf_xml, mf->poimf_mapsize); _PROP_FREE(mf, M_TEMP); } #endif /* !_KERNEL && !_STANDALONE */ /* * prop_object_retain -- * Increment the reference count on an object. */ void prop_object_retain(prop_object_t obj) { struct _prop_object *po = obj; uint32_t ncnt __unused; _PROP_ATOMIC_INC32_NV(&po->po_refcnt, ncnt); _PROP_ASSERT(ncnt != 0); } /* * prop_object_release_emergency * A direct free with prop_object_release failed. * Walk down the tree until a leaf is found and * free that. Do not recurse to avoid stack overflows. * * This is a slow edge condition, but necessary to * guarantee that an object can always be freed. */ static void prop_object_release_emergency(prop_object_t obj) { struct _prop_object *po; void (*unlock)(void); prop_object_t parent = NULL; uint32_t ocnt; for (;;) { po = obj; _PROP_ASSERT(obj); if (po->po_type->pot_lock != NULL) po->po_type->pot_lock(); /* Save pointerto unlock function */ unlock = po->po_type->pot_unlock; /* Dance a bit to make sure we always get the non-racy ocnt */ _PROP_ATOMIC_DEC32_NV(&po->po_refcnt, ocnt); ocnt++; _PROP_ASSERT(ocnt != 0); if (ocnt != 1) { if (unlock != NULL) unlock(); break; } _PROP_ASSERT(po->po_type); if ((po->po_type->pot_free)(NULL, &obj) == _PROP_OBJECT_FREE_DONE) { if (unlock != NULL) unlock(); break; } if (unlock != NULL) unlock(); parent = po; _PROP_ATOMIC_INC32(&po->po_refcnt); } _PROP_ASSERT(parent); /* One object was just freed. */ po = parent; (*po->po_type->pot_emergency_free)(parent); } /* * prop_object_release -- * Decrement the reference count on an object. * * Free the object if we are releasing the final * reference. */ void prop_object_release(prop_object_t obj) { struct _prop_object *po; struct _prop_stack stack; void (*unlock)(void); int ret; uint32_t ocnt; _prop_stack_init(&stack); do { do { po = obj; _PROP_ASSERT(obj); if (po->po_type->pot_lock != NULL) po->po_type->pot_lock(); /* Save pointer to object unlock function */ unlock = po->po_type->pot_unlock; _PROP_ATOMIC_DEC32_NV(&po->po_refcnt, ocnt); ocnt++; _PROP_ASSERT(ocnt != 0); if (ocnt != 1) { ret = 0; if (unlock != NULL) unlock(); break; } ret = (po->po_type->pot_free)(&stack, &obj); if (unlock != NULL) unlock(); if (ret == _PROP_OBJECT_FREE_DONE) break; _PROP_ATOMIC_INC32(&po->po_refcnt); } while (ret == _PROP_OBJECT_FREE_RECURSE); if (ret == _PROP_OBJECT_FREE_FAILED) prop_object_release_emergency(obj); } while (_prop_stack_pop(&stack, &obj, NULL, NULL, NULL)); } /* * prop_object_type -- * Return the type of an object. */ prop_type_t prop_object_type(prop_object_t obj) { struct _prop_object *po = obj; if (obj == NULL) return (PROP_TYPE_UNKNOWN); return (po->po_type->pot_type); } /* * prop_object_equals -- * Returns true if thw two objects are equivalent. */ bool prop_object_equals(prop_object_t obj1, prop_object_t obj2) { return (prop_object_equals_with_error(obj1, obj2, NULL)); } bool prop_object_equals_with_error(prop_object_t obj1, prop_object_t obj2, bool *error_flag) { struct _prop_object *po1; struct _prop_object *po2; void *stored_pointer1, *stored_pointer2; prop_object_t next_obj1, next_obj2; struct _prop_stack stack; _prop_object_equals_rv_t ret; _prop_stack_init(&stack); if (error_flag) *error_flag = false; start_subtree: stored_pointer1 = NULL; stored_pointer2 = NULL; po1 = obj1; po2 = obj2; if (po1->po_type != po2->po_type) return (false); continue_subtree: ret = (*po1->po_type->pot_equals)(obj1, obj2, &stored_pointer1, &stored_pointer2, &next_obj1, &next_obj2); if (ret == _PROP_OBJECT_EQUALS_FALSE) goto finish; if (ret == _PROP_OBJECT_EQUALS_TRUE) { if (!_prop_stack_pop(&stack, &obj1, &obj2, &stored_pointer1, &stored_pointer2)) return true; po1 = obj1; po2 = obj2; goto continue_subtree; } _PROP_ASSERT(ret == _PROP_OBJECT_EQUALS_RECURSE); if (!_prop_stack_push(&stack, obj1, obj2, stored_pointer1, stored_pointer2)) { if (error_flag) *error_flag = true; goto finish; } obj1 = next_obj1; obj2 = next_obj2; goto start_subtree; finish: while (_prop_stack_pop(&stack, &obj1, &obj2, NULL, NULL)) { po1 = obj1; (*po1->po_type->pot_equals_finish)(obj1, obj2); } return (false); } /* * prop_object_iterator_next -- * Return the next item during an iteration. */ prop_object_t prop_object_iterator_next(prop_object_iterator_t pi) { return ((*pi->pi_next_object)(pi)); } /* * prop_object_iterator_reset -- * Reset the iterator to the first object so as to restart * iteration. */ void prop_object_iterator_reset(prop_object_iterator_t pi) { (*pi->pi_reset)(pi); } /* * prop_object_iterator_release -- * Release the object iterator. */ void prop_object_iterator_release(prop_object_iterator_t pi) { prop_object_release(pi->pi_obj); _PROP_FREE(pi, M_TEMP); } |
| 965 964 128 129 129 128 131 131 131 2 61 61 60 61 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 197 198 199 200 201 202 203 204 205 206 207 208 209 210 211 212 213 214 215 216 217 218 219 220 221 222 223 224 225 226 227 228 229 230 231 232 233 234 235 236 237 238 239 240 241 242 243 244 245 246 247 248 249 250 251 252 253 254 255 256 257 258 259 260 261 262 263 264 265 266 267 268 269 270 271 272 273 274 275 276 277 278 279 280 281 282 283 284 285 286 287 288 289 | /* $NetBSD: ipi.c,v 1.30 2019/12/01 15:34:46 ad Exp $ */ /*- * Copyright (c) 2000, 2008, 2009, 2019 The NetBSD Foundation, Inc. * All rights reserved. * * This code is derived from software contributed to The NetBSD Foundation * by RedBack Networks Inc. * * Author: Bill Sommerfeld * * Redistribution and use in source and binary forms, with or without * modification, are permitted provided that the following conditions * are met: * 1. Redistributions of source code must retain the above copyright * notice, this list of conditions and the following disclaimer. * 2. Redistributions in binary form must reproduce the above copyright * notice, this list of conditions and the following disclaimer in the * documentation and/or other materials provided with the distribution. * * THIS SOFTWARE IS PROVIDED BY THE NETBSD FOUNDATION, INC. AND CONTRIBUTORS * ``AS IS'' AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED * TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR * PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE FOUNDATION OR CONTRIBUTORS * BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR * CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF * SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS * INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN * CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) * ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE * POSSIBILITY OF SUCH DAMAGE. */ #include <sys/cdefs.h> __KERNEL_RCSID(0, "$NetBSD: ipi.c,v 1.30 2019/12/01 15:34:46 ad Exp $"); #include "opt_mtrr.h" #include <sys/param.h> #include <sys/device.h> #include <sys/systm.h> #include <sys/atomic.h> #include <sys/intr.h> #include <sys/ipi.h> #include <sys/cpu.h> #include <sys/xcall.h> #ifdef MULTIPROCESSOR #include <machine/cpufunc.h> #include <machine/cpuvar.h> #include <machine/i82093var.h> #include <machine/i82489reg.h> #include <machine/i82489var.h> #include <machine/mtrr.h> #include <machine/gdt.h> #include "acpica.h" #include <x86/fpu.h> static void x86_ipi_ast(struct cpu_info *); static void x86_ipi_halt(struct cpu_info *); static void x86_ipi_kpreempt(struct cpu_info *); static void x86_ipi_xcall(struct cpu_info *); static void x86_ipi_generic(struct cpu_info *); #ifdef MTRR static void x86_ipi_reload_mtrr(struct cpu_info *); #else #define x86_ipi_reload_mtrr NULL #endif #if NACPICA > 0 void acpi_cpu_sleep(struct cpu_info *); #else #define acpi_cpu_sleep NULL #endif static void x86_ipi_synch_fpu(struct cpu_info *); void (* const ipifunc[X86_NIPI])(struct cpu_info *) = { x86_ipi_halt, /* X86_IPI_HALT */ x86_ipi_ast, /* X86_IPI_AST */ x86_ipi_generic, /* X86_IPI_GENERIC */ x86_ipi_synch_fpu, /* X86_IPI_SYNCH_FPU */ x86_ipi_reload_mtrr, /* X86_IPI_MTRR */ NULL, /* X86_IPI_GDT */ x86_ipi_xcall, /* X86_IPI_XCALL */ acpi_cpu_sleep, /* X86_IPI_ACPI_CPU_SLEEP */ x86_ipi_kpreempt /* X86_IPI_KPREEMPT */ }; /* * x86 IPI interface. */ int x86_send_ipi(struct cpu_info *ci, int ipimask) { uint32_t o, n; int ret = 0; /* Don't send IPI to CPU which isn't (yet) running. */ if (__predict_false((ci->ci_flags & CPUF_RUNNING) == 0)) return ENOENT; /* Set in new IPI bit, and capture previous state. */ for (o = 0;; o = n) { n = atomic_cas_32(&ci->ci_ipis, o, o | ipimask); if (__predict_true(o == n)) { break; } } /* If no IPI already pending, send one. */ if (o == 0) { ret = x86_ipi(LAPIC_IPI_VECTOR, ci->ci_cpuid, LAPIC_DLMODE_FIXED); if (ret != 0) { printf("ipi of %x from %s to %s failed\n", ipimask, device_xname(curcpu()->ci_dev), device_xname(ci->ci_dev)); } } return ret; } void x86_broadcast_ipi(int ipimask) { struct cpu_info *ci, *self = curcpu(); int count = 0; CPU_INFO_ITERATOR cii; for (CPU_INFO_FOREACH(cii, ci)) { if (ci == self) continue; if ((ci->ci_flags & CPUF_RUNNING) == 0) continue; atomic_or_32(&ci->ci_ipis, ipimask); count++; } if (!count) return; x86_ipi(LAPIC_IPI_VECTOR, LAPIC_DEST_ALLEXCL, LAPIC_DLMODE_FIXED); } void x86_ipi_handler(void) { struct cpu_info *ci = curcpu(); uint32_t pending; int bit; pending = atomic_swap_32(&ci->ci_ipis, 0); KDASSERT((pending >> X86_NIPI) == 0); while ((bit = ffs(pending)) != 0) { bit--; pending &= ~(1 << bit); ci->ci_ipi_events[bit].ev_count++; (*ipifunc[bit])(ci); } } /* * Common x86 IPI handlers. */ static void x86_ipi_halt(struct cpu_info *ci) { x86_disable_intr(); atomic_and_32(&ci->ci_flags, ~CPUF_RUNNING); for (;;) { x86_hlt(); } } static void x86_ipi_synch_fpu(struct cpu_info *ci) { panic("%s: impossible", __func__); } #ifdef MTRR static void x86_ipi_reload_mtrr(struct cpu_info *ci) { if (mtrr_funcs != NULL) { /* * mtrr_reload_cpu() is a macro in mtrr.h which picks * the appropriate function to use. */ mtrr_reload_cpu(ci); } } #endif static void x86_ipi_kpreempt(struct cpu_info *ci) { softint_trigger(1 << SIR_PREEMPT); } static void x86_ipi_ast(struct cpu_info *ci) { aston(ci->ci_onproc); } /* * MD support for xcall(9) interface. */ static void x86_ipi_xcall(struct cpu_info *ci) { xc_ipi_handler(); } static void x86_ipi_generic(struct cpu_info *ci) { ipi_cpu_handler(); } void xc_send_ipi(struct cpu_info *ci) { KASSERT(kpreempt_disabled()); KASSERT(curcpu() != ci); if (ci) { /* Unicast: remote CPU. */ x86_send_ipi(ci, X86_IPI_XCALL); } else { /* Broadcast: all, but local CPU (caller will handle it). */ x86_broadcast_ipi(X86_IPI_XCALL); } } void cpu_ipi(struct cpu_info *ci) { KASSERT(kpreempt_disabled()); KASSERT(curcpu() != ci); if (ci) { /* Unicast: remote CPU. */ x86_send_ipi(ci, X86_IPI_GENERIC); } else { /* Broadcast: all, but local CPU (caller will handle it). */ x86_broadcast_ipi(X86_IPI_GENERIC); } } #else int x86_send_ipi(struct cpu_info *ci, int ipimask) { return 0; } void x86_broadcast_ipi(int ipimask) { } void cpu_ipi(struct cpu_info *ci) { } #endif |
| 58 58 58 57 55 55 12 12 20 15 8 8 8 8 8 8 20 20 8 8 23 23 23 15 20 23 23 12 12 8 15 23 23 23 23 23 23 23 23 23 23 12 23 23 20 20 54 50 20 51 51 51 53 54 54 21 21 21 24 24 24 24 24 16 16 21 16 19 21 21 53 53 54 54 55 54 55 54 54 46 23 34 19 17 14 14 23 23 23 23 12 23 23 23 23 22 19 18 2 12 12 47 46 43 57 59 59 58 15 58 57 21 15 15 14 14 15 15 15 15 6 14 6 15 15 15 6 8 6 15 15 6 6 15 6 6 6 6 6 51 51 51 54 54 54 51 51 51 59 57 82 62 1086 1077 9 2 7 7 7 5 5 46 11 11 47 47 11 11 11 11 2 18 16 2 2 18 11 7 7 18 15 7 11 18 18 18 18 16 2 2 6 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 197 198 199 200 201 202 203 204 205 206 207 208 209 210 211 212 213 214 215 216 217 218 219 220 221 222 223 224 225 226 227 228 229 230 231 232 233 234 235 236 237 238 239 240 241 242 243 244 245 246 247 248 249 250 251 252 253 254 255 256 257 258 259 260 261 262 263 264 265 266 267 268 269 270 271 272 273 274 275 276 277 278 279 280 281 282 283 284 285 286 287 288 289 290 291 292 293 294 295 296 297 298 299 300 301 302 303 304 305 306 307 308 309 310 311 312 313 314 315 316 317 318 319 320 321 322 323 324 325 326 327 328 329 330 331 332 333 334 335 336 337 338 339 340 341 342 343 344 345 346 347 348 349 350 351 352 353 354 355 356 357 358 359 360 361 362 363 364 365 366 367 368 369 370 371 372 373 374 375 376 377 378 379 380 381 382 383 384 385 386 387 388 389 390 391 392 393 394 395 396 397 398 399 400 401 402 403 404 405 406 407 408 409 410 411 412 413 414 415 416 417 418 419 420 421 422 423 424 425 426 427 428 429 430 431 432 433 434 435 436 437 438 439 440 441 442 443 444 445 446 447 448 449 450 451 452 453 454 455 456 457 458 459 460 461 462 463 464 465 466 467 468 469 470 471 472 473 474 475 476 477 478 479 480 481 482 483 484 485 486 487 488 489 490 491 492 493 494 495 496 497 498 499 500 501 502 503 504 505 506 507 508 509 510 511 512 513 514 515 516 517 518 519 520 521 522 523 524 525 526 527 528 529 530 531 532 533 534 535 536 537 538 539 540 541 542 543 544 545 546 547 548 549 550 551 552 553 554 555 556 557 558 559 560 561 562 563 564 565 566 567 568 569 570 571 572 573 574 575 576 577 578 579 580 581 582 583 584 585 586 587 588 589 590 591 592 593 594 595 596 597 598 599 600 601 602 603 604 605 606 607 608 609 610 611 612 613 614 615 616 617 618 619 620 621 622 623 624 625 626 627 628 629 630 631 632 633 634 635 636 637 638 639 640 641 642 643 644 645 646 647 648 649 650 651 652 653 654 655 656 657 658 659 660 661 662 663 664 665 666 667 668 669 670 671 672 673 674 675 676 677 678 679 680 681 682 683 684 685 686 687 688 689 690 691 692 693 694 695 696 697 698 699 700 701 702 703 704 705 706 707 708 709 710 711 712 713 714 715 716 717 718 719 720 721 722 723 724 725 726 727 728 729 730 731 732 733 734 735 736 737 738 739 740 741 742 743 744 745 746 747 748 749 750 751 752 753 754 755 756 757 758 759 760 761 762 763 764 765 766 767 768 769 770 771 772 773 774 775 776 777 778 779 780 781 782 783 784 785 786 787 788 789 790 791 792 793 794 795 796 797 798 799 800 801 802 803 804 805 806 807 808 809 810 811 812 813 814 815 816 817 818 819 820 821 822 823 824 825 826 827 828 829 830 831 832 833 834 835 836 837 838 839 840 841 842 843 844 845 846 847 848 849 850 851 852 853 854 855 856 857 858 859 860 861 862 863 864 865 866 867 868 869 870 871 872 873 874 875 876 877 878 879 880 881 882 883 884 885 886 887 888 889 890 891 892 893 894 895 896 897 898 899 900 901 902 903 904 905 906 907 908 909 910 911 912 913 914 915 916 917 918 919 920 921 922 923 924 925 926 927 928 929 930 931 932 933 934 935 936 937 938 939 940 941 942 943 944 945 946 947 948 949 950 951 952 953 954 955 956 957 958 959 960 961 962 963 964 965 966 967 968 969 970 971 972 973 974 975 976 977 978 979 980 981 982 983 984 985 986 987 988 989 990 991 992 993 994 995 996 997 998 999 1000 1001 1002 1003 1004 1005 1006 1007 1008 1009 1010 1011 1012 1013 1014 1015 1016 1017 1018 1019 1020 1021 1022 1023 1024 1025 1026 1027 1028 1029 1030 1031 1032 1033 1034 1035 1036 1037 1038 1039 1040 1041 1042 1043 1044 1045 1046 1047 1048 1049 1050 1051 1052 1053 1054 1055 1056 1057 1058 1059 1060 1061 1062 1063 1064 1065 1066 1067 1068 1069 1070 1071 1072 1073 1074 1075 1076 1077 1078 1079 1080 1081 1082 1083 1084 1085 1086 1087 1088 1089 1090 1091 1092 1093 1094 1095 1096 1097 1098 1099 1100 1101 1102 1103 1104 1105 1106 1107 1108 1109 1110 1111 1112 1113 1114 1115 1116 1117 1118 1119 1120 1121 1122 1123 1124 1125 1126 1127 1128 1129 1130 1131 1132 1133 1134 1135 1136 1137 1138 1139 1140 1141 1142 1143 1144 1145 1146 1147 1148 1149 1150 1151 1152 1153 1154 1155 1156 1157 1158 1159 1160 1161 1162 1163 1164 1165 1166 1167 1168 1169 1170 1171 1172 1173 1174 1175 1176 1177 1178 1179 1180 1181 1182 1183 1184 1185 1186 1187 1188 1189 1190 1191 1192 1193 1194 1195 1196 1197 1198 1199 1200 1201 1202 1203 1204 1205 1206 1207 1208 1209 1210 1211 1212 1213 1214 1215 1216 1217 1218 1219 1220 1221 1222 1223 1224 1225 1226 1227 1228 1229 1230 1231 1232 1233 1234 1235 1236 1237 1238 1239 1240 1241 1242 1243 1244 1245 1246 1247 1248 1249 1250 1251 1252 1253 1254 1255 1256 1257 1258 1259 1260 1261 1262 1263 1264 1265 1266 1267 1268 1269 1270 1271 1272 1273 1274 1275 1276 1277 1278 1279 1280 1281 1282 1283 1284 1285 1286 1287 1288 1289 1290 1291 1292 1293 1294 1295 1296 1297 1298 1299 1300 1301 1302 1303 1304 1305 1306 1307 1308 1309 1310 1311 1312 1313 1314 1315 1316 1317 1318 1319 1320 1321 1322 1323 1324 1325 1326 1327 1328 1329 1330 1331 1332 1333 1334 1335 1336 1337 1338 1339 1340 1341 1342 1343 1344 1345 1346 1347 1348 1349 1350 1351 1352 1353 1354 1355 1356 1357 1358 1359 1360 1361 1362 1363 1364 1365 1366 1367 1368 1369 1370 1371 1372 1373 1374 1375 1376 1377 1378 1379 1380 1381 1382 1383 1384 1385 1386 1387 1388 1389 1390 1391 1392 1393 1394 1395 1396 1397 1398 1399 1400 1401 1402 1403 1404 1405 1406 1407 1408 1409 1410 1411 1412 1413 1414 1415 1416 1417 1418 1419 1420 1421 1422 1423 1424 1425 1426 1427 1428 1429 1430 1431 1432 1433 1434 1435 1436 1437 1438 1439 1440 1441 1442 1443 1444 1445 1446 1447 1448 1449 1450 1451 1452 1453 1454 1455 1456 1457 1458 1459 1460 1461 1462 1463 1464 1465 1466 1467 1468 1469 1470 1471 1472 1473 1474 1475 1476 1477 1478 1479 1480 1481 1482 1483 1484 1485 1486 1487 1488 1489 1490 1491 1492 1493 1494 1495 1496 1497 1498 1499 1500 1501 1502 1503 1504 1505 1506 1507 1508 1509 1510 1511 1512 1513 1514 1515 1516 1517 1518 1519 1520 1521 1522 1523 1524 1525 1526 1527 1528 1529 1530 1531 1532 1533 1534 1535 1536 1537 1538 1539 1540 1541 1542 1543 1544 1545 1546 1547 1548 1549 1550 1551 1552 1553 1554 1555 1556 1557 1558 1559 1560 1561 1562 1563 1564 1565 1566 1567 1568 1569 1570 1571 1572 1573 1574 1575 1576 1577 1578 1579 1580 1581 1582 1583 1584 1585 1586 1587 1588 1589 1590 1591 1592 1593 1594 1595 1596 1597 1598 1599 1600 1601 1602 1603 1604 1605 1606 1607 1608 1609 1610 1611 1612 1613 1614 1615 1616 1617 1618 1619 1620 1621 1622 1623 1624 1625 1626 1627 1628 1629 1630 1631 1632 1633 1634 1635 1636 1637 1638 1639 1640 1641 1642 1643 1644 1645 1646 1647 1648 1649 1650 1651 1652 1653 1654 1655 1656 1657 1658 1659 1660 1661 1662 1663 1664 1665 1666 1667 1668 1669 1670 1671 1672 1673 1674 1675 1676 1677 1678 1679 1680 1681 1682 1683 1684 1685 1686 1687 1688 1689 1690 1691 1692 1693 1694 1695 1696 1697 1698 1699 1700 1701 1702 1703 1704 1705 1706 1707 1708 1709 1710 1711 1712 1713 1714 1715 1716 1717 1718 1719 1720 1721 1722 1723 1724 1725 1726 1727 1728 1729 1730 1731 1732 1733 1734 1735 1736 1737 1738 1739 1740 1741 1742 1743 1744 1745 1746 1747 1748 1749 1750 1751 1752 1753 1754 1755 1756 1757 1758 1759 1760 1761 1762 1763 1764 1765 1766 1767 1768 1769 1770 1771 1772 1773 1774 1775 1776 1777 1778 1779 1780 1781 1782 1783 1784 1785 1786 1787 1788 1789 1790 1791 1792 1793 1794 1795 1796 1797 1798 1799 1800 1801 1802 1803 1804 1805 1806 1807 1808 1809 1810 1811 1812 1813 1814 1815 1816 1817 1818 1819 1820 1821 1822 1823 1824 1825 1826 1827 1828 1829 1830 1831 1832 1833 1834 1835 1836 1837 1838 1839 1840 1841 1842 1843 1844 1845 1846 1847 1848 1849 1850 1851 1852 1853 1854 1855 1856 1857 1858 1859 1860 1861 1862 1863 1864 1865 1866 1867 1868 1869 1870 1871 1872 1873 1874 1875 1876 1877 1878 1879 1880 1881 1882 1883 1884 1885 1886 1887 1888 1889 1890 1891 1892 1893 1894 1895 1896 1897 1898 1899 1900 1901 1902 1903 1904 1905 1906 1907 1908 1909 1910 1911 1912 1913 1914 1915 1916 1917 1918 1919 1920 1921 1922 1923 1924 1925 1926 1927 1928 1929 1930 1931 1932 1933 1934 1935 1936 1937 1938 1939 1940 1941 1942 1943 1944 1945 1946 1947 1948 1949 1950 1951 1952 1953 1954 1955 1956 1957 1958 1959 1960 1961 1962 1963 1964 1965 1966 1967 1968 1969 1970 1971 1972 1973 1974 1975 1976 1977 1978 1979 1980 1981 1982 1983 1984 1985 1986 1987 1988 1989 1990 1991 1992 1993 1994 1995 1996 1997 1998 1999 2000 2001 2002 2003 2004 2005 2006 2007 2008 2009 2010 2011 2012 2013 2014 2015 2016 2017 2018 2019 2020 2021 2022 2023 2024 2025 2026 2027 2028 2029 2030 2031 2032 2033 2034 2035 2036 2037 2038 2039 2040 2041 2042 2043 2044 2045 2046 2047 2048 2049 2050 2051 2052 2053 2054 2055 2056 2057 2058 2059 2060 2061 2062 2063 2064 2065 2066 2067 2068 2069 2070 2071 2072 2073 2074 2075 2076 2077 2078 2079 2080 2081 2082 2083 2084 2085 2086 2087 2088 2089 2090 2091 2092 2093 2094 2095 2096 2097 2098 2099 2100 2101 2102 2103 2104 2105 2106 2107 2108 2109 2110 2111 2112 2113 2114 2115 2116 2117 2118 2119 2120 2121 2122 2123 2124 2125 2126 2127 2128 2129 2130 2131 2132 2133 2134 2135 2136 2137 2138 2139 2140 2141 2142 2143 2144 2145 2146 2147 2148 2149 2150 2151 2152 2153 2154 2155 2156 2157 2158 2159 2160 2161 2162 2163 2164 2165 2166 2167 2168 2169 2170 2171 2172 2173 2174 2175 2176 2177 2178 2179 2180 2181 2182 2183 2184 2185 2186 2187 2188 2189 2190 2191 2192 2193 2194 2195 2196 2197 2198 2199 2200 2201 2202 2203 2204 2205 2206 2207 2208 2209 2210 2211 2212 2213 2214 2215 2216 2217 2218 2219 2220 2221 2222 2223 2224 2225 2226 2227 2228 2229 2230 2231 2232 2233 2234 2235 2236 2237 2238 2239 2240 2241 2242 2243 2244 2245 2246 2247 2248 2249 2250 2251 2252 2253 2254 2255 2256 2257 2258 2259 2260 2261 2262 2263 2264 2265 2266 2267 2268 2269 2270 2271 2272 2273 2274 2275 2276 2277 2278 2279 2280 2281 2282 2283 2284 2285 2286 2287 2288 2289 2290 2291 2292 2293 2294 2295 2296 2297 2298 2299 2300 2301 2302 2303 2304 2305 2306 2307 2308 2309 2310 2311 2312 2313 2314 2315 2316 2317 2318 2319 2320 2321 2322 2323 2324 2325 2326 2327 2328 2329 2330 2331 2332 2333 2334 2335 2336 2337 2338 2339 2340 2341 2342 2343 2344 2345 2346 2347 2348 2349 2350 2351 2352 2353 2354 2355 2356 2357 2358 2359 2360 2361 2362 2363 2364 2365 2366 2367 2368 2369 2370 2371 2372 2373 2374 2375 2376 2377 2378 2379 2380 2381 2382 2383 2384 2385 2386 2387 2388 2389 2390 2391 2392 2393 2394 2395 2396 2397 2398 2399 2400 2401 2402 2403 2404 2405 2406 2407 2408 2409 2410 2411 2412 2413 2414 2415 2416 2417 2418 2419 2420 2421 2422 2423 2424 2425 2426 2427 2428 2429 2430 2431 2432 2433 2434 2435 2436 2437 2438 2439 2440 2441 2442 2443 2444 2445 2446 2447 2448 2449 2450 2451 2452 2453 2454 2455 2456 2457 2458 2459 2460 2461 2462 2463 2464 2465 2466 2467 2468 2469 2470 2471 2472 2473 2474 2475 2476 2477 2478 2479 2480 2481 2482 2483 2484 2485 2486 2487 2488 2489 2490 2491 2492 2493 2494 2495 2496 2497 2498 2499 2500 2501 2502 2503 2504 2505 2506 2507 2508 2509 2510 2511 2512 2513 2514 2515 2516 2517 2518 2519 2520 2521 2522 2523 2524 2525 2526 2527 2528 2529 2530 2531 2532 2533 2534 2535 2536 2537 2538 2539 2540 2541 2542 2543 2544 2545 2546 2547 2548 2549 2550 2551 2552 2553 2554 2555 2556 2557 2558 2559 2560 2561 2562 2563 2564 2565 2566 2567 2568 2569 2570 2571 2572 2573 2574 2575 2576 2577 2578 2579 2580 2581 2582 2583 2584 2585 2586 2587 2588 2589 2590 2591 2592 2593 2594 2595 2596 2597 2598 2599 2600 2601 2602 2603 2604 2605 2606 2607 2608 2609 2610 2611 2612 2613 2614 2615 2616 2617 2618 2619 2620 2621 2622 2623 2624 2625 2626 2627 2628 2629 2630 2631 2632 2633 2634 2635 2636 2637 2638 2639 2640 2641 2642 2643 2644 2645 2646 2647 2648 2649 2650 2651 2652 2653 2654 2655 2656 2657 2658 2659 2660 2661 2662 2663 2664 2665 2666 2667 2668 2669 2670 2671 2672 2673 2674 2675 2676 2677 2678 2679 2680 2681 2682 2683 2684 2685 2686 2687 2688 2689 2690 2691 2692 2693 2694 2695 2696 2697 2698 2699 2700 2701 2702 2703 2704 2705 2706 2707 2708 2709 2710 2711 2712 2713 2714 2715 2716 2717 2718 2719 2720 2721 2722 2723 2724 2725 2726 2727 2728 2729 2730 2731 2732 2733 2734 2735 2736 2737 2738 2739 2740 2741 2742 2743 2744 2745 2746 2747 2748 2749 2750 2751 2752 2753 2754 2755 2756 2757 2758 2759 2760 2761 2762 2763 2764 2765 2766 2767 2768 2769 2770 2771 2772 2773 2774 2775 2776 2777 2778 2779 2780 2781 2782 2783 2784 2785 2786 2787 2788 2789 2790 2791 2792 2793 2794 2795 2796 2797 2798 2799 2800 2801 2802 2803 2804 2805 2806 2807 2808 2809 2810 2811 2812 2813 2814 2815 2816 2817 2818 2819 2820 2821 2822 2823 2824 2825 2826 2827 2828 2829 2830 2831 2832 2833 2834 2835 2836 2837 2838 2839 2840 2841 2842 2843 2844 2845 2846 2847 2848 2849 2850 2851 2852 2853 2854 2855 2856 2857 2858 2859 2860 2861 2862 2863 2864 2865 2866 2867 2868 2869 2870 2871 2872 2873 2874 2875 2876 2877 2878 2879 2880 2881 2882 2883 2884 2885 2886 2887 2888 2889 2890 2891 2892 2893 2894 2895 2896 2897 2898 2899 2900 2901 2902 2903 2904 2905 2906 2907 2908 2909 2910 2911 2912 2913 2914 2915 2916 2917 2918 2919 2920 2921 2922 2923 2924 2925 2926 2927 2928 2929 2930 2931 2932 2933 2934 2935 2936 2937 2938 2939 2940 2941 2942 2943 2944 2945 2946 2947 2948 2949 2950 2951 2952 2953 2954 2955 2956 2957 2958 2959 2960 2961 2962 2963 2964 2965 2966 2967 2968 2969 2970 2971 2972 2973 2974 2975 2976 2977 2978 2979 2980 2981 2982 2983 2984 2985 2986 2987 2988 2989 2990 2991 2992 2993 2994 2995 2996 2997 2998 2999 3000 3001 3002 3003 3004 3005 3006 3007 3008 3009 3010 3011 3012 3013 3014 3015 3016 3017 3018 3019 3020 3021 3022 3023 3024 3025 3026 3027 3028 3029 3030 3031 3032 3033 3034 3035 3036 3037 3038 3039 3040 3041 3042 3043 3044 3045 3046 3047 3048 3049 3050 3051 3052 3053 3054 3055 3056 3057 3058 3059 3060 3061 3062 3063 3064 3065 3066 3067 3068 3069 3070 3071 3072 3073 3074 3075 3076 3077 3078 3079 3080 3081 3082 3083 3084 3085 3086 3087 3088 3089 3090 3091 3092 3093 3094 3095 3096 3097 3098 3099 3100 3101 3102 3103 3104 3105 3106 3107 3108 3109 3110 3111 3112 3113 3114 3115 3116 3117 3118 3119 3120 3121 3122 3123 3124 3125 3126 3127 3128 3129 3130 3131 3132 3133 3134 3135 3136 3137 3138 3139 3140 3141 3142 3143 3144 3145 3146 3147 3148 3149 3150 3151 3152 3153 3154 3155 3156 3157 3158 3159 3160 3161 3162 3163 3164 3165 3166 3167 3168 3169 3170 3171 3172 3173 3174 3175 3176 3177 3178 3179 3180 3181 3182 3183 3184 3185 3186 3187 3188 3189 3190 3191 3192 3193 3194 3195 3196 3197 3198 3199 3200 3201 3202 3203 3204 3205 3206 3207 3208 3209 3210 3211 3212 3213 3214 3215 3216 3217 3218 3219 3220 3221 3222 3223 3224 3225 3226 3227 3228 3229 3230 3231 3232 3233 3234 3235 3236 3237 3238 3239 3240 3241 3242 3243 3244 3245 3246 3247 3248 3249 3250 3251 3252 3253 3254 3255 3256 3257 3258 3259 3260 3261 3262 3263 3264 3265 3266 3267 3268 3269 3270 3271 3272 3273 3274 3275 3276 3277 3278 3279 3280 3281 3282 3283 3284 3285 3286 3287 3288 3289 3290 3291 3292 3293 3294 3295 3296 3297 3298 3299 3300 3301 3302 3303 3304 3305 3306 3307 3308 3309 3310 3311 3312 3313 3314 3315 3316 3317 3318 3319 3320 3321 3322 3323 3324 3325 3326 3327 3328 3329 3330 3331 3332 3333 3334 3335 3336 3337 3338 3339 3340 3341 3342 3343 3344 3345 3346 3347 3348 3349 3350 3351 3352 3353 3354 3355 3356 3357 3358 3359 3360 3361 3362 3363 3364 3365 3366 3367 3368 3369 3370 3371 3372 3373 3374 3375 3376 3377 3378 3379 3380 3381 3382 3383 3384 3385 3386 3387 3388 3389 3390 3391 3392 3393 3394 3395 3396 3397 3398 3399 3400 3401 3402 3403 3404 3405 3406 3407 3408 3409 3410 3411 3412 3413 3414 3415 3416 3417 3418 3419 3420 3421 3422 3423 3424 3425 3426 3427 3428 3429 3430 3431 3432 3433 3434 3435 3436 3437 3438 3439 3440 3441 3442 3443 3444 3445 3446 3447 3448 3449 3450 3451 3452 3453 3454 3455 3456 3457 3458 3459 3460 3461 3462 3463 3464 3465 3466 3467 3468 3469 3470 3471 3472 3473 3474 3475 3476 3477 3478 3479 3480 3481 3482 3483 3484 3485 3486 3487 3488 3489 3490 3491 3492 3493 3494 3495 3496 3497 3498 3499 3500 3501 3502 3503 3504 3505 3506 3507 3508 3509 3510 3511 3512 3513 3514 3515 3516 3517 3518 3519 3520 3521 3522 3523 3524 3525 3526 3527 3528 3529 3530 3531 3532 3533 3534 3535 3536 3537 3538 3539 3540 3541 3542 3543 3544 3545 3546 3547 3548 3549 3550 3551 3552 3553 3554 3555 3556 3557 3558 3559 3560 3561 3562 3563 3564 3565 3566 3567 3568 3569 3570 3571 3572 3573 3574 3575 3576 3577 3578 3579 3580 3581 3582 3583 3584 3585 3586 3587 3588 3589 3590 3591 3592 3593 3594 3595 3596 3597 3598 3599 3600 3601 3602 3603 3604 3605 3606 3607 3608 3609 3610 3611 3612 3613 3614 3615 3616 3617 3618 3619 3620 3621 3622 3623 3624 3625 3626 3627 3628 3629 3630 3631 3632 3633 3634 3635 3636 3637 3638 3639 3640 3641 3642 3643 3644 3645 3646 3647 3648 3649 3650 3651 3652 3653 3654 3655 3656 3657 3658 3659 3660 3661 3662 3663 3664 3665 3666 3667 3668 3669 3670 3671 3672 3673 3674 3675 3676 3677 3678 3679 3680 3681 3682 3683 3684 3685 3686 3687 3688 3689 3690 3691 3692 3693 3694 3695 3696 3697 3698 3699 3700 3701 3702 3703 3704 3705 3706 3707 3708 3709 3710 3711 3712 3713 3714 3715 3716 3717 3718 3719 3720 3721 3722 3723 3724 3725 3726 3727 3728 3729 3730 3731 3732 3733 3734 3735 3736 3737 3738 3739 3740 3741 | /* $NetBSD: subr_autoconf.c,v 1.302 2022/08/12 16:16:12 riastradh Exp $ */ /* * Copyright (c) 1996, 2000 Christopher G. Demetriou * All rights reserved. * * Redistribution and use in source and binary forms, with or without * modification, are permitted provided that the following conditions * are met: * 1. Redistributions of source code must retain the above copyright * notice, this list of conditions and the following disclaimer. * 2. Redistributions in binary form must reproduce the above copyright * notice, this list of conditions and the following disclaimer in the * documentation and/or other materials provided with the distribution. * 3. All advertising materials mentioning features or use of this software * must display the following acknowledgement: * This product includes software developed for the * NetBSD Project. See http://www.NetBSD.org/ for * information about NetBSD. * 4. The name of the author may not be used to endorse or promote products * derived from this software without specific prior written permission. * * THIS SOFTWARE IS PROVIDED BY THE AUTHOR ``AS IS'' AND ANY EXPRESS OR * IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES * OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. * IN NO EVENT SHALL THE AUTHOR BE LIABLE FOR ANY DIRECT, INDIRECT, * INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT * NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, * DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY * THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT * (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF * THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. * * --(license Id: LICENSE.proto,v 1.1 2000/06/13 21:40:26 cgd Exp )-- */ /* * Copyright (c) 1992, 1993 * The Regents of the University of California. All rights reserved. * * This software was developed by the Computer Systems Engineering group * at Lawrence Berkeley Laboratory under DARPA contract BG 91-66 and * contributed to Berkeley. * * All advertising materials mentioning features or use of this software * must display the following acknowledgement: * This product includes software developed by the University of * California, Lawrence Berkeley Laboratories. * * Redistribution and use in source and binary forms, with or without * modification, are permitted provided that the following conditions * are met: * 1. Redistributions of source code must retain the above copyright * notice, this list of conditions and the following disclaimer. * 2. Redistributions in binary form must reproduce the above copyright * notice, this list of conditions and the following disclaimer in the * documentation and/or other materials provided with the distribution. * 3. Neither the name of the University nor the names of its contributors * may be used to endorse or promote products derived from this software * without specific prior written permission. * * THIS SOFTWARE IS PROVIDED BY THE REGENTS AND CONTRIBUTORS ``AS IS'' AND * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE * ARE DISCLAIMED. IN NO EVENT SHALL THE REGENTS OR CONTRIBUTORS BE LIABLE * FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL * DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS * OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) * HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT * LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY * OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF * SUCH DAMAGE. * * from: Header: subr_autoconf.c,v 1.12 93/02/01 19:31:48 torek Exp (LBL) * * @(#)subr_autoconf.c 8.3 (Berkeley) 5/17/94 */ #include <sys/cdefs.h> __KERNEL_RCSID(0, "$NetBSD: subr_autoconf.c,v 1.302 2022/08/12 16:16:12 riastradh Exp $"); #ifdef _KERNEL_OPT #include "opt_ddb.h" #include "drvctl.h" #endif #include <sys/param.h> #include <sys/device.h> #include <sys/device_impl.h> #include <sys/disklabel.h> #include <sys/conf.h> #include <sys/kauth.h> #include <sys/kmem.h> #include <sys/systm.h> #include <sys/kernel.h> #include <sys/errno.h> #include <sys/proc.h> #include <sys/reboot.h> #include <sys/kthread.h> #include <sys/buf.h> #include <sys/dirent.h> #include <sys/mount.h> #include <sys/namei.h> #include <sys/unistd.h> #include <sys/fcntl.h> #include <sys/lockf.h> #include <sys/callout.h> #include <sys/devmon.h> #include <sys/cpu.h> #include <sys/sysctl.h> #include <sys/stdarg.h> #include <sys/localcount.h> #include <sys/disk.h> #include <sys/rndsource.h> #include <machine/limits.h> /* * Autoconfiguration subroutines. */ /* * Device autoconfiguration timings are mixed into the entropy pool. */ static krndsource_t rnd_autoconf_source; /* * ioconf.c exports exactly two names: cfdata and cfroots. All system * devices and drivers are found via these tables. */ extern struct cfdata cfdata[]; extern const short cfroots[]; /* * List of all cfdriver structures. We use this to detect duplicates * when other cfdrivers are loaded. */ struct cfdriverlist allcfdrivers = LIST_HEAD_INITIALIZER(&allcfdrivers); extern struct cfdriver * const cfdriver_list_initial[]; /* * Initial list of cfattach's. */ extern const struct cfattachinit cfattachinit[]; /* * List of cfdata tables. We always have one such list -- the one * built statically when the kernel was configured. */ struct cftablelist allcftables = TAILQ_HEAD_INITIALIZER(allcftables); static struct cftable initcftable; #define ROOT ((device_t)NULL) struct matchinfo { cfsubmatch_t fn; device_t parent; const int *locs; void *aux; struct cfdata *match; int pri; }; struct alldevs_foray { int af_s; struct devicelist af_garbage; }; /* * Internal version of the cfargs structure; all versions are * canonicalized to this. */ struct cfargs_internal { union { cfsubmatch_t submatch;/* submatch function (direct config) */ cfsearch_t search; /* search function (indirect config) */ }; const char * iattr; /* interface attribute */ const int * locators; /* locators array */ devhandle_t devhandle; /* devhandle_t (by value) */ }; static char *number(char *, int); static void mapply(struct matchinfo *, cfdata_t); static void config_devdelete(device_t); static void config_devunlink(device_t, struct devicelist *); static void config_makeroom(int, struct cfdriver *); static void config_devlink(device_t); static void config_alldevs_enter(struct alldevs_foray *); static void config_alldevs_exit(struct alldevs_foray *); static void config_add_attrib_dict(device_t); static device_t config_attach_internal(device_t, cfdata_t, void *, cfprint_t, const struct cfargs_internal *); static void config_collect_garbage(struct devicelist *); static void config_dump_garbage(struct devicelist *); static void pmflock_debug(device_t, const char *, int); static device_t deviter_next1(deviter_t *); static void deviter_reinit(deviter_t *); struct deferred_config { TAILQ_ENTRY(deferred_config) dc_queue; device_t dc_dev; void (*dc_func)(device_t); }; TAILQ_HEAD(deferred_config_head, deferred_config); static struct deferred_config_head deferred_config_queue = TAILQ_HEAD_INITIALIZER(deferred_config_queue); static struct deferred_config_head interrupt_config_queue = TAILQ_HEAD_INITIALIZER(interrupt_config_queue); static int interrupt_config_threads = 8; static struct deferred_config_head mountroot_config_queue = TAILQ_HEAD_INITIALIZER(mountroot_config_queue); static int mountroot_config_threads = 2; static lwp_t **mountroot_config_lwpids; static size_t mountroot_config_lwpids_size; bool root_is_mounted = false; static void config_process_deferred(struct deferred_config_head *, device_t); /* Hooks to finalize configuration once all real devices have been found. */ struct finalize_hook { TAILQ_ENTRY(finalize_hook) f_list; int (*f_func)(device_t); device_t f_dev; }; static TAILQ_HEAD(, finalize_hook) config_finalize_list = TAILQ_HEAD_INITIALIZER(config_finalize_list); static int config_finalize_done; /* list of all devices */ static struct devicelist alldevs = TAILQ_HEAD_INITIALIZER(alldevs); static kmutex_t alldevs_lock __cacheline_aligned; static devgen_t alldevs_gen = 1; static int alldevs_nread = 0; static int alldevs_nwrite = 0; static bool alldevs_garbage = false; static struct devicelist config_pending = TAILQ_HEAD_INITIALIZER(config_pending); static kmutex_t config_misc_lock; static kcondvar_t config_misc_cv; static bool detachall = false; #define STREQ(s1, s2) \ (*(s1) == *(s2) && strcmp((s1), (s2)) == 0) static bool config_initialized = false; /* config_init() has been called. */ static int config_do_twiddle; static callout_t config_twiddle_ch; static void sysctl_detach_setup(struct sysctllog **); int no_devmon_insert(const char *, prop_dictionary_t); int (*devmon_insert_vec)(const char *, prop_dictionary_t) = no_devmon_insert; typedef int (*cfdriver_fn)(struct cfdriver *); static int frob_cfdrivervec(struct cfdriver * const *cfdriverv, cfdriver_fn drv_do, cfdriver_fn drv_undo, const char *style, bool dopanic) { void (*pr)(const char *, ...) __printflike(1, 2) = dopanic ? panic : printf; int i, error = 0, e2 __diagused; for (i = 0; cfdriverv[i] != NULL; i++) { if ((error = drv_do(cfdriverv[i])) != 0) { pr("configure: `%s' driver %s failed: %d", cfdriverv[i]->cd_name, style, error); goto bad; } } KASSERT(error == 0); return 0; bad: printf("\n"); for (i--; i >= 0; i--) { e2 = drv_undo(cfdriverv[i]); KASSERT(e2 == 0); } return error; } typedef int (*cfattach_fn)(const char *, struct cfattach *); static int frob_cfattachvec(const struct cfattachinit *cfattachv, cfattach_fn att_do, cfattach_fn att_undo, const char *style, bool dopanic) { const struct cfattachinit *cfai = NULL; void (*pr)(const char *, ...) __printflike(1, 2) = dopanic ? panic : printf; int j = 0, error = 0, e2 __diagused; for (cfai = &cfattachv[0]; cfai->cfai_name != NULL; cfai++) { for (j = 0; cfai->cfai_list[j] != NULL; j++) { if ((error = att_do(cfai->cfai_name, cfai->cfai_list[j])) != 0) { pr("configure: attachment `%s' " "of `%s' driver %s failed: %d", cfai->cfai_list[j]->ca_name, cfai->cfai_name, style, error); goto bad; } } } KASSERT(error == 0); return 0; bad: /* * Rollback in reverse order. dunno if super-important, but * do that anyway. Although the code looks a little like * someone did a little integration (in the math sense). */ printf("\n"); if (cfai) { bool last; for (last = false; last == false; ) { if (cfai == &cfattachv[0]) last = true; for (j--; j >= 0; j--) { e2 = att_undo(cfai->cfai_name, cfai->cfai_list[j]); KASSERT(e2 == 0); } if (!last) { cfai--; for (j = 0; cfai->cfai_list[j] != NULL; j++) ; } } } return error; } /* * Initialize the autoconfiguration data structures. Normally this * is done by configure(), but some platforms need to do this very * early (to e.g. initialize the console). */ void config_init(void) { KASSERT(config_initialized == false); mutex_init(&alldevs_lock, MUTEX_DEFAULT, IPL_VM); mutex_init(&config_misc_lock, MUTEX_DEFAULT, IPL_NONE); cv_init(&config_misc_cv, "cfgmisc"); callout_init(&config_twiddle_ch, CALLOUT_MPSAFE); frob_cfdrivervec(cfdriver_list_initial, config_cfdriver_attach, NULL, "bootstrap", true); frob_cfattachvec(cfattachinit, config_cfattach_attach, NULL, "bootstrap", true); initcftable.ct_cfdata = cfdata; TAILQ_INSERT_TAIL(&allcftables, &initcftable, ct_list); rnd_attach_source(&rnd_autoconf_source, "autoconf", RND_TYPE_UNKNOWN, RND_FLAG_COLLECT_TIME); config_initialized = true; } /* * Init or fini drivers and attachments. Either all or none * are processed (via rollback). It would be nice if this were * atomic to outside consumers, but with the current state of * locking ... */ int config_init_component(struct cfdriver * const *cfdriverv, const struct cfattachinit *cfattachv, struct cfdata *cfdatav) { int error; KERNEL_LOCK(1, NULL); if ((error = frob_cfdrivervec(cfdriverv, config_cfdriver_attach, config_cfdriver_detach, "init", false))!= 0) goto out; if ((error = frob_cfattachvec(cfattachv, config_cfattach_attach, config_cfattach_detach, "init", false)) != 0) { frob_cfdrivervec(cfdriverv, config_cfdriver_detach, NULL, "init rollback", true); goto out; } if ((error = config_cfdata_attach(cfdatav, 1)) != 0) { frob_cfattachvec(cfattachv, config_cfattach_detach, NULL, "init rollback", true); frob_cfdrivervec(cfdriverv, config_cfdriver_detach, NULL, "init rollback", true); goto out; } /* Success! */ error = 0; out: KERNEL_UNLOCK_ONE(NULL); return error; } int config_fini_component(struct cfdriver * const *cfdriverv, const struct cfattachinit *cfattachv, struct cfdata *cfdatav) { int error; KERNEL_LOCK(1, NULL); if ((error = config_cfdata_detach(cfdatav)) != 0) goto out; if ((error = frob_cfattachvec(cfattachv, config_cfattach_detach, config_cfattach_attach, "fini", false)) != 0) { if (config_cfdata_attach(cfdatav, 0) != 0) panic("config_cfdata fini rollback failed"); goto out; } if ((error = frob_cfdrivervec(cfdriverv, config_cfdriver_detach, config_cfdriver_attach, "fini", false)) != 0) { frob_cfattachvec(cfattachv, config_cfattach_attach, NULL, "fini rollback", true); if (config_cfdata_attach(cfdatav, 0) != 0) panic("config_cfdata fini rollback failed"); goto out; } /* Success! */ error = 0; out: KERNEL_UNLOCK_ONE(NULL); return error; } void config_init_mi(void) { if (!config_initialized) config_init(); sysctl_detach_setup(NULL); } void config_deferred(device_t dev) { KASSERT(KERNEL_LOCKED_P()); config_process_deferred(&deferred_config_queue, dev); config_process_deferred(&interrupt_config_queue, dev); config_process_deferred(&mountroot_config_queue, dev); } static void config_interrupts_thread(void *cookie) { struct deferred_config *dc; device_t dev; mutex_enter(&config_misc_lock); while ((dc = TAILQ_FIRST(&interrupt_config_queue)) != NULL) { TAILQ_REMOVE(&interrupt_config_queue, dc, dc_queue); mutex_exit(&config_misc_lock); dev = dc->dc_dev; (*dc->dc_func)(dev); if (!device_pmf_is_registered(dev)) aprint_debug_dev(dev, "WARNING: power management not supported\n"); config_pending_decr(dev); kmem_free(dc, sizeof(*dc)); mutex_enter(&config_misc_lock); } mutex_exit(&config_misc_lock); kthread_exit(0); } void config_create_interruptthreads(void) { int i; for (i = 0; i < interrupt_config_threads; i++) { (void)kthread_create(PRI_NONE, 0/*XXXSMP */, NULL, config_interrupts_thread, NULL, NULL, "configintr"); } } static void config_mountroot_thread(void *cookie) { struct deferred_config *dc; mutex_enter(&config_misc_lock); while ((dc = TAILQ_FIRST(&mountroot_config_queue)) != NULL) { TAILQ_REMOVE(&mountroot_config_queue, dc, dc_queue); mutex_exit(&config_misc_lock); (*dc->dc_func)(dc->dc_dev); kmem_free(dc, sizeof(*dc)); mutex_enter(&config_misc_lock); } mutex_exit(&config_misc_lock); kthread_exit(0); } void config_create_mountrootthreads(void) { int i; if (!root_is_mounted) root_is_mounted = true; mountroot_config_lwpids_size = sizeof(mountroot_config_lwpids) * mountroot_config_threads; mountroot_config_lwpids = kmem_alloc(mountroot_config_lwpids_size, KM_NOSLEEP); KASSERT(mountroot_config_lwpids); for (i = 0; i < mountroot_config_threads; i++) { mountroot_config_lwpids[i] = 0; (void)kthread_create(PRI_NONE, KTHREAD_MUSTJOIN/* XXXSMP */, NULL, config_mountroot_thread, NULL, &mountroot_config_lwpids[i], "configroot"); } } void config_finalize_mountroot(void) { int i, error; for (i = 0; i < mountroot_config_threads; i++) { if (mountroot_config_lwpids[i] == 0) continue; error = kthread_join(mountroot_config_lwpids[i]); if (error) printf("%s: thread %x joined with error %d\n", __func__, i, error); } kmem_free(mountroot_config_lwpids, mountroot_config_lwpids_size); } /* * Announce device attach/detach to userland listeners. */ int no_devmon_insert(const char *name, prop_dictionary_t p) { return ENODEV; } static void devmon_report_device(device_t dev, bool isattach) { prop_dictionary_t ev, dict = device_properties(dev); const char *parent; const char *what; const char *where; device_t pdev = device_parent(dev); /* If currently no drvctl device, just return */ if (devmon_insert_vec == no_devmon_insert) return; ev = prop_dictionary_create(); if (ev == NULL) return; what = (isattach ? "device-attach" : "device-detach"); parent = (pdev == NULL ? "root" : device_xname(pdev)); if (prop_dictionary_get_string(dict, "location", &where)) { prop_dictionary_set_string(ev, "location", where); aprint_debug("ev: %s %s at %s in [%s]\n", what, device_xname(dev), parent, where); } if (!prop_dictionary_set_string(ev, "device", device_xname(dev)) || !prop_dictionary_set_string(ev, "parent", parent)) { prop_object_release(ev); return; } if ((*devmon_insert_vec)(what, ev) != 0) prop_object_release(ev); } /* * Add a cfdriver to the system. */ int config_cfdriver_attach(struct cfdriver *cd) { struct cfdriver *lcd; /* Make sure this driver isn't already in the system. */ LIST_FOREACH(lcd, &allcfdrivers, cd_list) { if (STREQ(lcd->cd_name, cd->cd_name)) return EEXIST; } LIST_INIT(&cd->cd_attach); LIST_INSERT_HEAD(&allcfdrivers, cd, cd_list); return 0; } /* * Remove a cfdriver from the system. */ int config_cfdriver_detach(struct cfdriver *cd) { struct alldevs_foray af; int i, rc = 0; config_alldevs_enter(&af); /* Make sure there are no active instances. */ for (i = 0; i < cd->cd_ndevs; i++) { if (cd->cd_devs[i] != NULL) { rc = EBUSY; break; } } config_alldevs_exit(&af); if (rc != 0) return rc; /* ...and no attachments loaded. */ if (LIST_EMPTY(&cd->cd_attach) == 0) return EBUSY; LIST_REMOVE(cd, cd_list); KASSERT(cd->cd_devs == NULL); return 0; } /* * Look up a cfdriver by name. */ struct cfdriver * config_cfdriver_lookup(const char *name) { struct cfdriver *cd; LIST_FOREACH(cd, &allcfdrivers, cd_list) { if (STREQ(cd->cd_name, name)) return cd; } return NULL; } /* * Add a cfattach to the specified driver. */ int config_cfattach_attach(const char *driver, struct cfattach *ca) { struct cfattach *lca; struct cfdriver *cd; cd = config_cfdriver_lookup(driver); if (cd == NULL) return ESRCH; /* Make sure this attachment isn't already on this driver. */ LIST_FOREACH(lca, &cd->cd_attach, ca_list) { if (STREQ(lca->ca_name, ca->ca_name)) return EEXIST; } LIST_INSERT_HEAD(&cd->cd_attach, ca, ca_list); return 0; } /* * Remove a cfattach from the specified driver. */ int config_cfattach_detach(const char *driver, struct cfattach *ca) { struct alldevs_foray af; struct cfdriver *cd; device_t dev; int i, rc = 0; cd = config_cfdriver_lookup(driver); if (cd == NULL) return ESRCH; config_alldevs_enter(&af); /* Make sure there are no active instances. */ for (i = 0; i < cd->cd_ndevs; i++) { if ((dev = cd->cd_devs[i]) == NULL) continue; if (dev->dv_cfattach == ca) { rc = EBUSY; break; } } config_alldevs_exit(&af); if (rc != 0) return rc; LIST_REMOVE(ca, ca_list); return 0; } /* * Look up a cfattach by name. */ static struct cfattach * config_cfattach_lookup_cd(struct cfdriver *cd, const char *atname) { struct cfattach *ca; LIST_FOREACH(ca, &cd->cd_attach, ca_list) { if (STREQ(ca->ca_name, atname)) return ca; } return NULL; } /* * Look up a cfattach by driver/attachment name. */ struct cfattach * config_cfattach_lookup(const char *name, const char *atname) { struct cfdriver *cd; cd = config_cfdriver_lookup(name); if (cd == NULL) return NULL; return config_cfattach_lookup_cd(cd, atname); } /* * Apply the matching function and choose the best. This is used * a few times and we want to keep the code small. */ static void mapply(struct matchinfo *m, cfdata_t cf) { int pri; if (m->fn != NULL) { pri = (*m->fn)(m->parent, cf, m->locs, m->aux); } else { pri = config_match(m->parent, cf, m->aux); } if (pri > m->pri) { m->match = cf; m->pri = pri; } } int config_stdsubmatch(device_t parent, cfdata_t cf, const int *locs, void *aux) { const struct cfiattrdata *ci; const struct cflocdesc *cl; int nlocs, i; ci = cfiattr_lookup(cfdata_ifattr(cf), parent->dv_cfdriver); KASSERT(ci); nlocs = ci->ci_loclen; KASSERT(!nlocs || locs); for (i = 0; i < nlocs; i++) { cl = &ci->ci_locdesc[i]; if (cl->cld_defaultstr != NULL && cf->cf_loc[i] == cl->cld_default) continue; if (cf->cf_loc[i] == locs[i]) continue; return 0; } return config_match(parent, cf, aux); } /* * Helper function: check whether the driver supports the interface attribute * and return its descriptor structure. */ static const struct cfiattrdata * cfdriver_get_iattr(const struct cfdriver *cd, const char *ia) { const struct cfiattrdata * const *cpp; if (cd->cd_attrs == NULL) return 0; for (cpp = cd->cd_attrs; *cpp; cpp++) { if (STREQ((*cpp)->ci_name, ia)) { /* Match. */ return *cpp; } } return 0; } static int __diagused cfdriver_iattr_count(const struct cfdriver *cd) { const struct cfiattrdata * const *cpp; int i; if (cd->cd_attrs == NULL) return 0; for (i = 0, cpp = cd->cd_attrs; *cpp; cpp++) { i++; } return i; } /* * Lookup an interface attribute description by name. * If the driver is given, consider only its supported attributes. */ const struct cfiattrdata * cfiattr_lookup(const char *name, const struct cfdriver *cd) { const struct cfdriver *d; const struct cfiattrdata *ia; if (cd) return cfdriver_get_iattr(cd, name); LIST_FOREACH(d, &allcfdrivers, cd_list) { ia = cfdriver_get_iattr(d, name); if (ia) return ia; } return 0; } /* * Determine if `parent' is a potential parent for a device spec based * on `cfp'. */ static int cfparent_match(const device_t parent, const struct cfparent *cfp) { struct cfdriver *pcd; /* We don't match root nodes here. */ if (cfp == NULL) return 0; pcd = parent->dv_cfdriver; KASSERT(pcd != NULL); /* * First, ensure this parent has the correct interface * attribute. */ if (!cfdriver_get_iattr(pcd, cfp->cfp_iattr)) return 0; /* * If no specific parent device instance was specified (i.e. * we're attaching to the attribute only), we're done! */ if (cfp->cfp_parent == NULL) return 1; /* * Check the parent device's name. */ if (STREQ(pcd->cd_name, cfp->cfp_parent) == 0) return 0; /* not the same parent */ /* * Make sure the unit number matches. */ if (cfp->cfp_unit == DVUNIT_ANY || /* wildcard */ cfp->cfp_unit == parent->dv_unit) return 1; /* Unit numbers don't match. */ return 0; } /* * Helper for config_cfdata_attach(): check all devices whether it could be * parent any attachment in the config data table passed, and rescan. */ static void rescan_with_cfdata(const struct cfdata *cf) { device_t d; const struct cfdata *cf1; deviter_t di; KASSERT(KERNEL_LOCKED_P()); /* * "alldevs" is likely longer than a modules's cfdata, so make it * the outer loop. */ for (d = deviter_first(&di, 0); d != NULL; d = deviter_next(&di)) { if (!(d->dv_cfattach->ca_rescan)) continue; for (cf1 = cf; cf1->cf_name; cf1++) { if (!cfparent_match(d, cf1->cf_pspec)) continue; (*d->dv_cfattach->ca_rescan)(d, cfdata_ifattr(cf1), cf1->cf_loc); config_deferred(d); } } deviter_release(&di); } /* * Attach a supplemental config data table and rescan potential * parent devices if required. */ int config_cfdata_attach(cfdata_t cf, int scannow) { struct cftable *ct; KERNEL_LOCK(1, NULL); ct = kmem_alloc(sizeof(*ct), KM_SLEEP); ct->ct_cfdata = cf; TAILQ_INSERT_TAIL(&allcftables, ct, ct_list); if (scannow) rescan_with_cfdata(cf); KERNEL_UNLOCK_ONE(NULL); return 0; } /* * Helper for config_cfdata_detach: check whether a device is * found through any attachment in the config data table. */ static int dev_in_cfdata(device_t d, cfdata_t cf) { const struct cfdata *cf1; for (cf1 = cf; cf1->cf_name; cf1++) if (d->dv_cfdata == cf1) return 1; return 0; } /* * Detach a supplemental config data table. Detach all devices found * through that table (and thus keeping references to it) before. */ int config_cfdata_detach(cfdata_t cf) { device_t d; int error = 0; struct cftable *ct; deviter_t di; KERNEL_LOCK(1, NULL); for (d = deviter_first(&di, DEVITER_F_RW); d != NULL; d = deviter_next(&di)) { if (!dev_in_cfdata(d, cf)) continue; if ((error = config_detach(d, 0)) != 0) break; } deviter_release(&di); if (error) { aprint_error_dev(d, "unable to detach instance\n"); goto out; } TAILQ_FOREACH(ct, &allcftables, ct_list) { if (ct->ct_cfdata == cf) { TAILQ_REMOVE(&allcftables, ct, ct_list); kmem_free(ct, sizeof(*ct)); error = 0; goto out; } } /* not found -- shouldn't happen */ error = EINVAL; out: KERNEL_UNLOCK_ONE(NULL); return error; } /* * Invoke the "match" routine for a cfdata entry on behalf of * an external caller, usually a direct config "submatch" routine. */ int config_match(device_t parent, cfdata_t cf, void *aux) { struct cfattach *ca; KASSERT(KERNEL_LOCKED_P()); ca = config_cfattach_lookup(cf->cf_name, cf->cf_atname); if (ca == NULL) { /* No attachment for this entry, oh well. */ return 0; } return (*ca->ca_match)(parent, cf, aux); } /* * Invoke the "probe" routine for a cfdata entry on behalf of * an external caller, usually an indirect config "search" routine. */ int config_probe(device_t parent, cfdata_t cf, void *aux) { /* * This is currently a synonym for config_match(), but this * is an implementation detail; "match" and "probe" routines * have different behaviors. * * XXX config_probe() should return a bool, because there is * XXX no match score for probe -- it's either there or it's * XXX not, but some ports abuse the return value as a way * XXX to attach "critical" devices before "non-critical" * XXX devices. */ return config_match(parent, cf, aux); } static struct cfargs_internal * cfargs_canonicalize(const struct cfargs * const cfargs, struct cfargs_internal * const store) { struct cfargs_internal *args = store; memset(args, 0, sizeof(*args)); /* If none specified, are all-NULL pointers are good. */ if (cfargs == NULL) { return args; } /* * Only one arguments version is recognized at this time. */ if (cfargs->cfargs_version != CFARGS_VERSION) { panic("cfargs_canonicalize: unknown version %lu\n", (unsigned long)cfargs->cfargs_version); } /* * submatch and search are mutually-exclusive. */ if (cfargs->submatch != NULL && cfargs->search != NULL) { panic("cfargs_canonicalize: submatch and search are " "mutually-exclusive"); } if (cfargs->submatch != NULL) { args->submatch = cfargs->submatch; } else if (cfargs->search != NULL) { args->search = cfargs->search; } args->iattr = cfargs->iattr; args->locators = cfargs->locators; args->devhandle = cfargs->devhandle; return args; } /* * Iterate over all potential children of some device, calling the given * function (default being the child's match function) for each one. * Nonzero returns are matches; the highest value returned is considered * the best match. Return the `found child' if we got a match, or NULL * otherwise. The `aux' pointer is simply passed on through. * * Note that this function is designed so that it can be used to apply * an arbitrary function to all potential children (its return value * can be ignored). */ static cfdata_t config_search_internal(device_t parent, void *aux, const struct cfargs_internal * const args) { struct cftable *ct; cfdata_t cf; struct matchinfo m; KASSERT(config_initialized); KASSERT(!args->iattr || cfdriver_get_iattr(parent->dv_cfdriver, args->iattr)); KASSERT(args->iattr || cfdriver_iattr_count(parent->dv_cfdriver) < 2); m.fn = args->submatch; /* N.B. union */ m.parent = parent; m.locs = args->locators; m.aux = aux; m.match = NULL; m.pri = 0; TAILQ_FOREACH(ct, &allcftables, ct_list) { for (cf = ct->ct_cfdata; cf->cf_name; cf++) { /* We don't match root nodes here. */ if (!cf->cf_pspec) continue; /* * Skip cf if no longer eligible, otherwise scan * through parents for one matching `parent', and * try match function. */ if (cf->cf_fstate == FSTATE_FOUND) continue; if (cf->cf_fstate == FSTATE_DNOTFOUND || cf->cf_fstate == FSTATE_DSTAR) continue; /* * If an interface attribute was specified, * consider only children which attach to * that attribute. */ if (args->iattr != NULL && !STREQ(args->iattr, cfdata_ifattr(cf))) continue; if (cfparent_match(parent, cf->cf_pspec)) mapply(&m, cf); } } rnd_add_uint32(&rnd_autoconf_source, 0); return m.match; } cfdata_t config_search(device_t parent, void *aux, const struct cfargs *cfargs) { cfdata_t cf; struct cfargs_internal store; cf = config_search_internal(parent, aux, cfargs_canonicalize(cfargs, &store)); return cf; } /* * Find the given root device. * This is much like config_search, but there is no parent. * Don't bother with multiple cfdata tables; the root node * must always be in the initial table. */ cfdata_t config_rootsearch(cfsubmatch_t fn, const char *rootname, void *aux) { cfdata_t cf; const short *p; struct matchinfo m; m.fn = fn; m.parent = ROOT; m.aux = aux; m.match = NULL; m.pri = 0; m.locs = 0; /* * Look at root entries for matching name. We do not bother * with found-state here since only one root should ever be * searched (and it must be done first). */ for (p = cfroots; *p >= 0; p++) { cf = &cfdata[*p]; if (strcmp(cf->cf_name, rootname) == 0) mapply(&m, cf); } return m.match; } static const char * const msgs[] = { [QUIET] = "", [UNCONF] = " not configured\n", [UNSUPP] = " unsupported\n", }; /* * The given `aux' argument describes a device that has been found * on the given parent, but not necessarily configured. Locate the * configuration data for that device (using the submatch function * provided, or using candidates' cd_match configuration driver * functions) and attach it, and return its device_t. If the device was * not configured, call the given `print' function and return NULL. */ device_t config_found(device_t parent, void *aux, cfprint_t print, const struct cfargs * const cfargs) { cfdata_t cf; struct cfargs_internal store; const struct cfargs_internal * const args = cfargs_canonicalize(cfargs, &store); cf = config_search_internal(parent, aux, args); if (cf != NULL) { return config_attach_internal(parent, cf, aux, print, args); } if (print) { if (config_do_twiddle && cold) twiddle(); const int pret = (*print)(aux, device_xname(parent)); KASSERT(pret >= 0); KASSERT(pret < __arraycount(msgs)); KASSERT(msgs[pret] != NULL); aprint_normal("%s", msgs[pret]); } return NULL; } /* * As above, but for root devices. */ device_t config_rootfound(const char *rootname, void *aux) { cfdata_t cf; device_t dev = NULL; KERNEL_LOCK(1, NULL); if ((cf = config_rootsearch(NULL, rootname, aux)) != NULL) dev = config_attach(ROOT, cf, aux, NULL, CFARGS_NONE); else aprint_error("root device %s not configured\n", rootname); KERNEL_UNLOCK_ONE(NULL); return dev; } /* just like sprintf(buf, "%d") except that it works from the end */ static char * number(char *ep, int n) { *--ep = 0; while (n >= 10) { *--ep = (n % 10) + '0'; n /= 10; } *--ep = n + '0'; return ep; } /* * Expand the size of the cd_devs array if necessary. * * The caller must hold alldevs_lock. config_makeroom() may release and * re-acquire alldevs_lock, so callers should re-check conditions such * as alldevs_nwrite == 0 and alldevs_nread == 0 when config_makeroom() * returns. */ static void config_makeroom(int n, struct cfdriver *cd) { int ondevs, nndevs; device_t *osp, *nsp; KASSERT(mutex_owned(&alldevs_lock)); alldevs_nwrite++; for (nndevs = MAX(4, cd->cd_ndevs); nndevs <= n; nndevs += nndevs) ; while (n >= cd->cd_ndevs) { /* * Need to expand the array. */ ondevs = cd->cd_ndevs; osp = cd->cd_devs; /* * Release alldevs_lock around allocation, which may * sleep. */ mutex_exit(&alldevs_lock); nsp = kmem_alloc(sizeof(device_t) * nndevs, KM_SLEEP); mutex_enter(&alldevs_lock); /* * If another thread moved the array while we did * not hold alldevs_lock, try again. */ if (cd->cd_devs != osp) { mutex_exit(&alldevs_lock); kmem_free(nsp, sizeof(device_t) * nndevs); mutex_enter(&alldevs_lock); continue; } memset(nsp + ondevs, 0, sizeof(device_t) * (nndevs - ondevs)); if (ondevs != 0) memcpy(nsp, cd->cd_devs, sizeof(device_t) * ondevs); cd->cd_ndevs = nndevs; cd->cd_devs = nsp; if (ondevs != 0) { mutex_exit(&alldevs_lock); kmem_free(osp, sizeof(device_t) * ondevs); mutex_enter(&alldevs_lock); } } KASSERT(mutex_owned(&alldevs_lock)); alldevs_nwrite--; } /* * Put dev into the devices list. */ static void config_devlink(device_t dev) { mutex_enter(&alldevs_lock); KASSERT(device_cfdriver(dev)->cd_devs[dev->dv_unit] == dev); dev->dv_add_gen = alldevs_gen; /* It is safe to add a device to the tail of the list while * readers and writers are in the list. */ TAILQ_INSERT_TAIL(&alldevs, dev, dv_list); mutex_exit(&alldevs_lock); } static void config_devfree(device_t dev) { KASSERT(dev->dv_flags & DVF_PRIV_ALLOC); KASSERTMSG(dev->dv_pending == 0, "%d", dev->dv_pending); if (dev->dv_cfattach->ca_devsize > 0) kmem_free(dev->dv_private, dev->dv_cfattach->ca_devsize); kmem_free(dev, sizeof(*dev)); } /* * Caller must hold alldevs_lock. */ static void config_devunlink(device_t dev, struct devicelist *garbage) { struct device_garbage *dg = &dev->dv_garbage; cfdriver_t cd = device_cfdriver(dev); int i; KASSERT(mutex_owned(&alldevs_lock)); KASSERTMSG(dev->dv_pending == 0, "%d", dev->dv_pending); /* Unlink from device list. Link to garbage list. */ TAILQ_REMOVE(&alldevs, dev, dv_list); TAILQ_INSERT_TAIL(garbage, dev, dv_list); /* Remove from cfdriver's array. */ cd->cd_devs[dev->dv_unit] = NULL; /* * If the device now has no units in use, unlink its softc array. */ for (i = 0; i < cd->cd_ndevs; i++) { if (cd->cd_devs[i] != NULL) break; } /* Nothing found. Unlink, now. Deallocate, later. */ if (i == cd->cd_ndevs) { dg->dg_ndevs = cd->cd_ndevs; dg->dg_devs = cd->cd_devs; cd->cd_devs = NULL; cd->cd_ndevs = 0; } } static void config_devdelete(device_t dev) { struct device_garbage *dg = &dev->dv_garbage; device_lock_t dvl = device_getlock(dev); KASSERTMSG(dev->dv_pending == 0, "%d", dev->dv_pending); if (dg->dg_devs != NULL) kmem_free(dg->dg_devs, sizeof(device_t) * dg->dg_ndevs); localcount_fini(dev->dv_localcount); kmem_free(dev->dv_localcount, sizeof(*dev->dv_localcount)); cv_destroy(&dvl->dvl_cv); mutex_destroy(&dvl->dvl_mtx); KASSERT(dev->dv_properties != NULL); prop_object_release(dev->dv_properties); if (dev->dv_activity_handlers) panic("%s with registered handlers", __func__); if (dev->dv_locators) { size_t amount = *--dev->dv_locators; kmem_free(dev->dv_locators, amount); } config_devfree(dev); } static int config_unit_nextfree(cfdriver_t cd, cfdata_t cf) { int unit = cf->cf_unit; if (unit < 0) return -1; if (cf->cf_fstate == FSTATE_STAR) { for (; unit < cd->cd_ndevs; unit++) if (cd->cd_devs[unit] == NULL) break; /* * unit is now the unit of the first NULL device pointer, * or max(cd->cd_ndevs,cf->cf_unit). */ } else { if (unit < cd->cd_ndevs && cd->cd_devs[unit] != NULL) unit = -1; } return unit; } static int config_unit_alloc(device_t dev, cfdriver_t cd, cfdata_t cf) { struct alldevs_foray af; int unit; config_alldevs_enter(&af); for (;;) { unit = config_unit_nextfree(cd, cf); if (unit == -1) break; if (unit < cd->cd_ndevs) { cd->cd_devs[unit] = dev; dev->dv_unit = unit; break; } config_makeroom(unit, cd); } config_alldevs_exit(&af); return unit; } static device_t config_devalloc(const device_t parent, const cfdata_t cf, const struct cfargs_internal * const args) { cfdriver_t cd; cfattach_t ca; size_t lname, lunit; const char *xunit; int myunit; char num[10]; device_t dev; void *dev_private; const struct cfiattrdata *ia; device_lock_t dvl; cd = config_cfdriver_lookup(cf->cf_name); if (cd == NULL) return NULL; ca = config_cfattach_lookup_cd(cd, cf->cf_atname); if (ca == NULL) return NULL; /* get memory for all device vars */ KASSERT(ca->ca_flags & DVF_PRIV_ALLOC); if (ca->ca_devsize > 0) { dev_private = kmem_zalloc(ca->ca_devsize, KM_SLEEP); } else { dev_private = NULL; } dev = kmem_zalloc(sizeof(*dev), KM_SLEEP); dev->dv_handle = args->devhandle; dev->dv_class = cd->cd_class; dev->dv_cfdata = cf; dev->dv_cfdriver = cd; dev->dv_cfattach = ca; dev->dv_activity_count = 0; dev->dv_activity_handlers = NULL; dev->dv_private = dev_private; dev->dv_flags = ca->ca_flags; /* inherit flags from class */ dev->dv_attaching = curlwp; myunit = config_unit_alloc(dev, cd, cf); if (myunit == -1) { config_devfree(dev); return NULL; } /* compute length of name and decimal expansion of unit number */ lname = strlen(cd->cd_name); xunit = number(&num[sizeof(num)], myunit); lunit = &num[sizeof(num)] - xunit; if (lname + lunit > sizeof(dev->dv_xname)) panic("config_devalloc: device name too long"); dvl = device_getlock(dev); mutex_init(&dvl->dvl_mtx, MUTEX_DEFAULT, IPL_NONE); cv_init(&dvl->dvl_cv, "pmfsusp"); memcpy(dev->dv_xname, cd->cd_name, lname); memcpy(dev->dv_xname + lname, xunit, lunit); dev->dv_parent = parent; if (parent != NULL) dev->dv_depth = parent->dv_depth + 1; else dev->dv_depth = 0; dev->dv_flags |= DVF_ACTIVE; /* always initially active */ if (args->locators) { KASSERT(parent); /* no locators at root */ ia = cfiattr_lookup(cfdata_ifattr(cf), parent->dv_cfdriver); dev->dv_locators = kmem_alloc(sizeof(int) * (ia->ci_loclen + 1), KM_SLEEP); *dev->dv_locators++ = sizeof(int) * (ia->ci_loclen + 1); memcpy(dev->dv_locators, args->locators, sizeof(int) * ia->ci_loclen); } dev->dv_properties = prop_dictionary_create(); KASSERT(dev->dv_properties != NULL); prop_dictionary_set_string_nocopy(dev->dv_properties, "device-driver", dev->dv_cfdriver->cd_name); prop_dictionary_set_uint16(dev->dv_properties, "device-unit", dev->dv_unit); if (parent != NULL) { prop_dictionary_set_string(dev->dv_properties, "device-parent", device_xname(parent)); } dev->dv_localcount = kmem_zalloc(sizeof(*dev->dv_localcount), KM_SLEEP); localcount_init(dev->dv_localcount); if (dev->dv_cfdriver->cd_attrs != NULL) config_add_attrib_dict(dev); return dev; } /* * Create an array of device attach attributes and add it * to the device's dv_properties dictionary. * * <key>interface-attributes</key> * <array> * <dict> * <key>attribute-name</key> * <string>foo</string> * <key>locators</key> * <array> * <dict> * <key>loc-name</key> * <string>foo-loc1</string> * </dict> * <dict> * <key>loc-name</key> * <string>foo-loc2</string> * <key>default</key> * <string>foo-loc2-default</string> * </dict> * ... * </array> * </dict> * ... * </array> */ static void config_add_attrib_dict(device_t dev) { int i, j; const struct cfiattrdata *ci; prop_dictionary_t attr_dict, loc_dict; prop_array_t attr_array, loc_array; if ((attr_array = prop_array_create()) == NULL) return; for (i = 0; ; i++) { if ((ci = dev->dv_cfdriver->cd_attrs[i]) == NULL) break; if ((attr_dict = prop_dictionary_create()) == NULL) break; prop_dictionary_set_string_nocopy(attr_dict, "attribute-name", ci->ci_name); /* Create an array of the locator names and defaults */ if (ci->ci_loclen != 0 && (loc_array = prop_array_create()) != NULL) { for (j = 0; j < ci->ci_loclen; j++) { loc_dict = prop_dictionary_create(); if (loc_dict == NULL) continue; prop_dictionary_set_string_nocopy(loc_dict, "loc-name", ci->ci_locdesc[j].cld_name); if (ci->ci_locdesc[j].cld_defaultstr != NULL) prop_dictionary_set_string_nocopy( loc_dict, "default", ci->ci_locdesc[j].cld_defaultstr); prop_array_set(loc_array, j, loc_dict); prop_object_release(loc_dict); } prop_dictionary_set_and_rel(attr_dict, "locators", loc_array); } prop_array_add(attr_array, attr_dict); prop_object_release(attr_dict); } if (i == 0) prop_object_release(attr_array); else prop_dictionary_set_and_rel(dev->dv_properties, "interface-attributes", attr_array); return; } /* * Attach a found device. */ static device_t config_attach_internal(device_t parent, cfdata_t cf, void *aux, cfprint_t print, const struct cfargs_internal * const args) { device_t dev; struct cftable *ct; const char *drvname; bool deferred; KASSERT(KERNEL_LOCKED_P()); dev = config_devalloc(parent, cf, args); if (!dev) panic("config_attach: allocation of device softc failed"); /* XXX redundant - see below? */ if (cf->cf_fstate != FSTATE_STAR) { KASSERT(cf->cf_fstate == FSTATE_NOTFOUND); cf->cf_fstate = FSTATE_FOUND; } config_devlink(dev); if (config_do_twiddle && cold) twiddle(); else aprint_naive("Found "); /* * We want the next two printfs for normal, verbose, and quiet, * but not silent (in which case, we're twiddling, instead). */ if (parent == ROOT) { aprint_naive("%s (root)", device_xname(dev)); aprint_normal("%s (root)", device_xname(dev)); } else { aprint_naive("%s at %s", device_xname(dev), device_xname(parent)); aprint_normal("%s at %s", device_xname(dev), device_xname(parent)); if (print) (void) (*print)(aux, NULL); } /* * Before attaching, clobber any unfound devices that are * otherwise identical. * XXX code above is redundant? */ drvname = dev->dv_cfdriver->cd_name; TAILQ_FOREACH(ct, &allcftables, ct_list) { for (cf = ct->ct_cfdata; cf->cf_name; cf++) { if (STREQ(cf->cf_name, drvname) && cf->cf_unit == dev->dv_unit) { if (cf->cf_fstate == FSTATE_NOTFOUND) cf->cf_fstate = FSTATE_FOUND; } } } device_register(dev, aux); /* Let userland know */ devmon_report_device(dev, true); /* * Prevent detach until the driver's attach function, and all * deferred actions, have finished. */ config_pending_incr(dev); /* Call the driver's attach function. */ (*dev->dv_cfattach->ca_attach)(parent, dev, aux); /* * Allow other threads to acquire references to the device now * that the driver's attach function is done. */ mutex_enter(&config_misc_lock); KASSERT(dev->dv_attaching == curlwp); dev->dv_attaching = NULL; cv_broadcast(&config_misc_cv); mutex_exit(&config_misc_lock); /* * Synchronous parts of attach are done. Allow detach, unless * the driver's attach function scheduled deferred actions. */ config_pending_decr(dev); mutex_enter(&config_misc_lock); deferred = (dev->dv_pending != 0); mutex_exit(&config_misc_lock); if (!deferred && !device_pmf_is_registered(dev)) aprint_debug_dev(dev, "WARNING: power management not supported\n"); config_process_deferred(&deferred_config_queue, dev); device_register_post_config(dev, aux); rnd_add_uint32(&rnd_autoconf_source, 0); return dev; } device_t config_attach(device_t parent, cfdata_t cf, void *aux, cfprint_t print, const struct cfargs *cfargs) { struct cfargs_internal store; KASSERT(KERNEL_LOCKED_P()); return config_attach_internal(parent, cf, aux, print, cfargs_canonicalize(cfargs, &store)); } /* * As above, but for pseudo-devices. Pseudo-devices attached in this * way are silently inserted into the device tree, and their children * attached. * * Note that because pseudo-devices are attached silently, any information * the attach routine wishes to print should be prefixed with the device * name by the attach routine. */ device_t config_attach_pseudo(cfdata_t cf) { device_t dev; KERNEL_LOCK(1, NULL); struct cfargs_internal args = { }; dev = config_devalloc(ROOT, cf, &args); if (!dev) goto out; /* XXX mark busy in cfdata */ if (cf->cf_fstate != FSTATE_STAR) { KASSERT(cf->cf_fstate == FSTATE_NOTFOUND); cf->cf_fstate = FSTATE_FOUND; } config_devlink(dev); #if 0 /* XXXJRT not yet */ device_register(dev, NULL); /* like a root node */ #endif /* Let userland know */ devmon_report_device(dev, true); /* * Prevent detach until the driver's attach function, and all * deferred actions, have finished. */ config_pending_incr(dev); /* Call the driver's attach function. */ (*dev->dv_cfattach->ca_attach)(ROOT, dev, NULL); /* * Allow other threads to acquire references to the device now * that the driver's attach function is done. */ mutex_enter(&config_misc_lock); KASSERT(dev->dv_attaching == curlwp); dev->dv_attaching = NULL; cv_broadcast(&config_misc_cv); mutex_exit(&config_misc_lock); /* * Synchronous parts of attach are done. Allow detach, unless * the driver's attach function scheduled deferred actions. */ config_pending_decr(dev); config_process_deferred(&deferred_config_queue, dev); out: KERNEL_UNLOCK_ONE(NULL); return dev; } /* * Caller must hold alldevs_lock. */ static void config_collect_garbage(struct devicelist *garbage) { device_t dv; KASSERT(!cpu_intr_p()); KASSERT(!cpu_softintr_p()); KASSERT(mutex_owned(&alldevs_lock)); while (alldevs_nwrite == 0 && alldevs_nread == 0 && alldevs_garbage) { TAILQ_FOREACH(dv, &alldevs, dv_list) { if (dv->dv_del_gen != 0) break; } if (dv == NULL) { alldevs_garbage = false; break; } config_devunlink(dv, garbage); } KASSERT(mutex_owned(&alldevs_lock)); } static void config_dump_garbage(struct devicelist *garbage) { device_t dv; while ((dv = TAILQ_FIRST(garbage)) != NULL) { TAILQ_REMOVE(garbage, dv, dv_list); config_devdelete(dv); } } static int config_detach_enter(device_t dev) { struct lwp *l __diagused; int error = 0; mutex_enter(&config_misc_lock); /* * Wait until attach has fully completed, and until any * concurrent detach (e.g., drvctl racing with USB event * thread) has completed. * * Caller must hold alldevs_nread or alldevs_nwrite (e.g., via * deviter) to ensure the winner of the race doesn't free the * device leading the loser of the race into use-after-free. * * XXX Not all callers do this! */ while (dev->dv_pending || dev->dv_detaching) { KASSERTMSG(dev->dv_detaching != curlwp, "recursively detaching %s", device_xname(dev)); error = cv_wait_sig(&config_misc_cv, &config_misc_lock); if (error) goto out; } /* * Attach has completed, and no other concurrent detach is * running. Claim the device for detaching. This will cause * all new attempts to acquire references to block. */ KASSERTMSG((l = dev->dv_attaching) == NULL, "lwp %ld [%s] @ %p attaching", (long)l->l_lid, (l->l_name ? l->l_name : l->l_proc->p_comm), l); KASSERTMSG((l = dev->dv_detaching) == NULL, "lwp %ld [%s] @ %p detaching", (long)l->l_lid, (l->l_name ? l->l_name : l->l_proc->p_comm), l); dev->dv_detaching = curlwp; out: mutex_exit(&config_misc_lock); return error; } static void config_detach_exit(device_t dev) { struct lwp *l __diagused; mutex_enter(&config_misc_lock); KASSERTMSG((l = dev->dv_detaching) == curlwp, "lwp %ld [%s] @ %p detaching", (long)l->l_lid, (l->l_name ? l->l_name : l->l_proc->p_comm), l); dev->dv_detaching = NULL; cv_broadcast(&config_misc_cv); mutex_exit(&config_misc_lock); } /* * Detach a device. Optionally forced (e.g. because of hardware * removal) and quiet. Returns zero if successful, non-zero * (an error code) otherwise. * * Note that this code wants to be run from a process context, so * that the detach can sleep to allow processes which have a device * open to run and unwind their stacks. */ int config_detach(device_t dev, int flags) { struct alldevs_foray af; struct cftable *ct; cfdata_t cf; const struct cfattach *ca; struct cfdriver *cd; device_t d __diagused; int rv = 0; KERNEL_LOCK(1, NULL); cf = dev->dv_cfdata; KASSERTMSG((cf == NULL || cf->cf_fstate == FSTATE_FOUND || cf->cf_fstate == FSTATE_STAR), "config_detach: %s: bad device fstate: %d", device_xname(dev), cf ? cf->cf_fstate : -1); cd = dev->dv_cfdriver; KASSERT(cd != NULL); ca = dev->dv_cfattach; KASSERT(ca != NULL); /* * Only one detach at a time, please -- and not until fully * attached. */ rv = config_detach_enter(dev); if (rv) { KERNEL_UNLOCK_ONE(NULL); return rv; } mutex_enter(&alldevs_lock); if (dev->dv_del_gen != 0) { mutex_exit(&alldevs_lock); #ifdef DIAGNOSTIC printf("%s: %s is already detached\n", __func__, device_xname(dev)); #endif /* DIAGNOSTIC */ config_detach_exit(dev); KERNEL_UNLOCK_ONE(NULL); return ENOENT; } alldevs_nwrite++; mutex_exit(&alldevs_lock); /* * Call the driver's .ca_detach function, unless it has none or * we are skipping it because it's unforced shutdown time and * the driver didn't ask to detach on shutdown. */ if (!detachall && (flags & (DETACH_SHUTDOWN|DETACH_FORCE)) == DETACH_SHUTDOWN && (dev->dv_flags & DVF_DETACH_SHUTDOWN) == 0) { rv = EOPNOTSUPP; } else if (ca->ca_detach != NULL) { rv = (*ca->ca_detach)(dev, flags); } else rv = EOPNOTSUPP; /* * If it was not possible to detach the device, then we either * panic() (for the forced but failed case), or return an error. */ if (rv) { /* * Detach failed -- likely EOPNOTSUPP or EBUSY. Driver * must not have called config_detach_commit. */ KASSERTMSG(!dev->dv_detached, "%s committed to detaching and then backed out", device_xname(dev)); if (flags & DETACH_FORCE) { panic("config_detach: forced detach of %s failed (%d)", device_xname(dev), rv); } goto out; } /* * The device has now been successfully detached. */ /* * If .ca_detach didn't commit to detach, then do that for it. * This wakes any pending device_lookup_acquire calls so they * will fail. */ config_detach_commit(dev); /* * If it was possible to detach the device, ensure that the * device is deactivated. */ dev->dv_flags &= ~DVF_ACTIVE; /* XXXSMP */ /* * Wait for all device_lookup_acquire references -- mostly, for * all attempts to open the device -- to drain. It is the * responsibility of .ca_detach to ensure anything with open * references will be interrupted and release them promptly, * not block indefinitely. All new attempts to acquire * references will fail, as config_detach_commit has arranged * by now. */ mutex_enter(&config_misc_lock); localcount_drain(dev->dv_localcount, &config_misc_cv, &config_misc_lock); mutex_exit(&config_misc_lock); /* Let userland know */ devmon_report_device(dev, false); #ifdef DIAGNOSTIC /* * Sanity: If you're successfully detached, you should have no * children. (Note that because children must be attached * after parents, we only need to search the latter part of * the list.) */ mutex_enter(&alldevs_lock); for (d = TAILQ_NEXT(dev, dv_list); d != NULL; d = TAILQ_NEXT(d, dv_list)) { if (d->dv_parent == dev && d->dv_del_gen == 0) { printf("config_detach: detached device %s" " has children %s\n", device_xname(dev), device_xname(d)); panic("config_detach"); } } mutex_exit(&alldevs_lock); #endif /* notify the parent that the child is gone */ if (dev->dv_parent) { device_t p = dev->dv_parent; if (p->dv_cfattach->ca_childdetached) (*p->dv_cfattach->ca_childdetached)(p, dev); } /* * Mark cfdata to show that the unit can be reused, if possible. */ TAILQ_FOREACH(ct, &allcftables, ct_list) { for (cf = ct->ct_cfdata; cf->cf_name; cf++) { if (STREQ(cf->cf_name, cd->cd_name)) { if (cf->cf_fstate == FSTATE_FOUND && cf->cf_unit == dev->dv_unit) cf->cf_fstate = FSTATE_NOTFOUND; } } } if (dev->dv_cfdata != NULL && (flags & DETACH_QUIET) == 0) aprint_normal_dev(dev, "detached\n"); out: config_detach_exit(dev); config_alldevs_enter(&af); KASSERT(alldevs_nwrite != 0); --alldevs_nwrite; if (rv == 0 && dev->dv_del_gen == 0) { if (alldevs_nwrite == 0 && alldevs_nread == 0) config_devunlink(dev, &af.af_garbage); else { dev->dv_del_gen = alldevs_gen; alldevs_garbage = true; } } config_alldevs_exit(&af); KERNEL_UNLOCK_ONE(NULL); return rv; } /* * config_detach_commit(dev) * * Issued by a driver's .ca_detach routine to notify anyone * waiting in device_lookup_acquire that the driver is committed * to detaching the device, which allows device_lookup_acquire to * wake up and fail immediately. * * Safe to call multiple times -- idempotent. Must be called * during config_detach_enter/exit. Safe to use with * device_lookup because the device is not actually removed from * the table until after config_detach_exit. */ void config_detach_commit(device_t dev) { struct lwp *l __diagused; mutex_enter(&config_misc_lock); KASSERTMSG((l = dev->dv_detaching) == curlwp, "lwp %ld [%s] @ %p detaching", (long)l->l_lid, (l->l_name ? l->l_name : l->l_proc->p_comm), l); dev->dv_detached = true; cv_broadcast(&config_misc_cv); mutex_exit(&config_misc_lock); } int config_detach_children(device_t parent, int flags) { device_t dv; deviter_t di; int error = 0; KASSERT(KERNEL_LOCKED_P()); for (dv = deviter_first(&di, DEVITER_F_RW); dv != NULL; dv = deviter_next(&di)) { if (device_parent(dv) != parent) continue; if ((error = config_detach(dv, flags)) != 0) break; } deviter_release(&di); return error; } device_t shutdown_first(struct shutdown_state *s) { if (!s->initialized) { deviter_init(&s->di, DEVITER_F_SHUTDOWN|DEVITER_F_LEAVES_FIRST); s->initialized = true; } return shutdown_next(s); } device_t shutdown_next(struct shutdown_state *s) { device_t dv; while ((dv = deviter_next(&s->di)) != NULL && !device_is_active(dv)) ; if (dv == NULL) s->initialized = false; return dv; } bool config_detach_all(int how) { static struct shutdown_state s; device_t curdev; bool progress = false; int flags; KERNEL_LOCK(1, NULL); if ((how & (RB_NOSYNC|RB_DUMP)) != 0) goto out; if ((how & RB_POWERDOWN) == RB_POWERDOWN) flags = DETACH_SHUTDOWN | DETACH_POWEROFF; else flags = DETACH_SHUTDOWN; for (curdev = shutdown_first(&s); curdev != NULL; curdev = shutdown_next(&s)) { aprint_debug(" detaching %s, ", device_xname(curdev)); if (config_detach(curdev, flags) == 0) { progress = true; aprint_debug("success."); } else aprint_debug("failed."); } out: KERNEL_UNLOCK_ONE(NULL); return progress; } static bool device_is_ancestor_of(device_t ancestor, device_t descendant) { device_t dv; for (dv = descendant; dv != NULL; dv = device_parent(dv)) { if (device_parent(dv) == ancestor) return true; } return false; } int config_deactivate(device_t dev) { deviter_t di; const struct cfattach *ca; device_t descendant; int s, rv = 0, oflags; for (descendant = deviter_first(&di, DEVITER_F_ROOT_FIRST); descendant != NULL; descendant = deviter_next(&di)) { if (dev != descendant && !device_is_ancestor_of(dev, descendant)) continue; if ((descendant->dv_flags & DVF_ACTIVE) == 0) continue; ca = descendant->dv_cfattach; oflags = descendant->dv_flags; descendant->dv_flags &= ~DVF_ACTIVE; if (ca->ca_activate == NULL) continue; s = splhigh(); rv = (*ca->ca_activate)(descendant, DVACT_DEACTIVATE); splx(s); if (rv != 0) descendant->dv_flags = oflags; } deviter_release(&di); return rv; } /* * Defer the configuration of the specified device until all * of its parent's devices have been attached. */ void config_defer(device_t dev, void (*func)(device_t)) { struct deferred_config *dc; if (dev->dv_parent == NULL) panic("config_defer: can't defer config of a root device"); dc = kmem_alloc(sizeof(*dc), KM_SLEEP); config_pending_incr(dev); mutex_enter(&config_misc_lock); #ifdef DIAGNOSTIC struct deferred_config *odc; TAILQ_FOREACH(odc, &deferred_config_queue, dc_queue) { if (odc->dc_dev == dev) panic("config_defer: deferred twice"); } #endif dc->dc_dev = dev; dc->dc_func = func; TAILQ_INSERT_TAIL(&deferred_config_queue, dc, dc_queue); mutex_exit(&config_misc_lock); } /* * Defer some autoconfiguration for a device until after interrupts * are enabled. */ void config_interrupts(device_t dev, void (*func)(device_t)) { struct deferred_config *dc; /* * If interrupts are enabled, callback now. */ if (cold == 0) { (*func)(dev); return; } dc = kmem_alloc(sizeof(*dc), KM_SLEEP); config_pending_incr(dev); mutex_enter(&config_misc_lock); #ifdef DIAGNOSTIC struct deferred_config *odc; TAILQ_FOREACH(odc, &interrupt_config_queue, dc_queue) { if (odc->dc_dev == dev) panic("config_interrupts: deferred twice"); } #endif dc->dc_dev = dev; dc->dc_func = func; TAILQ_INSERT_TAIL(&interrupt_config_queue, dc, dc_queue); mutex_exit(&config_misc_lock); } /* * Defer some autoconfiguration for a device until after root file system * is mounted (to load firmware etc). */ void config_mountroot(device_t dev, void (*func)(device_t)) { struct deferred_config *dc; /* * If root file system is mounted, callback now. */ if (root_is_mounted) { (*func)(dev); return; } dc = kmem_alloc(sizeof(*dc), KM_SLEEP); mutex_enter(&config_misc_lock); #ifdef DIAGNOSTIC struct deferred_config *odc; TAILQ_FOREACH(odc, &mountroot_config_queue, dc_queue) { if (odc->dc_dev == dev) panic("%s: deferred twice", __func__); } #endif dc->dc_dev = dev; dc->dc_func = func; TAILQ_INSERT_TAIL(&mountroot_config_queue, dc, dc_queue); mutex_exit(&config_misc_lock); } /* * Process a deferred configuration queue. */ static void config_process_deferred(struct deferred_config_head *queue, device_t parent) { struct deferred_config *dc; KASSERT(KERNEL_LOCKED_P()); mutex_enter(&config_misc_lock); dc = TAILQ_FIRST(queue); while (dc) { if (parent == NULL || dc->dc_dev->dv_parent == parent) { TAILQ_REMOVE(queue, dc, dc_queue); mutex_exit(&config_misc_lock); (*dc->dc_func)(dc->dc_dev); config_pending_decr(dc->dc_dev); kmem_free(dc, sizeof(*dc)); mutex_enter(&config_misc_lock); /* Restart, queue might have changed */ dc = TAILQ_FIRST(queue); } else { dc = TAILQ_NEXT(dc, dc_queue); } } mutex_exit(&config_misc_lock); } /* * Manipulate the config_pending semaphore. */ void config_pending_incr(device_t dev) { mutex_enter(&config_misc_lock); KASSERTMSG(dev->dv_pending < INT_MAX, "%s: excess config_pending_incr", device_xname(dev)); if (dev->dv_pending++ == 0) TAILQ_INSERT_TAIL(&config_pending, dev, dv_pending_list); #ifdef DEBUG_AUTOCONF printf("%s: %s %d\n", __func__, device_xname(dev), dev->dv_pending); #endif mutex_exit(&config_misc_lock); } void config_pending_decr(device_t dev) { mutex_enter(&config_misc_lock); KASSERTMSG(dev->dv_pending > 0, "%s: excess config_pending_decr", device_xname(dev)); if (--dev->dv_pending == 0) { TAILQ_REMOVE(&config_pending, dev, dv_pending_list); cv_broadcast(&config_misc_cv); } #ifdef DEBUG_AUTOCONF printf("%s: %s %d\n", __func__, device_xname(dev), dev->dv_pending); #endif mutex_exit(&config_misc_lock); } /* * Register a "finalization" routine. Finalization routines are * called iteratively once all real devices have been found during * autoconfiguration, for as long as any one finalizer has done * any work. */ int config_finalize_register(device_t dev, int (*fn)(device_t)) { struct finalize_hook *f; int error = 0; KERNEL_LOCK(1, NULL); /* * If finalization has already been done, invoke the * callback function now. */ if (config_finalize_done) { while ((*fn)(dev) != 0) /* loop */ ; goto out; } /* Ensure this isn't already on the list. */ TAILQ_FOREACH(f, &config_finalize_list, f_list) { if (f->f_func == fn && f->f_dev == dev) { error = EEXIST; goto out; } } f = kmem_alloc(sizeof(*f), KM_SLEEP); f->f_func = fn; f->f_dev = dev; TAILQ_INSERT_TAIL(&config_finalize_list, f, f_list); /* Success! */ error = 0; out: KERNEL_UNLOCK_ONE(NULL); return error; } void config_finalize(void) { struct finalize_hook *f; struct pdevinit *pdev; extern struct pdevinit pdevinit[]; int errcnt, rv; /* * Now that device driver threads have been created, wait for * them to finish any deferred autoconfiguration. */ mutex_enter(&config_misc_lock); while (!TAILQ_EMPTY(&config_pending)) { device_t dev; int error; error = cv_timedwait(&config_misc_cv, &config_misc_lock, mstohz(1000)); if (error == EWOULDBLOCK) { aprint_debug("waiting for devices:"); TAILQ_FOREACH(dev, &config_pending, dv_pending_list) aprint_debug(" %s", device_xname(dev)); aprint_debug("\n"); } } mutex_exit(&config_misc_lock); KERNEL_LOCK(1, NULL); /* Attach pseudo-devices. */ for (pdev = pdevinit; pdev->pdev_attach != NULL; pdev++) (*pdev->pdev_attach)(pdev->pdev_count); /* Run the hooks until none of them does any work. */ do { rv = 0; TAILQ_FOREACH(f, &config_finalize_list, f_list) rv |= (*f->f_func)(f->f_dev); } while (rv != 0); config_finalize_done = 1; /* Now free all the hooks. */ while ((f = TAILQ_FIRST(&config_finalize_list)) != NULL) { TAILQ_REMOVE(&config_finalize_list, f, f_list); kmem_free(f, sizeof(*f)); } KERNEL_UNLOCK_ONE(NULL); errcnt = aprint_get_error_count(); if ((boothowto & (AB_QUIET|AB_SILENT)) != 0 && (boothowto & AB_VERBOSE) == 0) { mutex_enter(&config_misc_lock); if (config_do_twiddle) { config_do_twiddle = 0; printf_nolog(" done.\n"); } mutex_exit(&config_misc_lock); } if (errcnt != 0) { printf("WARNING: %d error%s while detecting hardware; " "check system log.\n", errcnt, errcnt == 1 ? "" : "s"); } } void config_twiddle_init(void) { if ((boothowto & (AB_SILENT|AB_VERBOSE)) == AB_SILENT) { config_do_twiddle = 1; } callout_setfunc(&config_twiddle_ch, config_twiddle_fn, NULL); } void config_twiddle_fn(void *cookie) { mutex_enter(&config_misc_lock); if (config_do_twiddle) { twiddle(); callout_schedule(&config_twiddle_ch, mstohz(100)); } mutex_exit(&config_misc_lock); } static void config_alldevs_enter(struct alldevs_foray *af) { TAILQ_INIT(&af->af_garbage); mutex_enter(&alldevs_lock); config_collect_garbage(&af->af_garbage); } static void config_alldevs_exit(struct alldevs_foray *af) { mutex_exit(&alldevs_lock); config_dump_garbage(&af->af_garbage); } /* * device_lookup: * * Look up a device instance for a given driver. * * Caller is responsible for ensuring the device's state is * stable, either by holding a reference already obtained with * device_lookup_acquire or by otherwise ensuring the device is * attached and can't be detached (e.g., holding an open device * node and ensuring *_detach calls vdevgone). * * XXX Find a way to assert this. * * Safe for use up to and including interrupt context at IPL_VM. * Never sleeps. */ device_t device_lookup(cfdriver_t cd, int unit) { device_t dv; mutex_enter(&alldevs_lock); if (unit < 0 || unit >= cd->cd_ndevs) dv = NULL; else if ((dv = cd->cd_devs[unit]) != NULL && dv->dv_del_gen != 0) dv = NULL; mutex_exit(&alldevs_lock); return dv; } /* * device_lookup_private: * * Look up a softc instance for a given driver. */ void * device_lookup_private(cfdriver_t cd, int unit) { return device_private(device_lookup(cd, unit)); } /* * device_lookup_acquire: * * Look up a device instance for a given driver, and return a * reference to it that must be released by device_release. * * => If the device is still attaching, blocks until *_attach has * returned. * * => If the device is detaching, blocks until *_detach has * returned. May succeed or fail in that case, depending on * whether *_detach has backed out (EBUSY) or committed to * detaching. * * May sleep. */ device_t device_lookup_acquire(cfdriver_t cd, int unit) { device_t dv; ASSERT_SLEEPABLE(); /* XXX This should have a pserialized fast path -- TBD. */ mutex_enter(&config_misc_lock); mutex_enter(&alldevs_lock); retry: if (unit < 0 || unit >= cd->cd_ndevs || (dv = cd->cd_devs[unit]) == NULL || dv->dv_del_gen != 0 || dv->dv_detached) { dv = NULL; } else { /* * Wait for the device to stabilize, if attaching or * detaching. Either way we must wait for *_attach or * *_detach to complete, and either way we must retry: * even if detaching, *_detach might fail (EBUSY) so * the device may still be there. */ if ((dv->dv_attaching != NULL && dv->dv_attaching != curlwp) || dv->dv_detaching != NULL) { mutex_exit(&alldevs_lock); cv_wait(&config_misc_cv, &config_misc_lock); mutex_enter(&alldevs_lock); goto retry; } localcount_acquire(dv->dv_localcount); } mutex_exit(&alldevs_lock); mutex_exit(&config_misc_lock); return dv; } /* * device_release: * * Release a reference to a device acquired with * device_lookup_acquire. */ void device_release(device_t dv) { localcount_release(dv->dv_localcount, &config_misc_cv, &config_misc_lock); } /* * device_find_by_xname: * * Returns the device of the given name or NULL if it doesn't exist. */ device_t device_find_by_xname(const char *name) { device_t dv; deviter_t di; for (dv = deviter_first(&di, 0); dv != NULL; dv = deviter_next(&di)) { if (strcmp(device_xname(dv), name) == 0) break; } deviter_release(&di); return dv; } /* * device_find_by_driver_unit: * * Returns the device of the given driver name and unit or * NULL if it doesn't exist. */ device_t device_find_by_driver_unit(const char *name, int unit) { struct cfdriver *cd; if ((cd = config_cfdriver_lookup(name)) == NULL) return NULL; return device_lookup(cd, unit); } static bool match_strcmp(const char * const s1, const char * const s2) { return strcmp(s1, s2) == 0; } static bool match_pmatch(const char * const s1, const char * const s2) { return pmatch(s1, s2, NULL) == 2; } static bool strarray_match_internal(const char ** const strings, unsigned int const nstrings, const char * const str, unsigned int * const indexp, bool (*match_fn)(const char *, const char *)) { unsigned int i; if (strings == NULL || nstrings == 0) { return false; } for (i = 0; i < nstrings; i++) { if ((*match_fn)(strings[i], str)) { *indexp = i; return true; } } return false; } static int strarray_match(const char ** const strings, unsigned int const nstrings, const char * const str) { unsigned int idx; if (strarray_match_internal(strings, nstrings, str, &idx, match_strcmp)) { return (int)(nstrings - idx); } return 0; } static int strarray_pmatch(const char ** const strings, unsigned int const nstrings, const char * const pattern) { unsigned int idx; if (strarray_match_internal(strings, nstrings, pattern, &idx, match_pmatch)) { return (int)(nstrings - idx); } return 0; } static int device_compatible_match_strarray_internal( const char **device_compats, int ndevice_compats, const struct device_compatible_entry *driver_compats, const struct device_compatible_entry **matching_entryp, int (*match_fn)(const char **, unsigned int, const char *)) { const struct device_compatible_entry *dce = NULL; int rv; if (ndevice_compats == 0 || device_compats == NULL || driver_compats == NULL) return 0; for (dce = driver_compats; dce->compat != NULL; dce++) { rv = (*match_fn)(device_compats, ndevice_compats, dce->compat); if (rv != 0) { if (matching_entryp != NULL) { *matching_entryp = dce; } return rv; } } return 0; } /* * device_compatible_match: * * Match a driver's "compatible" data against a device's * "compatible" strings. Returns resulted weighted by * which device "compatible" string was matched. */ int device_compatible_match(const char **device_compats, int ndevice_compats, const struct device_compatible_entry *driver_compats) { return device_compatible_match_strarray_internal(device_compats, ndevice_compats, driver_compats, NULL, strarray_match); } /* * device_compatible_pmatch: * * Like device_compatible_match(), but uses pmatch(9) to compare * the device "compatible" strings against patterns in the * driver's "compatible" data. */ int device_compatible_pmatch(const char **device_compats, int ndevice_compats, const struct device_compatible_entry *driver_compats) { return device_compatible_match_strarray_internal(device_compats, ndevice_compats, driver_compats, NULL, strarray_pmatch); } static int device_compatible_match_strlist_internal( const char * const device_compats, size_t const device_compatsize, const struct device_compatible_entry *driver_compats, const struct device_compatible_entry **matching_entryp, int (*match_fn)(const char *, size_t, const char *)) { const struct device_compatible_entry *dce = NULL; int rv; if (device_compats == NULL || device_compatsize == 0 || driver_compats == NULL) return 0; for (dce = driver_compats; dce->compat != NULL; dce++) { rv = (*match_fn)(device_compats, device_compatsize, dce->compat); if (rv != 0) { if (matching_entryp != NULL) { *matching_entryp = dce; } return rv; } } return 0; } /* * device_compatible_match_strlist: * * Like device_compatible_match(), but take the device * "compatible" strings as an OpenFirmware-style string * list. */ int device_compatible_match_strlist( const char * const device_compats, size_t const device_compatsize, const struct device_compatible_entry *driver_compats) { return device_compatible_match_strlist_internal(device_compats, device_compatsize, driver_compats, NULL, strlist_match); } /* * device_compatible_pmatch_strlist: * * Like device_compatible_pmatch(), but take the device * "compatible" strings as an OpenFirmware-style string * list. */ int device_compatible_pmatch_strlist( const char * const device_compats, size_t const device_compatsize, const struct device_compatible_entry *driver_compats) { return device_compatible_match_strlist_internal(device_compats, device_compatsize, driver_compats, NULL, strlist_pmatch); } static int device_compatible_match_id_internal( uintptr_t const id, uintptr_t const mask, uintptr_t const sentinel_id, const struct device_compatible_entry *driver_compats, const struct device_compatible_entry **matching_entryp) { const struct device_compatible_entry *dce = NULL; if (mask == 0) return 0; for (dce = driver_compats; dce->id != sentinel_id; dce++) { if ((id & mask) == dce->id) { if (matching_entryp != NULL) { *matching_entryp = dce; } return 1; } } return 0; } /* * device_compatible_match_id: * * Like device_compatible_match(), but takes a single * unsigned integer device ID. */ int device_compatible_match_id( uintptr_t const id, uintptr_t const sentinel_id, const struct device_compatible_entry *driver_compats) { return device_compatible_match_id_internal(id, (uintptr_t)-1, sentinel_id, driver_compats, NULL); } /* * device_compatible_lookup: * * Look up and return the device_compatible_entry, using the * same matching criteria used by device_compatible_match(). */ const struct device_compatible_entry * device_compatible_lookup(const char **device_compats, int ndevice_compats, const struct device_compatible_entry *driver_compats) { const struct device_compatible_entry *dce; if (device_compatible_match_strarray_internal(device_compats, ndevice_compats, driver_compats, &dce, strarray_match)) { return dce; } return NULL; } /* * device_compatible_plookup: * * Look up and return the device_compatible_entry, using the * same matching criteria used by device_compatible_pmatch(). */ const struct device_compatible_entry * device_compatible_plookup(const char **device_compats, int ndevice_compats, const struct device_compatible_entry *driver_compats) { const struct device_compatible_entry *dce; if (device_compatible_match_strarray_internal(device_compats, ndevice_compats, driver_compats, &dce, strarray_pmatch)) { return dce; } return NULL; } /* * device_compatible_lookup_strlist: * * Like device_compatible_lookup(), but take the device * "compatible" strings as an OpenFirmware-style string * list. */ const struct device_compatible_entry * device_compatible_lookup_strlist( const char * const device_compats, size_t const device_compatsize, const struct device_compatible_entry *driver_compats) { const struct device_compatible_entry *dce; if (device_compatible_match_strlist_internal(device_compats, device_compatsize, driver_compats, &dce, strlist_match)) { return dce; } return NULL; } /* * device_compatible_plookup_strlist: * * Like device_compatible_plookup(), but take the device * "compatible" strings as an OpenFirmware-style string * list. */ const struct device_compatible_entry * device_compatible_plookup_strlist( const char * const device_compats, size_t const device_compatsize, const struct device_compatible_entry *driver_compats) { const struct device_compatible_entry *dce; if (device_compatible_match_strlist_internal(device_compats, device_compatsize, driver_compats, &dce, strlist_pmatch)) { return dce; } return NULL; } /* * device_compatible_lookup_id: * * Like device_compatible_lookup(), but takes a single * unsigned integer device ID. */ const struct device_compatible_entry * device_compatible_lookup_id( uintptr_t const id, uintptr_t const sentinel_id, const struct device_compatible_entry *driver_compats) { const struct device_compatible_entry *dce; if (device_compatible_match_id_internal(id, (uintptr_t)-1, sentinel_id, driver_compats, &dce)) { return dce; } return NULL; } /* * Power management related functions. */ bool device_pmf_is_registered(device_t dev) { return (dev->dv_flags & DVF_POWER_HANDLERS) != 0; } bool device_pmf_driver_suspend(device_t dev, const pmf_qual_t *qual) { if ((dev->dv_flags & DVF_DRIVER_SUSPENDED) != 0) return true; if ((dev->dv_flags & DVF_CLASS_SUSPENDED) == 0) return false; if (pmf_qual_depth(qual) <= DEVACT_LEVEL_DRIVER && dev->dv_driver_suspend != NULL && !(*dev->dv_driver_suspend)(dev, qual)) return false; dev->dv_flags |= DVF_DRIVER_SUSPENDED; return true; } bool device_pmf_driver_resume(device_t dev, const pmf_qual_t *qual) { if ((dev->dv_flags & DVF_DRIVER_SUSPENDED) == 0) return true; if ((dev->dv_flags & DVF_BUS_SUSPENDED) != 0) return false; if (pmf_qual_depth(qual) <= DEVACT_LEVEL_DRIVER && dev->dv_driver_resume != NULL && !(*dev->dv_driver_resume)(dev, qual)) return false; dev->dv_flags &= ~DVF_DRIVER_SUSPENDED; return true; } bool device_pmf_driver_shutdown(device_t dev, int how) { if (*dev->dv_driver_shutdown != NULL && !(*dev->dv_driver_shutdown)(dev, how)) return false; return true; } bool device_pmf_driver_register(device_t dev, bool (*suspend)(device_t, const pmf_qual_t *), bool (*resume)(device_t, const pmf_qual_t *), bool (*shutdown)(device_t, int)) { dev->dv_driver_suspend = suspend; dev->dv_driver_resume = resume; dev->dv_driver_shutdown = shutdown; dev->dv_flags |= DVF_POWER_HANDLERS; return true; } void device_pmf_driver_deregister(device_t dev) { device_lock_t dvl = device_getlock(dev); dev->dv_driver_suspend = NULL; dev->dv_driver_resume = NULL; mutex_enter(&dvl->dvl_mtx); dev->dv_flags &= ~DVF_POWER_HANDLERS; while (dvl->dvl_nlock > 0 || dvl->dvl_nwait > 0) { /* Wake a thread that waits for the lock. That * thread will fail to acquire the lock, and then * it will wake the next thread that waits for the * lock, or else it will wake us. */ cv_signal(&dvl->dvl_cv); pmflock_debug(dev, __func__, __LINE__); cv_wait(&dvl->dvl_cv, &dvl->dvl_mtx); pmflock_debug(dev, __func__, __LINE__); } mutex_exit(&dvl->dvl_mtx); } bool device_pmf_driver_child_register(device_t dev) { device_t parent = device_parent(dev); if (parent == NULL || parent->dv_driver_child_register == NULL) return true; return (*parent->dv_driver_child_register)(dev); } void device_pmf_driver_set_child_register(device_t dev, bool (*child_register)(device_t)) { dev->dv_driver_child_register = child_register; } static void pmflock_debug(device_t dev, const char *func, int line) { #ifdef PMFLOCK_DEBUG device_lock_t dvl = device_getlock(dev); const char *curlwp_name; if (curlwp->l_name != NULL) curlwp_name = curlwp->l_name; else curlwp_name = curlwp->l_proc->p_comm; aprint_debug_dev(dev, "%s.%d, %s dvl_nlock %d dvl_nwait %d dv_flags %x\n", func, line, curlwp_name, dvl->dvl_nlock, dvl->dvl_nwait, dev->dv_flags); #endif /* PMFLOCK_DEBUG */ } static bool device_pmf_lock1(device_t dev) { device_lock_t dvl = device_getlock(dev); while (device_pmf_is_registered(dev) && dvl->dvl_nlock > 0 && dvl->dvl_holder != curlwp) { dvl->dvl_nwait++; pmflock_debug(dev, __func__, __LINE__); cv_wait(&dvl->dvl_cv, &dvl->dvl_mtx); pmflock_debug(dev, __func__, __LINE__); dvl->dvl_nwait--; } if (!device_pmf_is_registered(dev)) { pmflock_debug(dev, __func__, __LINE__); /* We could not acquire the lock, but some other thread may * wait for it, also. Wake that thread. */ cv_signal(&dvl->dvl_cv); return false; } dvl->dvl_nlock++; dvl->dvl_holder = curlwp; pmflock_debug(dev, __func__, __LINE__); return true; } bool device_pmf_lock(device_t dev) { bool rc; device_lock_t dvl = device_getlock(dev); mutex_enter(&dvl->dvl_mtx); rc = device_pmf_lock1(dev); mutex_exit(&dvl->dvl_mtx); return rc; } void device_pmf_unlock(device_t dev) { device_lock_t dvl = device_getlock(dev); KASSERT(dvl->dvl_nlock > 0); mutex_enter(&dvl->dvl_mtx); if (--dvl->dvl_nlock == 0) dvl->dvl_holder = NULL; cv_signal(&dvl->dvl_cv); pmflock_debug(dev, __func__, __LINE__); mutex_exit(&dvl->dvl_mtx); } device_lock_t device_getlock(device_t dev) { return &dev->dv_lock; } void * device_pmf_bus_private(device_t dev) { return dev->dv_bus_private; } bool device_pmf_bus_suspend(device_t dev, const pmf_qual_t *qual) { if ((dev->dv_flags & DVF_BUS_SUSPENDED) != 0) return true; if ((dev->dv_flags & DVF_CLASS_SUSPENDED) == 0 || (dev->dv_flags & DVF_DRIVER_SUSPENDED) == 0) return false; if (pmf_qual_depth(qual) <= DEVACT_LEVEL_BUS && dev->dv_bus_suspend != NULL && !(*dev->dv_bus_suspend)(dev, qual)) return false; dev->dv_flags |= DVF_BUS_SUSPENDED; return true; } bool device_pmf_bus_resume(device_t dev, const pmf_qual_t *qual) { if ((dev->dv_flags & DVF_BUS_SUSPENDED) == 0) return true; if (pmf_qual_depth(qual) <= DEVACT_LEVEL_BUS && dev->dv_bus_resume != NULL && !(*dev->dv_bus_resume)(dev, qual)) return false; dev->dv_flags &= ~DVF_BUS_SUSPENDED; return true; } bool device_pmf_bus_shutdown(device_t dev, int how) { if (*dev->dv_bus_shutdown != NULL && !(*dev->dv_bus_shutdown)(dev, how)) return false; return true; } void device_pmf_bus_register(device_t dev, void *priv, bool (*suspend)(device_t, const pmf_qual_t *), bool (*resume)(device_t, const pmf_qual_t *), bool (*shutdown)(device_t, int), void (*deregister)(device_t)) { dev->dv_bus_private = priv; dev->dv_bus_resume = resume; dev->dv_bus_suspend = suspend; dev->dv_bus_shutdown = shutdown; dev->dv_bus_deregister = deregister; } void device_pmf_bus_deregister(device_t dev) { if (dev->dv_bus_deregister == NULL) return; (*dev->dv_bus_deregister)(dev); dev->dv_bus_private = NULL; dev->dv_bus_suspend = NULL; dev->dv_bus_resume = NULL; dev->dv_bus_deregister = NULL; } void * device_pmf_class_private(device_t dev) { return dev->dv_class_private; } bool device_pmf_class_suspend(device_t dev, const pmf_qual_t *qual) { if ((dev->dv_flags & DVF_CLASS_SUSPENDED) != 0) return true; if (pmf_qual_depth(qual) <= DEVACT_LEVEL_CLASS && dev->dv_class_suspend != NULL && !(*dev->dv_class_suspend)(dev, qual)) return false; dev->dv_flags |= DVF_CLASS_SUSPENDED; return true; } bool device_pmf_class_resume(device_t dev, const pmf_qual_t *qual) { if ((dev->dv_flags & DVF_CLASS_SUSPENDED) == 0) return true; if ((dev->dv_flags & DVF_BUS_SUSPENDED) != 0 || (dev->dv_flags & DVF_DRIVER_SUSPENDED) != 0) return false; if (pmf_qual_depth(qual) <= DEVACT_LEVEL_CLASS && dev->dv_class_resume != NULL && !(*dev->dv_class_resume)(dev, qual)) return false; dev->dv_flags &= ~DVF_CLASS_SUSPENDED; return true; } void device_pmf_class_register(device_t dev, void *priv, bool (*suspend)(device_t, const pmf_qual_t *), bool (*resume)(device_t, const pmf_qual_t *), void (*deregister)(device_t)) { dev->dv_class_private = priv; dev->dv_class_suspend = suspend; dev->dv_class_resume = resume; dev->dv_class_deregister = deregister; } void device_pmf_class_deregister(device_t dev) { if (dev->dv_class_deregister == NULL) return; (*dev->dv_class_deregister)(dev); dev->dv_class_private = NULL; dev->dv_class_suspend = NULL; dev->dv_class_resume = NULL; dev->dv_class_deregister = NULL; } bool device_active(device_t dev, devactive_t type) { size_t i; if (dev->dv_activity_count == 0) return false; for (i = 0; i < dev->dv_activity_count; ++i) { if (dev->dv_activity_handlers[i] == NULL) break; (*dev->dv_activity_handlers[i])(dev, type); } return true; } bool device_active_register(device_t dev, void (*handler)(device_t, devactive_t)) { void (**new_handlers)(device_t, devactive_t); void (**old_handlers)(device_t, devactive_t); size_t i, old_size, new_size; int s; old_handlers = dev->dv_activity_handlers; old_size = dev->dv_activity_count; KASSERT(old_size == 0 || old_handlers != NULL); for (i = 0; i < old_size; ++i) { KASSERT(old_handlers[i] != handler); if (old_handlers[i] == NULL) { old_handlers[i] = handler; return true; } } new_size = old_size + 4; new_handlers = kmem_alloc(sizeof(void *) * new_size, KM_SLEEP); for (i = 0; i < old_size; ++i) new_handlers[i] = old_handlers[i]; new_handlers[old_size] = handler; for (i = old_size+1; i < new_size; ++i) new_handlers[i] = NULL; s = splhigh(); dev->dv_activity_count = new_size; dev->dv_activity_handlers = new_handlers; splx(s); if (old_size > 0) kmem_free(old_handlers, sizeof(void *) * old_size); return true; } void device_active_deregister(device_t dev, void (*handler)(device_t, devactive_t)) { void (**old_handlers)(device_t, devactive_t); size_t i, old_size; int s; old_handlers = dev->dv_activity_handlers; old_size = dev->dv_activity_count; for (i = 0; i < old_size; ++i) { if (old_handlers[i] == handler) break; if (old_handlers[i] == NULL) return; /* XXX panic? */ } if (i == old_size) return; /* XXX panic? */ for (; i < old_size - 1; ++i) { if ((old_handlers[i] = old_handlers[i + 1]) != NULL) continue; if (i == 0) { s = splhigh(); dev->dv_activity_count = 0; dev->dv_activity_handlers = NULL; splx(s); kmem_free(old_handlers, sizeof(void *) * old_size); } return; } old_handlers[i] = NULL; } /* Return true iff the device_t `dev' exists at generation `gen'. */ static bool device_exists_at(device_t dv, devgen_t gen) { return (dv->dv_del_gen == 0 || dv->dv_del_gen > gen) && dv->dv_add_gen <= gen; } static bool deviter_visits(const deviter_t *di, device_t dv) { return device_exists_at(dv, di->di_gen); } /* * Device Iteration * * deviter_t: a device iterator. Holds state for a "walk" visiting * each device_t's in the device tree. * * deviter_init(di, flags): initialize the device iterator `di' * to "walk" the device tree. deviter_next(di) will return * the first device_t in the device tree, or NULL if there are * no devices. * * `flags' is one or more of DEVITER_F_RW, indicating that the * caller intends to modify the device tree by calling * config_detach(9) on devices in the order that the iterator * returns them; DEVITER_F_ROOT_FIRST, asking for the devices * nearest the "root" of the device tree to be returned, first; * DEVITER_F_LEAVES_FIRST, asking for the devices furthest from * the root of the device tree, first; and DEVITER_F_SHUTDOWN, * indicating both that deviter_init() should not respect any * locks on the device tree, and that deviter_next(di) may run * in more than one LWP before the walk has finished. * * Only one DEVITER_F_RW iterator may be in the device tree at * once. * * DEVITER_F_SHUTDOWN implies DEVITER_F_RW. * * Results are undefined if the flags DEVITER_F_ROOT_FIRST and * DEVITER_F_LEAVES_FIRST are used in combination. * * deviter_first(di, flags): initialize the device iterator `di' * and return the first device_t in the device tree, or NULL * if there are no devices. The statement * * dv = deviter_first(di); * * is shorthand for * * deviter_init(di); * dv = deviter_next(di); * * deviter_next(di): return the next device_t in the device tree, * or NULL if there are no more devices. deviter_next(di) * is undefined if `di' was not initialized with deviter_init() or * deviter_first(). * * deviter_release(di): stops iteration (subsequent calls to * deviter_next() will return NULL), releases any locks and * resources held by the device iterator. * * Device iteration does not return device_t's in any particular * order. An iterator will never return the same device_t twice. * Device iteration is guaranteed to complete---i.e., if deviter_next(di) * is called repeatedly on the same `di', it will eventually return * NULL. It is ok to attach/detach devices during device iteration. */ void deviter_init(deviter_t *di, deviter_flags_t flags) { device_t dv; memset(di, 0, sizeof(*di)); if ((flags & DEVITER_F_SHUTDOWN) != 0) flags |= DEVITER_F_RW; mutex_enter(&alldevs_lock); if ((flags & DEVITER_F_RW) != 0) alldevs_nwrite++; else alldevs_nread++; di->di_gen = alldevs_gen++; di->di_flags = flags; switch (di->di_flags & (DEVITER_F_LEAVES_FIRST|DEVITER_F_ROOT_FIRST)) { case DEVITER_F_LEAVES_FIRST: TAILQ_FOREACH(dv, &alldevs, dv_list) { if (!deviter_visits(di, dv)) continue; di->di_curdepth = MAX(di->di_curdepth, dv->dv_depth); } break; case DEVITER_F_ROOT_FIRST: TAILQ_FOREACH(dv, &alldevs, dv_list) { if (!deviter_visits(di, dv)) continue; di->di_maxdepth = MAX(di->di_maxdepth, dv->dv_depth); } break; default: break; } deviter_reinit(di); mutex_exit(&alldevs_lock); } static void deviter_reinit(deviter_t *di) { KASSERT(mutex_owned(&alldevs_lock)); if ((di->di_flags & DEVITER_F_RW) != 0) di->di_prev = TAILQ_LAST(&alldevs, devicelist); else di->di_prev = TAILQ_FIRST(&alldevs); } device_t deviter_first(deviter_t *di, deviter_flags_t flags) { deviter_init(di, flags); return deviter_next(di); } static device_t deviter_next2(deviter_t *di) { device_t dv; KASSERT(mutex_owned(&alldevs_lock)); dv = di->di_prev; if (dv == NULL) return NULL; if ((di->di_flags & DEVITER_F_RW) != 0) di->di_prev = TAILQ_PREV(dv, devicelist, dv_list); else di->di_prev = TAILQ_NEXT(dv, dv_list); return dv; } static device_t deviter_next1(deviter_t *di) { device_t dv; KASSERT(mutex_owned(&alldevs_lock)); do { dv = deviter_next2(di); } while (dv != NULL && !deviter_visits(di, dv)); return dv; } device_t deviter_next(deviter_t *di) { device_t dv = NULL; mutex_enter(&alldevs_lock); switch (di->di_flags & (DEVITER_F_LEAVES_FIRST|DEVITER_F_ROOT_FIRST)) { case 0: dv = deviter_next1(di); break; case DEVITER_F_LEAVES_FIRST: while (di->di_curdepth >= 0) { if ((dv = deviter_next1(di)) == NULL) { di->di_curdepth--; deviter_reinit(di); } else if (dv->dv_depth == di->di_curdepth) break; } break; case DEVITER_F_ROOT_FIRST: while (di->di_curdepth <= di->di_maxdepth) { if ((dv = deviter_next1(di)) == NULL) { di->di_curdepth++; deviter_reinit(di); } else if (dv->dv_depth == di->di_curdepth) break; } break; default: break; } mutex_exit(&alldevs_lock); return dv; } void deviter_release(deviter_t *di) { bool rw = (di->di_flags & DEVITER_F_RW) != 0; mutex_enter(&alldevs_lock); if (rw) --alldevs_nwrite; else --alldevs_nread; /* XXX wake a garbage-collection thread */ mutex_exit(&alldevs_lock); } const char * cfdata_ifattr(const struct cfdata *cf) { return cf->cf_pspec->cfp_iattr; } bool ifattr_match(const char *snull, const char *t) { return (snull == NULL) || strcmp(snull, t) == 0; } void null_childdetached(device_t self, device_t child) { /* do nothing */ } static void sysctl_detach_setup(struct sysctllog **clog) { sysctl_createv(clog, 0, NULL, NULL, CTLFLAG_PERMANENT | CTLFLAG_READWRITE, CTLTYPE_BOOL, "detachall", SYSCTL_DESCR("Detach all devices at shutdown"), NULL, 0, &detachall, 0, CTL_KERN, CTL_CREATE, CTL_EOL); } |
| 2 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 197 198 199 200 201 202 203 204 205 206 207 208 209 210 211 212 213 214 215 216 217 218 219 220 221 222 223 224 225 226 227 228 229 230 231 232 233 234 235 236 237 238 239 240 241 242 243 244 245 246 247 248 249 250 251 252 253 254 255 256 257 258 259 260 261 262 263 264 265 266 267 268 269 270 271 272 273 274 275 276 277 278 279 280 281 282 283 284 285 286 287 288 289 290 291 292 293 294 295 296 297 298 299 300 301 302 303 304 305 306 307 308 309 310 311 312 313 314 315 316 317 318 319 320 321 322 323 324 325 326 327 328 329 330 331 332 333 334 335 336 337 338 339 340 341 342 343 344 345 346 347 348 349 350 351 352 353 354 355 356 357 358 359 360 361 362 363 364 365 366 367 368 369 370 371 372 373 374 375 376 377 378 379 380 381 382 383 384 385 386 387 388 389 390 391 392 393 394 395 396 397 398 399 400 401 402 403 404 405 406 407 408 409 410 411 412 413 414 415 416 417 418 419 420 421 422 423 424 425 426 427 428 429 430 431 432 433 434 435 436 437 438 439 440 441 442 443 444 445 446 447 448 449 450 451 452 453 454 455 456 457 458 459 460 461 462 463 464 465 466 467 468 469 470 471 472 473 474 475 476 477 478 479 480 481 482 483 484 485 486 487 488 489 490 491 492 493 494 495 496 497 498 499 500 501 502 503 504 505 506 507 508 509 510 511 512 513 514 515 516 517 518 519 520 521 522 523 524 525 526 527 528 529 530 531 532 533 534 535 536 537 538 539 540 541 542 543 544 545 546 547 548 549 550 551 552 553 554 555 556 557 558 559 560 561 562 563 564 565 566 567 568 569 570 571 572 573 574 575 576 577 578 579 580 581 582 583 584 585 586 587 588 589 590 591 592 593 594 595 596 597 598 599 600 601 602 603 604 605 606 607 608 609 610 611 612 613 614 615 616 617 618 619 620 621 622 623 624 625 626 627 628 629 630 631 632 633 634 635 636 637 638 639 640 641 642 643 644 645 646 647 648 649 650 651 652 653 654 655 656 657 658 659 660 661 662 663 664 665 666 667 668 669 670 671 672 673 674 675 676 677 678 679 680 681 682 683 684 685 686 687 688 689 690 691 692 693 694 695 696 697 698 699 700 701 702 703 704 705 706 707 708 709 710 711 712 713 714 715 716 717 718 719 720 721 722 723 724 725 726 727 728 729 730 731 732 733 734 735 736 737 738 739 740 741 742 743 744 745 746 747 748 749 750 751 752 753 754 755 756 757 758 759 760 761 762 763 764 765 766 767 768 769 770 771 772 773 774 775 776 777 778 779 780 781 782 783 784 785 786 787 788 789 790 791 792 793 794 795 796 797 798 799 800 801 802 803 804 805 806 807 808 809 810 811 812 813 814 | /* $NetBSD: rf_dagfuncs.c,v 1.35 2021/08/07 16:19:15 thorpej Exp $ */ /* * Copyright (c) 1995 Carnegie-Mellon University. * All rights reserved. * * Author: Mark Holland, William V. Courtright II * * Permission to use, copy, modify and distribute this software and * its documentation is hereby granted, provided that both the copyright * notice and this permission notice appear in all copies of the * software, derivative works or modified versions, and any portions * thereof, and that both notices appear in supporting documentation. * * CARNEGIE MELLON ALLOWS FREE USE OF THIS SOFTWARE IN ITS "AS IS" * CONDITION. CARNEGIE MELLON DISCLAIMS ANY LIABILITY OF ANY KIND * FOR ANY DAMAGES WHATSOEVER RESULTING FROM THE USE OF THIS SOFTWARE. * * Carnegie Mellon requests users of this software to return to * * Software Distribution Coordinator or Software.Distribution@CS.CMU.EDU * School of Computer Science * Carnegie Mellon University * Pittsburgh PA 15213-3890 * * any improvements or extensions that they make and grant Carnegie the * rights to redistribute these changes. */ /* * dagfuncs.c -- DAG node execution routines * * Rules: * 1. Every DAG execution function must eventually cause node->status to * get set to "good" or "bad", and "FinishNode" to be called. In the * case of nodes that complete immediately (xor, NullNodeFunc, etc), * the node execution function can do these two things directly. In * the case of nodes that have to wait for some event (a disk read to * complete, a lock to be released, etc) to occur before they can * complete, this is typically achieved by having whatever module * is doing the operation call GenericWakeupFunc upon completion. * 2. DAG execution functions should check the status in the DAG header * and NOP out their operations if the status is not "enable". However, * execution functions that release resources must be sure to release * them even when they NOP out the function that would use them. * Functions that acquire resources should go ahead and acquire them * even when they NOP, so that a downstream release node will not have * to check to find out whether or not the acquire was suppressed. */ #include <sys/cdefs.h> __KERNEL_RCSID(0, "$NetBSD: rf_dagfuncs.c,v 1.35 2021/08/07 16:19:15 thorpej Exp $"); #include <sys/param.h> #include <sys/ioctl.h> #include "rf_archs.h" #include "rf_raid.h" #include "rf_dag.h" #include "rf_layout.h" #include "rf_etimer.h" #include "rf_acctrace.h" #include "rf_diskqueue.h" #include "rf_dagfuncs.h" #include "rf_general.h" #include "rf_engine.h" #include "rf_dagutils.h" #include "rf_kintf.h" #if RF_INCLUDE_PARITYLOGGING > 0 #include "rf_paritylog.h" #endif /* RF_INCLUDE_PARITYLOGGING > 0 */ void (*rf_DiskReadFunc) (RF_DagNode_t *); void (*rf_DiskWriteFunc) (RF_DagNode_t *); void (*rf_DiskReadUndoFunc) (RF_DagNode_t *); void (*rf_DiskWriteUndoFunc) (RF_DagNode_t *); void (*rf_RegularXorUndoFunc) (RF_DagNode_t *); void (*rf_SimpleXorUndoFunc) (RF_DagNode_t *); void (*rf_RecoveryXorUndoFunc) (RF_DagNode_t *); /***************************************************************************** * main (only) configuration routine for this module ****************************************************************************/ int rf_ConfigureDAGFuncs(RF_ShutdownList_t **listp) { RF_ASSERT(((sizeof(long) == 8) && RF_LONGSHIFT == 3) || ((sizeof(long) == 4) && RF_LONGSHIFT == 2)); rf_DiskReadFunc = rf_DiskReadFuncForThreads; rf_DiskReadUndoFunc = rf_DiskUndoFunc; rf_DiskWriteFunc = rf_DiskWriteFuncForThreads; rf_DiskWriteUndoFunc = rf_DiskUndoFunc; rf_RegularXorUndoFunc = rf_NullNodeUndoFunc; rf_SimpleXorUndoFunc = rf_NullNodeUndoFunc; rf_RecoveryXorUndoFunc = rf_NullNodeUndoFunc; return (0); } /***************************************************************************** * the execution function associated with a terminate node ****************************************************************************/ void rf_TerminateFunc(RF_DagNode_t *node) { RF_ASSERT(node->dagHdr->numCommits == node->dagHdr->numCommitNodes); node->status = rf_good; rf_FinishNode(node, RF_THREAD_CONTEXT); } void rf_TerminateUndoFunc(RF_DagNode_t *node) { } /***************************************************************************** * execution functions associated with a mirror node * * parameters: * * 0 - physical disk address of data * 1 - buffer for holding read data * 2 - parity stripe ID * 3 - flags * 4 - physical disk address of mirror (parity) * ****************************************************************************/ void rf_DiskReadMirrorIdleFunc(RF_DagNode_t *node) { /* select the mirror copy with the shortest queue and fill in node * parameters with physical disk address */ rf_SelectMirrorDiskIdle(node); rf_DiskReadFunc(node); } #if (RF_INCLUDE_CHAINDECLUSTER > 0) || (RF_INCLUDE_INTERDECLUSTER > 0) || (RF_DEBUG_VALIDATE_DAG > 0) void rf_DiskReadMirrorPartitionFunc(RF_DagNode_t *node) { /* select the mirror copy with the shortest queue and fill in node * parameters with physical disk address */ rf_SelectMirrorDiskPartition(node); rf_DiskReadFunc(node); } #endif void rf_DiskReadMirrorUndoFunc(RF_DagNode_t *node) { } #if RF_INCLUDE_PARITYLOGGING > 0 /***************************************************************************** * the execution function associated with a parity log update node ****************************************************************************/ void rf_ParityLogUpdateFunc(RF_DagNode_t *node) { RF_PhysDiskAddr_t *pda = (RF_PhysDiskAddr_t *) node->params[0].p; void *bf = (void *) node->params[1].p; RF_ParityLogData_t *logData; #if RF_ACC_TRACE > 0 RF_AccTraceEntry_t *tracerec = node->dagHdr->tracerec; RF_Etimer_t timer; #endif if (node->dagHdr->status == rf_enable) { #if RF_ACC_TRACE > 0 RF_ETIMER_START(timer); #endif logData = rf_CreateParityLogData(RF_UPDATE, pda, bf, (RF_Raid_t *) (node->dagHdr->raidPtr), node->wakeFunc, node, node->dagHdr->tracerec, timer); if (logData) rf_ParityLogAppend(logData, RF_FALSE, NULL, RF_FALSE); else { #if RF_ACC_TRACE > 0 RF_ETIMER_STOP(timer); RF_ETIMER_EVAL(timer); tracerec->plog_us += RF_ETIMER_VAL_US(timer); #endif (node->wakeFunc) (node, ENOMEM); } } } /***************************************************************************** * the execution function associated with a parity log overwrite node ****************************************************************************/ void rf_ParityLogOverwriteFunc(RF_DagNode_t *node) { RF_PhysDiskAddr_t *pda = (RF_PhysDiskAddr_t *) node->params[0].p; void *bf = (void *) node->params[1].p; RF_ParityLogData_t *logData; #if RF_ACC_TRACE > 0 RF_AccTraceEntry_t *tracerec = node->dagHdr->tracerec; RF_Etimer_t timer; #endif if (node->dagHdr->status == rf_enable) { #if RF_ACC_TRACE > 0 RF_ETIMER_START(timer); #endif logData = rf_CreateParityLogData(RF_OVERWRITE, pda, bf, (RF_Raid_t *) (node->dagHdr->raidPtr), node->wakeFunc, node, node->dagHdr->tracerec, timer); if (logData) rf_ParityLogAppend(logData, RF_FALSE, NULL, RF_FALSE); else { #if RF_ACC_TRACE > 0 RF_ETIMER_STOP(timer); RF_ETIMER_EVAL(timer); tracerec->plog_us += RF_ETIMER_VAL_US(timer); #endif (node->wakeFunc) (node, ENOMEM); } } } void rf_ParityLogUpdateUndoFunc(RF_DagNode_t *node) { } void rf_ParityLogOverwriteUndoFunc(RF_DagNode_t *node) { } #endif /* RF_INCLUDE_PARITYLOGGING > 0 */ /***************************************************************************** * the execution function associated with a NOP node ****************************************************************************/ void rf_NullNodeFunc(RF_DagNode_t *node) { node->status = rf_good; rf_FinishNode(node, RF_THREAD_CONTEXT); } void rf_NullNodeUndoFunc(RF_DagNode_t *node) { node->status = rf_undone; rf_FinishNode(node, RF_THREAD_CONTEXT); } /***************************************************************************** * the execution function associated with a disk-read node ****************************************************************************/ void rf_DiskReadFuncForThreads(RF_DagNode_t *node) { RF_DiskQueueData_t *req; RF_PhysDiskAddr_t *pda = (RF_PhysDiskAddr_t *) node->params[0].p; void *bf = (void *) node->params[1].p; RF_StripeNum_t parityStripeID = (RF_StripeNum_t) node->params[2].v; unsigned priority = RF_EXTRACT_PRIORITY(node->params[3].v); unsigned which_ru = RF_EXTRACT_RU(node->params[3].v); RF_IoType_t iotype = (node->dagHdr->status == rf_enable) ? RF_IO_TYPE_READ : RF_IO_TYPE_NOP; RF_DiskQueue_t *dqs = ((RF_Raid_t *) (node->dagHdr->raidPtr))->Queues; req = rf_CreateDiskQueueData(iotype, pda->startSector, pda->numSector, bf, parityStripeID, which_ru, node->wakeFunc, node, #if RF_ACC_TRACE > 0 node->dagHdr->tracerec, #else NULL, #endif (void *) (node->dagHdr->raidPtr), 0, node->dagHdr->bp); node->dagFuncData = (void *) req; rf_DiskIOEnqueue(&(dqs[pda->col]), req, priority); } /***************************************************************************** * the execution function associated with a disk-write node ****************************************************************************/ void rf_DiskWriteFuncForThreads(RF_DagNode_t *node) { RF_DiskQueueData_t *req; RF_PhysDiskAddr_t *pda = (RF_PhysDiskAddr_t *) node->params[0].p; void *bf = (void *) node->params[1].p; RF_StripeNum_t parityStripeID = (RF_StripeNum_t) node->params[2].v; unsigned priority = RF_EXTRACT_PRIORITY(node->params[3].v); unsigned which_ru = RF_EXTRACT_RU(node->params[3].v); RF_IoType_t iotype = (node->dagHdr->status == rf_enable) ? RF_IO_TYPE_WRITE : RF_IO_TYPE_NOP; RF_DiskQueue_t *dqs = ((RF_Raid_t *) (node->dagHdr->raidPtr))->Queues; /* normal processing (rollaway or forward recovery) begins here */ req = rf_CreateDiskQueueData(iotype, pda->startSector, pda->numSector, bf, parityStripeID, which_ru, node->wakeFunc, node, #if RF_ACC_TRACE > 0 node->dagHdr->tracerec, #else NULL, #endif (void *) (node->dagHdr->raidPtr), 0, node->dagHdr->bp); node->dagFuncData = (void *) req; rf_DiskIOEnqueue(&(dqs[pda->col]), req, priority); } /***************************************************************************** * the undo function for disk nodes * Note: this is not a proper undo of a write node, only locks are released. * old data is not restored to disk! ****************************************************************************/ void rf_DiskUndoFunc(RF_DagNode_t *node) { RF_DiskQueueData_t *req; RF_PhysDiskAddr_t *pda = (RF_PhysDiskAddr_t *) node->params[0].p; RF_DiskQueue_t *dqs = ((RF_Raid_t *) (node->dagHdr->raidPtr))->Queues; req = rf_CreateDiskQueueData(RF_IO_TYPE_NOP, 0L, 0, NULL, 0L, 0, node->wakeFunc, node, #if RF_ACC_TRACE > 0 node->dagHdr->tracerec, #else NULL, #endif (void *) (node->dagHdr->raidPtr), 0, NULL); node->dagFuncData = (void *) req; rf_DiskIOEnqueue(&(dqs[pda->col]), req, RF_IO_NORMAL_PRIORITY); } /***************************************************************************** * Callback routine for DiskRead and DiskWrite nodes. When the disk * op completes, the routine is called to set the node status and * inform the execution engine that the node has fired. ****************************************************************************/ void rf_GenericWakeupFunc(void *v, int status) { RF_DagNode_t *node = v; switch (node->status) { case rf_fired: if (status) node->status = rf_bad; else node->status = rf_good; break; case rf_recover: /* probably should never reach this case */ if (status) node->status = rf_panic; else node->status = rf_undone; break; default: printf("rf_GenericWakeupFunc:"); printf("node->status is %d,", node->status); printf("status is %d \n", status); RF_PANIC(); break; } if (node->dagFuncData) rf_FreeDiskQueueData((RF_DiskQueueData_t *) node->dagFuncData); rf_FinishNode(node, RF_INTR_CONTEXT); } /***************************************************************************** * there are three distinct types of xor nodes: * A "regular xor" is used in the fault-free case where the access * spans a complete stripe unit. It assumes that the result buffer is * one full stripe unit in size, and uses the stripe-unit-offset * values that it computes from the PDAs to determine where within the * stripe unit to XOR each argument buffer. * * A "simple xor" is used in the fault-free case where the access * touches only a portion of one (or two, in some cases) stripe * unit(s). It assumes that all the argument buffers are of the same * size and have the same stripe unit offset. * * A "recovery xor" is used in the degraded-mode case. It's similar * to the regular xor function except that it takes the failed PDA as * an additional parameter, and uses it to determine what portions of * the argument buffers need to be xor'd into the result buffer, and * where in the result buffer they should go. ****************************************************************************/ /* xor the params together and store the result in the result field. * assume the result field points to a buffer that is the size of one * SU, and use the pda params to determine where within the buffer to * XOR the input buffers. */ void rf_RegularXorFunc(RF_DagNode_t *node) { RF_Raid_t *raidPtr = (RF_Raid_t *) node->params[node->numParams - 1].p; #if RF_ACC_TRACE > 0 RF_AccTraceEntry_t *tracerec = node->dagHdr->tracerec; RF_Etimer_t timer; #endif int i, retcode; retcode = 0; if (node->dagHdr->status == rf_enable) { /* don't do the XOR if the input is the same as the output */ #if RF_ACC_TRACE > 0 RF_ETIMER_START(timer); #endif for (i = 0; i < node->numParams - 1; i += 2) if (node->params[i + 1].p != node->results[0]) { retcode = rf_XorIntoBuffer(raidPtr, (RF_PhysDiskAddr_t *) node->params[i].p, (char *) node->params[i + 1].p, (char *) node->results[0]); } #if RF_ACC_TRACE > 0 RF_ETIMER_STOP(timer); RF_ETIMER_EVAL(timer); tracerec->xor_us += RF_ETIMER_VAL_US(timer); #endif } rf_GenericWakeupFunc(node, retcode); /* call wake func * explicitly since no * I/O in this node */ } /* xor the inputs into the result buffer, ignoring placement issues */ void rf_SimpleXorFunc(RF_DagNode_t *node) { RF_Raid_t *raidPtr = (RF_Raid_t *) node->params[node->numParams - 1].p; int i, retcode = 0; #if RF_ACC_TRACE > 0 RF_AccTraceEntry_t *tracerec = node->dagHdr->tracerec; RF_Etimer_t timer; #endif if (node->dagHdr->status == rf_enable) { #if RF_ACC_TRACE > 0 RF_ETIMER_START(timer); #endif /* don't do the XOR if the input is the same as the output */ for (i = 0; i < node->numParams - 1; i += 2) if (node->params[i + 1].p != node->results[0]) { retcode = rf_bxor((char *) node->params[i + 1].p, (char *) node->results[0], rf_RaidAddressToByte(raidPtr, ((RF_PhysDiskAddr_t *) node->params[i].p)->numSector)); } #if RF_ACC_TRACE > 0 RF_ETIMER_STOP(timer); RF_ETIMER_EVAL(timer); tracerec->xor_us += RF_ETIMER_VAL_US(timer); #endif } rf_GenericWakeupFunc(node, retcode); /* call wake func * explicitly since no * I/O in this node */ } /* this xor is used by the degraded-mode dag functions to recover lost * data. the second-to-last parameter is the PDA for the failed * portion of the access. the code here looks at this PDA and assumes * that the xor target buffer is equal in size to the number of * sectors in the failed PDA. It then uses the other PDAs in the * parameter list to determine where within the target buffer the * corresponding data should be xored. */ void rf_RecoveryXorFunc(RF_DagNode_t *node) { RF_Raid_t *raidPtr = (RF_Raid_t *) node->params[node->numParams - 1].p; RF_RaidLayout_t *layoutPtr = (RF_RaidLayout_t *) & raidPtr->Layout; RF_PhysDiskAddr_t *failedPDA = (RF_PhysDiskAddr_t *) node->params[node->numParams - 2].p; int i, retcode = 0; RF_PhysDiskAddr_t *pda; int suoffset, failedSUOffset = rf_StripeUnitOffset(layoutPtr, failedPDA->startSector); char *srcbuf, *destbuf; #if RF_ACC_TRACE > 0 RF_AccTraceEntry_t *tracerec = node->dagHdr->tracerec; RF_Etimer_t timer; #endif if (node->dagHdr->status == rf_enable) { #if RF_ACC_TRACE > 0 RF_ETIMER_START(timer); #endif for (i = 0; i < node->numParams - 2; i += 2) if (node->params[i + 1].p != node->results[0]) { pda = (RF_PhysDiskAddr_t *) node->params[i].p; srcbuf = (char *) node->params[i + 1].p; suoffset = rf_StripeUnitOffset(layoutPtr, pda->startSector); destbuf = ((char *) node->results[0]) + rf_RaidAddressToByte(raidPtr, suoffset - failedSUOffset); retcode = rf_bxor(srcbuf, destbuf, rf_RaidAddressToByte(raidPtr, pda->numSector)); } #if RF_ACC_TRACE > 0 RF_ETIMER_STOP(timer); RF_ETIMER_EVAL(timer); tracerec->xor_us += RF_ETIMER_VAL_US(timer); #endif } rf_GenericWakeupFunc(node, retcode); } /***************************************************************************** * The next three functions are utilities used by the above * xor-execution functions. ****************************************************************************/ /* * this is just a glorified buffer xor. targbuf points to a buffer * that is one full stripe unit in size. srcbuf points to a buffer * that may be less than 1 SU, but never more. When the access * described by pda is one SU in size (which by implication means it's * SU-aligned), all that happens is (targbuf) <- (srcbuf ^ targbuf). * When the access is less than one SU in size the XOR occurs on only * the portion of targbuf identified in the pda. */ int rf_XorIntoBuffer(RF_Raid_t *raidPtr, RF_PhysDiskAddr_t *pda, char *srcbuf, char *targbuf) { char *targptr; int sectPerSU = raidPtr->Layout.sectorsPerStripeUnit; int SUOffset = pda->startSector % sectPerSU; int length, retcode = 0; RF_ASSERT(pda->numSector <= sectPerSU); targptr = targbuf + rf_RaidAddressToByte(raidPtr, SUOffset); length = rf_RaidAddressToByte(raidPtr, pda->numSector); retcode = rf_bxor(srcbuf, targptr, length); return (retcode); } /* it really should be the case that the buffer pointers (returned by * malloc) are aligned to the natural word size of the machine, so * this is the only case we optimize for. The length should always be * a multiple of the sector size, so there should be no problem with * leftover bytes at the end. */ int rf_bxor(char *src, char *dest, int len) { unsigned mask = sizeof(long) - 1, retcode = 0; if (!(((unsigned long) src) & mask) && !(((unsigned long) dest) & mask) && !(len & mask)) { retcode = rf_longword_bxor((unsigned long *) src, (unsigned long *) dest, len >> RF_LONGSHIFT); } else { RF_ASSERT(0); } return (retcode); } /* When XORing in kernel mode, we need to map each user page to kernel * space before we can access it. We don't want to assume anything * about which input buffers are in kernel/user space, nor about their * alignment, so in each loop we compute the maximum number of bytes * that we can xor without crossing any page boundaries, and do only * this many bytes before the next remap. * * len - is in longwords */ int rf_longword_bxor(unsigned long *src, unsigned long *dest, int len) { unsigned long *end = src + len; unsigned long d0, d1, d2, d3, s0, s1, s2, s3; /* temps */ unsigned long *pg_src, *pg_dest; /* per-page source/dest pointers */ int longs_this_time;/* # longwords to xor in the current iteration */ pg_src = src; pg_dest = dest; if (!pg_src || !pg_dest) return (EFAULT); while (len >= 4) { longs_this_time = RF_MIN(len, RF_MIN(RF_BLIP(pg_src), RF_BLIP(pg_dest)) >> RF_LONGSHIFT); /* note len in longwords */ src += longs_this_time; dest += longs_this_time; len -= longs_this_time; while (longs_this_time >= 4) { d0 = pg_dest[0]; d1 = pg_dest[1]; d2 = pg_dest[2]; d3 = pg_dest[3]; s0 = pg_src[0]; s1 = pg_src[1]; s2 = pg_src[2]; s3 = pg_src[3]; pg_dest[0] = d0 ^ s0; pg_dest[1] = d1 ^ s1; pg_dest[2] = d2 ^ s2; pg_dest[3] = d3 ^ s3; pg_src += 4; pg_dest += 4; longs_this_time -= 4; } while (longs_this_time > 0) { /* cannot cross any page * boundaries here */ *pg_dest++ ^= *pg_src++; longs_this_time--; } /* either we're done, or we've reached a page boundary on one * (or possibly both) of the pointers */ if (len) { if (RF_PAGE_ALIGNED(src)) pg_src = src; if (RF_PAGE_ALIGNED(dest)) pg_dest = dest; if (!pg_src || !pg_dest) return (EFAULT); } } while (src < end) { *pg_dest++ ^= *pg_src++; src++; dest++; len--; if (RF_PAGE_ALIGNED(src)) pg_src = src; if (RF_PAGE_ALIGNED(dest)) pg_dest = dest; } RF_ASSERT(len == 0); return (0); } #if 0 /* dst = a ^ b ^ c; a may equal dst see comment above longword_bxor len is length in longwords */ int rf_longword_bxor3(unsigned long *dst, unsigned long *a, unsigned long *b, unsigned long *c, int len, void *bp) { unsigned long a0, a1, a2, a3, b0, b1, b2, b3; unsigned long *pg_a, *pg_b, *pg_c, *pg_dst; /* per-page source/dest * pointers */ int longs_this_time;/* # longs to xor in the current iteration */ char dst_is_a = 0; pg_a = a; pg_b = b; pg_c = c; if (a == dst) { pg_dst = pg_a; dst_is_a = 1; } else { pg_dst = dst; } /* align dest to cache line. Can't cross a pg boundary on dst here. */ while ((((unsigned long) pg_dst) & 0x1f)) { *pg_dst++ = *pg_a++ ^ *pg_b++ ^ *pg_c++; dst++; a++; b++; c++; if (RF_PAGE_ALIGNED(a)) { pg_a = a; if (!pg_a) return (EFAULT); } if (RF_PAGE_ALIGNED(b)) { pg_b = a; if (!pg_b) return (EFAULT); } if (RF_PAGE_ALIGNED(c)) { pg_c = a; if (!pg_c) return (EFAULT); } len--; } while (len > 4) { longs_this_time = RF_MIN(len, RF_MIN(RF_BLIP(a), RF_MIN(RF_BLIP(b), RF_MIN(RF_BLIP(c), RF_BLIP(dst)))) >> RF_LONGSHIFT); a += longs_this_time; b += longs_this_time; c += longs_this_time; dst += longs_this_time; len -= longs_this_time; while (longs_this_time >= 4) { a0 = pg_a[0]; longs_this_time -= 4; a1 = pg_a[1]; a2 = pg_a[2]; a3 = pg_a[3]; pg_a += 4; b0 = pg_b[0]; b1 = pg_b[1]; b2 = pg_b[2]; b3 = pg_b[3]; /* start dual issue */ a0 ^= b0; b0 = pg_c[0]; pg_b += 4; a1 ^= b1; a2 ^= b2; a3 ^= b3; b1 = pg_c[1]; a0 ^= b0; b2 = pg_c[2]; a1 ^= b1; b3 = pg_c[3]; a2 ^= b2; pg_dst[0] = a0; a3 ^= b3; pg_dst[1] = a1; pg_c += 4; pg_dst[2] = a2; pg_dst[3] = a3; pg_dst += 4; } while (longs_this_time > 0) { /* cannot cross any page * boundaries here */ *pg_dst++ = *pg_a++ ^ *pg_b++ ^ *pg_c++; longs_this_time--; } if (len) { if (RF_PAGE_ALIGNED(a)) { pg_a = a; if (!pg_a) return (EFAULT); if (dst_is_a) pg_dst = pg_a; } if (RF_PAGE_ALIGNED(b)) { pg_b = b; if (!pg_b) return (EFAULT); } if (RF_PAGE_ALIGNED(c)) { pg_c = c; if (!pg_c) return (EFAULT); } if (!dst_is_a) if (RF_PAGE_ALIGNED(dst)) { pg_dst = dst; if (!pg_dst) return (EFAULT); } } } while (len) { *pg_dst++ = *pg_a++ ^ *pg_b++ ^ *pg_c++; dst++; a++; b++; c++; if (RF_PAGE_ALIGNED(a)) { pg_a = a; if (!pg_a) return (EFAULT); if (dst_is_a) pg_dst = pg_a; } if (RF_PAGE_ALIGNED(b)) { pg_b = b; if (!pg_b) return (EFAULT); } if (RF_PAGE_ALIGNED(c)) { pg_c = c; if (!pg_c) return (EFAULT); } if (!dst_is_a) if (RF_PAGE_ALIGNED(dst)) { pg_dst = dst; if (!pg_dst) return (EFAULT); } len--; } return (0); } int rf_bxor3(unsigned char *dst, unsigned char *a, unsigned char *b, unsigned char *c, unsigned long len, void *bp) { RF_ASSERT(((RF_UL(dst) | RF_UL(a) | RF_UL(b) | RF_UL(c) | len) & 0x7) == 0); return (rf_longword_bxor3((unsigned long *) dst, (unsigned long *) a, (unsigned long *) b, (unsigned long *) c, len >> RF_LONGSHIFT, bp)); } #endif |
| 8 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 197 198 199 200 201 202 203 204 205 206 207 208 209 210 211 212 213 214 215 216 217 218 219 220 221 222 223 224 225 226 227 228 229 230 231 232 233 234 235 236 237 238 239 240 241 242 243 244 245 246 247 248 249 250 251 252 253 254 255 256 257 258 259 260 261 262 263 264 265 266 267 268 269 270 271 272 273 274 275 276 277 278 279 280 281 282 283 284 285 286 287 288 289 290 291 292 293 294 295 296 297 298 299 300 301 302 303 304 305 306 307 308 309 310 311 312 313 314 315 316 317 318 319 320 321 322 323 324 325 326 327 328 329 330 331 332 333 334 335 336 337 338 339 340 341 342 343 344 345 346 347 348 349 350 351 352 353 354 355 356 357 358 359 360 361 362 363 364 365 366 367 368 369 370 371 372 373 374 375 376 377 378 379 380 381 382 383 384 385 386 387 388 389 390 391 392 393 394 395 396 397 398 399 400 401 402 403 404 405 406 407 408 409 410 411 412 413 414 415 416 417 418 419 420 421 422 423 424 425 426 427 428 429 430 431 432 433 434 435 436 437 438 439 440 441 442 443 444 445 446 447 448 449 450 451 452 453 454 455 456 457 458 459 460 461 462 463 464 465 466 467 468 469 470 471 472 473 474 475 476 477 478 479 480 481 482 483 484 485 486 487 488 489 490 491 492 493 494 495 496 497 498 499 500 501 502 503 504 505 506 507 508 509 510 511 512 513 514 515 516 517 518 519 520 521 522 523 524 525 526 527 528 529 530 531 532 533 534 535 536 537 538 539 540 541 542 543 544 545 546 547 548 549 550 551 552 553 554 555 556 557 558 559 560 561 562 563 564 565 566 567 568 569 570 571 572 573 574 575 576 577 578 579 580 581 582 583 584 585 586 587 588 589 590 591 592 593 594 595 596 597 598 599 600 601 602 603 604 605 606 607 608 609 610 611 612 613 614 615 616 617 618 619 620 621 622 623 624 625 626 627 628 629 630 631 632 633 634 635 636 637 638 639 640 641 642 643 644 645 646 647 648 649 650 651 652 653 654 655 656 657 658 659 660 661 662 663 664 665 666 667 668 669 670 671 672 673 674 675 676 677 678 679 680 681 682 683 684 685 686 687 688 689 690 691 692 693 694 695 696 697 698 699 700 701 702 703 704 705 706 707 708 709 710 711 712 713 714 715 716 717 718 719 720 721 722 723 724 725 726 727 728 729 730 731 732 733 734 735 736 737 738 739 740 741 742 743 744 745 746 747 748 749 750 751 752 753 754 755 756 757 758 759 760 761 762 763 764 765 766 767 768 769 770 771 772 773 774 775 776 777 778 779 780 781 782 783 784 785 786 787 788 789 790 791 792 793 794 795 796 797 798 799 800 801 802 803 804 805 806 807 808 809 810 811 812 813 814 815 816 817 818 819 820 821 822 823 824 825 826 827 828 829 830 831 832 833 834 835 836 837 838 839 840 841 842 843 844 845 846 847 848 849 850 851 852 853 854 855 856 857 858 859 860 861 862 863 864 865 866 867 868 869 870 871 872 873 874 875 876 877 878 879 880 881 882 883 884 885 886 887 888 889 890 891 892 893 894 895 896 897 898 899 900 901 902 903 904 905 906 907 908 909 910 911 912 913 914 915 916 917 918 919 920 921 922 923 924 925 926 927 928 929 930 931 932 | /* $NetBSD: umcs.c,v 1.20 2022/06/26 21:35:53 riastradh Exp $ */ /* $FreeBSD: head/sys/dev/usb/serial/umcs.c 260559 2014-01-12 11:44:28Z hselasky $ */ /*- * Copyright (c) 2010 Lev Serebryakov <lev@FreeBSD.org>. * All rights reserved. * * Redistribution and use in source and binary forms, with or without * modification, are permitted provided that the following conditions * are met: * 1. Redistributions of source code must retain the above copyright * notice, this list of conditions and the following disclaimer. * 2. Redistributions in binary form must reproduce the above copyright * notice, this list of conditions and the following disclaimer in the * documentation and/or other materials provided with the distribution. * * THIS SOFTWARE IS PROVIDED BY THE AUTHOR AND CONTRIBUTORS ``AS IS'' AND * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE * ARE DISCLAIMED. IN NO EVENT SHALL THE AUTHOR OR CONTRIBUTORS BE LIABLE * FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL * DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS * OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) * HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT * LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY * OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF * SUCH DAMAGE. */ /* * This driver supports several multiport USB-to-RS232 serial adapters driven * by MosChip mos7820 and mos7840, bridge chips. * The adapters are sold under many different brand names. * * Datasheets are available at MosChip www site at * http://www.moschip.com. The datasheets don't contain full * programming information for the chip. * * It is nornal to have only two enabled ports in devices, based on * quad-port mos7840. * */ #include <sys/cdefs.h> __KERNEL_RCSID(0, "$NetBSD: umcs.c,v 1.20 2022/06/26 21:35:53 riastradh Exp $"); #include <sys/param.h> #include <sys/systm.h> #include <sys/atomic.h> #include <sys/kernel.h> #include <sys/conf.h> #include <sys/tty.h> #include <sys/device.h> #include <sys/kmem.h> #include <dev/usb/usb.h> #include <dev/usb/usbdi.h> #include <dev/usb/usbdi_util.h> #include <dev/usb/usbdevs.h> #include <dev/usb/usbdevs.h> #include <dev/usb/ucomvar.h> #include "umcs.h" #if 0 #define DPRINTF(ARG) printf ARG #else #define DPRINTF(ARG) #endif /* * Two-port devices (both with 7820 chip and 7840 chip configured as two-port) * have ports 0 and 2, with ports 1 and 3 omitted. * So, PHYSICAL port numbers on two-port device will be 0 and 2. * * We use an array of the following struct, indexed by ucom port index, * and include the physical port number in it. */ struct umcs7840_softc_oneport { device_t sc_port_ucom; /* ucom subdevice */ unsigned int sc_port_phys; /* physical port number */ uint8_t sc_port_lcr; /* local line control register */ uint8_t sc_port_mcr; /* local modem control register */ }; struct umcs7840_softc { device_t sc_dev; /* ourself */ enum { UMCS_INIT_NONE, UMCS_INIT_INITED } sc_init_state; struct usbd_interface *sc_iface; /* the usb interface */ struct usbd_device *sc_udev; /* the usb device */ struct usbd_pipe *sc_intr_pipe; /* interrupt pipe */ uint8_t *sc_intr_buf; /* buffer for interrupt xfer */ unsigned int sc_intr_buflen; /* size of buffer */ struct usb_task sc_change_task; /* async status changes */ volatile uint32_t sc_change_mask; /* mask of port changes */ struct umcs7840_softc_oneport sc_ports[UMCS7840_MAX_PORTS]; /* data for each port */ uint8_t sc_numports; /* number of ports (subunits) */ bool sc_init_done; /* special one time init in open */ bool sc_dying; /* we have been deactivated */ }; static int umcs7840_get_reg(struct umcs7840_softc *, uint8_t, uint8_t *); static int umcs7840_set_reg(struct umcs7840_softc *, uint8_t, uint8_t); static int umcs7840_get_UART_reg(struct umcs7840_softc *, uint8_t, uint8_t, uint8_t *); static int umcs7840_set_UART_reg(struct umcs7840_softc *, uint8_t, uint8_t, uint8_t ); static int umcs7840_calc_baudrate(uint32_t, uint16_t *, uint8_t *); static void umcs7840_dtr(struct umcs7840_softc *, int, bool); static void umcs7840_rts(struct umcs7840_softc *, int, bool); static void umcs7840_break(struct umcs7840_softc *, int, bool ); static int umcs7840_match(device_t, cfdata_t, void *); static void umcs7840_attach(device_t, device_t, void *); static int umcs7840_detach(device_t, int); static void umcs7840_intr(struct usbd_xfer *, void *, usbd_status); static void umcs7840_change_task(void *arg); static void umcs7840_childdet(device_t, device_t); static void umcs7840_get_status(void *, int, u_char *, u_char *); static void umcs7840_set(void *, int, int, int); static int umcs7840_param(void *, int, struct termios *); static int umcs7840_port_open(void *, int); static void umcs7840_port_close(void *, int); static const struct ucom_methods umcs7840_methods = { .ucom_get_status = umcs7840_get_status, .ucom_set = umcs7840_set, .ucom_param = umcs7840_param, .ucom_open = umcs7840_port_open, .ucom_close = umcs7840_port_close, }; static const struct usb_devno umcs7840_devs[] = { { USB_VENDOR_MOSCHIP, USB_PRODUCT_MOSCHIP_MCS7703 }, { USB_VENDOR_MOSCHIP, USB_PRODUCT_MOSCHIP_MCS7810 }, { USB_VENDOR_MOSCHIP, USB_PRODUCT_MOSCHIP_MCS7820 }, { USB_VENDOR_MOSCHIP, USB_PRODUCT_MOSCHIP_MCS7840 }, { USB_VENDOR_ATEN, USB_PRODUCT_ATEN_UC2324 } }; #define umcs7840_lookup(v, p) usb_lookup(umcs7840_devs, v, p) CFATTACH_DECL2_NEW(umcs, sizeof(struct umcs7840_softc), umcs7840_match, umcs7840_attach, umcs7840_detach, NULL, NULL, umcs7840_childdet); static inline int umcs7840_reg_sp(int phyport) { KASSERT(phyport >= 0 && phyport < 4); switch (phyport) { default: case 0: return MCS7840_DEV_REG_SP1; case 1: return MCS7840_DEV_REG_SP2; case 2: return MCS7840_DEV_REG_SP3; case 3: return MCS7840_DEV_REG_SP4; } } static inline int umcs7840_reg_ctrl(int phyport) { KASSERT(phyport >= 0 && phyport < 4); switch (phyport) { default: case 0: return MCS7840_DEV_REG_CONTROL1; case 1: return MCS7840_DEV_REG_CONTROL2; case 2: return MCS7840_DEV_REG_CONTROL3; case 3: return MCS7840_DEV_REG_CONTROL4; } } static int umcs7840_match(device_t dev, cfdata_t match, void *aux) { struct usb_attach_arg *uaa = aux; return umcs7840_lookup(uaa->uaa_vendor, uaa->uaa_product) != NULL ? UMATCH_VENDOR_PRODUCT : UMATCH_NONE; } static void umcs7840_attach(device_t parent, device_t self, void *aux) { struct umcs7840_softc *sc = device_private(self); struct usb_attach_arg *uaa = aux; struct usbd_device *dev = uaa->uaa_device; usb_interface_descriptor_t *id; usb_endpoint_descriptor_t *ed; char *devinfop; struct ucom_attach_args ucaa; int error, i, intr_addr; uint8_t data; sc->sc_dev = self; sc->sc_udev = uaa->uaa_device; sc->sc_dying = false; sc->sc_init_state = UMCS_INIT_NONE; if (usbd_set_config_index(sc->sc_udev, MCS7840_CONFIG_INDEX, 1) != 0) { aprint_error(": could not set configuration no\n"); sc->sc_dying = true; return; } /* get the first interface handle */ error = usbd_device2interface_handle(sc->sc_udev, MCS7840_IFACE_INDEX, &sc->sc_iface); if (error != 0) { aprint_error(": could not get interface handle\n"); sc->sc_dying = true; return; } /* * Get number of ports * Documentation (full datasheet) says, that number of ports is * set as MCS7840_DEV_MODE_SELECT24S bit in MODE R/Only * register. But vendor driver uses these undocumented * register & bit. * * Experiments show, that MODE register can have `0' * (4 ports) bit on 2-port device, so use vendor driver's way. * * Also, see notes in header file for these constants. */ error = umcs7840_get_reg(sc, MCS7840_DEV_REG_GPIO, &data); if (error == 0 && (data & MCS7840_DEV_GPIO_4PORTS) != 0) { sc->sc_numports = 4; /* physical port no are : 0, 1, 2, 3 */ } else { if (uaa->uaa_product == USB_PRODUCT_MOSCHIP_MCS7810) sc->sc_numports = 1; else { sc->sc_numports = 2; /* physical port no are : 0 and 2 */ } } devinfop = usbd_devinfo_alloc(dev, 0); aprint_normal(": %s\n", devinfop); usbd_devinfo_free(devinfop); aprint_verbose_dev(self, "found %d active ports\n", sc->sc_numports); if (!umcs7840_get_reg(sc, MCS7840_DEV_REG_MODE, &data)) { aprint_verbose_dev(self, "On-die confguration: RST: active %s, " "HRD: %s, PLL: %s, POR: %s, Ports: %s, EEPROM write %s, " "IrDA is %savailable\n", (data & MCS7840_DEV_MODE_RESET) ? "low" : "high", (data & MCS7840_DEV_MODE_SER_PRSNT) ? "yes" : "no", (data & MCS7840_DEV_MODE_PLLBYPASS) ? "bypassed" : "avail", (data & MCS7840_DEV_MODE_PORBYPASS) ? "bypassed" : "avail", (data & MCS7840_DEV_MODE_SELECT24S) ? "2" : "4", (data & MCS7840_DEV_MODE_EEPROMWR) ? "enabled" : "disabled", (data & MCS7840_DEV_MODE_IRDA) ? "" : "not "); } /* * Set up the interrupt pipe */ id = usbd_get_interface_descriptor(sc->sc_iface); intr_addr = -1; for (i = 0 ; i < id->bNumEndpoints ; i++) { ed = usbd_interface2endpoint_descriptor(sc->sc_iface, i); if (ed == NULL) continue; if (UE_GET_DIR(ed->bEndpointAddress) != UE_DIR_IN || UE_GET_XFERTYPE(ed->bmAttributes) != UE_INTERRUPT) continue; sc->sc_intr_buflen = UGETW(ed->wMaxPacketSize); intr_addr = ed->bEndpointAddress; break; } if (intr_addr < 0) { aprint_error_dev(self, "interrupt pipe not found\n"); sc->sc_dying = true; return; } if (sc->sc_intr_buflen == 0) { aprint_error_dev(self, "invalid interrupt endpoint" " (addr %d)\n", intr_addr); sc->sc_dying = true; return; } sc->sc_intr_buf = kmem_alloc(sc->sc_intr_buflen, KM_SLEEP); error = usbd_open_pipe_intr(sc->sc_iface, intr_addr, USBD_SHORT_XFER_OK, &sc->sc_intr_pipe, sc, sc->sc_intr_buf, sc->sc_intr_buflen, umcs7840_intr, 100); if (error) { aprint_error_dev(self, "cannot open interrupt pipe " "(addr %d): error %d\n", intr_addr, error); sc->sc_dying = true; return; } usb_init_task(&sc->sc_change_task, umcs7840_change_task, sc, USB_TASKQ_MPSAFE); usbd_add_drv_event(USB_EVENT_DRIVER_ATTACH, sc->sc_udev, sc->sc_dev); sc->sc_init_state = UMCS_INIT_INITED; memset(&ucaa, 0, sizeof(ucaa)); ucaa.ucaa_ibufsize = 256; ucaa.ucaa_obufsize = 256; ucaa.ucaa_ibufsizepad = 256; ucaa.ucaa_opkthdrlen = 0; ucaa.ucaa_device = sc->sc_udev; ucaa.ucaa_iface = sc->sc_iface; ucaa.ucaa_methods = &umcs7840_methods; ucaa.ucaa_arg = sc; for (i = 0; i < sc->sc_numports; i++) { ucaa.ucaa_bulkin = ucaa.ucaa_bulkout = -1; /* * On four port cards, endpoints are 0/1 for first, * 2/3 for second, ... * On two port cards, they are 0/1 for first, 4/5 for second. * On single port, just 0/1 will be used. */ int phyport = i * (sc->sc_numports == 2 ? 2 : 1); ed = usbd_interface2endpoint_descriptor(sc->sc_iface, phyport*2); if (ed == NULL) { aprint_error_dev(self, "no bulk in endpoint found for %d\n", i); sc->sc_dying = true; return; } ucaa.ucaa_bulkin = ed->bEndpointAddress; ed = usbd_interface2endpoint_descriptor(sc->sc_iface, phyport*2 + 1); if (ed == NULL) { aprint_error_dev(self, "no bulk out endpoint found for %d\n", i); return; } ucaa.ucaa_bulkout = ed->bEndpointAddress; ucaa.ucaa_portno = i; DPRINTF(("port %d physical port %d bulk-in %d bulk-out %d\n", i, phyport, ucaa.ucaa_bulkin, ucaa.ucaa_bulkout)); sc->sc_ports[i].sc_port_phys = phyport; sc->sc_ports[i].sc_port_ucom = config_found(self, &ucaa, ucomprint, CFARGS(.submatch = ucomsubmatch)); } } static int umcs7840_get_reg(struct umcs7840_softc *sc, uint8_t reg, uint8_t *data) { usb_device_request_t req; int err; req.bmRequestType = UT_READ_VENDOR_DEVICE; req.bRequest = MCS7840_RDREQ; USETW(req.wValue, 0); USETW(req.wIndex, reg); USETW(req.wLength, UMCS7840_READ_LENGTH); err = usbd_do_request(sc->sc_udev, &req, data); if (err) aprint_normal_dev(sc->sc_dev, "Reading register %d failed: %s\n", reg, usbd_errstr(err)); return err; } static int umcs7840_set_reg(struct umcs7840_softc *sc, uint8_t reg, uint8_t data) { usb_device_request_t req; int err; req.bmRequestType = UT_WRITE_VENDOR_DEVICE; req.bRequest = MCS7840_WRREQ; USETW(req.wValue, data); USETW(req.wIndex, reg); USETW(req.wLength, 0); err = usbd_do_request(sc->sc_udev, &req, 0); if (err) aprint_normal_dev(sc->sc_dev, "Writing register %d failed: %s\n", reg, usbd_errstr(err)); return err; } static int umcs7840_get_UART_reg(struct umcs7840_softc *sc, uint8_t portno, uint8_t reg, uint8_t *data) { usb_device_request_t req; uint16_t wVal; int err; /* portno is port number */ wVal = ((uint16_t)(portno + 1)) << 8; req.bmRequestType = UT_READ_VENDOR_DEVICE; req.bRequest = MCS7840_RDREQ; USETW(req.wValue, wVal); USETW(req.wIndex, reg); USETW(req.wLength, UMCS7840_READ_LENGTH); err = usbd_do_request(sc->sc_udev, &req, data); if (err) aprint_normal_dev(sc->sc_dev, "Reading UART %d register %d failed: %s\n", portno, reg, usbd_errstr(err)); return err; } static int umcs7840_set_UART_reg(struct umcs7840_softc *sc, uint8_t portno, uint8_t reg, uint8_t data) { usb_device_request_t req; int err; uint16_t wVal; /* portno is the physical port number */ wVal = ((uint16_t)(portno + 1)) << 8 | data; req.bmRequestType = UT_WRITE_VENDOR_DEVICE; req.bRequest = MCS7840_WRREQ; USETW(req.wValue, wVal); USETW(req.wIndex, reg); USETW(req.wLength, 0); err = usbd_do_request(sc->sc_udev, &req, NULL); if (err) aprint_error_dev(sc->sc_dev, "Writing UART %d register %d failed: %s\n", portno, reg, usbd_errstr(err)); return err; } static int umcs7840_set_baudrate(struct umcs7840_softc *sc, uint8_t portno, uint32_t rate) { int err; uint16_t divisor; uint8_t clk; uint8_t data; uint8_t physport = sc->sc_ports[portno].sc_port_phys; int spreg = umcs7840_reg_sp(physport); if (umcs7840_calc_baudrate(rate, &divisor, &clk)) { DPRINTF(("Port %d bad speed: %d\n", portno, rate)); return -1; } if (divisor == 0 || (clk & MCS7840_DEV_SPx_CLOCK_MASK) != clk) { DPRINTF(("Port %d bad speed calculation: %d\n", portno, rate)); return -1; } DPRINTF(("Port %d set speed: %d (%02x / %d)\n", portno, rate, clk, divisor)); /* Set clock source for standard BAUD frequences */ err = umcs7840_get_reg(sc, spreg, &data); if (err) return err; data &= MCS7840_DEV_SPx_CLOCK_MASK; data |= clk; err = umcs7840_set_reg(sc, spreg, data); if (err) return err; /* Set divider */ sc->sc_ports[portno].sc_port_lcr |= MCS7840_UART_LCR_DIVISORS; err = umcs7840_set_UART_reg(sc, physport, MCS7840_UART_REG_LCR, sc->sc_ports[portno].sc_port_lcr); if (err) return err; err = umcs7840_set_UART_reg(sc, physport, MCS7840_UART_REG_DLL, (uint8_t)(divisor & 0xff)); if (err) return err; err = umcs7840_set_UART_reg(sc, physport, MCS7840_UART_REG_DLM, (uint8_t)((divisor >> 8) & 0xff)); if (err) return err; /* Turn off access to DLL/DLM registers of UART */ sc->sc_ports[portno].sc_port_lcr &= ~MCS7840_UART_LCR_DIVISORS; err = umcs7840_set_UART_reg(sc, physport, MCS7840_UART_REG_LCR, sc->sc_ports[portno].sc_port_lcr); if (err) return err; return 0; } static int umcs7840_calc_baudrate(uint32_t rate, uint16_t *divisor, uint8_t *clk) { /* Maximum speeds for standard frequences, when PLL is not used */ static const uint32_t umcs7840_baudrate_divisors[] = {0, 115200, 230400, 403200, 460800, 806400, 921600, 1572864, 3145728,}; static const uint8_t umcs7840_baudrate_divisors_len = __arraycount(umcs7840_baudrate_divisors); uint8_t i = 0; if (rate > umcs7840_baudrate_divisors[umcs7840_baudrate_divisors_len - 1]) return -1; for (i = 0; i < umcs7840_baudrate_divisors_len - 1 && !(rate > umcs7840_baudrate_divisors[i] && rate <= umcs7840_baudrate_divisors[i + 1]); ++i); *divisor = umcs7840_baudrate_divisors[i + 1] / rate; /* 0x00 .. 0x70 */ *clk = i << MCS7840_DEV_SPx_CLOCK_SHIFT; return 0; } static int umcs7840_detach(device_t self, int flags) { struct umcs7840_softc *sc = device_private(self); int rv = 0, i; sc->sc_dying = true; /* close interrupt pipe */ if (sc->sc_intr_pipe != NULL) { usbd_abort_pipe(sc->sc_intr_pipe); usbd_close_pipe(sc->sc_intr_pipe); sc->sc_intr_pipe = NULL; } if (sc->sc_intr_buf != NULL) { kmem_free(sc->sc_intr_buf, sc->sc_intr_buflen); sc->sc_intr_buf = NULL; } if (sc->sc_init_state < UMCS_INIT_INITED) return 0; usb_rem_task_wait(sc->sc_udev, &sc->sc_change_task, USB_TASKQ_DRIVER, NULL); /* detach children */ for (i = 0; i < sc->sc_numports; i++) { if (sc->sc_ports[i].sc_port_ucom) { rv |= config_detach(sc->sc_ports[i].sc_port_ucom, flags); sc->sc_ports[i].sc_port_ucom = NULL; } } usbd_add_drv_event(USB_EVENT_DRIVER_DETACH, sc->sc_udev, sc->sc_dev); return rv; } static void umcs7840_childdet(device_t self, device_t child) { struct umcs7840_softc *sc = device_private(self); int i; for (i = 0; i < sc->sc_numports; i++) { if (child == sc->sc_ports[i].sc_port_ucom) { sc->sc_ports[i].sc_port_ucom = NULL; return; } } } static void umcs7840_get_status(void *self, int portno, u_char *lsr, u_char *msr) { struct umcs7840_softc *sc = self; uint8_t pn = sc->sc_ports[portno].sc_port_phys; uint8_t hw_lsr = 0; /* local line status register */ uint8_t hw_msr = 0; /* local modem status register */ if (sc->sc_dying) return; /* Read LSR & MSR */ umcs7840_get_UART_reg(sc, pn, MCS7840_UART_REG_LSR, &hw_lsr); umcs7840_get_UART_reg(sc, pn, MCS7840_UART_REG_MSR, &hw_msr); *lsr = hw_lsr; *msr = hw_msr; } static void umcs7840_set(void *self, int portno, int reg, int onoff) { struct umcs7840_softc *sc = self; if (sc->sc_dying) return; switch (reg) { case UCOM_SET_DTR: umcs7840_dtr(sc, portno, onoff); break; case UCOM_SET_RTS: umcs7840_rts(sc, portno, onoff); break; case UCOM_SET_BREAK: umcs7840_break(sc, portno, onoff); break; default: break; } } static int umcs7840_param(void *self, int portno, struct termios *t) { struct umcs7840_softc *sc = self; int pn = sc->sc_ports[portno].sc_port_phys; uint8_t lcr = sc->sc_ports[portno].sc_port_lcr; uint8_t mcr = sc->sc_ports[portno].sc_port_mcr; if (sc->sc_dying) return EIO; if (t->c_cflag & CSTOPB) { lcr |= MCS7840_UART_LCR_STOPB2; } else { lcr |= MCS7840_UART_LCR_STOPB1; } lcr &= ~MCS7840_UART_LCR_PARITYMASK; if (t->c_cflag & PARENB) { lcr |= MCS7840_UART_LCR_PARITYON; if (t->c_cflag & PARODD) { lcr = MCS7840_UART_LCR_PARITYODD; } else { lcr = MCS7840_UART_LCR_PARITYEVEN; } } else { lcr &= ~MCS7840_UART_LCR_PARITYON; } lcr &= ~MCS7840_UART_LCR_DATALENMASK; switch (t->c_cflag & CSIZE) { case CS5: lcr |= MCS7840_UART_LCR_DATALEN5; break; case CS6: lcr |= MCS7840_UART_LCR_DATALEN6; break; case CS7: lcr |= MCS7840_UART_LCR_DATALEN7; break; case CS8: lcr |= MCS7840_UART_LCR_DATALEN8; break; } if (t->c_cflag & CRTSCTS) mcr |= MCS7840_UART_MCR_CTSRTS; else mcr &= ~MCS7840_UART_MCR_CTSRTS; if (t->c_cflag & CLOCAL) mcr &= ~MCS7840_UART_MCR_DTRDSR; else mcr |= MCS7840_UART_MCR_DTRDSR; sc->sc_ports[portno].sc_port_lcr = lcr; umcs7840_set_UART_reg(sc, pn, MCS7840_UART_REG_LCR, sc->sc_ports[pn].sc_port_lcr); sc->sc_ports[portno].sc_port_mcr = mcr; umcs7840_set_UART_reg(sc, pn, MCS7840_UART_REG_MCR, sc->sc_ports[pn].sc_port_mcr); if (umcs7840_set_baudrate(sc, portno, t->c_ospeed)) return EIO; return 0; } static void umcs7840_dtr(struct umcs7840_softc *sc, int portno, bool onoff) { int pn = sc->sc_ports[portno].sc_port_phys; uint8_t mcr = sc->sc_ports[portno].sc_port_mcr; if (onoff) mcr |= MCS7840_UART_MCR_DTR; else mcr &= ~MCS7840_UART_MCR_DTR; sc->sc_ports[portno].sc_port_mcr = mcr; umcs7840_set_UART_reg(sc, pn, MCS7840_UART_REG_MCR, sc->sc_ports[pn].sc_port_mcr); } static void umcs7840_rts(struct umcs7840_softc *sc, int portno, bool onoff) { int pn = sc->sc_ports[portno].sc_port_phys; uint8_t mcr = sc->sc_ports[portno].sc_port_mcr; if (onoff) mcr |= MCS7840_UART_MCR_RTS; else mcr &= ~MCS7840_UART_MCR_RTS; sc->sc_ports[portno].sc_port_mcr = mcr; umcs7840_set_UART_reg(sc, pn, MCS7840_UART_REG_MCR, sc->sc_ports[pn].sc_port_mcr); } static void umcs7840_break(struct umcs7840_softc *sc, int portno, bool onoff) { int pn = sc->sc_ports[portno].sc_port_phys; uint8_t lcr = sc->sc_ports[portno].sc_port_lcr; if (onoff) lcr |= MCS7840_UART_LCR_BREAK; else lcr &= ~MCS7840_UART_LCR_BREAK; sc->sc_ports[portno].sc_port_lcr = lcr; umcs7840_set_UART_reg(sc, pn, MCS7840_UART_REG_LCR, sc->sc_ports[pn].sc_port_lcr); } static int umcs7840_port_open(void *self, int portno) { struct umcs7840_softc *sc = self; int pn = sc->sc_ports[portno].sc_port_phys; int spreg = umcs7840_reg_sp(pn); int ctrlreg = umcs7840_reg_ctrl(pn); uint8_t data; if (sc->sc_dying) return EIO; /* If it very first open, finish global configuration */ if (!sc->sc_init_done) { if (umcs7840_get_reg(sc, MCS7840_DEV_REG_CONTROL1, &data)) return EIO; data |= MCS7840_DEV_CONTROL1_DRIVER_DONE; if (umcs7840_set_reg(sc, MCS7840_DEV_REG_CONTROL1, data)) return EIO; sc->sc_init_done = 1; } /* Toggle reset bit on-off */ if (umcs7840_get_reg(sc, spreg, &data)) return EIO; data |= MCS7840_DEV_SPx_UART_RESET; if (umcs7840_set_reg(sc, spreg, data)) return EIO; data &= ~MCS7840_DEV_SPx_UART_RESET; if (umcs7840_set_reg(sc, spreg, data)) return EIO; /* Set RS-232 mode */ if (umcs7840_set_UART_reg(sc, pn, MCS7840_UART_REG_SCRATCHPAD, MCS7840_UART_SCRATCHPAD_RS232)) return EIO; /* Disable RX on time of initialization */ if (umcs7840_get_reg(sc, ctrlreg, &data)) return EIO; data |= MCS7840_DEV_CONTROLx_RX_DISABLE; if (umcs7840_set_reg(sc, ctrlreg, data)) return EIO; /* Disable all interrupts */ if (umcs7840_set_UART_reg(sc, pn, MCS7840_UART_REG_IER, 0)) return EIO; /* Reset FIFO -- documented */ if (umcs7840_set_UART_reg(sc, pn, MCS7840_UART_REG_FCR, 0)) return EIO; if (umcs7840_set_UART_reg(sc, pn, MCS7840_UART_REG_FCR, MCS7840_UART_FCR_ENABLE | MCS7840_UART_FCR_FLUSHRHR | MCS7840_UART_FCR_FLUSHTHR | MCS7840_UART_FCR_RTL_1_14)) return EIO; /* Set 8 bit, no parity, 1 stop bit -- documented */ sc->sc_ports[pn].sc_port_lcr = MCS7840_UART_LCR_DATALEN8 | MCS7840_UART_LCR_STOPB1; if (umcs7840_set_UART_reg(sc, pn, MCS7840_UART_REG_LCR, sc->sc_ports[pn].sc_port_lcr)) return EIO; /* * Enable DTR/RTS on modem control, enable modem interrupts -- * documented */ sc->sc_ports[pn].sc_port_mcr = MCS7840_UART_MCR_DTR | MCS7840_UART_MCR_RTS | MCS7840_UART_MCR_IE; if (umcs7840_set_UART_reg(sc, pn, MCS7840_UART_REG_MCR, sc->sc_ports[pn].sc_port_mcr)) return EIO; /* Clearing Bulkin and Bulkout FIFO */ if (umcs7840_get_reg(sc, spreg, &data)) return EIO; data |= MCS7840_DEV_SPx_RESET_OUT_FIFO | MCS7840_DEV_SPx_RESET_IN_FIFO; if (umcs7840_set_reg(sc, spreg, data)) return EIO; data &= ~(MCS7840_DEV_SPx_RESET_OUT_FIFO | MCS7840_DEV_SPx_RESET_IN_FIFO); if (umcs7840_set_reg(sc, spreg, data)) return EIO; /* Set speed 9600 */ if (umcs7840_set_baudrate(sc, portno, 9600)) return EIO; /* Finally enable all interrupts -- documented */ /* * Copied from vendor driver, I don't know why we should read LCR * here */ if (umcs7840_get_UART_reg(sc, pn, MCS7840_UART_REG_LCR, &sc->sc_ports[pn].sc_port_lcr)) return EIO; if (umcs7840_set_UART_reg(sc, pn, MCS7840_UART_REG_IER, MCS7840_UART_IER_RXSTAT | MCS7840_UART_IER_MODEM)) return EIO; /* Enable RX */ if (umcs7840_get_reg(sc, ctrlreg, &data)) return EIO; data &= ~MCS7840_DEV_CONTROLx_RX_DISABLE; if (umcs7840_set_reg(sc, ctrlreg, data)) return EIO; return 0; } static void umcs7840_port_close(void *self, int portno) { struct umcs7840_softc *sc = self; int pn = sc->sc_ports[portno].sc_port_phys; int ctrlreg = umcs7840_reg_ctrl(pn); uint8_t data; if (sc->sc_dying) return; umcs7840_set_UART_reg(sc, pn, MCS7840_UART_REG_MCR, 0); umcs7840_set_UART_reg(sc, pn, MCS7840_UART_REG_IER, 0); /* Disable RX */ if (umcs7840_get_reg(sc, ctrlreg, &data)) return; data |= MCS7840_DEV_CONTROLx_RX_DISABLE; if (umcs7840_set_reg(sc, ctrlreg, data)) return; } static void umcs7840_intr(struct usbd_xfer *xfer, void *priv, usbd_status status) { struct umcs7840_softc *sc = priv; u_char *buf = sc->sc_intr_buf; int actlen; int subunit; if (sc->sc_dying) return; if (status == USBD_NOT_STARTED || status == USBD_CANCELLED || status == USBD_IOERROR) return; if (status != USBD_NORMAL_COMPLETION) { aprint_error_dev(sc->sc_dev, "umcs7840_intr: abnormal status: %s\n", usbd_errstr(status)); usbd_clear_endpoint_stall_async(sc->sc_intr_pipe); return; } usbd_get_xfer_status(xfer, NULL, NULL, &actlen, NULL); if (actlen == 5 || actlen == 13) { uint32_t change_mask = 0; /* Check status of all ports */ for (subunit = 0; subunit < sc->sc_numports; subunit++) { uint8_t pn = sc->sc_ports[subunit].sc_port_phys; if (buf[pn] & MCS7840_UART_ISR_NOPENDING) continue; DPRINTF(("Port %d has pending interrupt: %02x " "(FIFO: %02x)\n", pn, buf[pn] & MCS7840_UART_ISR_INTMASK, buf[pn] & (~MCS7840_UART_ISR_INTMASK))); switch (buf[pn] & MCS7840_UART_ISR_INTMASK) { case MCS7840_UART_ISR_RXERR: case MCS7840_UART_ISR_RXHASDATA: case MCS7840_UART_ISR_RXTIMEOUT: case MCS7840_UART_ISR_MSCHANGE: change_mask |= (1U << subunit); break; default: /* Do nothing */ break; } } if (change_mask != 0) { atomic_or_32(&sc->sc_change_mask, change_mask); usb_add_task(sc->sc_udev, &sc->sc_change_task, USB_TASKQ_DRIVER); } } else { aprint_error_dev(sc->sc_dev, "Invalid interrupt data length %d", actlen); } } static void umcs7840_change_task(void *arg) { struct umcs7840_softc *sc = arg; uint32_t change_mask; int i; change_mask = atomic_swap_32(&sc->sc_change_mask, 0); for (i = 0; i < sc->sc_numports; i++) { if (ISSET(change_mask, (1U << i))) ucom_status_change(device_private( sc->sc_ports[i].sc_port_ucom)); } } |
| 2 2 1 1 2 2 1 2 2 1 1 2 1 1 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 197 198 199 200 201 202 203 204 205 206 207 208 209 210 211 212 213 214 215 216 217 218 219 220 221 222 223 224 225 226 227 228 229 230 231 232 233 234 235 236 237 238 239 240 241 242 243 244 245 246 247 248 249 250 251 252 253 254 255 256 257 258 259 260 261 262 263 264 265 266 267 268 269 270 271 272 273 274 275 276 277 278 279 280 281 282 283 284 285 286 287 288 289 290 291 292 293 294 295 296 297 298 299 300 301 302 303 304 305 306 307 308 309 310 311 312 313 314 315 316 317 318 319 320 321 322 323 324 325 326 327 328 329 330 331 332 333 334 335 336 337 338 339 340 341 342 343 344 345 346 347 348 349 350 351 352 353 354 355 356 357 358 359 360 361 362 363 364 365 366 367 368 369 370 371 372 373 374 375 376 377 378 379 380 381 382 383 384 385 386 387 388 389 390 391 392 393 394 395 396 397 398 399 400 401 402 403 404 405 406 407 408 409 410 411 412 413 414 415 416 417 418 419 420 421 422 423 424 425 426 427 428 429 430 431 432 433 434 435 436 437 438 439 440 441 442 443 444 445 446 447 448 449 450 451 452 453 454 455 456 457 458 459 460 461 462 463 464 465 466 467 468 469 470 471 472 473 474 475 476 477 478 479 480 481 482 483 484 485 486 487 488 489 490 491 492 493 494 495 496 497 498 499 500 501 502 503 504 505 506 507 508 509 510 511 512 513 514 515 516 517 518 519 520 521 522 523 524 525 526 527 528 529 530 531 532 533 534 535 536 537 538 539 540 541 542 543 544 545 546 547 548 549 550 551 552 553 554 555 556 557 558 559 560 561 562 563 564 565 566 567 568 569 570 571 572 573 574 575 576 577 578 579 580 581 582 583 584 585 586 587 588 589 590 591 592 593 594 595 596 597 598 599 600 601 602 603 604 605 606 607 608 609 610 611 612 613 614 615 616 617 618 619 620 621 622 623 624 625 626 627 628 629 630 631 632 633 634 635 636 637 638 639 640 641 642 643 644 645 646 647 648 649 650 651 652 653 654 655 656 657 658 659 660 661 662 663 664 665 666 667 668 669 670 | /* $NetBSD: btuart.c,v 1.30 2022/06/28 13:25:36 plunky Exp $ */ /*- * Copyright (c) 2006, 2007 KIYOHARA Takashi * All rights reserved. * * Redistribution and use in source and binary forms, with or without * modification, are permitted provided that the following conditions * are met: * 1. Redistributions of source code must retain the above copyright * notice, this list of conditions and the following disclaimer. * 2. Redistributions in binary form must reproduce the above copyright * notice, this list of conditions and the following disclaimer in the * documentation and/or other materials provided with the distribution. * * THIS SOFTWARE IS PROVIDED BY THE AUTHOR ``AS IS'' AND ANY EXPRESS OR * IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED * WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE * DISCLAIMED. IN NO EVENT SHALL THE AUTHOR BE LIABLE FOR ANY DIRECT, * INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES * (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR * SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) * HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, * STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN * ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE * POSSIBILITY OF SUCH DAMAGE. */ #include <sys/cdefs.h> __KERNEL_RCSID(0, "$NetBSD: btuart.c,v 1.30 2022/06/28 13:25:36 plunky Exp $"); #include <sys/param.h> #include <sys/conf.h> #include <sys/device.h> #include <sys/errno.h> #include <sys/fcntl.h> #include <sys/kauth.h> #include <sys/kernel.h> #include <sys/malloc.h> #include <sys/mbuf.h> #include <sys/proc.h> #include <sys/syslimits.h> #include <sys/systm.h> #include <sys/tty.h> #include <sys/bus.h> #include <sys/intr.h> #include <netbt/bluetooth.h> #include <netbt/hci.h> #include "ioconf.h" struct btuart_softc { device_t sc_dev; struct tty * sc_tp; /* tty pointer */ bool sc_enabled; /* device is enabled */ struct hci_unit *sc_unit; /* Bluetooth HCI handle */ struct bt_stats sc_stats; int sc_state; /* receive state */ int sc_want; /* how much we want */ struct mbuf * sc_rxp; /* incoming packet */ bool sc_xmit; /* transmit is active */ struct mbuf * sc_txp; /* outgoing packet */ /* transmit queues */ MBUFQ_HEAD() sc_cmdq; MBUFQ_HEAD() sc_aclq; MBUFQ_HEAD() sc_scoq; }; /* sc_state */ #define BTUART_RECV_PKT_TYPE 0 /* packet type */ #define BTUART_RECV_ACL_HDR 1 /* acl header */ #define BTUART_RECV_SCO_HDR 2 /* sco header */ #define BTUART_RECV_EVENT_HDR 3 /* event header */ #define BTUART_RECV_ACL_DATA 4 /* acl packet data */ #define BTUART_RECV_SCO_DATA 5 /* sco packet data */ #define BTUART_RECV_EVENT_DATA 6 /* event packet data */ static int btuart_match(device_t, cfdata_t, void *); static void btuart_attach(device_t, device_t, void *); static int btuart_detach(device_t, int); static int btuartopen(dev_t, struct tty *); static int btuartclose(struct tty *, int); static int btuartioctl(struct tty *, u_long, void *, int, struct lwp *); static int btuartinput(int, struct tty *); static int btuartstart(struct tty *); static int btuart_enable(device_t); static void btuart_disable(device_t); static void btuart_output_cmd(device_t, struct mbuf *); static void btuart_output_acl(device_t, struct mbuf *); static void btuart_output_sco(device_t, struct mbuf *); static void btuart_stats(device_t, struct bt_stats *, int); /* * It doesn't need to be exported, as only btuartattach() uses it, * but there's no "official" way to make it static. */ CFATTACH_DECL_NEW(btuart, sizeof(struct btuart_softc), btuart_match, btuart_attach, btuart_detach, NULL); static struct linesw btuart_disc = { .l_name = "btuart", .l_open = btuartopen, .l_close = btuartclose, .l_read = ttyerrio, .l_write = ttyerrio, .l_ioctl = btuartioctl, .l_rint = btuartinput, .l_start = btuartstart, .l_modem = ttymodem, .l_poll = ttyerrpoll, }; static const struct hci_if btuart_hci = { .enable = btuart_enable, .disable = btuart_disable, .output_cmd = btuart_output_cmd, .output_acl = btuart_output_acl, .output_sco = btuart_output_sco, .get_stats = btuart_stats, .ipl = IPL_TTY, }; /***************************************************************************** * * autoconf(9) functions */ /* * pseudo-device attach routine. */ void btuartattach(int num __unused) { int error; error = ttyldisc_attach(&btuart_disc); if (error) { aprint_error("%s: unable to register line discipline, " "error = %d\n", btuart_cd.cd_name, error); return; } error = config_cfattach_attach(btuart_cd.cd_name, &btuart_ca); if (error) { aprint_error("%s: unable to register cfattach, error = %d\n", btuart_cd.cd_name, error); config_cfdriver_detach(&btuart_cd); (void) ttyldisc_detach(&btuart_disc); } } /* * Autoconf match routine. */ static int btuart_match(device_t self __unused, cfdata_t cfdata __unused, void *arg __unused) { /* pseudo-device; always present */ return 1; } /* * Autoconf attach routine. * Called by config_attach_pseudo(9) when we open the line discipline. */ static void btuart_attach(device_t parent __unused, device_t self, void *aux __unused) { struct btuart_softc *sc = device_private(self); sc->sc_dev = self; MBUFQ_INIT(&sc->sc_cmdq); MBUFQ_INIT(&sc->sc_aclq); MBUFQ_INIT(&sc->sc_scoq); /* Attach Bluetooth unit */ sc->sc_unit = hci_attach_pcb(&btuart_hci, self, 0); if (sc->sc_unit == NULL) aprint_error_dev(self, "HCI attach failed\n"); } /* * Autoconf detach routine. * Called when we close the line discipline. */ static int btuart_detach(device_t self, int flags __unused) { struct btuart_softc *sc = device_private(self); btuart_disable(self); if (sc->sc_unit) { hci_detach_pcb(sc->sc_unit); sc->sc_unit = NULL; } return 0; } /***************************************************************************** * * Line discipline functions. */ static int btuartopen(dev_t devno __unused, struct tty *tp) { struct btuart_softc *sc; device_t dev; cfdata_t cfdata; struct lwp *l = curlwp; /* XXX */ int error, unit, s; error = kauth_authorize_device(l->l_cred, KAUTH_DEVICE_BLUETOOTH_BTUART, KAUTH_ARG(KAUTH_REQ_DEVICE_BLUETOOTH_BTUART_ADD), NULL, NULL, NULL); if (error) return (error); s = spltty(); if (tp->t_linesw == &btuart_disc) { sc = tp->t_sc; if (sc != NULL) { splx(s); return EBUSY; } } cfdata = malloc(sizeof(struct cfdata), M_DEVBUF, M_WAITOK); for (unit = 0; unit < btuart_cd.cd_ndevs; unit++) if (device_lookup(&btuart_cd, unit) == NULL) break; cfdata->cf_name = btuart_cd.cd_name; cfdata->cf_atname = btuart_cd.cd_name; cfdata->cf_unit = unit; cfdata->cf_fstate = FSTATE_STAR; dev = config_attach_pseudo(cfdata); if (dev == NULL) { free(cfdata, M_DEVBUF); splx(s); return EIO; } sc = device_private(dev); aprint_normal_dev(dev, "major %llu minor %llu\n", (unsigned long long)major(tp->t_dev), (unsigned long long)minor(tp->t_dev)); sc->sc_tp = tp; tp->t_sc = sc; mutex_spin_enter(&tty_lock); ttyflush(tp, FREAD | FWRITE); mutex_spin_exit(&tty_lock); splx(s); return 0; } static int btuartclose(struct tty *tp, int flag __unused) { struct btuart_softc *sc = tp->t_sc; cfdata_t cfdata; int s; s = spltty(); mutex_spin_enter(&tty_lock); ttyflush(tp, FREAD | FWRITE); mutex_spin_exit(&tty_lock); /* XXX */ ttyldisc_release(tp->t_linesw); tp->t_linesw = ttyldisc_default(); if (sc != NULL) { tp->t_sc = NULL; if (sc->sc_tp == tp) { cfdata = device_cfdata(sc->sc_dev); config_detach(sc->sc_dev, 0); free(cfdata, M_DEVBUF); } } splx(s); return 0; } static int btuartioctl(struct tty *tp, u_long cmd, void *data __unused, int flag __unused, struct lwp *l __unused) { struct btuart_softc *sc = tp->t_sc; int error; /* * XXX * This function can be called without KERNEL_LOCK when caller's * struct cdevsw is set D_MPSAFE. Is KERNEL_LOCK required? */ if (sc == NULL || tp != sc->sc_tp) return EPASSTHROUGH; switch(cmd) { default: error = EPASSTHROUGH; break; } return error; } static int btuartinput(int c, struct tty *tp) { struct btuart_softc *sc = tp->t_sc; struct mbuf *m = sc->sc_rxp; int space = 0; if (!sc->sc_enabled) return 0; c &= TTY_CHARMASK; /* If we already started a packet, find the trailing end of it. */ if (m) { while (m->m_next) m = m->m_next; space = M_TRAILINGSPACE(m); } if (space == 0) { if (m == NULL) { /* new packet */ MGETHDR(m, M_DONTWAIT, MT_DATA); if (m == NULL) { aprint_error_dev(sc->sc_dev, "out of memory\n"); sc->sc_stats.err_rx++; return 0; /* (lost sync) */ } sc->sc_rxp = m; m->m_pkthdr.len = m->m_len = 0; space = MHLEN; sc->sc_state = BTUART_RECV_PKT_TYPE; sc->sc_want = 1; } else { /* extend mbuf */ MGET(m->m_next, M_DONTWAIT, MT_DATA); if (m->m_next == NULL) { aprint_error_dev(sc->sc_dev, "out of memory\n"); sc->sc_stats.err_rx++; return 0; /* (lost sync) */ } m = m->m_next; m->m_len = 0; space = MLEN; if (sc->sc_want > MINCLSIZE) { MCLGET(m, M_DONTWAIT); if (m->m_flags & M_EXT) space = MCLBYTES; } } } mtod(m, uint8_t *)[m->m_len++] = c; sc->sc_rxp->m_pkthdr.len++; sc->sc_stats.byte_rx++; sc->sc_want--; if (sc->sc_want > 0) return 0; /* want more */ switch (sc->sc_state) { case BTUART_RECV_PKT_TYPE: /* Got packet type */ switch (c) { case HCI_ACL_DATA_PKT: sc->sc_state = BTUART_RECV_ACL_HDR; sc->sc_want = sizeof(hci_acldata_hdr_t) - 1; break; case HCI_SCO_DATA_PKT: sc->sc_state = BTUART_RECV_SCO_HDR; sc->sc_want = sizeof(hci_scodata_hdr_t) - 1; break; case HCI_EVENT_PKT: sc->sc_state = BTUART_RECV_EVENT_HDR; sc->sc_want = sizeof(hci_event_hdr_t) - 1; break; default: aprint_error_dev(sc->sc_dev, "Unknown packet type=%#x!\n", c); sc->sc_stats.err_rx++; m_freem(sc->sc_rxp); sc->sc_rxp = NULL; return 0; /* (lost sync) */ } break; /* * we assume (correctly of course :) that the packet headers all fit * into a single pkthdr mbuf */ case BTUART_RECV_ACL_HDR: /* Got ACL Header */ sc->sc_state = BTUART_RECV_ACL_DATA; sc->sc_want = mtod(m, hci_acldata_hdr_t *)->length; sc->sc_want = le16toh(sc->sc_want); break; case BTUART_RECV_SCO_HDR: /* Got SCO Header */ sc->sc_state = BTUART_RECV_SCO_DATA; sc->sc_want = mtod(m, hci_scodata_hdr_t *)->length; break; case BTUART_RECV_EVENT_HDR: /* Got Event Header */ sc->sc_state = BTUART_RECV_EVENT_DATA; sc->sc_want = mtod(m, hci_event_hdr_t *)->length; break; case BTUART_RECV_ACL_DATA: /* ACL Packet Complete */ if (!hci_input_acl(sc->sc_unit, sc->sc_rxp)) sc->sc_stats.err_rx++; sc->sc_stats.acl_rx++; sc->sc_rxp = m = NULL; break; case BTUART_RECV_SCO_DATA: /* SCO Packet Complete */ if (!hci_input_sco(sc->sc_unit, sc->sc_rxp)) sc->sc_stats.err_rx++; sc->sc_stats.sco_rx++; sc->sc_rxp = m = NULL; break; case BTUART_RECV_EVENT_DATA: /* Event Packet Complete */ if (!hci_input_event(sc->sc_unit, sc->sc_rxp)) sc->sc_stats.err_rx++; sc->sc_stats.evt_rx++; sc->sc_rxp = m = NULL; break; default: panic("%s: invalid state %d!\n", device_xname(sc->sc_dev), sc->sc_state); } return 0; } static int btuartstart(struct tty *tp) { struct btuart_softc *sc = tp->t_sc; struct mbuf *m; int count, rlen; uint8_t *rptr; if (!sc->sc_enabled) return 0; m = sc->sc_txp; if (m == NULL) { if (MBUFQ_FIRST(&sc->sc_cmdq)) { MBUFQ_DEQUEUE(&sc->sc_cmdq, m); sc->sc_stats.cmd_tx++; } else if (MBUFQ_FIRST(&sc->sc_scoq)) { MBUFQ_DEQUEUE(&sc->sc_scoq, m); sc->sc_stats.sco_tx++; } else if (MBUFQ_FIRST(&sc->sc_aclq)) { MBUFQ_DEQUEUE(&sc->sc_aclq, m); sc->sc_stats.acl_tx++; } else { sc->sc_xmit = false; return 0; /* no more to send */ } sc->sc_txp = m; sc->sc_xmit = true; } count = 0; rlen = 0; rptr = mtod(m, uint8_t *); for(;;) { if (rlen >= m->m_len) { m = m->m_next; if (m == NULL) { m = sc->sc_txp; sc->sc_txp = NULL; if (M_GETCTX(m, void *) == NULL) m_freem(m); else if (!hci_complete_sco(sc->sc_unit, m)) sc->sc_stats.err_tx++; break; } rlen = 0; rptr = mtod(m, uint8_t *); continue; } if (putc(*rptr++, &tp->t_outq) < 0) { m_adj(m, rlen); break; } rlen++; count++; } sc->sc_stats.byte_tx += count; if (tp->t_outq.c_cc != 0 && tp->t_oproc != NULL) (*tp->t_oproc)(tp); return 0; } /***************************************************************************** * * bluetooth(9) functions */ static int btuart_enable(device_t self) { struct btuart_softc *sc = device_private(self); int s; if (sc->sc_enabled) return 0; s = spltty(); sc->sc_enabled = true; sc->sc_xmit = false; splx(s); return 0; } static void btuart_disable(device_t self) { struct btuart_softc *sc = device_private(self); int s; if (!sc->sc_enabled) return; s = spltty(); if (sc->sc_rxp) { m_freem(sc->sc_rxp); sc->sc_rxp = NULL; } if (sc->sc_txp) { m_freem(sc->sc_txp); sc->sc_txp = NULL; } MBUFQ_DRAIN(&sc->sc_cmdq); MBUFQ_DRAIN(&sc->sc_aclq); MBUFQ_DRAIN(&sc->sc_scoq); sc->sc_enabled = false; splx(s); } static void btuart_output_cmd(device_t self, struct mbuf *m) { struct btuart_softc *sc = device_private(self); int s; KASSERT(sc->sc_enabled); M_SETCTX(m, NULL); s = spltty(); MBUFQ_ENQUEUE(&sc->sc_cmdq, m); if (!sc->sc_xmit) btuartstart(sc->sc_tp); splx(s); } static void btuart_output_acl(device_t self, struct mbuf *m) { struct btuart_softc *sc = device_private(self); int s; KASSERT(sc->sc_enabled); M_SETCTX(m, NULL); s = spltty(); MBUFQ_ENQUEUE(&sc->sc_aclq, m); if (!sc->sc_xmit) btuartstart(sc->sc_tp); splx(s); } static void btuart_output_sco(device_t self, struct mbuf *m) { struct btuart_softc *sc = device_private(self); int s; KASSERT(sc->sc_enabled); s = spltty(); MBUFQ_ENQUEUE(&sc->sc_scoq, m); if (!sc->sc_xmit) btuartstart(sc->sc_tp); splx(s); } static void btuart_stats(device_t self, struct bt_stats *dest, int flush) { struct btuart_softc *sc = device_private(self); int s; s = spltty(); memcpy(dest, &sc->sc_stats, sizeof(struct bt_stats)); if (flush) memset(&sc->sc_stats, 0, sizeof(struct bt_stats)); splx(s); } |
| 1705 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 | /* $NetBSD: hash.h,v 1.8 2014/09/05 05:46:15 matt Exp $ */ /*- * Copyright (c) 2001 The NetBSD Foundation, Inc. * All rights reserved. * * This code is derived from software contributed to The NetBSD Foundation * by Luke Mewburn. * * Redistribution and use in source and binary forms, with or without * modification, are permitted provided that the following conditions * are met: * 1. Redistributions of source code must retain the above copyright * notice, this list of conditions and the following disclaimer. * 2. Redistributions in binary form must reproduce the above copyright * notice, this list of conditions and the following disclaimer in the * documentation and/or other materials provided with the distribution. * * THIS SOFTWARE IS PROVIDED BY THE NETBSD FOUNDATION, INC. AND CONTRIBUTORS * ``AS IS'' AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED * TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR * PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE FOUNDATION OR CONTRIBUTORS * BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR * CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF * SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS * INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN * CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) * ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE * POSSIBILITY OF SUCH DAMAGE. */ #ifndef _SYS_HASH_H_ #define _SYS_HASH_H_ #include <sys/types.h> #ifdef __HAVE_MACHINE_HASH_H #include <machine/hash.h> #endif #ifndef __HAVE_HASH32_BUF /* not overridden by MD hash */ #define HASH32_BUF_INIT 5381 /* * uint32_t * hash32_buf(const void *bf, size_t len, uint32_t hash) * return a 32 bit hash of the binary buffer buf (size len), * seeded with an initial hash value of hash (usually HASH32_BUF_INIT). */ static __inline uint32_t hash32_buf(const void *bf, size_t len, uint32_t hash) { const uint8_t *s = (const uint8_t *)bf; while (len-- != 0) /* "nemesi": k=257, r=r*257 */ hash = hash * 257 + *s++; return (hash * 257); } #endif /* __HAVE_HASH32_BUF */ #ifndef __HAVE_HASH32_STR /* not overridden by MD hash */ #define HASH32_STR_INIT 5381 /* * uint32_t * hash32_str(const void *bf, uint32_t hash) * return a 32 bit hash of NUL terminated ASCII string buf, * seeded with an initial hash value of hash (usually HASH32_STR_INIT). */ static __inline uint32_t hash32_str(const void *bf, uint32_t hash) { const uint8_t *s = (const uint8_t *)bf; uint8_t c; while ((c = *s++) != 0) hash = hash * 33 + c; /* "perl": k=33, r=r+r/32 */ return (hash + (hash >> 5)); } /* * uint32_t * hash32_strn(const void *bf, size_t len, uint32_t hash) * return a 32 bit hash of NUL terminated ASCII string buf up to * a maximum of len bytes, * seeded with an initial hash value of hash (usually HASH32_STR_INIT). */ static __inline uint32_t hash32_strn(const void *bf, size_t len, uint32_t hash) { const uint8_t *s = (const uint8_t *)bf; uint8_t c; while ((c = *s++) != 0 && len-- != 0) hash = hash * 33 + c; /* "perl": k=33, r=r+r/32 */ return (hash + (hash >> 5)); } #endif /* __HAVE_HASH32_STR */ __BEGIN_DECLS uint32_t murmurhash2(const void *, size_t, uint32_t); __END_DECLS #endif /* !_SYS_HASH_H_ */ |
| 30 91 69 44 1289 2 34 247 1 8 584 22 250 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 197 198 199 200 201 202 203 204 205 206 207 208 209 210 211 212 213 214 215 216 217 218 219 220 221 222 223 224 225 226 227 228 229 230 231 232 233 234 235 236 237 238 239 240 241 242 243 244 245 246 247 248 249 250 251 252 253 254 255 256 257 258 259 260 261 262 263 264 265 266 267 268 269 270 271 272 273 274 275 276 277 278 279 280 281 282 283 284 285 286 287 288 289 290 291 292 293 294 295 296 297 298 299 300 301 302 303 304 305 306 307 308 309 310 311 312 313 314 315 316 317 318 319 320 321 322 323 324 325 326 327 328 329 330 331 332 333 334 335 336 337 338 339 340 341 342 343 344 345 346 347 348 349 350 351 352 353 354 355 356 357 358 359 360 361 362 363 364 365 366 367 368 369 370 371 372 373 374 375 376 377 378 379 380 381 382 383 384 385 386 387 388 389 390 391 392 393 394 395 396 397 398 399 400 401 402 403 404 405 406 407 408 409 410 411 412 413 414 415 416 417 418 419 420 421 422 423 424 425 426 427 428 429 430 431 432 433 434 435 436 437 438 439 440 441 442 443 444 445 446 447 448 449 450 451 452 453 454 455 456 457 458 459 460 461 462 463 464 465 466 467 468 469 | /* $NetBSD: ktrace.h,v 1.68 2022/06/29 22:10:43 riastradh Exp $ */ /* * Copyright (c) 1988, 1993 * The Regents of the University of California. All rights reserved. * * Redistribution and use in source and binary forms, with or without * modification, are permitted provided that the following conditions * are met: * 1. Redistributions of source code must retain the above copyright * notice, this list of conditions and the following disclaimer. * 2. Redistributions in binary form must reproduce the above copyright * notice, this list of conditions and the following disclaimer in the * documentation and/or other materials provided with the distribution. * 3. Neither the name of the University nor the names of its contributors * may be used to endorse or promote products derived from this software * without specific prior written permission. * * THIS SOFTWARE IS PROVIDED BY THE REGENTS AND CONTRIBUTORS ``AS IS'' AND * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE * ARE DISCLAIMED. IN NO EVENT SHALL THE REGENTS OR CONTRIBUTORS BE LIABLE * FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL * DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS * OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) * HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT * LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY * OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF * SUCH DAMAGE. * * @(#)ktrace.h 8.2 (Berkeley) 2/19/95 */ #ifndef _SYS_KTRACE_H_ #define _SYS_KTRACE_H_ #include <sys/mutex.h> #include <sys/lwp.h> #include <sys/signal.h> #include <sys/time.h> #include <sys/uio.h> /* * operations to ktrace system call (KTROP(op)) */ #define KTROP_SET 0 /* set trace points */ #define KTROP_CLEAR 1 /* clear trace points */ #define KTROP_CLEARFILE 2 /* stop all tracing to file */ #define KTROP_MASK 0x3 #define KTROP(o) ((o)&KTROP_MASK) /* macro to extract operation */ /* * flags (ORed in with operation) */ #define KTRFLAG_DESCEND 4 /* perform op on all children too */ /* * ktrace record header */ struct ktr_header { int ktr_len; /* length of record minus length of old header */ #if BYTE_ORDER == LITTLE_ENDIAN short ktr_type; /* trace record type */ short ktr_version; /* trace record version */ #else short ktr_version; /* trace record version */ short ktr_type; /* trace record type */ #endif pid_t ktr_pid; /* process id */ char ktr_comm[MAXCOMLEN+1]; /* command name */ union { struct { /* v0 */ struct { int32_t tv_sec; long tv_usec; } _tv; const void *_buf; } _v0; struct { /* v1 */ struct { int32_t tv_sec; long tv_nsec; } _ts; lwpid_t _lid; } _v1; struct { /* v2 */ struct timespec _ts; lwpid_t _lid; } _v2; } _v; }; #define ktr_lid _v._v2._lid #define ktr_olid _v._v1._lid #define ktr_time _v._v2._ts #define ktr_otv _v._v0._tv #define ktr_ots _v._v1._ts #define ktr_ts _v._v2._ts #define ktr_unused _v._v0._buf #define KTR_SHIMLEN offsetof(struct ktr_header, ktr_pid) /* * Test for kernel trace point */ #define KTRPOINT(p, type) \ (((p)->p_traceflag & (1<<(type))) != 0) /* * ktrace record types */ /* * KTR_SYSCALL - system call record */ #define KTR_SYSCALL 1 struct ktr_syscall { int ktr_code; /* syscall number */ int ktr_argsize; /* size of arguments */ /* * followed by ktr_argsize/sizeof(register_t) "register_t"s */ }; /* * KTR_SYSRET - return from system call record */ #define KTR_SYSRET 2 struct ktr_sysret { short ktr_code; short ktr_eosys; /* XXX unused */ int ktr_error; __register_t ktr_retval; __register_t ktr_retval_1; }; /* * KTR_NAMEI - namei record */ #define KTR_NAMEI 3 /* record contains pathname */ /* * KTR_GENIO - trace generic process i/o */ #define KTR_GENIO 4 struct ktr_genio { int ktr_fd; enum uio_rw ktr_rw; /* * followed by data successfully read/written */ }; /* * KTR_PSIG - trace processed signal */ #define KTR_PSIG 5 struct ktr_psig { int signo; sig_t action; sigset_t mask; int code; /* * followed by optional siginfo_t */ }; /* * KTR_CSW - trace context switches */ #define KTR_CSW 6 struct ktr_csw { int out; /* 1 if switch out, 0 if switch in */ int user; /* 1 if usermode (ivcsw), 0 if kernel (vcsw) */ }; /* * KTR_EMUL - emulation change */ #define KTR_EMUL 7 /* record contains emulation name */ /* * KTR_USER - user record */ #define KTR_USER 8 #define KTR_USER_MAXIDLEN 20 #define KTR_USER_MAXLEN 2048 /* maximum length of passed data */ struct ktr_user { char ktr_id[KTR_USER_MAXIDLEN]; /* string id of caller */ /* * Followed by ktr_len - sizeof(struct ktr_user) of user data. */ }; /* * KTR_EXEC_ARG, KTR_EXEC_ENV - Arguments and environment from exec */ #define KTR_EXEC_ARG 10 #define KTR_EXEC_ENV 11 /* record contains arg/env string */ /* * KTR_SAUPCALL - scheduler activated upcall. * * The structure is no longer used, but retained for compatibility. */ #define KTR_SAUPCALL 13 struct ktr_saupcall { int ktr_type; int ktr_nevent; int ktr_nint; void *ktr_sas; void *ktr_ap; /* * followed by nevent sa_t's from sas[] */ }; /* * KTR_MIB - MIB name and data */ #define KTR_MIB 14 /* Record contains MIB name */ /* * KTR_EXEC_FD - Opened file descriptor from exec */ #define KTR_EXEC_FD 15 struct ktr_execfd { int ktr_fd; u_int ktr_dtype; /* one of DTYPE_* constants */ }; /* * kernel trace points (in p_traceflag) */ #define KTRFAC_MASK 0x00ffffff #define KTRFAC_SYSCALL (1<<KTR_SYSCALL) #define KTRFAC_SYSRET (1<<KTR_SYSRET) #define KTRFAC_NAMEI (1<<KTR_NAMEI) #define KTRFAC_GENIO (1<<KTR_GENIO) #define KTRFAC_PSIG (1<<KTR_PSIG) #define KTRFAC_CSW (1<<KTR_CSW) #define KTRFAC_EMUL (1<<KTR_EMUL) #define KTRFAC_USER (1<<KTR_USER) #define KTRFAC_EXEC_ARG (1<<KTR_EXEC_ARG) #define KTRFAC_EXEC_ENV (1<<KTR_EXEC_ENV) #define KTRFAC_MIB (1<<KTR_MIB) #define KTRFAC_EXEC_FD (1<<KTR_EXEC_FD) #define __KTRACE_FLAG_BITS \ "\177\020" \ "b\1SYSCALL\0" \ "b\2SYSRET\0" \ "b\3NAMEI\0" \ "b\4GENIO\0" \ "b\5PSIG\0" \ "b\6CSW\0" \ "b\7EMUL\0" \ "b\10USER\0" \ "b\12EXEC_ARG\0" \ "b\13EXEC_ENV\0" \ "b\15SAUPCALL\0" \ "b\16MIB\0" \ "b\17EXEC_FD\0" \ "f\30\4VERSION\0" \ "b\36TRC_EMUL\0" \ "b\37INHERIT\0" \ "b\40PERSISTENT\0" /* * trace flags (also in p_traceflags) */ #define KTRFAC_PERSISTENT 0x80000000 /* persistent trace across sugid exec (exclusive) */ #define KTRFAC_INHERIT 0x40000000 /* pass trace flags to children */ #define KTRFAC_TRC_EMUL 0x10000000 /* ktrace KTR_EMUL before next trace */ #define KTRFAC_VER_MASK 0x0f000000 /* record version mask */ #define KTRFAC_VER_SHIFT 24 /* record version shift */ #define KTRFAC_VERSION(tf) (((tf) & KTRFAC_VER_MASK) >> KTRFAC_VER_SHIFT) #define KTRFACv0 (0 << KTRFAC_VER_SHIFT) #define KTRFACv1 (1 << KTRFAC_VER_SHIFT) #define KTRFACv2 (2 << KTRFAC_VER_SHIFT) #ifndef _KERNEL #include <sys/cdefs.h> __BEGIN_DECLS int ktrace(const char *, int, int, pid_t); int fktrace(int, int, int, pid_t); int utrace(const char *, void *, size_t); __END_DECLS #else struct syncobj; void ktrinit(void); void ktrderef(struct proc *); void ktradref(struct proc *); extern kmutex_t ktrace_lock; extern int ktrace_on; int ktruser(const char *, void *, size_t, int); bool ktr_point(int); void ktr_csw(int, int, const struct syncobj *); void ktr_emul(void); void ktr_geniov(int, enum uio_rw, struct iovec *, size_t, int); void ktr_genio(int, enum uio_rw, const void *, size_t, int); void ktr_mibio(int, enum uio_rw, const void *, size_t, int); void ktr_namei(const char *, size_t); void ktr_namei2(const char *, size_t, const char *, size_t); void ktr_psig(int, sig_t, const sigset_t *, const ksiginfo_t *); void ktr_syscall(register_t, const register_t [], int); void ktr_sysret(register_t, int, register_t *); void ktr_kuser(const char *, const void *, size_t); void ktr_mib(const int *a , u_int b); void ktr_execarg(const void *, size_t); void ktr_execenv(const void *, size_t); void ktr_execfd(int, u_int); int ktrace_common(lwp_t *, int, int, int, file_t **); static __inline int ktrenter(lwp_t *l) { if ((l->l_pflag & LP_KTRACTIVE) != 0) return 1; l->l_pflag |= LP_KTRACTIVE; return 0; } static __inline void ktrexit(lwp_t *l) { l->l_pflag &= ~LP_KTRACTIVE; } static __inline bool ktrpoint(int fac) { return __predict_false(ktrace_on) && __predict_false(ktr_point(1 << fac)); } static __inline void ktrcsw(int a, int b, const struct syncobj *c) { if (__predict_false(ktrace_on)) ktr_csw(a, b, c); } static __inline void ktremul(void) { if (__predict_false(ktrace_on)) ktr_emul(); } static __inline void ktrgenio(int a, enum uio_rw b, const void *c, size_t d, int e) { if (__predict_false(ktrace_on)) ktr_genio(a, b, c, d, e); } static __inline void ktrgeniov(int a, enum uio_rw b, struct iovec *c, int d, int e) { if (__predict_false(ktrace_on)) ktr_geniov(a, b, c, d, e); } static __inline void ktrmibio(int a, enum uio_rw b, const void *c, size_t d, int e) { if (__predict_false(ktrace_on)) ktr_mibio(a, b, c, d, e); } static __inline void ktrnamei(const char *a, size_t b) { if (__predict_false(ktrace_on)) ktr_namei(a, b); } static __inline void ktrnamei2(const char *a, size_t b, const char *c, size_t d) { if (__predict_false(ktrace_on)) ktr_namei2(a, b, c, d); } static __inline void ktrpsig(int a, sig_t b, const sigset_t *c, const ksiginfo_t * d) { if (__predict_false(ktrace_on)) ktr_psig(a, b, c, d); } static __inline void ktrsyscall(register_t code, const register_t args[], int narg) { if (__predict_false(ktrace_on)) ktr_syscall(code, args, narg); } static __inline void ktrsysret(register_t a, int b, register_t *c) { if (__predict_false(ktrace_on)) ktr_sysret(a, b, c); } static __inline void ktrkuser(const char *a, const void *b, size_t c) { if (__predict_false(ktrace_on)) ktr_kuser(a, b, c); } static __inline void ktrmib(const int *a , u_int b) { if (__predict_false(ktrace_on)) ktr_mib(a, b); } static __inline void ktrexecarg(const void *a, size_t b) { if (__predict_false(ktrace_on)) ktr_execarg(a, b); } static __inline void ktrexecenv(const void *a, size_t b) { if (__predict_false(ktrace_on)) ktr_execenv(a, b); } static __inline void ktrexecfd(int fd, u_int dtype) { if (__predict_false(ktrace_on)) ktr_execfd(fd, dtype); } struct ktrace_entry; int ktealloc(struct ktrace_entry **, void **, lwp_t *, int, size_t); void ktesethdrlen(struct ktrace_entry *, size_t); void ktraddentry(lwp_t *, struct ktrace_entry *, int); /* Flags for ktraddentry (3rd arg) */ #define KTA_NOWAIT 0x0000 #define KTA_WAITOK 0x0001 #define KTA_LARGE 0x0002 #endif /* !_KERNEL */ #endif /* _SYS_KTRACE_H_ */ |
| 8 8 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 197 198 199 200 201 202 203 204 205 206 207 208 209 210 211 212 213 214 215 216 217 218 219 220 221 222 223 224 225 226 227 228 229 230 231 232 233 234 235 236 237 238 239 240 241 242 243 244 245 246 247 248 249 250 251 252 253 254 255 256 257 258 259 260 261 262 263 264 265 266 267 268 269 270 271 272 273 274 275 276 277 278 279 280 281 282 283 284 285 286 287 288 289 290 291 292 293 294 295 296 297 298 299 300 301 302 303 304 305 306 307 308 309 310 311 312 313 314 315 316 317 318 319 320 321 322 323 324 325 326 327 328 329 330 331 332 333 334 335 336 337 338 339 340 341 342 343 344 345 346 347 348 349 350 351 352 353 354 355 356 357 358 359 360 361 362 363 364 365 366 367 368 369 370 371 372 373 374 375 376 377 378 379 380 381 382 383 384 385 386 387 388 389 390 391 392 393 394 395 396 397 398 399 400 401 402 403 404 405 406 407 408 409 410 411 412 413 414 415 416 417 418 419 420 421 422 423 424 425 426 427 428 429 430 431 432 433 434 435 436 437 438 439 440 441 442 443 444 445 446 447 448 449 450 451 452 453 454 455 456 457 458 459 460 461 462 463 464 465 466 467 468 469 470 471 472 473 474 475 476 477 478 479 480 481 482 483 484 485 486 487 488 489 490 491 492 493 494 495 496 497 498 499 500 501 502 503 504 505 506 507 508 509 510 511 512 513 514 515 516 517 518 519 520 521 522 523 524 525 526 527 528 529 530 531 532 533 534 535 536 537 538 539 540 541 542 543 544 545 546 547 548 549 550 551 552 553 554 555 556 557 558 559 560 561 562 563 564 565 566 567 568 569 570 571 572 573 574 575 576 577 578 579 580 581 582 583 584 585 586 587 588 589 590 591 592 593 594 595 596 597 598 599 600 601 602 603 604 605 606 607 608 609 610 611 612 613 614 615 616 617 618 619 620 621 622 623 624 625 626 627 628 629 630 631 632 633 634 635 636 637 638 639 640 641 642 643 644 645 646 647 648 649 650 651 652 653 654 655 656 657 658 659 660 661 662 663 664 665 666 667 668 669 670 671 672 673 674 675 676 677 678 679 680 681 682 683 684 685 686 687 688 689 690 691 692 693 694 695 696 697 698 699 700 701 702 703 704 705 706 707 708 709 710 711 712 713 714 715 716 717 718 719 720 721 722 723 724 725 726 727 728 729 730 731 732 733 734 735 736 737 738 739 740 741 742 743 744 745 746 747 748 749 750 751 752 753 754 755 756 757 758 759 760 761 762 763 764 765 766 767 768 769 770 771 772 773 774 775 776 777 778 779 780 781 782 783 784 785 786 787 788 789 790 791 792 793 794 795 796 797 798 799 800 801 802 803 804 805 806 807 808 809 810 811 812 813 814 815 816 817 818 819 820 821 822 823 824 825 826 827 828 829 830 831 832 833 834 835 836 837 838 839 840 841 842 843 844 845 846 847 848 849 850 851 852 853 854 855 856 857 858 859 860 861 862 863 864 865 866 867 868 869 870 871 872 873 874 875 876 877 878 879 880 881 882 883 884 885 886 887 888 889 890 891 892 893 894 895 896 897 898 899 900 901 902 903 904 905 906 907 908 909 910 911 912 913 914 915 916 917 918 919 920 921 922 923 924 925 926 927 928 929 930 931 932 933 934 935 936 937 938 939 940 941 942 943 944 945 946 947 948 949 950 951 952 953 954 955 956 957 958 959 960 961 962 963 964 965 966 967 968 969 970 971 972 973 974 975 976 977 978 979 980 981 982 983 984 985 986 987 988 989 990 991 992 993 994 995 996 997 998 999 1000 1001 1002 1003 1004 1005 1006 1007 1008 1009 1010 1011 1012 1013 1014 1015 1016 1017 1018 1019 1020 1021 1022 1023 1024 1025 1026 1027 1028 1029 1030 1031 1032 1033 1034 1035 1036 1037 1038 1039 1040 1041 1042 1043 1044 1045 1046 1047 1048 1049 1050 1051 1052 1053 1054 1055 1056 1057 1058 1059 1060 1061 1062 1063 1064 1065 1066 1067 1068 1069 1070 1071 1072 1073 1074 1075 1076 1077 1078 1079 1080 1081 1082 1083 1084 1085 1086 1087 1088 1089 1090 1091 1092 1093 1094 1095 1096 1097 1098 1099 1100 1101 1102 1103 1104 1105 1106 1107 1108 1109 1110 1111 1112 1113 1114 1115 1116 1117 1118 1119 1120 1121 1122 1123 1124 1125 1126 1127 1128 1129 1130 1131 1132 1133 1134 1135 1136 1137 1138 1139 1140 1141 1142 1143 1144 1145 1146 1147 1148 1149 1150 1151 1152 1153 1154 1155 1156 1157 1158 1159 1160 1161 1162 1163 1164 1165 1166 1167 1168 1169 1170 1171 1172 1173 1174 1175 1176 1177 1178 1179 1180 1181 1182 1183 1184 1185 1186 1187 1188 1189 1190 1191 1192 1193 1194 1195 1196 1197 1198 1199 1200 1201 1202 1203 1204 1205 1206 1207 1208 1209 1210 1211 1212 1213 1214 1215 1216 1217 1218 1219 1220 1221 1222 1223 1224 1225 1226 1227 1228 1229 1230 1231 1232 1233 1234 1235 1236 1237 1238 1239 1240 1241 1242 1243 1244 1245 1246 1247 1248 1249 1250 1251 1252 1253 1254 1255 1256 1257 1258 1259 1260 1261 1262 1263 1264 1265 1266 1267 1268 1269 1270 1271 1272 1273 1274 1275 1276 1277 1278 1279 1280 1281 1282 1283 1284 1285 1286 1287 1288 1289 1290 1291 1292 1293 1294 1295 1296 1297 1298 1299 1300 1301 1302 1303 1304 1305 1306 1307 1308 1309 1310 1311 1312 1313 1314 1315 1316 1317 1318 1319 1320 1321 1322 1323 1324 1325 1326 1327 1328 1329 1330 1331 1332 1333 1334 1335 1336 1337 1338 1339 1340 1341 1342 1343 1344 1345 1346 1347 1348 1349 1350 1351 1352 1353 1354 1355 1356 1357 1358 1359 1360 1361 1362 1363 1364 1365 1366 1367 1368 1369 1370 1371 1372 1373 1374 1375 1376 1377 1378 1379 1380 1381 1382 1383 1384 1385 1386 1387 1388 1389 1390 1391 1392 1393 1394 1395 1396 1397 1398 1399 1400 1401 1402 1403 1404 1405 1406 1407 1408 1409 1410 1411 1412 1413 1414 1415 1416 1417 1418 1419 1420 1421 1422 1423 1424 1425 1426 1427 1428 1429 1430 1431 1432 1433 1434 1435 1436 1437 1438 1439 1440 1441 1442 1443 1444 1445 1446 1447 1448 1449 1450 1451 1452 1453 1454 1455 1456 1457 1458 1459 1460 1461 1462 1463 1464 1465 1466 1467 1468 1469 1470 1471 1472 1473 1474 1475 1476 1477 1478 1479 1480 1481 1482 1483 1484 1485 1486 1487 1488 1489 1490 1491 1492 1493 1494 1495 1496 1497 1498 1499 1500 1501 1502 1503 1504 1505 1506 1507 1508 1509 1510 1511 1512 1513 1514 1515 1516 1517 1518 1519 1520 1521 1522 1523 1524 1525 1526 1527 1528 1529 1530 1531 1532 1533 1534 1535 1536 1537 1538 1539 1540 1541 1542 1543 1544 1545 1546 1547 1548 1549 1550 1551 1552 1553 1554 1555 1556 1557 1558 1559 1560 1561 1562 1563 1564 1565 1566 1567 1568 1569 1570 1571 1572 1573 1574 1575 1576 1577 1578 1579 1580 1581 1582 1583 1584 1585 1586 1587 1588 1589 1590 1591 1592 1593 1594 1595 1596 1597 1598 1599 1600 1601 1602 1603 1604 1605 1606 1607 1608 1609 1610 1611 1612 1613 1614 1615 1616 1617 1618 1619 1620 1621 1622 1623 1624 1625 1626 1627 1628 1629 1630 1631 1632 1633 1634 1635 1636 1637 1638 1639 1640 1641 1642 1643 1644 1645 1646 1647 1648 1649 1650 1651 1652 1653 1654 1655 1656 1657 1658 1659 1660 1661 1662 1663 1664 1665 1666 1667 1668 1669 1670 1671 1672 1673 1674 1675 1676 1677 1678 1679 1680 1681 1682 1683 1684 1685 1686 1687 1688 1689 1690 1691 1692 1693 1694 1695 1696 1697 1698 1699 1700 1701 1702 1703 1704 1705 1706 1707 1708 1709 1710 1711 1712 1713 1714 1715 1716 1717 1718 1719 1720 1721 1722 1723 1724 1725 1726 1727 1728 1729 1730 1731 1732 1733 1734 1735 1736 1737 1738 1739 1740 1741 1742 1743 1744 1745 1746 1747 1748 1749 1750 1751 1752 1753 1754 1755 1756 1757 1758 1759 1760 1761 1762 1763 1764 1765 1766 1767 1768 1769 1770 1771 1772 1773 1774 1775 1776 1777 1778 1779 1780 1781 1782 1783 1784 1785 1786 1787 1788 1789 1790 1791 1792 1793 1794 1795 1796 1797 1798 1799 1800 1801 1802 1803 1804 1805 1806 1807 1808 1809 1810 1811 1812 1813 1814 1815 1816 1817 1818 1819 1820 1821 1822 1823 1824 1825 1826 1827 1828 1829 1830 1831 1832 1833 1834 1835 1836 1837 1838 1839 1840 1841 1842 1843 1844 1845 1846 1847 1848 1849 1850 1851 1852 1853 1854 1855 1856 1857 1858 1859 1860 1861 1862 1863 1864 1865 1866 1867 1868 1869 1870 1871 1872 1873 1874 1875 1876 1877 1878 1879 1880 1881 1882 1883 1884 1885 1886 1887 1888 1889 1890 1891 1892 1893 1894 1895 1896 1897 1898 1899 1900 1901 1902 1903 1904 1905 1906 1907 1908 1909 1910 1911 1912 1913 1914 1915 1916 1917 1918 1919 1920 1921 1922 1923 1924 1925 1926 1927 1928 1929 1930 1931 1932 1933 1934 1935 1936 1937 1938 1939 1940 1941 1942 1943 1944 1945 1946 1947 1948 1949 1950 1951 1952 1953 1954 1955 1956 1957 1958 1959 1960 1961 1962 1963 1964 1965 1966 1967 1968 1969 1970 1971 1972 1973 1974 1975 1976 1977 1978 1979 1980 1981 1982 1983 1984 1985 1986 1987 1988 1989 1990 1991 1992 1993 1994 1995 1996 1997 1998 1999 2000 2001 2002 2003 2004 2005 2006 2007 2008 2009 2010 2011 2012 2013 2014 2015 2016 2017 2018 2019 2020 2021 2022 2023 2024 2025 2026 2027 2028 2029 2030 2031 2032 2033 2034 2035 2036 2037 2038 2039 2040 2041 2042 2043 2044 2045 2046 2047 2048 2049 2050 2051 2052 2053 2054 2055 2056 2057 2058 2059 2060 2061 2062 2063 2064 2065 2066 2067 2068 2069 2070 2071 2072 2073 2074 2075 2076 2077 2078 2079 2080 2081 2082 2083 2084 2085 2086 2087 2088 2089 2090 2091 2092 2093 2094 2095 2096 2097 2098 2099 2100 2101 2102 2103 2104 2105 2106 2107 2108 2109 2110 2111 2112 2113 2114 2115 2116 2117 2118 2119 2120 2121 2122 2123 2124 2125 2126 2127 2128 2129 2130 2131 2132 2133 2134 2135 2136 2137 2138 2139 2140 2141 2142 2143 2144 2145 2146 2147 2148 2149 2150 2151 2152 2153 2154 2155 2156 2157 2158 2159 2160 2161 2162 2163 2164 2165 2166 2167 2168 2169 2170 2171 2172 2173 2174 2175 2176 2177 2178 2179 2180 2181 2182 2183 2184 2185 2186 2187 2188 2189 2190 2191 2192 2193 2194 2195 2196 2197 2198 2199 2200 2201 2202 2203 2204 2205 2206 2207 2208 2209 2210 2211 2212 2213 2214 2215 2216 2217 2218 2219 2220 2221 2222 2223 2224 2225 2226 2227 2228 2229 2230 2231 2232 2233 2234 2235 2236 2237 2238 2239 2240 2241 2242 2243 2244 2245 2246 2247 2248 2249 2250 2251 2252 2253 2254 2255 2256 2257 2258 2259 2260 2261 2262 2263 2264 2265 2266 2267 2268 2269 2270 2271 2272 2273 2274 2275 2276 2277 2278 2279 2280 2281 2282 2283 2284 2285 2286 2287 2288 2289 2290 2291 2292 2293 2294 2295 2296 2297 2298 2299 2300 2301 2302 2303 2304 2305 2306 2307 2308 2309 2310 2311 2312 2313 2314 2315 2316 2317 2318 2319 2320 2321 2322 2323 2324 2325 2326 2327 2328 2329 2330 2331 2332 2333 2334 2335 2336 2337 2338 2339 2340 2341 2342 2343 2344 2345 2346 2347 2348 2349 2350 2351 2352 2353 2354 2355 2356 2357 2358 2359 2360 2361 2362 2363 2364 2365 2366 2367 2368 2369 2370 2371 2372 2373 2374 2375 2376 2377 2378 2379 2380 2381 2382 2383 2384 2385 2386 2387 2388 2389 2390 2391 2392 2393 2394 2395 2396 2397 2398 2399 2400 2401 2402 2403 2404 2405 2406 2407 2408 2409 2410 2411 2412 2413 2414 2415 2416 2417 2418 2419 2420 2421 2422 2423 2424 2425 2426 2427 2428 2429 2430 2431 2432 2433 2434 2435 2436 2437 2438 2439 2440 2441 2442 2443 2444 2445 2446 2447 2448 2449 2450 2451 2452 2453 2454 2455 2456 2457 2458 2459 2460 2461 2462 2463 2464 2465 2466 2467 2468 2469 2470 2471 2472 2473 2474 2475 2476 2477 2478 2479 2480 2481 2482 2483 2484 2485 2486 2487 2488 2489 2490 2491 2492 2493 2494 2495 2496 2497 2498 2499 2500 2501 2502 2503 2504 2505 2506 2507 2508 2509 2510 2511 2512 2513 2514 2515 2516 2517 2518 2519 2520 2521 2522 2523 2524 2525 2526 2527 2528 2529 2530 2531 2532 2533 2534 2535 2536 2537 2538 2539 2540 2541 2542 2543 2544 2545 2546 2547 2548 2549 2550 2551 2552 2553 2554 2555 2556 2557 2558 2559 2560 2561 2562 2563 2564 2565 2566 2567 2568 2569 2570 2571 2572 2573 2574 2575 2576 2577 2578 2579 2580 2581 2582 2583 2584 2585 2586 2587 2588 2589 2590 2591 2592 2593 2594 2595 2596 2597 2598 2599 2600 2601 2602 2603 2604 2605 2606 2607 2608 2609 2610 2611 2612 2613 2614 2615 2616 2617 2618 2619 2620 2621 2622 2623 2624 2625 2626 2627 2628 2629 2630 2631 2632 2633 2634 2635 2636 2637 2638 2639 2640 2641 2642 2643 2644 2645 2646 2647 2648 2649 2650 2651 2652 2653 2654 2655 2656 2657 2658 2659 2660 2661 2662 2663 2664 2665 2666 2667 2668 2669 2670 2671 2672 2673 2674 2675 2676 2677 2678 2679 2680 2681 2682 2683 2684 2685 2686 2687 2688 2689 2690 2691 2692 2693 2694 2695 2696 2697 2698 2699 2700 2701 2702 2703 2704 2705 2706 2707 2708 2709 2710 2711 2712 2713 2714 2715 2716 2717 2718 2719 2720 2721 2722 2723 2724 2725 2726 2727 2728 2729 2730 2731 2732 2733 2734 2735 2736 2737 2738 2739 2740 2741 2742 2743 2744 2745 2746 2747 2748 2749 2750 2751 2752 2753 2754 2755 2756 2757 2758 2759 2760 2761 2762 2763 2764 2765 2766 2767 2768 2769 2770 2771 2772 2773 2774 2775 2776 2777 2778 2779 2780 2781 2782 2783 2784 2785 2786 2787 2788 2789 2790 2791 2792 2793 2794 2795 2796 2797 2798 2799 2800 2801 2802 2803 2804 2805 2806 2807 2808 2809 2810 2811 2812 2813 2814 2815 2816 2817 2818 2819 2820 2821 2822 2823 2824 2825 2826 2827 2828 2829 2830 2831 2832 2833 2834 2835 2836 2837 2838 2839 2840 2841 2842 2843 2844 2845 2846 2847 2848 2849 2850 2851 2852 2853 2854 2855 2856 2857 2858 2859 2860 2861 2862 2863 2864 2865 2866 2867 2868 2869 2870 2871 2872 2873 2874 2875 2876 2877 2878 2879 2880 2881 2882 2883 2884 2885 2886 2887 2888 2889 2890 2891 2892 2893 2894 2895 2896 2897 2898 2899 2900 2901 2902 2903 2904 2905 2906 2907 2908 2909 2910 2911 2912 2913 2914 2915 2916 2917 2918 2919 2920 2921 2922 2923 2924 2925 2926 2927 2928 2929 2930 2931 2932 2933 2934 2935 2936 2937 2938 2939 2940 2941 2942 2943 2944 2945 2946 2947 2948 2949 2950 2951 2952 2953 2954 2955 2956 2957 2958 2959 2960 2961 2962 2963 2964 2965 2966 2967 2968 2969 2970 2971 2972 2973 2974 2975 2976 2977 2978 2979 2980 2981 2982 2983 2984 2985 2986 2987 2988 2989 2990 2991 2992 2993 2994 2995 2996 2997 2998 2999 3000 3001 3002 3003 3004 3005 3006 3007 3008 3009 3010 3011 3012 3013 3014 3015 3016 3017 3018 3019 3020 3021 3022 3023 3024 3025 3026 3027 3028 3029 3030 3031 3032 3033 3034 3035 3036 3037 3038 3039 3040 3041 3042 3043 3044 3045 3046 3047 3048 3049 3050 3051 3052 3053 3054 3055 3056 3057 3058 3059 3060 3061 3062 3063 3064 3065 3066 3067 3068 3069 3070 3071 3072 3073 3074 3075 3076 3077 3078 3079 3080 3081 3082 3083 3084 3085 3086 3087 3088 3089 3090 3091 3092 3093 3094 3095 3096 3097 3098 3099 3100 3101 3102 3103 3104 3105 3106 3107 3108 3109 3110 3111 3112 3113 3114 3115 3116 3117 3118 3119 3120 3121 3122 3123 3124 3125 3126 3127 3128 3129 3130 3131 3132 3133 3134 3135 3136 3137 3138 3139 3140 3141 3142 3143 3144 3145 3146 3147 3148 3149 3150 3151 3152 3153 3154 3155 3156 3157 3158 3159 3160 3161 3162 3163 3164 3165 3166 3167 3168 3169 3170 3171 3172 3173 3174 3175 3176 3177 3178 3179 3180 3181 3182 3183 3184 3185 3186 3187 3188 3189 3190 3191 3192 3193 3194 3195 3196 3197 3198 3199 3200 3201 3202 3203 3204 3205 3206 3207 3208 3209 3210 3211 3212 3213 3214 3215 3216 3217 3218 3219 3220 3221 3222 3223 3224 | /* $NetBSD: uaudio.c,v 1.174 2022/06/28 05:22:13 skrll Exp $ */ /* * Copyright (c) 1999, 2012 The NetBSD Foundation, Inc. * All rights reserved. * * This code is derived from software contributed to The NetBSD Foundation * by Lennart Augustsson (lennart@augustsson.net) at * Carlstedt Research & Technology, and Matthew R. Green (mrg@eterna.com.au). * * Redistribution and use in source and binary forms, with or without * modification, are permitted provided that the following conditions * are met: * 1. Redistributions of source code must retain the above copyright * notice, this list of conditions and the following disclaimer. * 2. Redistributions in binary form must reproduce the above copyright * notice, this list of conditions and the following disclaimer in the * documentation and/or other materials provided with the distribution. * * THIS SOFTWARE IS PROVIDED BY THE NETBSD FOUNDATION, INC. AND CONTRIBUTORS * ``AS IS'' AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED * TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR * PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE FOUNDATION OR CONTRIBUTORS * BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR * CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF * SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS * INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN * CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) * ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE * POSSIBILITY OF SUCH DAMAGE. */ /* * USB audio specs: http://www.usb.org/developers/docs/devclass_docs/audio10.pdf * http://www.usb.org/developers/docs/devclass_docs/frmts10.pdf * http://www.usb.org/developers/docs/devclass_docs/termt10.pdf */ #include <sys/cdefs.h> __KERNEL_RCSID(0, "$NetBSD: uaudio.c,v 1.174 2022/06/28 05:22:13 skrll Exp $"); #ifdef _KERNEL_OPT #include "opt_usb.h" #endif #include <sys/param.h> #include <sys/systm.h> #include <sys/kernel.h> #include <sys/malloc.h> #include <sys/device.h> #include <sys/ioctl.h> #include <sys/file.h> #include <sys/reboot.h> /* for bootverbose */ #include <sys/select.h> #include <sys/proc.h> #include <sys/vnode.h> #include <sys/poll.h> #include <sys/module.h> #include <sys/bus.h> #include <sys/cpu.h> #include <sys/atomic.h> #include <sys/audioio.h> #include <dev/audio/audio_if.h> #include <dev/usb/usb.h> #include <dev/usb/usbdi.h> #include <dev/usb/usbdivar.h> #include <dev/usb/usbdi_util.h> #include <dev/usb/usb_quirks.h> #include <dev/usb/usbdevs.h> #include <dev/usb/uaudioreg.h> /* #define UAUDIO_DEBUG */ /* #define UAUDIO_MULTIPLE_ENDPOINTS */ #ifdef UAUDIO_DEBUG #define DPRINTF(x,y...) do { \ if (uaudiodebug) { \ struct lwp *l = curlwp; \ printf("%s[%d:%d]: "x, __func__, l->l_proc->p_pid, l->l_lid, y); \ } \ } while (0) #define DPRINTFN_CLEAN(n,x...) do { \ if (uaudiodebug > (n)) \ printf(x); \ } while (0) #define DPRINTFN(n,x,y...) do { \ if (uaudiodebug > (n)) { \ struct lwp *l = curlwp; \ printf("%s[%d:%d]: "x, __func__, l->l_proc->p_pid, l->l_lid, y); \ } \ } while (0) int uaudiodebug = 0; #else #define DPRINTF(x,y...) #define DPRINTFN_CLEAN(n,x...) #define DPRINTFN(n,x,y...) #endif #define UAUDIO_NCHANBUFS 6 /* number of outstanding request */ #define UAUDIO_NFRAMES 10 /* ms of sound in each request */ #define MIX_MAX_CHAN 8 struct mixerctl { uint16_t wValue[MIX_MAX_CHAN]; /* using nchan */ uint16_t wIndex; uint8_t nchan; uint8_t type; #define MIX_ON_OFF 1 #define MIX_SIGNED_16 2 #define MIX_UNSIGNED_16 3 #define MIX_SIGNED_8 4 #define MIX_SELECTOR 5 #define MIX_SIZE(n) ((n) == MIX_SIGNED_16 || (n) == MIX_UNSIGNED_16 ? 2 : 1) #define MIX_UNSIGNED(n) ((n) == MIX_UNSIGNED_16) int minval, maxval; u_int delta; u_int mul; uint8_t class; char ctlname[MAX_AUDIO_DEV_LEN]; const char *ctlunit; }; #define MAKE(h,l) (((h) << 8) | (l)) struct as_info { uint8_t alt; uint8_t encoding; uint8_t attributes; /* Copy of bmAttributes of * usb_audio_streaming_endpoint_descriptor */ struct usbd_interface * ifaceh; const usb_interface_descriptor_t *idesc; const usb_endpoint_descriptor_audio_t *edesc; const usb_endpoint_descriptor_audio_t *edesc1; const struct usb_audio_streaming_type1_descriptor *asf1desc; struct audio_format *aformat; int sc_busy; /* currently used */ }; struct chan { void (*intr)(void *); /* DMA completion intr handler */ void *arg; /* arg for intr() */ struct usbd_pipe *pipe; struct usbd_pipe *sync_pipe; u_int sample_size; u_int sample_rate; u_int bytes_per_frame; u_int fraction; /* fraction/1000 is the extra samples/frame */ u_int residue; /* accumulates the fractional samples */ u_char *start; /* upper layer buffer start */ u_char *end; /* upper layer buffer end */ u_char *cur; /* current position in upper layer buffer */ int blksize; /* chunk size to report up */ int transferred; /* transferred bytes not reported up */ int altidx; /* currently used altidx */ int curchanbuf; struct chanbuf { struct chan *chan; struct usbd_xfer *xfer; u_char *buffer; uint16_t sizes[UAUDIO_NFRAMES]; uint16_t offsets[UAUDIO_NFRAMES]; uint16_t size; } chanbufs[UAUDIO_NCHANBUFS]; struct uaudio_softc *sc; /* our softc */ }; /* * The MI USB audio subsystem is now MP-SAFE and expects sc_intr_lock to be * held on entry the callbacks passed to uaudio_trigger_{in,out}put */ struct uaudio_softc { device_t sc_dev; /* base device */ kmutex_t sc_lock; kmutex_t sc_intr_lock; struct usbd_device *sc_udev; /* USB device */ int sc_ac_iface; /* Audio Control interface */ struct usbd_interface * sc_ac_ifaceh; struct chan sc_playchan; /* play channel */ struct chan sc_recchan; /* record channel */ int sc_nullalt; int sc_audio_rev; struct as_info *sc_alts; /* alternate settings */ int sc_nalts; /* # of alternate settings */ int sc_altflags; #define HAS_8 0x01 #define HAS_16 0x02 #define HAS_8U 0x04 #define HAS_ALAW 0x08 #define HAS_MULAW 0x10 #define UA_NOFRAC 0x20 /* don't do sample rate adjustment */ #define HAS_24 0x40 int sc_mode; /* play/record capability */ struct mixerctl *sc_ctls; /* mixer controls */ int sc_nctls; /* # of mixer controls */ device_t sc_audiodev; struct audio_format *sc_formats; int sc_nformats; u_int sc_channel_config; char sc_dying; struct audio_device sc_adev; }; struct terminal_list { int size; uint16_t terminals[1]; }; #define TERMINAL_LIST_SIZE(N) (offsetof(struct terminal_list, terminals) \ + sizeof(uint16_t) * (N)) struct io_terminal { union { const uaudio_cs_descriptor_t *desc; const struct usb_audio_input_terminal *it; const struct usb_audio_output_terminal *ot; const struct usb_audio_mixer_unit *mu; const struct usb_audio_selector_unit *su; const struct usb_audio_feature_unit *fu; const struct usb_audio_processing_unit *pu; const struct usb_audio_extension_unit *eu; } d; int inputs_size; struct terminal_list **inputs; /* list of source input terminals */ struct terminal_list *output; /* list of destination output terminals */ int direct; /* directly connected to an output terminal */ }; #define UAC_OUTPUT 0 #define UAC_INPUT 1 #define UAC_EQUAL 2 #define UAC_RECORD 3 #define UAC_NCLASSES 4 #ifdef UAUDIO_DEBUG Static const char *uac_names[] = { AudioCoutputs, AudioCinputs, AudioCequalization, AudioCrecord, }; #endif #ifdef UAUDIO_DEBUG Static void uaudio_dump_tml (struct terminal_list *tml); #endif Static usbd_status uaudio_identify_ac (struct uaudio_softc *, const usb_config_descriptor_t *); Static usbd_status uaudio_identify_as (struct uaudio_softc *, const usb_config_descriptor_t *); Static usbd_status uaudio_process_as (struct uaudio_softc *, const char *, int *, int, const usb_interface_descriptor_t *); Static void uaudio_add_alt(struct uaudio_softc *, const struct as_info *); Static const usb_interface_descriptor_t *uaudio_find_iface (const char *, int, int *, int); Static void uaudio_mixer_add_ctl(struct uaudio_softc *, struct mixerctl *); Static char *uaudio_id_name (struct uaudio_softc *, const struct io_terminal *, int); #ifdef UAUDIO_DEBUG Static void uaudio_dump_cluster(const struct usb_audio_cluster *); #endif Static struct usb_audio_cluster uaudio_get_cluster (int, const struct io_terminal *); Static void uaudio_add_input (struct uaudio_softc *, const struct io_terminal *, int); Static void uaudio_add_output (struct uaudio_softc *, const struct io_terminal *, int); Static void uaudio_add_mixer (struct uaudio_softc *, const struct io_terminal *, int); Static void uaudio_add_selector (struct uaudio_softc *, const struct io_terminal *, int); #ifdef UAUDIO_DEBUG Static const char *uaudio_get_terminal_name(int); #endif Static int uaudio_determine_class (const struct io_terminal *, struct mixerctl *); Static const char *uaudio_feature_name (const struct io_terminal *, struct mixerctl *); Static void uaudio_add_feature (struct uaudio_softc *, const struct io_terminal *, int); Static void uaudio_add_processing_updown (struct uaudio_softc *, const struct io_terminal *, int); Static void uaudio_add_processing (struct uaudio_softc *, const struct io_terminal *, int); Static void uaudio_add_extension (struct uaudio_softc *, const struct io_terminal *, int); Static struct terminal_list *uaudio_merge_terminal_list (const struct io_terminal *); Static struct terminal_list *uaudio_io_terminaltype (int, struct io_terminal *, int); Static usbd_status uaudio_identify (struct uaudio_softc *, const usb_config_descriptor_t *); Static int uaudio_signext(int, int); Static int uaudio_value2bsd(struct mixerctl *, int); Static int uaudio_bsd2value(struct mixerctl *, int); Static int uaudio_get(struct uaudio_softc *, int, int, int, int, int); Static int uaudio_ctl_get (struct uaudio_softc *, int, struct mixerctl *, int); Static void uaudio_set (struct uaudio_softc *, int, int, int, int, int, int); Static void uaudio_ctl_set (struct uaudio_softc *, int, struct mixerctl *, int, int); Static usbd_status uaudio_set_speed(struct uaudio_softc *, int, u_int); Static usbd_status uaudio_chan_open(struct uaudio_softc *, struct chan *); Static void uaudio_chan_abort(struct uaudio_softc *, struct chan *); Static void uaudio_chan_close(struct uaudio_softc *, struct chan *); Static usbd_status uaudio_chan_alloc_buffers (struct uaudio_softc *, struct chan *); Static void uaudio_chan_free_buffers(struct uaudio_softc *, struct chan *); Static void uaudio_chan_init (struct chan *, int, const struct audio_params *, int); Static void uaudio_chan_set_param(struct chan *, u_char *, u_char *, int); Static void uaudio_chan_ptransfer(struct chan *); Static void uaudio_chan_pintr (struct usbd_xfer *, void *, usbd_status); Static void uaudio_chan_rtransfer(struct chan *); Static void uaudio_chan_rintr (struct usbd_xfer *, void *, usbd_status); Static int uaudio_open(void *, int); Static int uaudio_query_format(void *, audio_format_query_t *); Static int uaudio_set_format (void *, int, const audio_params_t *, const audio_params_t *, audio_filter_reg_t *, audio_filter_reg_t *); Static int uaudio_round_blocksize(void *, int, int, const audio_params_t *); Static int uaudio_trigger_output (void *, void *, void *, int, void (*)(void *), void *, const audio_params_t *); Static int uaudio_trigger_input (void *, void *, void *, int, void (*)(void *), void *, const audio_params_t *); Static int uaudio_halt_in_dma(void *); Static int uaudio_halt_out_dma(void *); Static void uaudio_halt_in_dma_unlocked(struct uaudio_softc *); Static void uaudio_halt_out_dma_unlocked(struct uaudio_softc *); Static int uaudio_getdev(void *, struct audio_device *); Static int uaudio_mixer_set_port(void *, mixer_ctrl_t *); Static int uaudio_mixer_get_port(void *, mixer_ctrl_t *); Static int uaudio_query_devinfo(void *, mixer_devinfo_t *); Static int uaudio_get_props(void *); Static void uaudio_get_locks(void *, kmutex_t **, kmutex_t **); Static const struct audio_hw_if uaudio_hw_if = { .open = uaudio_open, .query_format = uaudio_query_format, .set_format = uaudio_set_format, .round_blocksize = uaudio_round_blocksize, .halt_output = uaudio_halt_out_dma, .halt_input = uaudio_halt_in_dma, .getdev = uaudio_getdev, .set_port = uaudio_mixer_set_port, .get_port = uaudio_mixer_get_port, .query_devinfo = uaudio_query_devinfo, .get_props = uaudio_get_props, .trigger_output = uaudio_trigger_output, .trigger_input = uaudio_trigger_input, .get_locks = uaudio_get_locks, }; static int uaudio_match(device_t, cfdata_t, void *); static void uaudio_attach(device_t, device_t, void *); static int uaudio_detach(device_t, int); static void uaudio_childdet(device_t, device_t); static int uaudio_activate(device_t, enum devact); CFATTACH_DECL2_NEW(uaudio, sizeof(struct uaudio_softc), uaudio_match, uaudio_attach, uaudio_detach, uaudio_activate, NULL, uaudio_childdet); static int uaudio_match(device_t parent, cfdata_t match, void *aux) { struct usbif_attach_arg *uiaa = aux; /* Trigger on the control interface. */ if (uiaa->uiaa_class != UICLASS_AUDIO || uiaa->uiaa_subclass != UISUBCLASS_AUDIOCONTROL || (usbd_get_quirks(uiaa->uiaa_device)->uq_flags & UQ_BAD_AUDIO)) return UMATCH_NONE; return UMATCH_IFACECLASS_IFACESUBCLASS; } static void uaudio_attach(device_t parent, device_t self, void *aux) { struct uaudio_softc *sc = device_private(self); struct usbif_attach_arg *uiaa = aux; usb_interface_descriptor_t *id; usb_config_descriptor_t *cdesc; char *devinfop; usbd_status err; int i, j, found; sc->sc_dev = self; sc->sc_udev = uiaa->uiaa_device; mutex_init(&sc->sc_lock, MUTEX_DEFAULT, IPL_NONE); mutex_init(&sc->sc_intr_lock, MUTEX_DEFAULT, IPL_SOFTUSB); strlcpy(sc->sc_adev.name, "USB audio", sizeof(sc->sc_adev.name)); strlcpy(sc->sc_adev.version, "", sizeof(sc->sc_adev.version)); snprintf(sc->sc_adev.config, sizeof(sc->sc_adev.config), "usb:%08x", sc->sc_udev->ud_cookie.cookie); aprint_naive("\n"); aprint_normal("\n"); devinfop = usbd_devinfo_alloc(uiaa->uiaa_device, 0); aprint_normal_dev(self, "%s\n", devinfop); usbd_devinfo_free(devinfop); cdesc = usbd_get_config_descriptor(sc->sc_udev); if (cdesc == NULL) { aprint_error_dev(self, "failed to get configuration descriptor\n"); return; } err = uaudio_identify(sc, cdesc); if (err) { aprint_error_dev(self, "audio descriptors make no sense, error=%d\n", err); return; } sc->sc_ac_ifaceh = uiaa->uiaa_iface; /* Pick up the AS interface. */ for (i = 0; i < uiaa->uiaa_nifaces; i++) { if (uiaa->uiaa_ifaces[i] == NULL) continue; id = usbd_get_interface_descriptor(uiaa->uiaa_ifaces[i]); if (id == NULL) continue; found = 0; for (j = 0; j < sc->sc_nalts; j++) { if (id->bInterfaceNumber == sc->sc_alts[j].idesc->bInterfaceNumber) { sc->sc_alts[j].ifaceh = uiaa->uiaa_ifaces[i]; found = 1; } } if (found) uiaa->uiaa_ifaces[i] = NULL; } for (j = 0; j < sc->sc_nalts; j++) { if (sc->sc_alts[j].ifaceh == NULL) { aprint_error_dev(self, "alt %d missing AS interface(s)\n", j); return; } } aprint_normal_dev(self, "audio rev %d.%02x\n", sc->sc_audio_rev >> 8, sc->sc_audio_rev & 0xff); sc->sc_playchan.sc = sc->sc_recchan.sc = sc; sc->sc_playchan.altidx = -1; sc->sc_recchan.altidx = -1; if (usbd_get_quirks(sc->sc_udev)->uq_flags & UQ_AU_NO_FRAC) sc->sc_altflags |= UA_NOFRAC; #ifndef UAUDIO_DEBUG if (bootverbose) #endif aprint_normal_dev(self, "%d mixer controls\n", sc->sc_nctls); usbd_add_drv_event(USB_EVENT_DRIVER_ATTACH, sc->sc_udev, sc->sc_dev); DPRINTF("%s", "doing audio_attach_mi\n"); sc->sc_audiodev = audio_attach_mi(&uaudio_hw_if, sc, sc->sc_dev); if (!pmf_device_register(self, NULL, NULL)) aprint_error_dev(self, "couldn't establish power handler\n"); return; } static int uaudio_activate(device_t self, enum devact act) { struct uaudio_softc *sc = device_private(self); switch (act) { case DVACT_DEACTIVATE: sc->sc_dying = 1; return 0; default: return EOPNOTSUPP; } } static void uaudio_childdet(device_t self, device_t child) { struct uaudio_softc *sc = device_private(self); KASSERT(sc->sc_audiodev == child); sc->sc_audiodev = NULL; } static int uaudio_detach(device_t self, int flags) { struct uaudio_softc *sc = device_private(self); int rv; sc->sc_dying = 1; pmf_device_deregister(self); /* Wait for outstanding requests to complete. */ uaudio_halt_out_dma_unlocked(sc); uaudio_halt_in_dma_unlocked(sc); if (sc->sc_audiodev != NULL) { rv = config_detach(sc->sc_audiodev, flags); if (rv) return rv; } usbd_add_drv_event(USB_EVENT_DRIVER_DETACH, sc->sc_udev, sc->sc_dev); if (sc->sc_formats != NULL) kmem_free(sc->sc_formats, sizeof(struct audio_format) * sc->sc_nformats); mutex_destroy(&sc->sc_lock); mutex_destroy(&sc->sc_intr_lock); return 0; } Static int uaudio_query_format(void *addr, audio_format_query_t *afp) { struct uaudio_softc *sc; sc = addr; return audio_query_format(sc->sc_formats, sc->sc_nformats, afp); } Static const usb_interface_descriptor_t * uaudio_find_iface(const char *tbuf, int size, int *offsp, int subtype) { const usb_interface_descriptor_t *d; while (*offsp + sizeof(*d) <= size) { d = (const void *)(tbuf + *offsp); DPRINTFN(3, "%d + %d <= %d type %d class %d/%d iface %d\n", *offsp, d->bLength, size, d->bDescriptorType, d->bInterfaceClass, d->bInterfaceSubClass, d->bInterfaceNumber); *offsp += d->bLength; if (d->bDescriptorType == UDESC_INTERFACE && d->bInterfaceClass == UICLASS_AUDIO && d->bInterfaceSubClass == subtype) return d; } return NULL; } Static void uaudio_mixer_add_ctl(struct uaudio_softc *sc, struct mixerctl *mc) { int res; size_t len; struct mixerctl *nmc; if (mc->class < UAC_NCLASSES) { DPRINTF("adding %s.%s\n", uac_names[mc->class], mc->ctlname); } else { DPRINTF("adding %s\n", mc->ctlname); } len = sizeof(*mc) * (sc->sc_nctls + 1); nmc = kmem_alloc(len, KM_SLEEP); /* Copy old data, if there was any */ if (sc->sc_nctls != 0) { memcpy(nmc, sc->sc_ctls, sizeof(*mc) * (sc->sc_nctls)); kmem_free(sc->sc_ctls, sizeof(*mc) * sc->sc_nctls); } sc->sc_ctls = nmc; mc->delta = 0; if (mc->type == MIX_ON_OFF) { mc->minval = 0; mc->maxval = 1; } else if (mc->type == MIX_SELECTOR) { ; } else { /* Determine min and max values. */ mc->minval = uaudio_signext(mc->type, uaudio_get(sc, GET_MIN, UT_READ_CLASS_INTERFACE, mc->wValue[0], mc->wIndex, MIX_SIZE(mc->type))); mc->maxval = 1 + uaudio_signext(mc->type, uaudio_get(sc, GET_MAX, UT_READ_CLASS_INTERFACE, mc->wValue[0], mc->wIndex, MIX_SIZE(mc->type))); mc->mul = mc->maxval - mc->minval; if (mc->mul == 0) mc->mul = 1; res = uaudio_get(sc, GET_RES, UT_READ_CLASS_INTERFACE, mc->wValue[0], mc->wIndex, MIX_SIZE(mc->type)); if (res > 0) mc->delta = (res * 255 + mc->mul/2) / mc->mul; } sc->sc_ctls[sc->sc_nctls++] = *mc; #ifdef UAUDIO_DEBUG if (uaudiodebug > 2) { int i; DPRINTFN_CLEAN(2, "wValue=%04x", mc->wValue[0]); for (i = 1; i < mc->nchan; i++) DPRINTFN_CLEAN(2, ",%04x", mc->wValue[i]); DPRINTFN_CLEAN(2, " wIndex=%04x type=%d name='%s' unit='%s' " "min=%d max=%d\n", mc->wIndex, mc->type, mc->ctlname, mc->ctlunit, mc->minval, mc->maxval); } #endif } Static char * uaudio_id_name(struct uaudio_softc *sc, const struct io_terminal *iot, int id) { static char tbuf[32]; snprintf(tbuf, sizeof(tbuf), "i%d", id); return tbuf; } #ifdef UAUDIO_DEBUG Static void uaudio_dump_cluster(const struct usb_audio_cluster *cl) { static const char *channel_names[16] = { "LEFT", "RIGHT", "CENTER", "LFE", "LEFT_SURROUND", "RIGHT_SURROUND", "LEFT_CENTER", "RIGHT_CENTER", "SURROUND", "LEFT_SIDE", "RIGHT_SIDE", "TOP", "RESERVED12", "RESERVED13", "RESERVED14", "RESERVED15", }; int cc, i, first; cc = UGETW(cl->wChannelConfig); printf("cluster: bNrChannels=%u wChannelConfig=%#.4x", cl->bNrChannels, cc); first = TRUE; for (i = 0; cc != 0; i++) { if (cc & 1) { printf("%c%s", first ? '<' : ',', channel_names[i]); first = FALSE; } cc = cc >> 1; } printf("> iChannelNames=%u", cl->iChannelNames); } #endif Static struct usb_audio_cluster uaudio_get_cluster(int id, const struct io_terminal *iot) { struct usb_audio_cluster r; const uaudio_cs_descriptor_t *dp; int i; for (i = 0; i < 25; i++) { /* avoid infinite loops */ dp = iot[id].d.desc; if (dp == 0) goto bad; switch (dp->bDescriptorSubtype) { case UDESCSUB_AC_INPUT: r.bNrChannels = iot[id].d.it->bNrChannels; USETW(r.wChannelConfig, UGETW(iot[id].d.it->wChannelConfig)); r.iChannelNames = iot[id].d.it->iChannelNames; return r; case UDESCSUB_AC_OUTPUT: id = iot[id].d.ot->bSourceId; break; case UDESCSUB_AC_MIXER: r = *(const struct usb_audio_cluster *) &iot[id].d.mu->baSourceId[iot[id].d.mu->bNrInPins]; return r; case UDESCSUB_AC_SELECTOR: /* XXX This is not really right */ id = iot[id].d.su->baSourceId[0]; break; case UDESCSUB_AC_FEATURE: id = iot[id].d.fu->bSourceId; break; case UDESCSUB_AC_PROCESSING: r = *(const struct usb_audio_cluster *) &iot[id].d.pu->baSourceId[iot[id].d.pu->bNrInPins]; return r; case UDESCSUB_AC_EXTENSION: r = *(const struct usb_audio_cluster *) &iot[id].d.eu->baSourceId[iot[id].d.eu->bNrInPins]; return r; default: goto bad; } } bad: aprint_error("uaudio_get_cluster: bad data\n"); memset(&r, 0, sizeof(r)); return r; } Static void uaudio_add_input(struct uaudio_softc *sc, const struct io_terminal *iot, int id) { const struct usb_audio_input_terminal *d; d = iot[id].d.it; #ifdef UAUDIO_DEBUG DPRINTFN(2,"bTerminalId=%d wTerminalType=0x%04x " "bAssocTerminal=%d bNrChannels=%d wChannelConfig=%d " "iChannelNames=%d iTerminal=%d\n", d->bTerminalId, UGETW(d->wTerminalType), d->bAssocTerminal, d->bNrChannels, UGETW(d->wChannelConfig), d->iChannelNames, d->iTerminal); #endif /* If USB input terminal, record wChannelConfig */ if ((UGETW(d->wTerminalType) & 0xff00) != 0x0100) return; sc->sc_channel_config = UGETW(d->wChannelConfig); } Static void uaudio_add_output(struct uaudio_softc *sc, const struct io_terminal *iot, int id) { #ifdef UAUDIO_DEBUG const struct usb_audio_output_terminal *d; d = iot[id].d.ot; DPRINTFN(2,"bTerminalId=%d wTerminalType=0x%04x " "bAssocTerminal=%d bSourceId=%d iTerminal=%d\n", d->bTerminalId, UGETW(d->wTerminalType), d->bAssocTerminal, d->bSourceId, d->iTerminal); #endif } Static void uaudio_add_mixer(struct uaudio_softc *sc, const struct io_terminal *iot, int id) { const struct usb_audio_mixer_unit *d; const struct usb_audio_mixer_unit_1 *d1; int c, chs, ichs, ochs, i, o, bno, p, mo, mc, k; const uByte *bm; struct mixerctl mix; d = iot[id].d.mu; DPRINTFN(2,"bUnitId=%d bNrInPins=%d\n", d->bUnitId, d->bNrInPins); /* Compute the number of input channels */ ichs = 0; for (i = 0; i < d->bNrInPins; i++) ichs += uaudio_get_cluster(d->baSourceId[i], iot).bNrChannels; /* and the number of output channels */ d1 = (const struct usb_audio_mixer_unit_1 *)&d->baSourceId[d->bNrInPins]; ochs = d1->bNrChannels; DPRINTFN(2,"ichs=%d ochs=%d\n", ichs, ochs); bm = d1->bmControls; mix.wIndex = MAKE(d->bUnitId, sc->sc_ac_iface); uaudio_determine_class(&iot[id], &mix); mix.type = MIX_SIGNED_16; mix.ctlunit = AudioNvolume; #define _BIT(bno) ((bm[bno / 8] >> (7 - bno % 8)) & 1) for (p = i = 0; i < d->bNrInPins; i++) { chs = uaudio_get_cluster(d->baSourceId[i], iot).bNrChannels; mc = 0; for (c = 0; c < chs; c++) { mo = 0; for (o = 0; o < ochs; o++) { bno = (p + c) * ochs + o; if (_BIT(bno)) mo++; } if (mo == 1) mc++; } if (mc == chs && chs <= MIX_MAX_CHAN) { k = 0; for (c = 0; c < chs; c++) for (o = 0; o < ochs; o++) { bno = (p + c) * ochs + o; if (_BIT(bno)) mix.wValue[k++] = MAKE(p+c+1, o+1); } snprintf(mix.ctlname, sizeof(mix.ctlname), "mix%d-%s", d->bUnitId, uaudio_id_name(sc, iot, d->baSourceId[i])); mix.nchan = chs; uaudio_mixer_add_ctl(sc, &mix); } else { /* XXX */ } #undef _BIT p += chs; } } Static void uaudio_add_selector(struct uaudio_softc *sc, const struct io_terminal *iot, int id) { const struct usb_audio_selector_unit *d; struct mixerctl mix; int i, wp; d = iot[id].d.su; DPRINTFN(2,"bUnitId=%d bNrInPins=%d\n", d->bUnitId, d->bNrInPins); mix.wIndex = MAKE(d->bUnitId, sc->sc_ac_iface); mix.wValue[0] = MAKE(0, 0); uaudio_determine_class(&iot[id], &mix); mix.nchan = 1; mix.type = MIX_SELECTOR; mix.ctlunit = ""; mix.minval = 1; mix.maxval = d->bNrInPins; mix.mul = mix.maxval - mix.minval; wp = snprintf(mix.ctlname, MAX_AUDIO_DEV_LEN, "sel%d-", d->bUnitId); for (i = 1; i <= d->bNrInPins; i++) { wp += snprintf(mix.ctlname + wp, MAX_AUDIO_DEV_LEN - wp, "i%d", d->baSourceId[i - 1]); if (wp > MAX_AUDIO_DEV_LEN - 1) break; } uaudio_mixer_add_ctl(sc, &mix); } #ifdef UAUDIO_DEBUG Static const char * uaudio_get_terminal_name(int terminal_type) { static char tbuf[100]; switch (terminal_type) { /* USB terminal types */ case UAT_UNDEFINED: return "UAT_UNDEFINED"; case UAT_STREAM: return "UAT_STREAM"; case UAT_VENDOR: return "UAT_VENDOR"; /* input terminal types */ case UATI_UNDEFINED: return "UATI_UNDEFINED"; case UATI_MICROPHONE: return "UATI_MICROPHONE"; case UATI_DESKMICROPHONE: return "UATI_DESKMICROPHONE"; case UATI_PERSONALMICROPHONE: return "UATI_PERSONALMICROPHONE"; case UATI_OMNIMICROPHONE: return "UATI_OMNIMICROPHONE"; case UATI_MICROPHONEARRAY: return "UATI_MICROPHONEARRAY"; case UATI_PROCMICROPHONEARR: return "UATI_PROCMICROPHONEARR"; /* output terminal types */ case UATO_UNDEFINED: return "UATO_UNDEFINED"; case UATO_SPEAKER: return "UATO_SPEAKER"; case UATO_HEADPHONES: return "UATO_HEADPHONES"; case UATO_DISPLAYAUDIO: return "UATO_DISPLAYAUDIO"; case UATO_DESKTOPSPEAKER: return "UATO_DESKTOPSPEAKER"; case UATO_ROOMSPEAKER: return "UATO_ROOMSPEAKER"; case UATO_COMMSPEAKER: return "UATO_COMMSPEAKER"; case UATO_SUBWOOFER: return "UATO_SUBWOOFER"; /* bidir terminal types */ case UATB_UNDEFINED: return "UATB_UNDEFINED"; case UATB_HANDSET: return "UATB_HANDSET"; case UATB_HEADSET: return "UATB_HEADSET"; case UATB_SPEAKERPHONE: return "UATB_SPEAKERPHONE"; case UATB_SPEAKERPHONEESUP: return "UATB_SPEAKERPHONEESUP"; case UATB_SPEAKERPHONEECANC: return "UATB_SPEAKERPHONEECANC"; /* telephony terminal types */ case UATT_UNDEFINED: return "UATT_UNDEFINED"; case UATT_PHONELINE: return "UATT_PHONELINE"; case UATT_TELEPHONE: return "UATT_TELEPHONE"; case UATT_DOWNLINEPHONE: return "UATT_DOWNLINEPHONE"; /* external terminal types */ case UATE_UNDEFINED: return "UATE_UNDEFINED"; case UATE_ANALOGCONN: return "UATE_ANALOGCONN"; case UATE_LINECONN: return "UATE_LINECONN"; case UATE_LEGACYCONN: return "UATE_LEGACYCONN"; case UATE_DIGITALAUIFC: return "UATE_DIGITALAUIFC"; case UATE_SPDIF: return "UATE_SPDIF"; case UATE_1394DA: return "UATE_1394DA"; case UATE_1394DV: return "UATE_1394DV"; /* embedded function terminal types */ case UATF_UNDEFINED: return "UATF_UNDEFINED"; case UATF_CALIBNOISE: return "UATF_CALIBNOISE"; case UATF_EQUNOISE: return "UATF_EQUNOISE"; case UATF_CDPLAYER: return "UATF_CDPLAYER"; case UATF_DAT: return "UATF_DAT"; case UATF_DCC: return "UATF_DCC"; case UATF_MINIDISK: return "UATF_MINIDISK"; case UATF_ANALOGTAPE: return "UATF_ANALOGTAPE"; case UATF_PHONOGRAPH: return "UATF_PHONOGRAPH"; case UATF_VCRAUDIO: return "UATF_VCRAUDIO"; case UATF_VIDEODISCAUDIO: return "UATF_VIDEODISCAUDIO"; case UATF_DVDAUDIO: return "UATF_DVDAUDIO"; case UATF_TVTUNERAUDIO: return "UATF_TVTUNERAUDIO"; case UATF_SATELLITE: return "UATF_SATELLITE"; case UATF_CABLETUNER: return "UATF_CABLETUNER"; case UATF_DSS: return "UATF_DSS"; case UATF_RADIORECV: return "UATF_RADIORECV"; case UATF_RADIOXMIT: return "UATF_RADIOXMIT"; case UATF_MULTITRACK: return "UATF_MULTITRACK"; case UATF_SYNTHESIZER: return "UATF_SYNTHESIZER"; default: snprintf(tbuf, sizeof(tbuf), "unknown type (%#.4x)", terminal_type); return tbuf; } } #endif Static int uaudio_determine_class(const struct io_terminal *iot, struct mixerctl *mix) { int terminal_type; if (iot == NULL || iot->output == NULL) { mix->class = UAC_OUTPUT; return 0; } terminal_type = 0; if (iot->output->size == 1) terminal_type = iot->output->terminals[0]; /* * If the only output terminal is USB, * the class is UAC_RECORD. */ if ((terminal_type & 0xff00) == (UAT_UNDEFINED & 0xff00)) { mix->class = UAC_RECORD; if (iot->inputs_size == 1 && iot->inputs[0] != NULL && iot->inputs[0]->size == 1) return iot->inputs[0]->terminals[0]; else return 0; } /* * If the ultimate destination of the unit is just one output * terminal and the unit is connected to the output terminal * directly, the class is UAC_OUTPUT. */ if (terminal_type != 0 && iot->direct) { mix->class = UAC_OUTPUT; return terminal_type; } /* * If the unit is connected to just one input terminal, * the class is UAC_INPUT. */ if (iot->inputs_size == 1 && iot->inputs[0] != NULL && iot->inputs[0]->size == 1) { mix->class = UAC_INPUT; return iot->inputs[0]->terminals[0]; } /* * Otherwise, the class is UAC_OUTPUT. */ mix->class = UAC_OUTPUT; return terminal_type; } Static const char * uaudio_feature_name(const struct io_terminal *iot, struct mixerctl *mix) { int terminal_type; terminal_type = uaudio_determine_class(iot, mix); if (mix->class == UAC_RECORD && terminal_type == 0) return AudioNmixerout; DPRINTF("terminal_type=%s\n", uaudio_get_terminal_name(terminal_type)); switch (terminal_type) { case UAT_STREAM: return AudioNdac; case UATI_MICROPHONE: case UATI_DESKMICROPHONE: case UATI_PERSONALMICROPHONE: case UATI_OMNIMICROPHONE: case UATI_MICROPHONEARRAY: case UATI_PROCMICROPHONEARR: return AudioNmicrophone; case UATO_SPEAKER: case UATO_DESKTOPSPEAKER: case UATO_ROOMSPEAKER: case UATO_COMMSPEAKER: return AudioNspeaker; case UATO_HEADPHONES: return AudioNheadphone; case UATO_SUBWOOFER: return AudioNlfe; /* telephony terminal types */ case UATT_UNDEFINED: case UATT_PHONELINE: case UATT_TELEPHONE: case UATT_DOWNLINEPHONE: return "phone"; case UATE_ANALOGCONN: case UATE_LINECONN: case UATE_LEGACYCONN: return AudioNline; case UATE_DIGITALAUIFC: case UATE_SPDIF: case UATE_1394DA: case UATE_1394DV: return AudioNaux; case UATF_CDPLAYER: return AudioNcd; case UATF_SYNTHESIZER: return AudioNfmsynth; case UATF_VIDEODISCAUDIO: case UATF_DVDAUDIO: case UATF_TVTUNERAUDIO: return AudioNvideo; case UAT_UNDEFINED: case UAT_VENDOR: case UATI_UNDEFINED: /* output terminal types */ case UATO_UNDEFINED: case UATO_DISPLAYAUDIO: /* bidir terminal types */ case UATB_UNDEFINED: case UATB_HANDSET: case UATB_HEADSET: case UATB_SPEAKERPHONE: case UATB_SPEAKERPHONEESUP: case UATB_SPEAKERPHONEECANC: /* external terminal types */ case UATE_UNDEFINED: /* embedded function terminal types */ case UATF_UNDEFINED: case UATF_CALIBNOISE: case UATF_EQUNOISE: case UATF_DAT: case UATF_DCC: case UATF_MINIDISK: case UATF_ANALOGTAPE: case UATF_PHONOGRAPH: case UATF_VCRAUDIO: case UATF_SATELLITE: case UATF_CABLETUNER: case UATF_DSS: case UATF_RADIORECV: case UATF_RADIOXMIT: case UATF_MULTITRACK: case 0xffff: default: DPRINTF("'master' for %#.4x\n", terminal_type); return AudioNmaster; } return AudioNmaster; } Static void uaudio_add_feature(struct uaudio_softc *sc, const struct io_terminal *iot, int id) { const struct usb_audio_feature_unit *d; const uByte *ctls; int ctlsize; int nchan; u_int fumask, mmask, cmask; struct mixerctl mix; int chan, ctl, i, unit; const char *mixername; #define GET(i) (ctls[(i)*ctlsize] | \ (ctlsize > 1 ? ctls[(i)*ctlsize+1] << 8 : 0)) d = iot[id].d.fu; ctls = d->bmaControls; ctlsize = d->bControlSize; if (ctlsize == 0) { DPRINTF("ignoring feature %d with controlSize of zero\n", id); return; } nchan = (d->bLength - 7) / ctlsize; mmask = GET(0); /* Figure out what we can control */ for (cmask = 0, chan = 1; chan < nchan; chan++) { DPRINTFN(9,"chan=%d mask=%x\n", chan, GET(chan)); cmask |= GET(chan); } DPRINTFN(1,"bUnitId=%d, " "%d channels, mmask=0x%04x, cmask=0x%04x\n", d->bUnitId, nchan, mmask, cmask); if (nchan > MIX_MAX_CHAN) nchan = MIX_MAX_CHAN; unit = d->bUnitId; mix.wIndex = MAKE(unit, sc->sc_ac_iface); for (ctl = MUTE_CONTROL; ctl < LOUDNESS_CONTROL; ctl++) { fumask = FU_MASK(ctl); DPRINTFN(4,"ctl=%d fumask=0x%04x\n", ctl, fumask); if (mmask & fumask) { mix.nchan = 1; mix.wValue[0] = MAKE(ctl, 0); } else if (cmask & fumask) { mix.nchan = nchan - 1; for (i = 1; i < nchan; i++) { if (GET(i) & fumask) mix.wValue[i-1] = MAKE(ctl, i); else mix.wValue[i-1] = -1; } } else { continue; } #undef GET mixername = uaudio_feature_name(&iot[id], &mix); switch (ctl) { case MUTE_CONTROL: mix.type = MIX_ON_OFF; mix.ctlunit = ""; snprintf(mix.ctlname, sizeof(mix.ctlname), "%s.%s", mixername, AudioNmute); break; case VOLUME_CONTROL: mix.type = MIX_SIGNED_16; mix.ctlunit = AudioNvolume; strlcpy(mix.ctlname, mixername, sizeof(mix.ctlname)); break; case BASS_CONTROL: mix.type = MIX_SIGNED_8; mix.ctlunit = AudioNbass; snprintf(mix.ctlname, sizeof(mix.ctlname), "%s.%s", mixername, AudioNbass); break; case MID_CONTROL: mix.type = MIX_SIGNED_8; mix.ctlunit = AudioNmid; snprintf(mix.ctlname, sizeof(mix.ctlname), "%s.%s", mixername, AudioNmid); break; case TREBLE_CONTROL: mix.type = MIX_SIGNED_8; mix.ctlunit = AudioNtreble; snprintf(mix.ctlname, sizeof(mix.ctlname), "%s.%s", mixername, AudioNtreble); break; case GRAPHIC_EQUALIZER_CONTROL: continue; /* XXX don't add anything */ break; case AGC_CONTROL: mix.type = MIX_ON_OFF; mix.ctlunit = ""; snprintf(mix.ctlname, sizeof(mix.ctlname), "%s.%s", mixername, AudioNagc); break; case DELAY_CONTROL: mix.type = MIX_UNSIGNED_16; mix.ctlunit = "4 ms"; snprintf(mix.ctlname, sizeof(mix.ctlname), "%s.%s", mixername, AudioNdelay); break; case BASS_BOOST_CONTROL: mix.type = MIX_ON_OFF; mix.ctlunit = ""; snprintf(mix.ctlname, sizeof(mix.ctlname), "%s.%s", mixername, AudioNbassboost); break; case LOUDNESS_CONTROL: mix.type = MIX_ON_OFF; mix.ctlunit = ""; snprintf(mix.ctlname, sizeof(mix.ctlname), "%s.%s", mixername, AudioNloudness); break; } uaudio_mixer_add_ctl(sc, &mix); } } Static void uaudio_add_processing_updown(struct uaudio_softc *sc, const struct io_terminal *iot, int id) { const struct usb_audio_processing_unit *d; const struct usb_audio_processing_unit_1 *d1; const struct usb_audio_processing_unit_updown *ud; struct mixerctl mix; int i; d = iot[id].d.pu; d1 = (const struct usb_audio_processing_unit_1 *) &d->baSourceId[d->bNrInPins]; ud = (const struct usb_audio_processing_unit_updown *) &d1->bmControls[d1->bControlSize]; DPRINTFN(2,"bUnitId=%d bNrModes=%d\n", d->bUnitId, ud->bNrModes); if (!(d1->bmControls[0] & UA_PROC_MASK(UD_MODE_SELECT_CONTROL))) { DPRINTF("%s", "no mode select\n"); return; } mix.wIndex = MAKE(d->bUnitId, sc->sc_ac_iface); mix.nchan = 1; mix.wValue[0] = MAKE(UD_MODE_SELECT_CONTROL, 0); uaudio_determine_class(&iot[id], &mix); mix.type = MIX_ON_OFF; /* XXX */ mix.ctlunit = ""; snprintf(mix.ctlname, sizeof(mix.ctlname), "pro%d-mode", d->bUnitId); for (i = 0; i < ud->bNrModes; i++) { DPRINTFN(2,"i=%d bm=%#x\n", i, UGETW(ud->waModes[i])); /* XXX */ } uaudio_mixer_add_ctl(sc, &mix); } Static void uaudio_add_processing(struct uaudio_softc *sc, const struct io_terminal *iot, int id) { const struct usb_audio_processing_unit *d; const struct usb_audio_processing_unit_1 *d1; int ptype; struct mixerctl mix; d = iot[id].d.pu; d1 = (const struct usb_audio_processing_unit_1 *) &d->baSourceId[d->bNrInPins]; ptype = UGETW(d->wProcessType); DPRINTFN(2,"wProcessType=%d bUnitId=%d " "bNrInPins=%d\n", ptype, d->bUnitId, d->bNrInPins); if (d1->bmControls[0] & UA_PROC_ENABLE_MASK) { mix.wIndex = MAKE(d->bUnitId, sc->sc_ac_iface); mix.nchan = 1; mix.wValue[0] = MAKE(XX_ENABLE_CONTROL, 0); uaudio_determine_class(&iot[id], &mix); mix.type = MIX_ON_OFF; mix.ctlunit = ""; snprintf(mix.ctlname, sizeof(mix.ctlname), "pro%d.%d-enable", d->bUnitId, ptype); uaudio_mixer_add_ctl(sc, &mix); } switch(ptype) { case UPDOWNMIX_PROCESS: uaudio_add_processing_updown(sc, iot, id); break; case DOLBY_PROLOGIC_PROCESS: case P3D_STEREO_EXTENDER_PROCESS: case REVERBATION_PROCESS: case CHORUS_PROCESS: case DYN_RANGE_COMP_PROCESS: default: #ifdef UAUDIO_DEBUG aprint_debug( "uaudio_add_processing: unit %d, type=%d not impl.\n", d->bUnitId, ptype); #endif break; } } Static void uaudio_add_extension(struct uaudio_softc *sc, const struct io_terminal *iot, int id) { const struct usb_audio_extension_unit *d; const struct usb_audio_extension_unit_1 *d1; struct mixerctl mix; d = iot[id].d.eu; d1 = (const struct usb_audio_extension_unit_1 *) &d->baSourceId[d->bNrInPins]; DPRINTFN(2,"bUnitId=%d bNrInPins=%d\n", d->bUnitId, d->bNrInPins); if (usbd_get_quirks(sc->sc_udev)->uq_flags & UQ_AU_NO_XU) return; if (d1->bmControls[0] & UA_EXT_ENABLE_MASK) { mix.wIndex = MAKE(d->bUnitId, sc->sc_ac_iface); mix.nchan = 1; mix.wValue[0] = MAKE(UA_EXT_ENABLE, 0); uaudio_determine_class(&iot[id], &mix); mix.type = MIX_ON_OFF; mix.ctlunit = ""; snprintf(mix.ctlname, sizeof(mix.ctlname), "ext%d-enable", d->bUnitId); uaudio_mixer_add_ctl(sc, &mix); } } Static struct terminal_list* uaudio_merge_terminal_list(const struct io_terminal *iot) { struct terminal_list *tml; uint16_t *ptm; int i, len; len = 0; if (iot->inputs == NULL) return NULL; for (i = 0; i < iot->inputs_size; i++) { if (iot->inputs[i] != NULL) len += iot->inputs[i]->size; } tml = malloc(TERMINAL_LIST_SIZE(len), M_TEMP, M_NOWAIT); if (tml == NULL) { aprint_error("uaudio_merge_terminal_list: no memory\n"); return NULL; } tml->size = 0; ptm = tml->terminals; for (i = 0; i < iot->inputs_size; i++) { if (iot->inputs[i] == NULL) continue; if (iot->inputs[i]->size > len) break; memcpy(ptm, iot->inputs[i]->terminals, iot->inputs[i]->size * sizeof(uint16_t)); tml->size += iot->inputs[i]->size; ptm += iot->inputs[i]->size; len -= iot->inputs[i]->size; } return tml; } Static struct terminal_list * uaudio_io_terminaltype(int outtype, struct io_terminal *iot, int id) { struct terminal_list *tml; struct io_terminal *it; int src_id, i; it = &iot[id]; if (it->output != NULL) { /* already has outtype? */ for (i = 0; i < it->output->size; i++) if (it->output->terminals[i] == outtype) return uaudio_merge_terminal_list(it); tml = malloc(TERMINAL_LIST_SIZE(it->output->size + 1), M_TEMP, M_NOWAIT); if (tml == NULL) { aprint_error("uaudio_io_terminaltype: no memory\n"); return uaudio_merge_terminal_list(it); } memcpy(tml, it->output, TERMINAL_LIST_SIZE(it->output->size)); tml->terminals[it->output->size] = outtype; tml->size++; free(it->output, M_TEMP); it->output = tml; if (it->inputs != NULL) { for (i = 0; i < it->inputs_size; i++) if (it->inputs[i] != NULL) free(it->inputs[i], M_TEMP); free(it->inputs, M_TEMP); } it->inputs_size = 0; it->inputs = NULL; } else { /* end `iot[id] != NULL' */ it->inputs_size = 0; it->inputs = NULL; it->output = malloc(TERMINAL_LIST_SIZE(1), M_TEMP, M_NOWAIT); if (it->output == NULL) { aprint_error("uaudio_io_terminaltype: no memory\n"); return NULL; } it->output->terminals[0] = outtype; it->output->size = 1; it->direct = FALSE; } switch (it->d.desc->bDescriptorSubtype) { case UDESCSUB_AC_INPUT: it->inputs = malloc(sizeof(struct terminal_list *), M_TEMP, M_NOWAIT); if (it->inputs == NULL) { aprint_error("uaudio_io_terminaltype: no memory\n"); return NULL; } tml = malloc(TERMINAL_LIST_SIZE(1), M_TEMP, M_NOWAIT); if (tml == NULL) { aprint_error("uaudio_io_terminaltype: no memory\n"); free(it->inputs, M_TEMP); it->inputs = NULL; return NULL; } it->inputs[0] = tml; tml->terminals[0] = UGETW(it->d.it->wTerminalType); tml->size = 1; it->inputs_size = 1; return uaudio_merge_terminal_list(it); case UDESCSUB_AC_FEATURE: src_id = it->d.fu->bSourceId; it->inputs = malloc(sizeof(struct terminal_list *), M_TEMP, M_NOWAIT); if (it->inputs == NULL) { aprint_error("uaudio_io_terminaltype: no memory\n"); return uaudio_io_terminaltype(outtype, iot, src_id); } it->inputs[0] = uaudio_io_terminaltype(outtype, iot, src_id); it->inputs_size = 1; return uaudio_merge_terminal_list(it); case UDESCSUB_AC_OUTPUT: it->inputs = malloc(sizeof(struct terminal_list *), M_TEMP, M_NOWAIT); if (it->inputs == NULL) { aprint_error("uaudio_io_terminaltype: no memory\n"); return NULL; } src_id = it->d.ot->bSourceId; it->inputs[0] = uaudio_io_terminaltype(outtype, iot, src_id); it->inputs_size = 1; iot[src_id].direct = TRUE; return NULL; case UDESCSUB_AC_MIXER: it->inputs_size = 0; it->inputs = malloc(sizeof(struct terminal_list *) * it->d.mu->bNrInPins, M_TEMP, M_NOWAIT); if (it->inputs == NULL) { aprint_error("uaudio_io_terminaltype: no memory\n"); return NULL; } for (i = 0; i < it->d.mu->bNrInPins; i++) { src_id = it->d.mu->baSourceId[i]; it->inputs[i] = uaudio_io_terminaltype(outtype, iot, src_id); it->inputs_size++; } return uaudio_merge_terminal_list(it); case UDESCSUB_AC_SELECTOR: it->inputs_size = 0; it->inputs = malloc(sizeof(struct terminal_list *) * it->d.su->bNrInPins, M_TEMP, M_NOWAIT); if (it->inputs == NULL) { aprint_error("uaudio_io_terminaltype: no memory\n"); return NULL; } for (i = 0; i < it->d.su->bNrInPins; i++) { src_id = it->d.su->baSourceId[i]; it->inputs[i] = uaudio_io_terminaltype(outtype, iot, src_id); it->inputs_size++; } return uaudio_merge_terminal_list(it); case UDESCSUB_AC_PROCESSING: it->inputs_size = 0; it->inputs = malloc(sizeof(struct terminal_list *) * it->d.pu->bNrInPins, M_TEMP, M_NOWAIT); if (it->inputs == NULL) { aprint_error("uaudio_io_terminaltype: no memory\n"); return NULL; } for (i = 0; i < it->d.pu->bNrInPins; i++) { src_id = it->d.pu->baSourceId[i]; it->inputs[i] = uaudio_io_terminaltype(outtype, iot, src_id); it->inputs_size++; } return uaudio_merge_terminal_list(it); case UDESCSUB_AC_EXTENSION: it->inputs_size = 0; it->inputs = malloc(sizeof(struct terminal_list *) * it->d.eu->bNrInPins, M_TEMP, M_NOWAIT); if (it->inputs == NULL) { aprint_error("uaudio_io_terminaltype: no memory\n"); return NULL; } for (i = 0; i < it->d.eu->bNrInPins; i++) { src_id = it->d.eu->baSourceId[i]; it->inputs[i] = uaudio_io_terminaltype(outtype, iot, src_id); it->inputs_size++; } return uaudio_merge_terminal_list(it); case UDESCSUB_AC_HEADER: default: return NULL; } } Static usbd_status uaudio_identify(struct uaudio_softc *sc, const usb_config_descriptor_t *cdesc) { usbd_status err; err = uaudio_identify_ac(sc, cdesc); if (err) return err; return uaudio_identify_as(sc, cdesc); } Static void uaudio_add_alt(struct uaudio_softc *sc, const struct as_info *ai) { size_t len; struct as_info *nai; len = sizeof(*ai) * (sc->sc_nalts + 1); nai = kmem_alloc(len, KM_SLEEP); /* Copy old data, if there was any */ if (sc->sc_nalts != 0) { memcpy(nai, sc->sc_alts, sizeof(*ai) * (sc->sc_nalts)); kmem_free(sc->sc_alts, sizeof(*ai) * sc->sc_nalts); } sc->sc_alts = nai; DPRINTFN(2,"adding alt=%d, enc=%d\n", ai->alt, ai->encoding); sc->sc_alts[sc->sc_nalts++] = *ai; } Static usbd_status uaudio_process_as(struct uaudio_softc *sc, const char *tbuf, int *offsp, int size, const usb_interface_descriptor_t *id) { const struct usb_audio_streaming_interface_descriptor *asid; const struct usb_audio_streaming_type1_descriptor *asf1d; const usb_endpoint_descriptor_audio_t *ed; const usb_endpoint_descriptor_audio_t *epdesc1; const struct usb_audio_streaming_endpoint_descriptor *sed; int format, chan __unused, prec, enc; int dir, type, sync, epcount; struct as_info ai; const char *format_str __unused; const uaudio_cs_descriptor_t *desc; DPRINTF("offset = %d < %d\n", *offsp, size); epcount = 0; asid = NULL; asf1d = NULL; ed = NULL; epdesc1 = NULL; sed = NULL; while (*offsp < size) { desc = (const uaudio_cs_descriptor_t *)(tbuf + *offsp); if (*offsp + desc->bLength > size) return USBD_INVAL; switch (desc->bDescriptorType) { case UDESC_CS_INTERFACE: switch (desc->bDescriptorSubtype) { case AS_GENERAL: if (asid != NULL) goto ignore; asid = (const struct usb_audio_streaming_interface_descriptor *) desc; DPRINTF("asid: bTerminalLink=%d wFormatTag=%d bLength=%d\n", asid->bTerminalLink, UGETW(asid->wFormatTag), asid->bLength); break; case FORMAT_TYPE: if (asf1d != NULL) goto ignore; asf1d = (const struct usb_audio_streaming_type1_descriptor *) desc; DPRINTF("asf1d: bDescriptorType=%d bDescriptorSubtype=%d\n", asf1d->bDescriptorType, asf1d->bDescriptorSubtype); if (asf1d->bFormatType != FORMAT_TYPE_I) { aprint_normal_dev(sc->sc_dev, "ignored setting with type %d format\n", asf1d->bFormatType); return USBD_NORMAL_COMPLETION; } break; default: goto ignore; } break; case UDESC_ENDPOINT: epcount++; if (epcount > id->bNumEndpoints) goto ignore; switch (epcount) { case 1: ed = (const usb_endpoint_descriptor_audio_t *) desc; DPRINTF("endpoint[0] bLength=%d bDescriptorType=%d " "bEndpointAddress=%d bmAttributes=%#x wMaxPacketSize=%d " "bInterval=%d bRefresh=%d bSynchAddress=%d\n", ed->bLength, ed->bDescriptorType, ed->bEndpointAddress, ed->bmAttributes, UGETW(ed->wMaxPacketSize), ed->bInterval, ed->bRefresh, ed->bSynchAddress); if (UE_GET_XFERTYPE(ed->bmAttributes) != UE_ISOCHRONOUS) return USBD_INVAL; break; case 2: epdesc1 = (const usb_endpoint_descriptor_audio_t *) desc; DPRINTF("endpoint[1] bLength=%d " "bDescriptorType=%d bEndpointAddress=%d " "bmAttributes=%#x wMaxPacketSize=%d bInterval=%d " "bRefresh=%d bSynchAddress=%d\n", epdesc1->bLength, epdesc1->bDescriptorType, epdesc1->bEndpointAddress, epdesc1->bmAttributes, UGETW(epdesc1->wMaxPacketSize), epdesc1->bInterval, epdesc1->bRefresh, epdesc1->bSynchAddress); if (epdesc1->bSynchAddress != 0) { aprint_error_dev(sc->sc_dev, "invalid endpoint: bSynchAddress=0\n"); return USBD_INVAL; } if (UE_GET_XFERTYPE(epdesc1->bmAttributes) != UE_ISOCHRONOUS) { aprint_error_dev(sc->sc_dev, "invalid endpoint: bmAttributes=%#x\n", epdesc1->bmAttributes); return USBD_INVAL; } if (epdesc1->bEndpointAddress != ed->bSynchAddress) { aprint_error_dev(sc->sc_dev, "invalid endpoint addresses: " "ep[0]->bSynchAddress=%#x " "ep[1]->bEndpointAddress=%#x\n", ed->bSynchAddress, epdesc1->bEndpointAddress); return USBD_INVAL; } /* UE_GET_ADDR(epdesc1->bEndpointAddress), and epdesc1->bRefresh */ break; default: goto ignore; } break; case UDESC_CS_ENDPOINT: switch (desc->bDescriptorSubtype) { case AS_GENERAL: if (sed != NULL) goto ignore; sed = (const struct usb_audio_streaming_endpoint_descriptor *) desc; DPRINTF(" streadming_endpoint: offset=%d bLength=%d\n", *offsp, sed->bLength); break; default: goto ignore; } break; case UDESC_INTERFACE: case UDESC_DEVICE: goto leave; default: ignore: aprint_normal_dev(sc->sc_dev, "ignored descriptor type %d subtype %d\n", desc->bDescriptorType, desc->bDescriptorSubtype); break; } *offsp += desc->bLength; } leave: if (asid == NULL) { DPRINTF("%s", "No streaming interface descriptor found\n"); return USBD_INVAL; } if (asf1d == NULL) { DPRINTF("%s", "No format type descriptor found\n"); return USBD_INVAL; } if (ed == NULL) { DPRINTF("%s", "No endpoint descriptor found\n"); return USBD_INVAL; } if (sed == NULL) { DPRINTF("%s", "No streaming endpoint descriptor found\n"); return USBD_INVAL; } dir = UE_GET_DIR(ed->bEndpointAddress); type = UE_GET_ISO_TYPE(ed->bmAttributes); if ((usbd_get_quirks(sc->sc_udev)->uq_flags & UQ_AU_INP_ASYNC) && dir == UE_DIR_IN && type == UE_ISO_ADAPT) type = UE_ISO_ASYNC; /* We can't handle endpoints that need a sync pipe yet. */ sync = FALSE; if (dir == UE_DIR_IN && type == UE_ISO_ADAPT) { sync = TRUE; #ifndef UAUDIO_MULTIPLE_ENDPOINTS aprint_normal_dev(sc->sc_dev, "ignored input endpoint of type adaptive\n"); return USBD_NORMAL_COMPLETION; #endif } if (dir != UE_DIR_IN && type == UE_ISO_ASYNC) { sync = TRUE; #ifndef UAUDIO_MULTIPLE_ENDPOINTS aprint_normal_dev(sc->sc_dev, "ignored output endpoint of type async\n"); return USBD_NORMAL_COMPLETION; #endif } #ifdef UAUDIO_MULTIPLE_ENDPOINTS if (sync && id->bNumEndpoints <= 1) { aprint_error_dev(sc->sc_dev, "a sync-pipe endpoint but no other endpoint\n"); return USBD_INVAL; } #endif if (!sync && id->bNumEndpoints > 1) { aprint_error_dev(sc->sc_dev, "non sync-pipe endpoint but multiple endpoints\n"); return USBD_INVAL; } format = UGETW(asid->wFormatTag); chan = asf1d->bNrChannels; prec = asf1d->bBitResolution; if (prec != 8 && prec != 16 && prec != 24) { aprint_normal_dev(sc->sc_dev, "ignored setting with precision %d\n", prec); return USBD_NORMAL_COMPLETION; } switch (format) { case UA_FMT_PCM: if (prec == 8) { sc->sc_altflags |= HAS_8; } else if (prec == 16) { sc->sc_altflags |= HAS_16; } else if (prec == 24) { sc->sc_altflags |= HAS_24; } enc = AUDIO_ENCODING_SLINEAR_LE; format_str = "pcm"; break; case UA_FMT_PCM8: enc = AUDIO_ENCODING_ULINEAR_LE; sc->sc_altflags |= HAS_8U; format_str = "pcm8"; break; case UA_FMT_ALAW: enc = AUDIO_ENCODING_ALAW; sc->sc_altflags |= HAS_ALAW; format_str = "alaw"; break; case UA_FMT_MULAW: enc = AUDIO_ENCODING_ULAW; sc->sc_altflags |= HAS_MULAW; format_str = "mulaw"; break; case UA_FMT_IEEE_FLOAT: default: aprint_normal_dev(sc->sc_dev, "ignored setting with format %d\n", format); return USBD_NORMAL_COMPLETION; } #ifdef UAUDIO_DEBUG aprint_debug_dev(sc->sc_dev, "%s: %dch, %d/%dbit, %s,", dir == UE_DIR_IN ? "recording" : "playback", chan, prec, asf1d->bSubFrameSize * 8, format_str); if (asf1d->bSamFreqType == UA_SAMP_CONTNUOUS) { aprint_debug(" %d-%dHz\n", UA_SAMP_LO(asf1d), UA_SAMP_HI(asf1d)); } else { int r; aprint_debug(" %d", UA_GETSAMP(asf1d, 0)); for (r = 1; r < asf1d->bSamFreqType; r++) aprint_debug(",%d", UA_GETSAMP(asf1d, r)); aprint_debug("Hz\n"); } #endif ai.alt = id->bAlternateSetting; ai.encoding = enc; ai.attributes = sed->bmAttributes; ai.idesc = id; ai.edesc = ed; ai.edesc1 = epdesc1; ai.asf1desc = asf1d; ai.sc_busy = 0; ai.aformat = NULL; ai.ifaceh = NULL; uaudio_add_alt(sc, &ai); #ifdef UAUDIO_DEBUG if (ai.attributes & UA_SED_FREQ_CONTROL) DPRINTFN(1, "%s", "FREQ_CONTROL\n"); if (ai.attributes & UA_SED_PITCH_CONTROL) DPRINTFN(1, "%s", "PITCH_CONTROL\n"); #endif sc->sc_mode |= (dir == UE_DIR_OUT) ? AUMODE_PLAY : AUMODE_RECORD; return USBD_NORMAL_COMPLETION; } Static usbd_status uaudio_identify_as(struct uaudio_softc *sc, const usb_config_descriptor_t *cdesc) { const usb_interface_descriptor_t *id; const char *tbuf; struct audio_format *auf; const struct usb_audio_streaming_type1_descriptor *t1desc; int size, offs; int i, j; size = UGETW(cdesc->wTotalLength); tbuf = (const char *)cdesc; /* Locate the AudioStreaming interface descriptor. */ offs = 0; id = uaudio_find_iface(tbuf, size, &offs, UISUBCLASS_AUDIOSTREAM); if (id == NULL) return USBD_INVAL; /* Loop through all the alternate settings. */ while (offs <= size) { DPRINTFN(2, "interface=%d offset=%d\n", id->bInterfaceNumber, offs); switch (id->bNumEndpoints) { case 0: DPRINTFN(2, "AS null alt=%d\n", id->bAlternateSetting); sc->sc_nullalt = id->bAlternateSetting; break; case 1: #ifdef UAUDIO_MULTIPLE_ENDPOINTS case 2: #endif uaudio_process_as(sc, tbuf, &offs, size, id); break; default: aprint_error_dev(sc->sc_dev, "ignored audio interface with %d endpoints\n", id->bNumEndpoints); break; } id = uaudio_find_iface(tbuf, size, &offs, UISUBCLASS_AUDIOSTREAM); if (id == NULL) break; } if (offs > size) return USBD_INVAL; DPRINTF("%d alts available\n", sc->sc_nalts); if (sc->sc_mode == 0) { aprint_error_dev(sc->sc_dev, "no usable endpoint found\n"); return USBD_INVAL; } /* build audio_format array */ sc->sc_formats = kmem_zalloc(sizeof(struct audio_format) * sc->sc_nalts, KM_SLEEP); sc->sc_nformats = sc->sc_nalts; for (i = 0; i < sc->sc_nalts; i++) { auf = &sc->sc_formats[i]; t1desc = sc->sc_alts[i].asf1desc; if (UE_GET_DIR(sc->sc_alts[i].edesc->bEndpointAddress) == UE_DIR_OUT) auf->mode = AUMODE_PLAY; else auf->mode = AUMODE_RECORD; auf->encoding = sc->sc_alts[i].encoding; auf->validbits = t1desc->bBitResolution; auf->precision = t1desc->bSubFrameSize * 8; auf->channels = t1desc->bNrChannels; auf->channel_mask = sc->sc_channel_config; auf->frequency_type = t1desc->bSamFreqType; if (t1desc->bSamFreqType == UA_SAMP_CONTNUOUS) { auf->frequency[0] = UA_SAMP_LO(t1desc); auf->frequency[1] = UA_SAMP_HI(t1desc); } else { for (j = 0; j < t1desc->bSamFreqType; j++) { if (j >= AUFMT_MAX_FREQUENCIES) { aprint_error("%s: please increase " "AUFMT_MAX_FREQUENCIES to %d\n", __func__, t1desc->bSamFreqType); auf->frequency_type = AUFMT_MAX_FREQUENCIES; break; } auf->frequency[j] = UA_GETSAMP(t1desc, j); } } sc->sc_alts[i].aformat = auf; } return USBD_NORMAL_COMPLETION; } #ifdef UAUDIO_DEBUG Static void uaudio_dump_tml(struct terminal_list *tml) { if (tml == NULL) { printf("NULL"); } else { int i; for (i = 0; i < tml->size; i++) printf("%s ", uaudio_get_terminal_name (tml->terminals[i])); } printf("\n"); } #endif Static usbd_status uaudio_identify_ac(struct uaudio_softc *sc, const usb_config_descriptor_t *cdesc) { struct io_terminal* iot; const usb_interface_descriptor_t *id; const struct usb_audio_control_descriptor *acdp; const uaudio_cs_descriptor_t *dp; const struct usb_audio_output_terminal *pot; struct terminal_list *tml; const char *tbuf, *ibuf, *ibufend; int size, offs, ndps, i, j; size = UGETW(cdesc->wTotalLength); tbuf = (const char *)cdesc; /* Locate the AudioControl interface descriptor. */ offs = 0; id = uaudio_find_iface(tbuf, size, &offs, UISUBCLASS_AUDIOCONTROL); if (id == NULL) return USBD_INVAL; if (offs + sizeof(*acdp) > size) return USBD_INVAL; sc->sc_ac_iface = id->bInterfaceNumber; DPRINTFN(2,"AC interface is %d\n", sc->sc_ac_iface); /* A class-specific AC interface header should follow. */ ibuf = tbuf + offs; ibufend = tbuf + size; acdp = (const struct usb_audio_control_descriptor *)ibuf; if (acdp->bDescriptorType != UDESC_CS_INTERFACE || acdp->bDescriptorSubtype != UDESCSUB_AC_HEADER) return USBD_INVAL; if (!(usbd_get_quirks(sc->sc_udev)->uq_flags & UQ_BAD_ADC) && UGETW(acdp->bcdADC) != UAUDIO_VERSION) return USBD_INVAL; sc->sc_audio_rev = UGETW(acdp->bcdADC); DPRINTFN(2, "found AC header, vers=%03x\n", sc->sc_audio_rev); sc->sc_nullalt = -1; /* Scan through all the AC specific descriptors */ dp = (const uaudio_cs_descriptor_t *)ibuf; ndps = 0; iot = malloc(sizeof(struct io_terminal) * 256, M_TEMP, M_NOWAIT | M_ZERO); if (iot == NULL) { aprint_error("%s: no memory\n", __func__); return USBD_NOMEM; } for (;;) { ibuf += dp->bLength; if (ibuf >= ibufend) break; dp = (const uaudio_cs_descriptor_t *)ibuf; if (ibuf + dp->bLength > ibufend) { free(iot, M_TEMP); return USBD_INVAL; } if (dp->bDescriptorType != UDESC_CS_INTERFACE) break; i = ((const struct usb_audio_input_terminal *)dp)->bTerminalId; iot[i].d.desc = dp; if (i > ndps) ndps = i; } ndps++; /* construct io_terminal */ for (i = 0; i < ndps; i++) { dp = iot[i].d.desc; if (dp == NULL) continue; if (dp->bDescriptorSubtype != UDESCSUB_AC_OUTPUT) continue; pot = iot[i].d.ot; tml = uaudio_io_terminaltype(UGETW(pot->wTerminalType), iot, i); if (tml != NULL) free(tml, M_TEMP); } #ifdef UAUDIO_DEBUG for (i = 0; i < 256; i++) { struct usb_audio_cluster cluster; if (iot[i].d.desc == NULL) continue; printf("id %d:\t", i); switch (iot[i].d.desc->bDescriptorSubtype) { case UDESCSUB_AC_INPUT: printf("AC_INPUT type=%s\n", uaudio_get_terminal_name (UGETW(iot[i].d.it->wTerminalType))); printf("\t"); cluster = uaudio_get_cluster(i, iot); uaudio_dump_cluster(&cluster); printf("\n"); break; case UDESCSUB_AC_OUTPUT: printf("AC_OUTPUT type=%s ", uaudio_get_terminal_name (UGETW(iot[i].d.ot->wTerminalType))); printf("src=%d\n", iot[i].d.ot->bSourceId); break; case UDESCSUB_AC_MIXER: printf("AC_MIXER src="); for (j = 0; j < iot[i].d.mu->bNrInPins; j++) printf("%d ", iot[i].d.mu->baSourceId[j]); printf("\n\t"); cluster = uaudio_get_cluster(i, iot); uaudio_dump_cluster(&cluster); printf("\n"); break; case UDESCSUB_AC_SELECTOR: printf("AC_SELECTOR src="); for (j = 0; j < iot[i].d.su->bNrInPins; j++) printf("%d ", iot[i].d.su->baSourceId[j]); printf("\n"); break; case UDESCSUB_AC_FEATURE: printf("AC_FEATURE src=%d\n", iot[i].d.fu->bSourceId); break; case UDESCSUB_AC_PROCESSING: printf("AC_PROCESSING src="); for (j = 0; j < iot[i].d.pu->bNrInPins; j++) printf("%d ", iot[i].d.pu->baSourceId[j]); printf("\n\t"); cluster = uaudio_get_cluster(i, iot); uaudio_dump_cluster(&cluster); printf("\n"); break; case UDESCSUB_AC_EXTENSION: printf("AC_EXTENSION src="); for (j = 0; j < iot[i].d.eu->bNrInPins; j++) printf("%d ", iot[i].d.eu->baSourceId[j]); printf("\n\t"); cluster = uaudio_get_cluster(i, iot); uaudio_dump_cluster(&cluster); printf("\n"); break; default: printf("unknown audio control (subtype=%d)\n", iot[i].d.desc->bDescriptorSubtype); } for (j = 0; j < iot[i].inputs_size; j++) { printf("\tinput%d: ", j); uaudio_dump_tml(iot[i].inputs[j]); } printf("\toutput: "); uaudio_dump_tml(iot[i].output); } #endif for (i = 0; i < ndps; i++) { dp = iot[i].d.desc; if (dp == NULL) continue; DPRINTF("id=%d subtype=%d\n", i, dp->bDescriptorSubtype); switch (dp->bDescriptorSubtype) { case UDESCSUB_AC_HEADER: aprint_error("uaudio_identify_ac: unexpected AC header\n"); break; case UDESCSUB_AC_INPUT: uaudio_add_input(sc, iot, i); break; case UDESCSUB_AC_OUTPUT: uaudio_add_output(sc, iot, i); break; case UDESCSUB_AC_MIXER: uaudio_add_mixer(sc, iot, i); break; case UDESCSUB_AC_SELECTOR: uaudio_add_selector(sc, iot, i); break; case UDESCSUB_AC_FEATURE: uaudio_add_feature(sc, iot, i); break; case UDESCSUB_AC_PROCESSING: uaudio_add_processing(sc, iot, i); break; case UDESCSUB_AC_EXTENSION: uaudio_add_extension(sc, iot, i); break; default: aprint_error( "uaudio_identify_ac: bad AC desc subtype=0x%02x\n", dp->bDescriptorSubtype); break; } } /* delete io_terminal */ for (i = 0; i < 256; i++) { if (iot[i].d.desc == NULL) continue; if (iot[i].inputs != NULL) { for (j = 0; j < iot[i].inputs_size; j++) { if (iot[i].inputs[j] != NULL) free(iot[i].inputs[j], M_TEMP); } free(iot[i].inputs, M_TEMP); } if (iot[i].output != NULL) free(iot[i].output, M_TEMP); iot[i].d.desc = NULL; } free(iot, M_TEMP); return USBD_NORMAL_COMPLETION; } Static int uaudio_query_devinfo(void *addr, mixer_devinfo_t *mi) { struct uaudio_softc *sc; struct mixerctl *mc; int n, nctls, i; DPRINTFN(7, "index=%d\n", mi->index); sc = addr; if (sc->sc_dying) return EIO; n = mi->index; nctls = sc->sc_nctls; switch (n) { case UAC_OUTPUT: mi->type = AUDIO_MIXER_CLASS; mi->mixer_class = UAC_OUTPUT; mi->next = mi->prev = AUDIO_MIXER_LAST; strlcpy(mi->label.name, AudioCoutputs, sizeof(mi->label.name)); return 0; case UAC_INPUT: mi->type = AUDIO_MIXER_CLASS; mi->mixer_class = UAC_INPUT; mi->next = mi->prev = AUDIO_MIXER_LAST; strlcpy(mi->label.name, AudioCinputs, sizeof(mi->label.name)); return 0; case UAC_EQUAL: mi->type = AUDIO_MIXER_CLASS; mi->mixer_class = UAC_EQUAL; mi->next = mi->prev = AUDIO_MIXER_LAST; strlcpy(mi->label.name, AudioCequalization, sizeof(mi->label.name)); return 0; case UAC_RECORD: mi->type = AUDIO_MIXER_CLASS; mi->mixer_class = UAC_RECORD; mi->next = mi->prev = AUDIO_MIXER_LAST; strlcpy(mi->label.name, AudioCrecord, sizeof(mi->label.name)); return 0; default: break; } n -= UAC_NCLASSES; if (n < 0 || n >= nctls) return ENXIO; mc = &sc->sc_ctls[n]; strlcpy(mi->label.name, mc->ctlname, sizeof(mi->label.name)); mi->mixer_class = mc->class; mi->next = mi->prev = AUDIO_MIXER_LAST; /* XXX */ switch (mc->type) { case MIX_ON_OFF: mi->type = AUDIO_MIXER_ENUM; mi->un.e.num_mem = 2; strlcpy(mi->un.e.member[0].label.name, AudioNoff, sizeof(mi->un.e.member[0].label.name)); mi->un.e.member[0].ord = 0; strlcpy(mi->un.e.member[1].label.name, AudioNon, sizeof(mi->un.e.member[1].label.name)); mi->un.e.member[1].ord = 1; break; case MIX_SELECTOR: mi->type = AUDIO_MIXER_ENUM; mi->un.e.num_mem = mc->maxval - mc->minval + 1; for (i = 0; i <= mc->maxval - mc->minval; i++) { snprintf(mi->un.e.member[i].label.name, sizeof(mi->un.e.member[i].label.name), "%d", i + mc->minval); mi->un.e.member[i].ord = i + mc->minval; } break; default: mi->type = AUDIO_MIXER_VALUE; strncpy(mi->un.v.units.name, mc->ctlunit, MAX_AUDIO_DEV_LEN); mi->un.v.num_channels = mc->nchan; mi->un.v.delta = mc->delta; break; } return 0; } Static int uaudio_open(void *addr, int flags) { struct uaudio_softc *sc; sc = addr; DPRINTF("sc=%p\n", sc); if (sc->sc_dying) return EIO; if ((flags & FWRITE) && !(sc->sc_mode & AUMODE_PLAY)) return EACCES; if ((flags & FREAD) && !(sc->sc_mode & AUMODE_RECORD)) return EACCES; return 0; } Static int uaudio_halt_out_dma(void *addr) { struct uaudio_softc *sc = addr; DPRINTF("%s", "enter\n"); mutex_exit(&sc->sc_intr_lock); uaudio_halt_out_dma_unlocked(sc); mutex_enter(&sc->sc_intr_lock); return 0; } Static void uaudio_halt_out_dma_unlocked(struct uaudio_softc *sc) { if (sc->sc_playchan.pipe != NULL) { uaudio_chan_abort(sc, &sc->sc_playchan); uaudio_chan_free_buffers(sc, &sc->sc_playchan); uaudio_chan_close(sc, &sc->sc_playchan); sc->sc_playchan.intr = NULL; } } Static int uaudio_halt_in_dma(void *addr) { struct uaudio_softc *sc = addr; DPRINTF("%s", "enter\n"); mutex_exit(&sc->sc_intr_lock); uaudio_halt_in_dma_unlocked(sc); mutex_enter(&sc->sc_intr_lock); return 0; } Static void uaudio_halt_in_dma_unlocked(struct uaudio_softc *sc) { if (sc->sc_recchan.pipe != NULL) { uaudio_chan_abort(sc, &sc->sc_recchan); uaudio_chan_free_buffers(sc, &sc->sc_recchan); uaudio_chan_close(sc, &sc->sc_recchan); sc->sc_recchan.intr = NULL; } } Static int uaudio_getdev(void *addr, struct audio_device *retp) { struct uaudio_softc *sc; DPRINTF("%s", "\n"); sc = addr; if (sc->sc_dying) return EIO; *retp = sc->sc_adev; return 0; } /* * Make sure the block size is large enough to hold all outstanding transfers. */ Static int uaudio_round_blocksize(void *addr, int blk, int mode, const audio_params_t *param) { struct uaudio_softc *sc; int b; sc = addr; DPRINTF("blk=%d mode=%s\n", blk, mode == AUMODE_PLAY ? "AUMODE_PLAY" : "AUMODE_RECORD"); /* chan.bytes_per_frame can be 0. */ if (mode == AUMODE_PLAY || sc->sc_recchan.bytes_per_frame <= 0) { b = param->sample_rate * UAUDIO_NFRAMES * UAUDIO_NCHANBUFS; /* * This does not make accurate value in the case * of b % USB_FRAMES_PER_SECOND != 0 */ b /= USB_FRAMES_PER_SECOND; b *= param->precision / 8 * param->channels; } else { /* * use wMaxPacketSize in bytes_per_frame. * See uaudio_set_format() and uaudio_chan_init() */ b = sc->sc_recchan.bytes_per_frame * UAUDIO_NFRAMES * UAUDIO_NCHANBUFS; } if (b <= 0) b = 1; blk = blk <= b ? b : blk / b * b; #ifdef DIAGNOSTIC if (blk <= 0) { aprint_debug("uaudio_round_blocksize: blk=%d\n", blk); blk = 512; } #endif DPRINTF("resultant blk=%d\n", blk); return blk; } Static int uaudio_get_props(void *addr) { struct uaudio_softc *sc; int props; sc = addr; props = 0; if ((sc->sc_mode & AUMODE_PLAY)) props |= AUDIO_PROP_PLAYBACK; if ((sc->sc_mode & AUMODE_RECORD)) props |= AUDIO_PROP_CAPTURE; /* XXX I'm not sure all bidirectional devices support FULLDUP&INDEP */ if (props == (AUDIO_PROP_PLAYBACK | AUDIO_PROP_CAPTURE)) props |= AUDIO_PROP_FULLDUPLEX | AUDIO_PROP_INDEPENDENT; return props; } Static void uaudio_get_locks(void *addr, kmutex_t **intr, kmutex_t **thread) { struct uaudio_softc *sc; sc = addr; *intr = &sc->sc_intr_lock; *thread = &sc->sc_lock; } Static int uaudio_get(struct uaudio_softc *sc, int which, int type, int wValue, int wIndex, int len) { usb_device_request_t req; uint8_t data[4]; usbd_status err; int val; if (wValue == -1) return 0; req.bmRequestType = type; req.bRequest = which; USETW(req.wValue, wValue); USETW(req.wIndex, wIndex); USETW(req.wLength, len); DPRINTFN(2,"type=0x%02x req=0x%02x wValue=0x%04x " "wIndex=0x%04x len=%d\n", type, which, wValue, wIndex, len); err = usbd_do_request(sc->sc_udev, &req, data); if (err) { DPRINTF("err=%s\n", usbd_errstr(err)); return -1; } switch (len) { case 1: val = data[0]; break; case 2: val = data[0] | (data[1] << 8); break; default: DPRINTF("bad length=%d\n", len); return -1; } DPRINTFN(2,"val=%d\n", val); return val; } Static void uaudio_set(struct uaudio_softc *sc, int which, int type, int wValue, int wIndex, int len, int val) { usb_device_request_t req; uint8_t data[4]; int err __unused; if (wValue == -1) return; req.bmRequestType = type; req.bRequest = which; USETW(req.wValue, wValue); USETW(req.wIndex, wIndex); USETW(req.wLength, len); switch (len) { case 1: data[0] = val; break; case 2: data[0] = val; data[1] = val >> 8; break; default: return; } DPRINTFN(2,"type=0x%02x req=0x%02x wValue=0x%04x " "wIndex=0x%04x len=%d, val=%d\n", type, which, wValue, wIndex, len, val & 0xffff); err = usbd_do_request(sc->sc_udev, &req, data); #ifdef UAUDIO_DEBUG if (err) DPRINTF("err=%d\n", err); #endif } Static int uaudio_signext(int type, int val) { if (!MIX_UNSIGNED(type)) { if (MIX_SIZE(type) == 2) val = (int16_t)val; else val = (int8_t)val; } return val; } Static int uaudio_value2bsd(struct mixerctl *mc, int val) { DPRINTFN(5, "type=%03x val=%d min=%d max=%d ", mc->type, val, mc->minval, mc->maxval); if (mc->type == MIX_ON_OFF) { val = (val != 0); } else if (mc->type == MIX_SELECTOR) { if (val < mc->minval || val > mc->maxval) val = mc->minval; } else val = ((uaudio_signext(mc->type, val) - mc->minval) * 255 + mc->mul/2) / mc->mul; DPRINTFN_CLEAN(5, "val'=%d\n", val); return val; } int uaudio_bsd2value(struct mixerctl *mc, int val) { DPRINTFN(5,"type=%03x val=%d min=%d max=%d ", mc->type, val, mc->minval, mc->maxval); if (mc->type == MIX_ON_OFF) { val = (val != 0); } else if (mc->type == MIX_SELECTOR) { if (val < mc->minval || val > mc->maxval) val = mc->minval; } else val = (val + mc->delta/2) * mc->mul / 255 + mc->minval; DPRINTFN_CLEAN(5, "val'=%d\n", val); return val; } Static int uaudio_ctl_get(struct uaudio_softc *sc, int which, struct mixerctl *mc, int chan) { int val; DPRINTFN(5,"which=%d chan=%d\n", which, chan); mutex_exit(&sc->sc_lock); val = uaudio_get(sc, which, UT_READ_CLASS_INTERFACE, mc->wValue[chan], mc->wIndex, MIX_SIZE(mc->type)); mutex_enter(&sc->sc_lock); return uaudio_value2bsd(mc, val); } Static void uaudio_ctl_set(struct uaudio_softc *sc, int which, struct mixerctl *mc, int chan, int val) { val = uaudio_bsd2value(mc, val); mutex_exit(&sc->sc_lock); uaudio_set(sc, which, UT_WRITE_CLASS_INTERFACE, mc->wValue[chan], mc->wIndex, MIX_SIZE(mc->type), val); mutex_enter(&sc->sc_lock); } Static int uaudio_mixer_get_port(void *addr, mixer_ctrl_t *cp) { struct uaudio_softc *sc; struct mixerctl *mc; int i, n, vals[MIX_MAX_CHAN], val; DPRINTFN(2, "index=%d\n", cp->dev); sc = addr; if (sc->sc_dying) return EIO; n = cp->dev - UAC_NCLASSES; if (n < 0 || n >= sc->sc_nctls) return ENXIO; mc = &sc->sc_ctls[n]; if (mc->type == MIX_ON_OFF) { if (cp->type != AUDIO_MIXER_ENUM) return EINVAL; cp->un.ord = uaudio_ctl_get(sc, GET_CUR, mc, 0); } else if (mc->type == MIX_SELECTOR) { if (cp->type != AUDIO_MIXER_ENUM) return EINVAL; cp->un.ord = uaudio_ctl_get(sc, GET_CUR, mc, 0); } else { if (cp->type != AUDIO_MIXER_VALUE) return EINVAL; if (cp->un.value.num_channels != 1 && cp->un.value.num_channels != mc->nchan) return EINVAL; for (i = 0; i < mc->nchan; i++) vals[i] = uaudio_ctl_get(sc, GET_CUR, mc, i); if (cp->un.value.num_channels == 1 && mc->nchan != 1) { for (val = 0, i = 0; i < mc->nchan; i++) val += vals[i]; vals[0] = val / mc->nchan; } for (i = 0; i < cp->un.value.num_channels; i++) cp->un.value.level[i] = vals[i]; } return 0; } Static int uaudio_mixer_set_port(void *addr, mixer_ctrl_t *cp) { struct uaudio_softc *sc; struct mixerctl *mc; int i, n, vals[MIX_MAX_CHAN]; DPRINTFN(2, "index = %d\n", cp->dev); sc = addr; if (sc->sc_dying) return EIO; n = cp->dev - UAC_NCLASSES; if (n < 0 || n >= sc->sc_nctls) return ENXIO; mc = &sc->sc_ctls[n]; if (mc->type == MIX_ON_OFF) { if (cp->type != AUDIO_MIXER_ENUM) return EINVAL; uaudio_ctl_set(sc, SET_CUR, mc, 0, cp->un.ord); } else if (mc->type == MIX_SELECTOR) { if (cp->type != AUDIO_MIXER_ENUM) return EINVAL; uaudio_ctl_set(sc, SET_CUR, mc, 0, cp->un.ord); } else { if (cp->type != AUDIO_MIXER_VALUE) return EINVAL; if (cp->un.value.num_channels == 1) for (i = 0; i < mc->nchan; i++) vals[i] = cp->un.value.level[0]; else if (cp->un.value.num_channels == mc->nchan) for (i = 0; i < mc->nchan; i++) vals[i] = cp->un.value.level[i]; else return EINVAL; for (i = 0; i < mc->nchan; i++) uaudio_ctl_set(sc, SET_CUR, mc, i, vals[i]); } return 0; } Static int uaudio_trigger_input(void *addr, void *start, void *end, int blksize, void (*intr)(void *), void *arg, const audio_params_t *param) { struct uaudio_softc *sc; struct chan *ch; usbd_status err; int i; sc = addr; if (sc->sc_dying) return EIO; mutex_exit(&sc->sc_intr_lock); DPRINTFN(3, "sc=%p start=%p end=%p " "blksize=%d\n", sc, start, end, blksize); ch = &sc->sc_recchan; uaudio_chan_set_param(ch, start, end, blksize); DPRINTFN(3, "sample_size=%d bytes/frame=%d " "fraction=0.%03d\n", ch->sample_size, ch->bytes_per_frame, ch->fraction); err = uaudio_chan_open(sc, ch); if (err) { mutex_enter(&sc->sc_intr_lock); device_printf(sc->sc_dev,"%s open channel err=%s\n",__func__, usbd_errstr(err)); return EIO; } err = uaudio_chan_alloc_buffers(sc, ch); if (err) { uaudio_chan_close(sc, ch); device_printf(sc->sc_dev,"%s alloc buffers err=%s\n",__func__, usbd_errstr(err)); mutex_enter(&sc->sc_intr_lock); return EIO; } ch->intr = intr; ch->arg = arg; /* * Start as half as many channels for recording as for playback. * This stops playback from stuttering in full-duplex operation. */ for (i = 0; i < UAUDIO_NCHANBUFS / 2; i++) { uaudio_chan_rtransfer(ch); } mutex_enter(&sc->sc_intr_lock); return 0; } Static int uaudio_trigger_output(void *addr, void *start, void *end, int blksize, void (*intr)(void *), void *arg, const audio_params_t *param) { struct uaudio_softc *sc; struct chan *ch; usbd_status err; int i; sc = addr; if (sc->sc_dying) return EIO; mutex_exit(&sc->sc_intr_lock); DPRINTFN(3, "sc=%p start=%p end=%p " "blksize=%d\n", sc, start, end, blksize); ch = &sc->sc_playchan; uaudio_chan_set_param(ch, start, end, blksize); DPRINTFN(3, "sample_size=%d bytes/frame=%d " "fraction=0.%03d\n", ch->sample_size, ch->bytes_per_frame, ch->fraction); err = uaudio_chan_open(sc, ch); if (err) { mutex_enter(&sc->sc_intr_lock); device_printf(sc->sc_dev,"%s open channel err=%s\n",__func__, usbd_errstr(err)); return EIO; } err = uaudio_chan_alloc_buffers(sc, ch); if (err) { uaudio_chan_close(sc, ch); device_printf(sc->sc_dev,"%s alloc buffers err=%s\n",__func__, usbd_errstr(err)); mutex_enter(&sc->sc_intr_lock); return EIO; } ch->intr = intr; ch->arg = arg; for (i = 0; i < UAUDIO_NCHANBUFS; i++) uaudio_chan_ptransfer(ch); mutex_enter(&sc->sc_intr_lock); return 0; } /* Set up a pipe for a channel. */ Static usbd_status uaudio_chan_open(struct uaudio_softc *sc, struct chan *ch) { struct as_info *as; usb_device_descriptor_t *ddesc; int endpt; usbd_status err; as = &sc->sc_alts[ch->altidx]; endpt = as->edesc->bEndpointAddress; DPRINTF("endpt=0x%02x, speed=%d, alt=%d\n", endpt, ch->sample_rate, as->alt); /* Set alternate interface corresponding to the mode. */ err = usbd_set_interface(as->ifaceh, as->alt); if (err) return err; /* * Roland SD-90 freezes by a SAMPLING_FREQ_CONTROL request. */ ddesc = usbd_get_device_descriptor(sc->sc_udev); if ((UGETW(ddesc->idVendor) != USB_VENDOR_ROLAND) && (UGETW(ddesc->idProduct) != USB_PRODUCT_ROLAND_SD90)) { err = uaudio_set_speed(sc, endpt, ch->sample_rate); if (err) { DPRINTF("set_speed failed err=%s\n", usbd_errstr(err)); } } DPRINTF("create pipe to 0x%02x\n", endpt); err = usbd_open_pipe(as->ifaceh, endpt, USBD_MPSAFE, &ch->pipe); if (err) return err; if (as->edesc1 != NULL) { endpt = as->edesc1->bEndpointAddress; DPRINTF("create sync-pipe to 0x%02x\n", endpt); err = usbd_open_pipe(as->ifaceh, endpt, USBD_MPSAFE, &ch->sync_pipe); } return err; } Static void uaudio_chan_abort(struct uaudio_softc *sc, struct chan *ch) { struct usbd_pipe *pipe; struct as_info *as; as = &sc->sc_alts[ch->altidx]; as->sc_busy = 0; if (sc->sc_nullalt >= 0) { DPRINTF("set null alt=%d\n", sc->sc_nullalt); usbd_set_interface(as->ifaceh, sc->sc_nullalt); } pipe = ch->pipe; if (pipe) { usbd_abort_pipe(pipe); } pipe = ch->sync_pipe; if (pipe) { usbd_abort_pipe(pipe); } } Static void uaudio_chan_close(struct uaudio_softc *sc, struct chan *ch) { struct usbd_pipe *pipe; pipe = atomic_swap_ptr(&ch->pipe, NULL); if (pipe) { usbd_close_pipe(pipe); } pipe = atomic_swap_ptr(&ch->sync_pipe, NULL); if (pipe) { usbd_close_pipe(pipe); } } Static usbd_status uaudio_chan_alloc_buffers(struct uaudio_softc *sc, struct chan *ch) { int i, size; size = (ch->bytes_per_frame + ch->sample_size) * UAUDIO_NFRAMES; for (i = 0; i < UAUDIO_NCHANBUFS; i++) { struct usbd_xfer *xfer; int err = usbd_create_xfer(ch->pipe, size, 0, UAUDIO_NFRAMES, &xfer); if (err) goto bad; ch->chanbufs[i].xfer = xfer; ch->chanbufs[i].buffer = usbd_get_buffer(xfer); ch->chanbufs[i].chan = ch; } return USBD_NORMAL_COMPLETION; bad: while (--i >= 0) /* implicit buffer free */ usbd_destroy_xfer(ch->chanbufs[i].xfer); return USBD_NOMEM; } Static void uaudio_chan_free_buffers(struct uaudio_softc *sc, struct chan *ch) { int i; for (i = 0; i < UAUDIO_NCHANBUFS; i++) usbd_destroy_xfer(ch->chanbufs[i].xfer); } Static void uaudio_chan_ptransfer(struct chan *ch) { struct chanbuf *cb; int i, n, size, residue, total; if (ch->sc->sc_dying) return; /* Pick the next channel buffer. */ cb = &ch->chanbufs[ch->curchanbuf]; if (++ch->curchanbuf >= UAUDIO_NCHANBUFS) ch->curchanbuf = 0; /* Compute the size of each frame in the next transfer. */ residue = ch->residue; total = 0; for (i = 0; i < UAUDIO_NFRAMES; i++) { size = ch->bytes_per_frame; residue += ch->fraction; if (residue >= USB_FRAMES_PER_SECOND) { if ((ch->sc->sc_altflags & UA_NOFRAC) == 0) size += ch->sample_size; residue -= USB_FRAMES_PER_SECOND; } cb->sizes[i] = size; total += size; } ch->residue = residue; cb->size = total; /* * Transfer data from upper layer buffer to channel buffer, taking * care of wrapping the upper layer buffer. */ n = uimin(total, ch->end - ch->cur); memcpy(cb->buffer, ch->cur, n); ch->cur += n; if (ch->cur >= ch->end) ch->cur = ch->start; if (total > n) { total -= n; memcpy(cb->buffer + n, ch->cur, total); ch->cur += total; } #ifdef UAUDIO_DEBUG if (uaudiodebug > 8) { DPRINTF("buffer=%p, residue=0.%03d\n", cb->buffer, ch->residue); for (i = 0; i < UAUDIO_NFRAMES; i++) { DPRINTF(" [%d] length %d\n", i, cb->sizes[i]); } } #endif //DPRINTFN(5, "ptransfer xfer=%p\n", cb->xfer); /* Fill the request */ usbd_setup_isoc_xfer(cb->xfer, cb, cb->sizes, UAUDIO_NFRAMES, 0, uaudio_chan_pintr); usbd_status err = usbd_transfer(cb->xfer); if (err != USBD_IN_PROGRESS && err != USBD_NORMAL_COMPLETION) device_printf(ch->sc->sc_dev, "ptransfer error %d\n", err); } Static void uaudio_chan_pintr(struct usbd_xfer *xfer, void *priv, usbd_status status) { struct chanbuf *cb; struct chan *ch; uint32_t count; cb = priv; ch = cb->chan; /* Return if we are aborting. */ if (status == USBD_CANCELLED) return; if (status != USBD_NORMAL_COMPLETION) device_printf(ch->sc->sc_dev, "pintr error: %s\n", usbd_errstr(status)); usbd_get_xfer_status(xfer, NULL, NULL, &count, NULL); DPRINTFN(5, "count=%d, transferred=%d\n", count, ch->transferred); #ifdef DIAGNOSTIC if (count != cb->size) { device_printf(ch->sc->sc_dev, "uaudio_chan_pintr: count(%d) != size(%d), status(%d)\n", count, cb->size, status); } #endif mutex_enter(&ch->sc->sc_intr_lock); ch->transferred += cb->size; /* Call back to upper layer */ while (ch->transferred >= ch->blksize) { ch->transferred -= ch->blksize; DPRINTFN(5, "call %p(%p)\n", ch->intr, ch->arg); ch->intr(ch->arg); } mutex_exit(&ch->sc->sc_intr_lock); /* start next transfer */ uaudio_chan_ptransfer(ch); } Static void uaudio_chan_rtransfer(struct chan *ch) { struct chanbuf *cb; int i, size, residue, total; if (ch->sc->sc_dying) return; /* Pick the next channel buffer. */ cb = &ch->chanbufs[ch->curchanbuf]; if (++ch->curchanbuf >= UAUDIO_NCHANBUFS) ch->curchanbuf = 0; /* Compute the size of each frame in the next transfer. */ residue = ch->residue; total = 0; for (i = 0; i < UAUDIO_NFRAMES; i++) { size = ch->bytes_per_frame; cb->sizes[i] = size; cb->offsets[i] = total; total += size; } ch->residue = residue; cb->size = total; #ifdef UAUDIO_DEBUG if (uaudiodebug > 8) { DPRINTF("buffer=%p, residue=0.%03d\n", cb->buffer, ch->residue); for (i = 0; i < UAUDIO_NFRAMES; i++) { DPRINTF(" [%d] length %d\n", i, cb->sizes[i]); } } #endif DPRINTFN(5, "transfer xfer=%p\n", cb->xfer); /* Fill the request */ usbd_setup_isoc_xfer(cb->xfer, cb, cb->sizes, UAUDIO_NFRAMES, 0, uaudio_chan_rintr); usbd_status err = usbd_transfer(cb->xfer); if (err != USBD_IN_PROGRESS && err != USBD_NORMAL_COMPLETION) device_printf(ch->sc->sc_dev, "rtransfer error %d\n", err); } Static void uaudio_chan_rintr(struct usbd_xfer *xfer, void *priv, usbd_status status) { struct chanbuf *cb; struct chan *ch; uint32_t count; int i, n, frsize; cb = priv; ch = cb->chan; /* Return if we are aborting. */ if (status == USBD_CANCELLED) return; if (status != USBD_NORMAL_COMPLETION && status != USBD_SHORT_XFER) device_printf(ch->sc->sc_dev, "rintr error: %s\n", usbd_errstr(status)); usbd_get_xfer_status(xfer, NULL, NULL, &count, NULL); DPRINTFN(5, "count=%d, transferred=%d\n", count, ch->transferred); /* count < cb->size is normal for asynchronous source */ #ifdef DIAGNOSTIC if (count > cb->size) { device_printf(ch->sc->sc_dev, "uaudio_chan_rintr: count(%d) > size(%d) status(%d)\n", count, cb->size, status); } #endif /* * Transfer data from channel buffer to upper layer buffer, taking * care of wrapping the upper layer buffer. */ for (i = 0; i < UAUDIO_NFRAMES; i++) { frsize = cb->sizes[i]; n = uimin(frsize, ch->end - ch->cur); memcpy(ch->cur, cb->buffer + cb->offsets[i], n); ch->cur += n; if (ch->cur >= ch->end) ch->cur = ch->start; if (frsize > n) { memcpy(ch->cur, cb->buffer + cb->offsets[i] + n, frsize - n); ch->cur += frsize - n; } } /* Call back to upper layer */ mutex_enter(&ch->sc->sc_intr_lock); ch->transferred += count; while (ch->transferred >= ch->blksize) { ch->transferred -= ch->blksize; DPRINTFN(5, "call %p(%p)\n", ch->intr, ch->arg); ch->intr(ch->arg); } mutex_exit(&ch->sc->sc_intr_lock); /* start next transfer */ uaudio_chan_rtransfer(ch); } Static void uaudio_chan_init(struct chan *ch, int altidx, const struct audio_params *param, int maxpktsize) { int samples_per_frame, sample_size; ch->altidx = altidx; sample_size = param->precision * param->channels / 8; samples_per_frame = param->sample_rate / USB_FRAMES_PER_SECOND; ch->sample_size = sample_size; ch->sample_rate = param->sample_rate; if (maxpktsize == 0) { ch->fraction = param->sample_rate % USB_FRAMES_PER_SECOND; ch->bytes_per_frame = samples_per_frame * sample_size; } else { ch->fraction = 0; ch->bytes_per_frame = maxpktsize; } ch->residue = 0; } Static void uaudio_chan_set_param(struct chan *ch, u_char *start, u_char *end, int blksize) { ch->start = start; ch->end = end; ch->cur = start; ch->blksize = blksize; ch->transferred = 0; ch->curchanbuf = 0; } Static int uaudio_set_format(void *addr, int setmode, const audio_params_t *play, const audio_params_t *rec, audio_filter_reg_t *pfil, audio_filter_reg_t *rfil) { struct uaudio_softc *sc; int paltidx, raltidx; sc = addr; paltidx = -1; raltidx = -1; if (sc->sc_dying) return EIO; if ((setmode & AUMODE_PLAY) && sc->sc_playchan.altidx != -1) { sc->sc_alts[sc->sc_playchan.altidx].sc_busy = 0; } if ((setmode & AUMODE_RECORD) && sc->sc_recchan.altidx != -1) { sc->sc_alts[sc->sc_recchan.altidx].sc_busy = 0; } /* Some uaudio devices are unidirectional. Don't try to find a matching mode for the unsupported direction. */ setmode &= sc->sc_mode; if ((setmode & AUMODE_PLAY)) { paltidx = audio_indexof_format(sc->sc_formats, sc->sc_nformats, AUMODE_PLAY, play); /* Transfer should have halted */ uaudio_chan_init(&sc->sc_playchan, paltidx, play, 0); } if ((setmode & AUMODE_RECORD)) { raltidx = audio_indexof_format(sc->sc_formats, sc->sc_nformats, AUMODE_RECORD, rec); /* Transfer should have halted */ uaudio_chan_init(&sc->sc_recchan, raltidx, rec, UGETW(sc->sc_alts[raltidx].edesc->wMaxPacketSize)); } if ((setmode & AUMODE_PLAY) && sc->sc_playchan.altidx != -1) { sc->sc_alts[sc->sc_playchan.altidx].sc_busy = 1; } if ((setmode & AUMODE_RECORD) && sc->sc_recchan.altidx != -1) { sc->sc_alts[sc->sc_recchan.altidx].sc_busy = 1; } DPRINTF("use altidx=p%d/r%d, altno=p%d/r%d\n", sc->sc_playchan.altidx, sc->sc_recchan.altidx, (sc->sc_playchan.altidx >= 0) ?sc->sc_alts[sc->sc_playchan.altidx].idesc->bAlternateSetting : -1, (sc->sc_recchan.altidx >= 0) ? sc->sc_alts[sc->sc_recchan.altidx].idesc->bAlternateSetting : -1); return 0; } Static usbd_status uaudio_set_speed(struct uaudio_softc *sc, int endpt, u_int speed) { usb_device_request_t req; usbd_status err; uint8_t data[3]; DPRINTFN(5, "endpt=%d speed=%u\n", endpt, speed); req.bmRequestType = UT_WRITE_CLASS_ENDPOINT; req.bRequest = SET_CUR; USETW2(req.wValue, SAMPLING_FREQ_CONTROL, 0); USETW(req.wIndex, endpt); USETW(req.wLength, 3); data[0] = speed; data[1] = speed >> 8; data[2] = speed >> 16; err = usbd_do_request(sc->sc_udev, &req, data); return err; } #ifdef _MODULE MODULE(MODULE_CLASS_DRIVER, uaudio, NULL); static const struct cfiattrdata audiobuscf_iattrdata = { "audiobus", 0, { { NULL, NULL, 0 }, } }; static const struct cfiattrdata * const uaudio_attrs[] = { &audiobuscf_iattrdata, NULL }; CFDRIVER_DECL(uaudio, DV_DULL, uaudio_attrs); extern struct cfattach uaudio_ca; static int uaudioloc[6/*USBIFIFCF_NLOCS*/] = { -1/*USBIFIFCF_PORT_DEFAULT*/, -1/*USBIFIFCF_CONFIGURATION_DEFAULT*/, -1/*USBIFIFCF_INTERFACE_DEFAULT*/, -1/*USBIFIFCF_VENDOR_DEFAULT*/, -1/*USBIFIFCF_PRODUCT_DEFAULT*/, -1/*USBIFIFCF_RELEASE_DEFAULT*/}; static struct cfparent uhubparent = { "usbifif", NULL, DVUNIT_ANY }; static struct cfdata uaudio_cfdata[] = { { .cf_name = "uaudio", .cf_atname = "uaudio", .cf_unit = 0, .cf_fstate = FSTATE_STAR, .cf_loc = uaudioloc, .cf_flags = 0, .cf_pspec = &uhubparent, }, { NULL } }; static int uaudio_modcmd(modcmd_t cmd, void *arg) { int err; switch (cmd) { case MODULE_CMD_INIT: err = config_cfdriver_attach(&uaudio_cd); if (err) { return err; } err = config_cfattach_attach("uaudio", &uaudio_ca); if (err) { config_cfdriver_detach(&uaudio_cd); return err; } err = config_cfdata_attach(uaudio_cfdata, 1); if (err) { config_cfattach_detach("uaudio", &uaudio_ca); config_cfdriver_detach(&uaudio_cd); return err; } return 0; case MODULE_CMD_FINI: err = config_cfdata_detach(uaudio_cfdata); if (err) return err; config_cfattach_detach("uaudio", &uaudio_ca); config_cfdriver_detach(&uaudio_cd); return 0; default: return ENOTTY; } } #endif |
| 2 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 | /* $NetBSD: dnvlist.c,v 1.4 2018/09/08 14:32:25 christos Exp $ */ /*- * SPDX-License-Identifier: BSD-2-Clause-FreeBSD * * Copyright (c) 2013 The FreeBSD Foundation * All rights reserved. * * This software was developed by Pawel Jakub Dawidek under sponsorship from * the FreeBSD Foundation. * * Redistribution and use in source and binary forms, with or without * modification, are permitted provided that the following conditions * are met: * 1. Redistributions of source code must retain the above copyright * notice, this list of conditions and the following disclaimer. * 2. Redistributions in binary form must reproduce the above copyright * notice, this list of conditions and the following disclaimer in the * documentation and/or other materials provided with the distribution. * * THIS SOFTWARE IS PROVIDED BY THE AUTHORS AND CONTRIBUTORS ``AS IS'' AND * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE * ARE DISCLAIMED. IN NO EVENT SHALL THE AUTHORS OR CONTRIBUTORS BE LIABLE * FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL * DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS * OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) * HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT * LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY * OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF * SUCH DAMAGE. */ #include <sys/cdefs.h> #ifdef __FreeBSD__ __FBSDID("$FreeBSD: head/sys/contrib/libnv/dnvlist.c 328474 2018-01-27 12:58:21Z oshogbo $"); #else __RCSID("$NetBSD: dnvlist.c,v 1.4 2018/09/08 14:32:25 christos Exp $"); #endif #if defined(_KERNEL) || defined(_STANDALONE) #include <sys/types.h> #include <sys/param.h> #include <sys/kernel.h> #include <sys/systm.h> #include <sys/malloc.h> #ifdef __FreeBSD__ #include <machine/stdarg.h> #endif #else #include <stdarg.h> #include <stdbool.h> #include <stdint.h> #include <stdlib.h> #endif #ifdef __FreeBSD__ #include <sys/dnv.h> #include <sys/nv.h> #else #include "dnv.h" #include "nv.h" #endif #include "nv_impl.h" #define DNVLIST_GET(ftype, type) \ ftype \ dnvlist_get_##type(const nvlist_t *nvl, const char *name, ftype defval) \ { \ \ if (nvlist_exists_##type(nvl, name)) \ return (nvlist_get_##type(nvl, name)); \ else \ return (defval); \ } DNVLIST_GET(bool, bool) DNVLIST_GET(uint64_t, number) DNVLIST_GET(const char *, string) DNVLIST_GET(const nvlist_t *, nvlist) #if !defined(_KERNEL) && !defined(_STANDALONE) DNVLIST_GET(int, descriptor) #endif #undef DNVLIST_GET const void * dnvlist_get_binary(const nvlist_t *nvl, const char *name, size_t *sizep, const void *defval, size_t defsize) { const void *value; if (nvlist_exists_binary(nvl, name)) value = nvlist_get_binary(nvl, name, sizep); else { if (sizep != NULL) *sizep = defsize; value = defval; } return (value); } #define DNVLIST_TAKE(ftype, type) \ ftype \ dnvlist_take_##type(nvlist_t *nvl, const char *name, ftype defval) \ { \ \ if (nvlist_exists_##type(nvl, name)) \ return (nvlist_take_##type(nvl, name)); \ else \ return (defval); \ } DNVLIST_TAKE(bool, bool) DNVLIST_TAKE(uint64_t, number) DNVLIST_TAKE(char *, string) DNVLIST_TAKE(nvlist_t *, nvlist) #if !defined(_KERNEL) && !defined(_STANDALONE) DNVLIST_TAKE(int, descriptor) #endif #undef DNVLIST_TAKE void * dnvlist_take_binary(nvlist_t *nvl, const char *name, size_t *sizep, void *defval, size_t defsize) { void *value; if (nvlist_exists_binary(nvl, name)) value = nvlist_take_binary(nvl, name, sizep); else { if (sizep != NULL) *sizep = defsize; value = defval; } return (value); } |
| 8 8 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 197 198 199 200 201 202 203 204 205 206 207 208 209 210 211 212 213 214 215 216 217 218 219 220 221 222 223 224 225 226 227 228 229 230 231 232 233 234 235 236 237 238 239 240 241 242 243 244 245 246 247 248 249 250 251 252 253 254 255 256 257 258 259 260 261 262 263 264 265 266 267 268 269 270 271 272 273 274 275 276 277 278 279 280 281 282 283 284 285 286 287 288 289 290 291 292 293 294 295 296 297 298 299 300 301 302 303 304 305 306 307 308 309 310 311 312 313 314 315 316 317 318 319 320 321 322 323 324 325 326 327 328 329 330 331 332 333 334 335 336 337 338 339 340 341 342 343 344 345 346 347 348 349 350 351 352 353 354 355 356 357 358 359 360 361 362 363 364 365 366 367 368 369 370 371 372 373 374 375 376 377 378 379 380 381 382 383 384 385 386 387 388 389 390 391 392 393 394 395 396 397 398 399 400 401 402 403 404 405 406 407 408 | /* $NetBSD: auvitek.c,v 1.13 2022/03/13 12:49:36 riastradh Exp $ */ /*- * Copyright (c) 2010 Jared D. McNeill <jmcneill@invisible.ca> * All rights reserved. * * Redistribution and use in source and binary forms, with or without * modification, are permitted provided that the following conditions * are met: * 1. Redistributions of source code must retain the above copyright * notice, this list of conditions and the following disclaimer. * 2. Redistributions in binary form must reproduce the above copyright * notice, this list of conditions and the following disclaimer in the * documentation and/or other materials provided with the distribution. * * THIS SOFTWARE IS PROVIDED BY THE NETBSD FOUNDATION, INC. AND CONTRIBUTORS * ``AS IS'' AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED * TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR * PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE FOUNDATION OR CONTRIBUTORS * BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR * CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF * SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS * INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN * CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) * ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE * POSSIBILITY OF SUCH DAMAGE. */ /* * Auvitek AU0828 USB controller */ #include <sys/cdefs.h> __KERNEL_RCSID(0, "$NetBSD: auvitek.c,v 1.13 2022/03/13 12:49:36 riastradh Exp $"); #include <sys/types.h> #include <sys/param.h> #include <sys/systm.h> #include <sys/device.h> #include <sys/conf.h> #include <sys/bus.h> #include <sys/module.h> #include <dev/usb/usb.h> #include <dev/usb/usbdi.h> #include <dev/usb/usbdi_util.h> #include <dev/usb/usbdevs.h> #include <dev/usb/auvitekreg.h> #include <dev/usb/auvitekvar.h> static int auvitek_match(device_t, cfdata_t, void *); static void auvitek_attach(device_t, device_t, void *); static int auvitek_detach(device_t, int); static int auvitek_rescan(device_t, const char *, const int *); static void auvitek_childdet(device_t, device_t); static int auvitek_activate(device_t, enum devact); CFATTACH_DECL2_NEW(auvitek, sizeof(struct auvitek_softc), auvitek_match, auvitek_attach, auvitek_detach, auvitek_activate, auvitek_rescan, auvitek_childdet); static const struct { uint16_t vendor; uint16_t product; const char * name; enum auvitek_board board; } auvitek_devices[] = { { 0x2040, 0x7200, "WinTV HVR-950Q", AUVITEK_BOARD_HVR_950Q }, { 0x2040, 0x7240, "WinTV HVR-850", AUVITEK_BOARD_HVR_850 }, }; static int auvitek_match(device_t parent, cfdata_t match, void *opaque) { struct usb_attach_arg *uaa = opaque; unsigned int i; for (i = 0; i < __arraycount(auvitek_devices); i++) { if (auvitek_devices[i].vendor == uaa->uaa_vendor && auvitek_devices[i].product == uaa->uaa_product) return UMATCH_VENDOR_PRODUCT; } return UMATCH_NONE; } static void auvitek_attach(device_t parent, device_t self, void *opaque) { struct auvitek_softc *sc = device_private(self); struct usb_attach_arg *uaa = opaque; struct usbd_device *dev = uaa->uaa_device; usb_endpoint_descriptor_t *ed; usbd_status err; unsigned int i; uint8_t nep; aprint_naive("\n"); aprint_normal(": AU0828\n"); sc->sc_dev = self; sc->sc_udev = dev; sc->sc_uport = uaa->uaa_port; for (i = 0; i < __arraycount(auvitek_devices); i++) { if (auvitek_devices[i].vendor == uaa->uaa_vendor && auvitek_devices[i].product == uaa->uaa_product) break; } KASSERT(i != __arraycount(auvitek_devices)); sc->sc_descr = auvitek_devices[i].name; sc->sc_board = auvitek_devices[i].board; sc->sc_dying = sc->sc_running = 0; mutex_init(&sc->sc_subdev_lock, MUTEX_DEFAULT, IPL_NONE); err = usbd_set_config_index(dev, 0, 1); if (err) { aprint_error_dev(self, "couldn't set config index: %s\n", usbd_errstr(err)); return; } err = usbd_device2interface_handle(dev, 0, &sc->sc_isoc_iface); if (err) { aprint_error_dev(self, "couldn't get interface handle: %s\n", usbd_errstr(err)); return; } err = usbd_device2interface_handle(dev, 3, &sc->sc_bulk_iface); if (err) { aprint_error_dev(self, "couldn't get interface handle: %s\n", usbd_errstr(err)); return; } sc->sc_ax.ax_sc = sc->sc_ab.ab_sc = sc; sc->sc_ax.ax_endpt = sc->sc_ab.ab_endpt = -1; err = usbd_set_interface(sc->sc_isoc_iface, AUVITEK_XFER_ALTNO); if (err) { aprint_error_dev(self, "couldn't set interface: %s\n", usbd_errstr(err)); return; } nep = 0; usbd_endpoint_count(sc->sc_isoc_iface, &nep); for (i = 0; i < nep; i++) { int dir, type; ed = usbd_interface2endpoint_descriptor(sc->sc_isoc_iface, i); if (ed == NULL) { aprint_error_dev(self, "couldn't read endpoint descriptor %d\n", i); continue; } dir = UE_GET_DIR(ed->bEndpointAddress); type = UE_GET_XFERTYPE(ed->bmAttributes); if (dir == UE_DIR_IN && type == UE_ISOCHRONOUS && sc->sc_ax.ax_endpt == -1) { sc->sc_ax.ax_endpt = ed->bEndpointAddress; sc->sc_ax.ax_maxpktlen = UE_GET_SIZE(UGETW(ed->wMaxPacketSize)) * (UE_GET_TRANS(UGETW(ed->wMaxPacketSize)) + 1); } } err = usbd_set_interface(sc->sc_isoc_iface, 0); if (err) { aprint_error_dev(self, "couldn't set interface: %s\n", usbd_errstr(err)); return; } if (sc->sc_ax.ax_endpt == -1) { aprint_error_dev(self, "couldn't find isoc endpoint\n"); sc->sc_dying = 1; return; } if (sc->sc_ax.ax_maxpktlen == 0) { aprint_error_dev(self, "couldn't determine packet length\n"); sc->sc_dying = 1; return; } aprint_debug_dev(self, "isoc endpoint 0x%02x size %d\n", sc->sc_ax.ax_endpt, sc->sc_ax.ax_maxpktlen); nep = 0; usbd_endpoint_count(sc->sc_bulk_iface, &nep); for (i = 0; i < nep; i++) { int dir, type; ed = usbd_interface2endpoint_descriptor(sc->sc_bulk_iface, i); if (ed == NULL) { aprint_error_dev(self, "couldn't read endpoint descriptor %d\n", i); continue; } dir = UE_GET_DIR(ed->bEndpointAddress); type = UE_GET_XFERTYPE(ed->bmAttributes); if (dir == UE_DIR_IN && type == UE_BULK && sc->sc_ab.ab_endpt == -1) { sc->sc_ab.ab_endpt = ed->bEndpointAddress; } } if (sc->sc_ab.ab_endpt == -1) { aprint_error_dev(self, "couldn't find bulk endpoint\n"); sc->sc_dying = 1; return; } aprint_debug_dev(self, "bulk endpoint 0x%02x size %d\n", sc->sc_ab.ab_endpt, AUVITEK_BULK_BUFLEN); auvitek_board_init(sc); auvitek_i2c_attach(sc); sc->sc_au8522 = au8522_open(self, &sc->sc_i2c, 0x8e >> 1, auvitek_board_get_if_frequency(sc)); if (sc->sc_au8522 == NULL) { aprint_error_dev(sc->sc_dev, "couldn't initialize decoder\n"); sc->sc_dying = 1; return; } config_mountroot(self, auvitek_attach_tuner); auvitek_video_attach(sc); auvitek_audio_attach(sc); auvitek_dtv_attach(sc); } void auvitek_attach_tuner(device_t self) { struct auvitek_softc *sc = device_private(self); mutex_enter(&sc->sc_subdev_lock); if (sc->sc_xc5k == NULL) { sc->sc_xc5k = xc5k_open(sc->sc_dev, &sc->sc_i2c, 0xc2 >> 1, auvitek_board_tuner_reset, sc, auvitek_board_get_if_frequency(sc), FE_ATSC); } mutex_exit(&sc->sc_subdev_lock); } static int auvitek_detach(device_t self, int flags) { struct auvitek_softc *sc = device_private(self); int error; sc->sc_dying = 1; error = config_detach_children(self, flags); if (error) { /* * XXX Should ask autoconf to block open with * .d_cfdriver until we're done, instead of setting * this and then rolling it back. */ sc->sc_dying = 0; return error; } pmf_device_deregister(self); auvitek_dtv_detach(sc, flags); auvitek_audio_detach(sc, flags); auvitek_video_detach(sc, flags); if (sc->sc_xc5k) xc5k_close(sc->sc_xc5k); if (sc->sc_au8522) au8522_close(sc->sc_au8522); auvitek_i2c_detach(sc, flags); mutex_destroy(&sc->sc_subdev_lock); return 0; } int auvitek_activate(device_t self, enum devact act) { struct auvitek_softc *sc = device_private(self); switch (act) { case DVACT_DEACTIVATE: sc->sc_dying = 1; return 0; default: return 0; } } static int auvitek_rescan(device_t self, const char *ifattr, const int *locs) { struct auvitek_softc *sc = device_private(self); auvitek_video_rescan(sc, ifattr, locs); auvitek_dtv_rescan(sc, ifattr, locs); auvitek_i2c_rescan(sc, ifattr, locs); return 0; } static void auvitek_childdet(device_t self, device_t child) { struct auvitek_softc *sc = device_private(self); auvitek_video_childdet(sc, child); auvitek_audio_childdet(sc, child); auvitek_dtv_childdet(sc, child); } uint8_t auvitek_read_1(struct auvitek_softc *sc, uint16_t reg) { usb_device_request_t req; usbd_status err; int actlen; uint8_t data; req.bmRequestType = UT_READ_VENDOR_DEVICE; req.bRequest = AU0828_CMD_REQUEST_IN; USETW(req.wValue, 0); USETW(req.wIndex, reg); USETW(req.wLength, sizeof(data)); KERNEL_LOCK(1, curlwp); err = usbd_do_request_flags(sc->sc_udev, &req, &data, 0, &actlen, USBD_DEFAULT_TIMEOUT); KERNEL_UNLOCK_ONE(curlwp); if (err) printf("%s: read failed: %s\n", device_xname(sc->sc_dev), usbd_errstr(err)); return data; } void auvitek_write_1(struct auvitek_softc *sc, uint16_t reg, uint8_t data) { usb_device_request_t req; usbd_status err; int actlen; req.bmRequestType = UT_WRITE_VENDOR_DEVICE; req.bRequest = AU0828_CMD_REQUEST_OUT; USETW(req.wValue, data); USETW(req.wIndex, reg); USETW(req.wLength, 0); KERNEL_LOCK(1, curlwp); err = usbd_do_request_flags(sc->sc_udev, &req, NULL, 0, &actlen, USBD_DEFAULT_TIMEOUT); KERNEL_UNLOCK_ONE(curlwp); if (err) printf("%s: write failed: %s\n", device_xname(sc->sc_dev), usbd_errstr(err)); } MODULE(MODULE_CLASS_DRIVER, auvitek, "au8522,xc5k"); #ifdef _MODULE #include "ioconf.c" #endif static int auvitek_modcmd(modcmd_t cmd, void *opaque) { switch (cmd) { case MODULE_CMD_INIT: #ifdef _MODULE return config_init_component(cfdriver_ioconf_auvitek, cfattach_ioconf_auvitek, cfdata_ioconf_auvitek); #else return 0; #endif case MODULE_CMD_FINI: #ifdef _MODULE return config_fini_component(cfdriver_ioconf_auvitek, cfattach_ioconf_auvitek, cfdata_ioconf_auvitek); #else return 0; #endif default: return ENOTTY; } } |
| 13 13 13 13 13 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 197 198 199 200 201 202 203 204 205 206 207 208 209 210 211 212 213 214 215 216 217 218 219 220 221 222 223 224 225 226 227 228 229 230 231 232 233 234 235 236 237 238 239 240 241 242 243 244 245 246 247 248 249 250 251 252 253 254 255 256 257 258 259 260 261 262 263 264 265 266 267 268 269 270 271 272 273 274 275 276 277 278 279 280 281 282 283 284 285 286 287 288 289 290 291 292 293 294 295 296 297 298 299 300 301 302 303 304 305 306 307 308 309 310 311 312 313 314 315 316 317 318 319 320 321 322 323 324 325 326 327 328 329 330 331 332 333 334 335 336 337 338 339 340 341 342 343 344 345 346 347 348 349 350 351 352 353 354 355 356 357 358 359 360 361 362 363 364 365 366 367 368 369 370 371 372 373 374 375 376 377 378 379 380 381 382 383 384 385 386 387 388 389 390 391 392 393 394 395 396 397 398 399 400 401 402 403 404 405 406 407 408 409 410 411 412 413 414 415 416 417 418 419 420 421 422 423 424 425 426 427 428 429 430 431 432 433 434 435 436 437 438 439 440 441 442 443 444 445 446 447 448 449 450 451 452 453 454 455 456 457 458 459 460 461 462 463 464 465 466 467 468 469 470 471 472 473 474 475 476 477 478 479 480 481 482 483 484 485 486 487 488 489 490 491 492 493 494 495 496 497 498 499 500 501 502 503 504 505 506 507 508 509 510 511 512 513 514 515 516 517 518 519 520 521 522 523 524 525 526 527 528 529 530 531 532 533 534 535 536 537 538 539 540 541 542 543 544 545 546 547 548 549 550 551 552 553 554 555 556 557 558 559 560 561 562 563 564 565 566 567 568 569 570 571 572 573 574 575 576 577 578 579 580 581 | /* $NetBSD: kern_fileassoc.c,v 1.36 2014/07/10 15:00:28 christos Exp $ */ /*- * Copyright (c) 2006 Elad Efrat <elad@NetBSD.org> * All rights reserved. * * Redistribution and use in source and binary forms, with or without * modification, are permitted provided that the following conditions * are met: * 1. Redistributions of source code must retain the above copyright * notice, this list of conditions and the following disclaimer. * 2. Redistributions in binary form must reproduce the above copyright * notice, this list of conditions and the following disclaimer in the * documentation and/or other materials provided with the distribution. * 3. The name of the author may not be used to endorse or promote products * derived from this software without specific prior written permission. * * THIS SOFTWARE IS PROVIDED BY THE AUTHOR ``AS IS'' AND ANY EXPRESS OR * IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES * OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. * IN NO EVENT SHALL THE AUTHOR BE LIABLE FOR ANY DIRECT, INDIRECT, * INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT * NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, * DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY * THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT * (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF * THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. */ #include <sys/cdefs.h> __KERNEL_RCSID(0, "$NetBSD: kern_fileassoc.c,v 1.36 2014/07/10 15:00:28 christos Exp $"); #include "opt_fileassoc.h" #include <sys/param.h> #include <sys/mount.h> #include <sys/queue.h> #include <sys/vnode.h> #include <sys/errno.h> #include <sys/fileassoc.h> #include <sys/specificdata.h> #include <sys/hash.h> #include <sys/kmem.h> #include <sys/once.h> #define FILEASSOC_INITIAL_TABLESIZE 128 static specificdata_domain_t fileassoc_domain = NULL; static specificdata_key_t fileassoc_mountspecific_key; static ONCE_DECL(control); /* * Assoc entry. * Includes the assoc name for identification and private clear callback. */ struct fileassoc { LIST_ENTRY(fileassoc) assoc_list; const char *assoc_name; /* Name. */ fileassoc_cleanup_cb_t assoc_cleanup_cb; /* Clear callback. */ specificdata_key_t assoc_key; }; static LIST_HEAD(, fileassoc) fileassoc_list; /* An entry in the per-mount hash table. */ struct fileassoc_file { fhandle_t *faf_handle; /* File handle */ specificdata_reference faf_data; /* Assoc data. */ u_int faf_nassocs; /* # of assocs. */ LIST_ENTRY(fileassoc_file) faf_list; /* List pointer. */ }; LIST_HEAD(fileassoc_hash_entry, fileassoc_file); struct fileassoc_table { struct fileassoc_hash_entry *tbl_hash; u_long tbl_mask; /* Hash table mask. */ size_t tbl_nslots; /* Number of slots. */ size_t tbl_nused; /* # of used slots. */ specificdata_reference tbl_data; }; /* * Hashing function: Takes a number modulus the mask to give back an * index into the hash table. */ #define FILEASSOC_HASH(tbl, handle) \ (hash32_buf((handle), FHANDLE_SIZE(handle), HASH32_BUF_INIT) \ & ((tbl)->tbl_mask)) static void * file_getdata(struct fileassoc_file *faf, const struct fileassoc *assoc) { return specificdata_getspecific(fileassoc_domain, &faf->faf_data, assoc->assoc_key); } static void file_setdata(struct fileassoc_file *faf, const struct fileassoc *assoc, void *data) { specificdata_setspecific(fileassoc_domain, &faf->faf_data, assoc->assoc_key, data); } static void file_cleanup(struct fileassoc_file *faf, const struct fileassoc *assoc) { fileassoc_cleanup_cb_t cb; void *data; cb = assoc->assoc_cleanup_cb; if (cb == NULL) { return; } data = file_getdata(faf, assoc); (*cb)(data); } static void file_free(struct fileassoc_file *faf) { struct fileassoc *assoc; LIST_REMOVE(faf, faf_list); LIST_FOREACH(assoc, &fileassoc_list, assoc_list) { file_cleanup(faf, assoc); } vfs_composefh_free(faf->faf_handle); specificdata_fini(fileassoc_domain, &faf->faf_data); kmem_free(faf, sizeof(*faf)); } static void table_dtor(void *v) { struct fileassoc_table *tbl = v; u_long i; /* Remove all entries from the table and lists */ for (i = 0; i < tbl->tbl_nslots; i++) { struct fileassoc_file *faf; while ((faf = LIST_FIRST(&tbl->tbl_hash[i])) != NULL) { file_free(faf); } } /* Remove hash table and sysctl node */ hashdone(tbl->tbl_hash, HASH_LIST, tbl->tbl_mask); specificdata_fini(fileassoc_domain, &tbl->tbl_data); kmem_free(tbl, sizeof(*tbl)); } /* * Initialize the fileassoc subsystem. */ static int fileassoc_init(void) { int error; error = mount_specific_key_create(&fileassoc_mountspecific_key, table_dtor); if (error) { return error; } fileassoc_domain = specificdata_domain_create(); return 0; } /* * Register a new assoc. */ int fileassoc_register(const char *name, fileassoc_cleanup_cb_t cleanup_cb, fileassoc_t *result) { int error; specificdata_key_t key; struct fileassoc *assoc; error = RUN_ONCE(&control, fileassoc_init); if (error) { return error; } error = specificdata_key_create(fileassoc_domain, &key, NULL); if (error) { return error; } assoc = kmem_alloc(sizeof(*assoc), KM_SLEEP); assoc->assoc_name = name; assoc->assoc_cleanup_cb = cleanup_cb; assoc->assoc_key = key; LIST_INSERT_HEAD(&fileassoc_list, assoc, assoc_list); *result = assoc; return 0; } /* * Deregister an assoc. */ int fileassoc_deregister(fileassoc_t assoc) { LIST_REMOVE(assoc, assoc_list); specificdata_key_delete(fileassoc_domain, assoc->assoc_key); kmem_free(assoc, sizeof(*assoc)); return 0; } /* * Get the hash table for the specified device. */ static struct fileassoc_table * fileassoc_table_lookup(struct mount *mp) { int error; error = RUN_ONCE(&control, fileassoc_init); if (error) { return NULL; } return mount_getspecific(mp, fileassoc_mountspecific_key); } /* * Perform a lookup on a hash table. If hint is non-zero then use the value * of the hint as the identifier instead of performing a lookup for the * fileid. */ static struct fileassoc_file * fileassoc_file_lookup(struct vnode *vp, fhandle_t *hint) { struct fileassoc_table *tbl; struct fileassoc_hash_entry *hash_entry; struct fileassoc_file *faf; size_t indx; fhandle_t *th; int error; tbl = fileassoc_table_lookup(vp->v_mount); if (tbl == NULL) { return NULL; } if (hint == NULL) { error = vfs_composefh_alloc(vp, &th); if (error) return (NULL); } else { th = hint; } indx = FILEASSOC_HASH(tbl, th); hash_entry = &(tbl->tbl_hash[indx]); LIST_FOREACH(faf, hash_entry, faf_list) { if (((FHANDLE_FILEID(faf->faf_handle)->fid_len == FHANDLE_FILEID(th)->fid_len)) && (memcmp(FHANDLE_FILEID(faf->faf_handle), FHANDLE_FILEID(th), (FHANDLE_FILEID(th))->fid_len) == 0)) { break; } } if (hint == NULL) vfs_composefh_free(th); return faf; } /* * Return assoc data associated with a vnode. */ void * fileassoc_lookup(struct vnode *vp, fileassoc_t assoc) { struct fileassoc_file *faf; faf = fileassoc_file_lookup(vp, NULL); if (faf == NULL) return (NULL); return file_getdata(faf, assoc); } static struct fileassoc_table * fileassoc_table_resize(struct fileassoc_table *tbl) { struct fileassoc_table *newtbl; u_long i; /* * Allocate a new table. Like the condition in fileassoc_file_add(), * this is also temporary -- just double the number of slots. */ newtbl = kmem_zalloc(sizeof(*newtbl), KM_SLEEP); newtbl->tbl_nslots = (tbl->tbl_nslots * 2); if (newtbl->tbl_nslots < tbl->tbl_nslots) newtbl->tbl_nslots = tbl->tbl_nslots; newtbl->tbl_hash = hashinit(newtbl->tbl_nslots, HASH_LIST, true, &newtbl->tbl_mask); newtbl->tbl_nused = 0; specificdata_init(fileassoc_domain, &newtbl->tbl_data); /* XXX we need to make sure nothing uses fileassoc here! */ for (i = 0; i < tbl->tbl_nslots; i++) { struct fileassoc_file *faf; while ((faf = LIST_FIRST(&tbl->tbl_hash[i])) != NULL) { struct fileassoc_hash_entry *hash_entry; size_t indx; LIST_REMOVE(faf, faf_list); indx = FILEASSOC_HASH(newtbl, faf->faf_handle); hash_entry = &(newtbl->tbl_hash[indx]); LIST_INSERT_HEAD(hash_entry, faf, faf_list); newtbl->tbl_nused++; } } if (tbl->tbl_nused != newtbl->tbl_nused) panic("fileassoc_table_resize: inconsistency detected! " "needed %zu entries, got %zu", tbl->tbl_nused, newtbl->tbl_nused); hashdone(tbl->tbl_hash, HASH_LIST, tbl->tbl_mask); specificdata_fini(fileassoc_domain, &tbl->tbl_data); kmem_free(tbl, sizeof(*tbl)); return (newtbl); } /* * Create a new fileassoc table. */ static struct fileassoc_table * fileassoc_table_add(struct mount *mp) { struct fileassoc_table *tbl; /* Check for existing table for device. */ tbl = fileassoc_table_lookup(mp); if (tbl != NULL) return (tbl); /* Allocate and initialize a table. */ tbl = kmem_zalloc(sizeof(*tbl), KM_SLEEP); tbl->tbl_nslots = FILEASSOC_INITIAL_TABLESIZE; tbl->tbl_hash = hashinit(tbl->tbl_nslots, HASH_LIST, true, &tbl->tbl_mask); tbl->tbl_nused = 0; specificdata_init(fileassoc_domain, &tbl->tbl_data); mount_setspecific(mp, fileassoc_mountspecific_key, tbl); return (tbl); } /* * Delete a table. */ int fileassoc_table_delete(struct mount *mp) { struct fileassoc_table *tbl; tbl = fileassoc_table_lookup(mp); if (tbl == NULL) return (EEXIST); mount_setspecific(mp, fileassoc_mountspecific_key, NULL); table_dtor(tbl); return (0); } /* * Run a callback for each assoc in a table. */ int fileassoc_table_run(struct mount *mp, fileassoc_t assoc, fileassoc_cb_t cb, void *cookie) { struct fileassoc_table *tbl; u_long i; tbl = fileassoc_table_lookup(mp); if (tbl == NULL) return (EEXIST); for (i = 0; i < tbl->tbl_nslots; i++) { struct fileassoc_file *faf; LIST_FOREACH(faf, &tbl->tbl_hash[i], faf_list) { void *data; data = file_getdata(faf, assoc); if (data != NULL) cb(data, cookie); } } return (0); } /* * Clear a table for a given assoc. */ int fileassoc_table_clear(struct mount *mp, fileassoc_t assoc) { struct fileassoc_table *tbl; u_long i; tbl = fileassoc_table_lookup(mp); if (tbl == NULL) return (EEXIST); for (i = 0; i < tbl->tbl_nslots; i++) { struct fileassoc_file *faf; LIST_FOREACH(faf, &tbl->tbl_hash[i], faf_list) { file_cleanup(faf, assoc); file_setdata(faf, assoc, NULL); } } return (0); } /* * Add a file entry to a table. */ static struct fileassoc_file * fileassoc_file_add(struct vnode *vp, fhandle_t *hint) { struct fileassoc_table *tbl; struct fileassoc_hash_entry *hash_entry; struct fileassoc_file *faf; size_t indx; fhandle_t *th; int error; if (hint == NULL) { error = vfs_composefh_alloc(vp, &th); if (error) return (NULL); } else th = hint; faf = fileassoc_file_lookup(vp, th); if (faf != NULL) { if (hint == NULL) vfs_composefh_free(th); return (faf); } tbl = fileassoc_table_lookup(vp->v_mount); if (tbl == NULL) { tbl = fileassoc_table_add(vp->v_mount); } indx = FILEASSOC_HASH(tbl, th); hash_entry = &(tbl->tbl_hash[indx]); faf = kmem_zalloc(sizeof(*faf), KM_SLEEP); faf->faf_handle = th; specificdata_init(fileassoc_domain, &faf->faf_data); LIST_INSERT_HEAD(hash_entry, faf, faf_list); /* * This decides when we need to resize the table. For now, * resize it whenever we "filled" up the number of slots it * has. That's not really true unless of course we had zero * collisions. Think positive! :) */ if (++(tbl->tbl_nused) == tbl->tbl_nslots) { struct fileassoc_table *newtbl; newtbl = fileassoc_table_resize(tbl); mount_setspecific(vp->v_mount, fileassoc_mountspecific_key, newtbl); } return (faf); } /* * Delete a file entry from a table. */ int fileassoc_file_delete(struct vnode *vp) { struct fileassoc_table *tbl; struct fileassoc_file *faf; /* Pre-check if fileassoc is used. XXX */ if (!fileassoc_domain) { return ENOENT; } KERNEL_LOCK(1, NULL); faf = fileassoc_file_lookup(vp, NULL); if (faf == NULL) { KERNEL_UNLOCK_ONE(NULL); return (ENOENT); } file_free(faf); tbl = fileassoc_table_lookup(vp->v_mount); KASSERT(tbl != NULL); --(tbl->tbl_nused); /* XXX gc? */ KERNEL_UNLOCK_ONE(NULL); return (0); } /* * Add an assoc to a vnode. */ int fileassoc_add(struct vnode *vp, fileassoc_t assoc, void *data) { struct fileassoc_file *faf; void *olddata; faf = fileassoc_file_lookup(vp, NULL); if (faf == NULL) { faf = fileassoc_file_add(vp, NULL); if (faf == NULL) return (ENOTDIR); } olddata = file_getdata(faf, assoc); if (olddata != NULL) return (EEXIST); file_setdata(faf, assoc, data); faf->faf_nassocs++; return (0); } /* * Clear an assoc from a vnode. */ int fileassoc_clear(struct vnode *vp, fileassoc_t assoc) { struct fileassoc_file *faf; faf = fileassoc_file_lookup(vp, NULL); if (faf == NULL) return (ENOENT); file_cleanup(faf, assoc); file_setdata(faf, assoc, NULL); --(faf->faf_nassocs); /* XXX gc? */ return (0); } |
| 32 69 3 66 98 98 98 2 96 98 98 4 94 6 92 95 4 2 96 98 85 85 85 85 85 85 85 85 83 2 82 3 82 85 84 1 85 85 85 85 85 85 84 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 197 198 199 200 201 202 203 204 205 206 207 208 209 210 211 212 213 214 215 216 217 218 219 220 221 222 223 224 225 226 227 228 229 230 231 232 233 234 235 236 237 238 239 240 241 242 243 244 245 246 247 248 249 250 251 252 253 254 255 256 257 258 259 260 261 262 263 264 265 266 267 268 269 270 271 272 273 274 275 276 277 278 279 280 281 282 283 284 285 286 287 288 289 290 291 292 293 294 295 296 297 298 299 300 301 302 303 304 305 306 307 308 309 310 311 312 313 314 315 316 317 318 319 320 321 322 323 324 325 326 327 328 329 330 331 332 333 334 335 336 337 338 339 340 341 342 343 344 345 346 347 348 349 350 351 352 353 354 355 356 357 358 359 360 361 362 363 364 365 366 367 368 369 370 371 372 373 374 375 376 377 378 379 380 381 382 383 384 385 386 387 388 389 390 391 392 393 394 395 396 397 398 399 400 401 402 403 404 405 406 407 408 409 410 411 412 413 414 415 416 417 418 419 420 421 422 423 424 425 426 427 428 429 430 431 432 433 434 435 436 437 438 439 440 441 442 443 444 445 446 447 448 449 450 451 452 453 454 455 456 457 458 459 460 461 462 463 464 465 466 467 468 469 470 471 472 473 474 475 476 477 478 479 480 481 482 483 484 485 486 487 488 489 490 491 492 493 494 495 496 497 498 499 500 501 502 503 504 505 506 507 508 509 510 511 512 513 514 515 516 517 518 519 520 521 522 523 524 525 526 527 528 529 530 531 532 533 534 535 536 537 538 539 540 541 542 543 544 545 546 547 548 549 550 551 552 553 554 555 556 557 558 559 560 561 562 563 564 565 566 567 568 569 570 571 572 573 574 575 576 577 578 579 580 581 582 583 584 585 586 587 588 589 590 591 592 593 594 595 596 597 598 599 600 601 602 603 604 605 606 607 608 609 610 611 612 613 614 615 616 617 618 619 620 621 622 623 624 625 626 627 628 629 630 631 632 633 634 635 636 637 638 639 640 641 642 643 644 645 646 647 648 649 650 651 652 653 654 655 656 657 658 659 | /* $NetBSD: kern_fork.c,v 1.229 2022/07/01 09:54:36 prlw1 Exp $ */ /*- * Copyright (c) 1999, 2001, 2004, 2006, 2007, 2008, 2019 * The NetBSD Foundation, Inc. * All rights reserved. * * This code is derived from software contributed to The NetBSD Foundation * by Jason R. Thorpe of the Numerical Aerospace Simulation Facility, * NASA Ames Research Center, by Charles M. Hannum, and by Andrew Doran. * * Redistribution and use in source and binary forms, with or without * modification, are permitted provided that the following conditions * are met: * 1. Redistributions of source code must retain the above copyright * notice, this list of conditions and the following disclaimer. * 2. Redistributions in binary form must reproduce the above copyright * notice, this list of conditions and the following disclaimer in the * documentation and/or other materials provided with the distribution. * * THIS SOFTWARE IS PROVIDED BY THE NETBSD FOUNDATION, INC. AND CONTRIBUTORS * ``AS IS'' AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED * TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR * PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE FOUNDATION OR CONTRIBUTORS * BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR * CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF * SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS * INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN * CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) * ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE * POSSIBILITY OF SUCH DAMAGE. */ /* * Copyright (c) 1982, 1986, 1989, 1991, 1993 * The Regents of the University of California. All rights reserved. * (c) UNIX System Laboratories, Inc. * All or some portions of this file are derived from material licensed * to the University of California by American Telephone and Telegraph * Co. or Unix System Laboratories, Inc. and are reproduced herein with * the permission of UNIX System Laboratories, Inc. * * Redistribution and use in source and binary forms, with or without * modification, are permitted provided that the following conditions * are met: * 1. Redistributions of source code must retain the above copyright * notice, this list of conditions and the following disclaimer. * 2. Redistributions in binary form must reproduce the above copyright * notice, this list of conditions and the following disclaimer in the * documentation and/or other materials provided with the distribution. * 3. Neither the name of the University nor the names of its contributors * may be used to endorse or promote products derived from this software * without specific prior written permission. * * THIS SOFTWARE IS PROVIDED BY THE REGENTS AND CONTRIBUTORS ``AS IS'' AND * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE * ARE DISCLAIMED. IN NO EVENT SHALL THE REGENTS OR CONTRIBUTORS BE LIABLE * FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL * DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS * OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) * HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT * LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY * OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF * SUCH DAMAGE. * * @(#)kern_fork.c 8.8 (Berkeley) 2/14/95 */ #include <sys/cdefs.h> __KERNEL_RCSID(0, "$NetBSD: kern_fork.c,v 1.229 2022/07/01 09:54:36 prlw1 Exp $"); #include "opt_ktrace.h" #include "opt_dtrace.h" #include <sys/param.h> #include <sys/systm.h> #include <sys/filedesc.h> #include <sys/kernel.h> #include <sys/pool.h> #include <sys/mount.h> #include <sys/proc.h> #include <sys/ras.h> #include <sys/resourcevar.h> #include <sys/vnode.h> #include <sys/file.h> #include <sys/acct.h> #include <sys/ktrace.h> #include <sys/sched.h> #include <sys/signalvar.h> #include <sys/syscall.h> #include <sys/kauth.h> #include <sys/atomic.h> #include <sys/syscallargs.h> #include <sys/uidinfo.h> #include <sys/sdt.h> #include <sys/ptrace.h> /* * DTrace SDT provider definitions */ SDT_PROVIDER_DECLARE(proc); SDT_PROBE_DEFINE3(proc, kernel, , create, "struct proc *", /* new process */ "struct proc *", /* parent process */ "int" /* flags */); u_int nprocs __cacheline_aligned = 1; /* process 0 */ /* * Number of ticks to sleep if fork() would fail due to process hitting * limits. Exported in miliseconds to userland via sysctl. */ int forkfsleep = 0; int sys_fork(struct lwp *l, const void *v, register_t *retval) { return fork1(l, 0, SIGCHLD, NULL, 0, NULL, NULL, retval); } /* * vfork(2) system call compatible with 4.4BSD (i.e. BSD with Mach VM). * Address space is not shared, but parent is blocked until child exit. */ int sys_vfork(struct lwp *l, const void *v, register_t *retval) { return fork1(l, FORK_PPWAIT, SIGCHLD, NULL, 0, NULL, NULL, retval); } /* * New vfork(2) system call for NetBSD, which implements original 3BSD vfork(2) * semantics. Address space is shared, and parent is blocked until child exit. */ int sys___vfork14(struct lwp *l, const void *v, register_t *retval) { return fork1(l, FORK_PPWAIT|FORK_SHAREVM, SIGCHLD, NULL, 0, NULL, NULL, retval); } /* * Linux-compatible __clone(2) system call. */ int sys___clone(struct lwp *l, const struct sys___clone_args *uap, register_t *retval) { /* { syscallarg(int) flags; syscallarg(void *) stack; } */ int flags, sig; /* * We don't support the CLONE_PTRACE flag. */ if (SCARG(uap, flags) & (CLONE_PTRACE)) return EINVAL; /* * Linux enforces CLONE_VM with CLONE_SIGHAND, do same. */ if (SCARG(uap, flags) & CLONE_SIGHAND && (SCARG(uap, flags) & CLONE_VM) == 0) return EINVAL; flags = 0; if (SCARG(uap, flags) & CLONE_VM) flags |= FORK_SHAREVM; if (SCARG(uap, flags) & CLONE_FS) flags |= FORK_SHARECWD; if (SCARG(uap, flags) & CLONE_FILES) flags |= FORK_SHAREFILES; if (SCARG(uap, flags) & CLONE_SIGHAND) flags |= FORK_SHARESIGS; if (SCARG(uap, flags) & CLONE_VFORK) flags |= FORK_PPWAIT; sig = SCARG(uap, flags) & CLONE_CSIGNAL; if (sig < 0 || sig >= _NSIG) return EINVAL; /* * Note that the Linux API does not provide a portable way of * specifying the stack area; the caller must know if the stack * grows up or down. So, we pass a stack size of 0, so that the * code that makes this adjustment is a noop. */ return fork1(l, flags, sig, SCARG(uap, stack), 0, NULL, NULL, retval); } /* * Print the 'table full' message once per 10 seconds. */ static struct timeval fork_tfmrate = { 10, 0 }; /* * Check if a process is traced and shall inform about FORK events. */ static inline bool tracefork(struct proc *p, int flags) { return (p->p_slflag & (PSL_TRACEFORK|PSL_TRACED)) == (PSL_TRACEFORK|PSL_TRACED) && (flags & FORK_PPWAIT) == 0; } /* * Check if a process is traced and shall inform about VFORK events. */ static inline bool tracevfork(struct proc *p, int flags) { return (p->p_slflag & (PSL_TRACEVFORK|PSL_TRACED)) == (PSL_TRACEVFORK|PSL_TRACED) && (flags & FORK_PPWAIT) != 0; } /* * Check if a process is traced and shall inform about VFORK_DONE events. */ static inline bool tracevforkdone(struct proc *p, int flags) { return (p->p_slflag & (PSL_TRACEVFORK_DONE|PSL_TRACED)) == (PSL_TRACEVFORK_DONE|PSL_TRACED) && (flags & FORK_PPWAIT); } /* * General fork call. Note that another LWP in the process may call exec() * or exit() while we are forking. It's safe to continue here, because * neither operation will complete until all LWPs have exited the process. */ int fork1(struct lwp *l1, int flags, int exitsig, void *stack, size_t stacksize, void (*func)(void *), void *arg, register_t *retval) { struct proc *p1, *p2, *parent; struct plimit *p1_lim; uid_t uid; struct lwp *l2; int count; vaddr_t uaddr; int tnprocs; int error = 0; p1 = l1->l_proc; uid = kauth_cred_getuid(l1->l_cred); tnprocs = atomic_inc_uint_nv(&nprocs); /* * Although process entries are dynamically created, we still keep * a global limit on the maximum number we will create. */ if (__predict_false(tnprocs >= maxproc)) error = -1; else error = kauth_authorize_process(l1->l_cred, KAUTH_PROCESS_FORK, p1, KAUTH_ARG(tnprocs), NULL, NULL); if (error) { static struct timeval lasttfm; atomic_dec_uint(&nprocs); if (ratecheck(&lasttfm, &fork_tfmrate)) tablefull("proc", "increase kern.maxproc or NPROC"); if (forkfsleep) kpause("forkmx", false, forkfsleep, NULL); return EAGAIN; } /* * Enforce limits. */ count = chgproccnt(uid, 1); if (__predict_false(count > p1->p_rlimit[RLIMIT_NPROC].rlim_cur)) { if (kauth_authorize_process(l1->l_cred, KAUTH_PROCESS_RLIMIT, p1, KAUTH_ARG(KAUTH_REQ_PROCESS_RLIMIT_BYPASS), &p1->p_rlimit[RLIMIT_NPROC], KAUTH_ARG(RLIMIT_NPROC)) != 0) { (void)chgproccnt(uid, -1); atomic_dec_uint(&nprocs); if (forkfsleep) kpause("forkulim", false, forkfsleep, NULL); return EAGAIN; } } /* * Allocate virtual address space for the U-area now, while it * is still easy to abort the fork operation if we're out of * kernel virtual address space. */ uaddr = uvm_uarea_alloc(); if (__predict_false(uaddr == 0)) { (void)chgproccnt(uid, -1); atomic_dec_uint(&nprocs); return ENOMEM; } /* Allocate new proc. */ p2 = proc_alloc(); if (p2 == NULL) { /* We were unable to allocate a process ID. */ uvm_uarea_free(uaddr); mutex_enter(p1->p_lock); uid = kauth_cred_getuid(p1->p_cred); (void)chgproccnt(uid, -1); mutex_exit(p1->p_lock); atomic_dec_uint(&nprocs); return EAGAIN; } /* * We are now committed to the fork. From here on, we may * block on resources, but resource allocation may NOT fail. */ /* * Make a proc table entry for the new process. * Start by zeroing the section of proc that is zero-initialized, * then copy the section that is copied directly from the parent. */ memset(&p2->p_startzero, 0, (unsigned) ((char *)&p2->p_endzero - (char *)&p2->p_startzero)); memcpy(&p2->p_startcopy, &p1->p_startcopy, (unsigned) ((char *)&p2->p_endcopy - (char *)&p2->p_startcopy)); TAILQ_INIT(&p2->p_sigpend.sp_info); LIST_INIT(&p2->p_lwps); LIST_INIT(&p2->p_sigwaiters); /* * Duplicate sub-structures as needed. * Increase reference counts on shared objects. * Inherit flags we want to keep. The flags related to SIGCHLD * handling are important in order to keep a consistent behaviour * for the child after the fork. If we are a 32-bit process, the * child will be too. */ p2->p_flag = p1->p_flag & (PK_SUGID | PK_NOCLDWAIT | PK_CLDSIGIGN | PK_32); p2->p_emul = p1->p_emul; p2->p_execsw = p1->p_execsw; if (flags & FORK_SYSTEM) { /* * Mark it as a system process. Set P_NOCLDWAIT so that * children are reparented to init(8) when they exit. * init(8) can easily wait them out for us. */ p2->p_flag |= (PK_SYSTEM | PK_NOCLDWAIT); } mutex_init(&p2->p_stmutex, MUTEX_DEFAULT, IPL_HIGH); mutex_init(&p2->p_auxlock, MUTEX_DEFAULT, IPL_NONE); rw_init(&p2->p_reflock); cv_init(&p2->p_waitcv, "wait"); cv_init(&p2->p_lwpcv, "lwpwait"); /* * Share a lock between the processes if they are to share signal * state: we must synchronize access to it. */ if (flags & FORK_SHARESIGS) { p2->p_lock = p1->p_lock; mutex_obj_hold(p1->p_lock); } else p2->p_lock = mutex_obj_alloc(MUTEX_DEFAULT, IPL_NONE); kauth_proc_fork(p1, p2); p2->p_raslist = NULL; #if defined(__HAVE_RAS) ras_fork(p1, p2); #endif /* bump references to the text vnode (for procfs) */ p2->p_textvp = p1->p_textvp; if (p2->p_textvp) vref(p2->p_textvp); if (p1->p_path) p2->p_path = kmem_strdupsize(p1->p_path, NULL, KM_SLEEP); else p2->p_path = NULL; if (flags & FORK_SHAREFILES) fd_share(p2); else if (flags & FORK_CLEANFILES) p2->p_fd = fd_init(NULL); else p2->p_fd = fd_copy(); /* XXX racy */ p2->p_mqueue_cnt = p1->p_mqueue_cnt; if (flags & FORK_SHARECWD) cwdshare(p2); else p2->p_cwdi = cwdinit(); /* * Note: p_limit (rlimit stuff) is copy-on-write, so normally * we just need increase pl_refcnt. */ p1_lim = p1->p_limit; if (!p1_lim->pl_writeable) { lim_addref(p1_lim); p2->p_limit = p1_lim; } else { p2->p_limit = lim_copy(p1_lim); } if (flags & FORK_PPWAIT) { /* Mark ourselves as waiting for a child. */ p2->p_lflag = PL_PPWAIT; l1->l_vforkwaiting = true; p2->p_vforklwp = l1; } else { p2->p_lflag = 0; l1->l_vforkwaiting = false; } p2->p_sflag = 0; p2->p_slflag = 0; parent = (flags & FORK_NOWAIT) ? initproc : p1; p2->p_pptr = parent; p2->p_ppid = parent->p_pid; LIST_INIT(&p2->p_children); p2->p_aio = NULL; #ifdef KTRACE /* * Copy traceflag and tracefile if enabled. * If not inherited, these were zeroed above. */ if (p1->p_traceflag & KTRFAC_INHERIT) { mutex_enter(&ktrace_lock); p2->p_traceflag = p1->p_traceflag; if ((p2->p_tracep = p1->p_tracep) != NULL) ktradref(p2); mutex_exit(&ktrace_lock); } #endif /* * Create signal actions for the child process. */ p2->p_sigacts = sigactsinit(p1, flags & FORK_SHARESIGS); mutex_enter(p1->p_lock); p2->p_sflag |= (p1->p_sflag & (PS_STOPFORK | PS_STOPEXEC | PS_NOCLDSTOP)); sched_proc_fork(p1, p2); mutex_exit(p1->p_lock); p2->p_stflag = p1->p_stflag; /* * p_stats. * Copy parts of p_stats, and zero out the rest. */ p2->p_stats = pstatscopy(p1->p_stats); /* * Set up the new process address space. */ uvm_proc_fork(p1, p2, (flags & FORK_SHAREVM) ? true : false); /* * Finish creating the child process. * It will return through a different path later. */ lwp_create(l1, p2, uaddr, (flags & FORK_PPWAIT) ? LWP_VFORK : 0, stack, stacksize, (func != NULL) ? func : child_return, arg, &l2, l1->l_class, &l1->l_sigmask, &l1->l_sigstk); /* * Inherit l_private from the parent. * Note that we cannot use lwp_setprivate() here since that * also sets the CPU TLS register, which is incorrect if the * process has changed that without letting the kernel know. */ l2->l_private = l1->l_private; /* * If emulation has a process fork hook, call it now. */ if (p2->p_emul->e_proc_fork) (*p2->p_emul->e_proc_fork)(p2, l1, flags); /* * ...and finally, any other random fork hooks that subsystems * might have registered. */ doforkhooks(p2, p1); SDT_PROBE(proc, kernel, , create, p2, p1, flags, 0, 0); /* * It's now safe for the scheduler and other processes to see the * child process. */ mutex_enter(&proc_lock); if (p1->p_session->s_ttyvp != NULL && p1->p_lflag & PL_CONTROLT) p2->p_lflag |= PL_CONTROLT; LIST_INSERT_HEAD(&parent->p_children, p2, p_sibling); p2->p_exitsig = exitsig; /* signal for parent on exit */ /* * Trace fork(2) and vfork(2)-like events on demand in a debugger. */ if (tracefork(p1, flags) || tracevfork(p1, flags)) { proc_changeparent(p2, p1->p_pptr); SET(p2->p_slflag, PSL_TRACEDCHILD); } p2->p_oppid = p1->p_pid; /* Remember the original parent id. */ LIST_INSERT_AFTER(p1, p2, p_pglist); LIST_INSERT_HEAD(&allproc, p2, p_list); p2->p_trace_enabled = trace_is_enabled(p2); #ifdef __HAVE_SYSCALL_INTERN (*p2->p_emul->e_syscall_intern)(p2); #endif /* * Update stats now that we know the fork was successful. */ KPREEMPT_DISABLE(l1); CPU_COUNT(CPU_COUNT_FORKS, 1); if (flags & FORK_PPWAIT) CPU_COUNT(CPU_COUNT_FORKS_PPWAIT, 1); if (flags & FORK_SHAREVM) CPU_COUNT(CPU_COUNT_FORKS_SHAREVM, 1); KPREEMPT_ENABLE(l1); if (ktrpoint(KTR_EMUL)) p2->p_traceflag |= KTRFAC_TRC_EMUL; /* * Notify any interested parties about the new process. */ if (!SLIST_EMPTY(&p1->p_klist)) { mutex_exit(&proc_lock); knote_proc_fork(p1, p2); mutex_enter(&proc_lock); } /* * Make child runnable, set start time, and add to run queue except * if the parent requested the child to start in SSTOP state. */ mutex_enter(p2->p_lock); /* * Start profiling. */ if ((p2->p_stflag & PST_PROFIL) != 0) { mutex_spin_enter(&p2->p_stmutex); startprofclock(p2); mutex_spin_exit(&p2->p_stmutex); } getmicrotime(&p2->p_stats->p_start); p2->p_acflag = AFORK; lwp_lock(l2); KASSERT(p2->p_nrlwps == 1); KASSERT(l2->l_stat == LSIDL); if (p2->p_sflag & PS_STOPFORK) { p2->p_nrlwps = 0; p2->p_stat = SSTOP; p2->p_waited = 0; p1->p_nstopchild++; l2->l_stat = LSSTOP; KASSERT(l2->l_wchan == NULL); lwp_unlock(l2); } else { p2->p_nrlwps = 1; p2->p_stat = SACTIVE; setrunnable(l2); /* LWP now unlocked */ } /* * Return child pid to parent process, * marking us as parent via retval[1]. */ if (retval != NULL) { retval[0] = p2->p_pid; retval[1] = 0; } mutex_exit(p2->p_lock); /* * Let the parent know that we are tracing its child. */ if (tracefork(p1, flags) || tracevfork(p1, flags)) { mutex_enter(p1->p_lock); eventswitch(TRAP_CHLD, tracefork(p1, flags) ? PTRACE_FORK : PTRACE_VFORK, retval[0]); mutex_enter(&proc_lock); } /* * Preserve synchronization semantics of vfork. If waiting for * child to exec or exit, sleep until it clears p_vforkwaiting. */ while (l1->l_vforkwaiting) cv_wait(&l1->l_waitcv, &proc_lock); /* * Let the parent know that we are tracing its child. */ if (tracevforkdone(p1, flags)) { mutex_enter(p1->p_lock); eventswitch(TRAP_CHLD, PTRACE_VFORK_DONE, retval[0]); } else mutex_exit(&proc_lock); return 0; } /* * MI code executed in each newly spawned process before returning to userland. */ void child_return(void *arg) { struct lwp *l = curlwp; struct proc *p = l->l_proc; if ((p->p_slflag & (PSL_TRACED|PSL_TRACEDCHILD)) == (PSL_TRACED|PSL_TRACEDCHILD)) { eventswitchchild(p, TRAP_CHLD, ISSET(p->p_lflag, PL_PPWAIT) ? PTRACE_VFORK : PTRACE_FORK); } md_child_return(l); /* * Return SYS_fork for all fork types, including vfork(2) and clone(2). * * This approach simplifies the code and avoids extra locking. */ ktrsysret(SYS_fork, 0, 0); } |
| 1 1 1 1 1 1 1 1 5 3 3 3 17 17 9 9 27 3 25 25 7 27 4 7 1 6 6 7 7 7 5 1 1 1 1 4 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 197 198 199 200 201 202 203 204 205 206 207 208 209 210 211 212 213 214 215 216 217 218 219 220 221 222 223 224 225 226 227 228 229 230 231 232 233 234 235 236 237 238 239 240 241 242 243 244 245 246 247 248 249 250 251 252 253 254 255 256 257 258 259 260 261 262 263 264 265 266 267 268 269 270 271 272 273 274 275 276 277 278 279 280 281 282 283 284 285 286 287 288 289 290 291 292 293 294 295 296 297 298 299 300 301 302 303 304 305 306 307 308 309 310 311 312 313 314 315 316 317 318 319 320 321 322 323 324 325 326 327 328 329 330 331 332 333 334 335 336 337 338 339 340 341 342 343 344 345 346 347 348 349 350 351 352 353 354 355 356 357 358 359 360 361 362 363 364 365 366 367 368 369 370 371 372 373 374 375 376 377 378 379 380 381 382 383 384 385 386 387 388 389 390 391 392 393 394 395 396 397 398 399 400 401 402 403 404 405 406 407 408 409 410 411 412 413 414 415 416 417 418 419 420 421 422 423 424 425 426 427 428 429 430 431 432 433 434 435 436 437 438 439 440 441 442 443 444 445 446 447 448 449 450 451 452 453 454 455 456 457 458 459 460 461 462 463 464 465 466 467 468 469 470 471 472 473 474 475 476 477 478 479 480 481 482 483 484 485 486 487 488 489 490 491 492 493 494 495 496 497 498 | /* $NetBSD: tty_subr.c,v 1.43 2019/12/27 09:41:51 msaitoh Exp $ */ /* * Copyright (c) 1993, 1994 Theo de Raadt * All rights reserved. * * Per Lindqvist <pgd@compuram.bbt.se> supplied an almost fully working * set of true clist functions that this is very loosely based on. * * Redistribution and use in source and binary forms, with or without * modification, are permitted provided that the following conditions * are met: * 1. Redistributions of source code must retain the above copyright * notice, this list of conditions and the following disclaimer. * 2. Redistributions in binary form must reproduce the above copyright * notice, this list of conditions and the following disclaimer in the * documentation and/or other materials provided with the distribution. * * THIS SOFTWARE IS PROVIDED BY THE AUTHOR ``AS IS'' AND ANY EXPRESS OR * IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES * OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. * IN NO EVENT SHALL THE AUTHOR BE LIABLE FOR ANY DIRECT, INDIRECT, * INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT * NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, * DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY * THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT * (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF * THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. */ #include <sys/cdefs.h> __KERNEL_RCSID(0, "$NetBSD: tty_subr.c,v 1.43 2019/12/27 09:41:51 msaitoh Exp $"); #include <sys/param.h> #include <sys/systm.h> #include <sys/buf.h> #include <sys/ioctl.h> #include <sys/tty.h> #include <sys/kmem.h> /* * At compile time, choose: * There are two ways the TTY_QUOTE bit can be stored. If QBITS is * defined we allocate an array of bits -- 1/8th as much memory but * setbit(), clrbit(), and isset() take more CPU. If QBITS is * undefined, we just use an array of bytes. * * If TTY_QUOTE functionality isn't required by a line discipline, * it can free c_cq and set it to NULL. This speeds things up, * and also does not use any extra memory. This is useful for (say) * a SLIP line discipline that wants a 32K ring buffer for data * but doesn't need quoting. */ #define QBITS #ifdef QBITS #define QMEM(n) ((((n)-1)/NBBY)+1) #else #define QMEM(n) (n) #endif #ifdef QBITS static void clrbits(u_char *, unsigned int, unsigned int); #endif /* * Initialize a particular clist. Ok, they are really ring buffers, * of the specified length, with/without quoting support. */ int clalloc(struct clist *clp, int size, int quot) { clp->c_cs = kmem_zalloc(size, KM_SLEEP); if (quot) clp->c_cq = kmem_zalloc(QMEM(size), KM_SLEEP); else clp->c_cq = NULL; clp->c_cf = clp->c_cl = NULL; clp->c_ce = clp->c_cs + size; clp->c_cn = size; clp->c_cc = 0; return (0); } void clfree(struct clist *clp) { if (clp->c_cs) kmem_free(clp->c_cs, clp->c_cn); if (clp->c_cq) kmem_free(clp->c_cq, QMEM(clp->c_cn)); clp->c_cs = clp->c_cq = NULL; } /* * Get a character from a clist. */ int getc(struct clist *clp) { int c = -1; int s; s = spltty(); if (clp->c_cc == 0) goto out; c = *clp->c_cf & 0xff; if (clp->c_cq) { #ifdef QBITS if (isset(clp->c_cq, clp->c_cf - clp->c_cs) ) c |= TTY_QUOTE; #else if (*(clp->c_cf - clp->c_cs + clp->c_cq)) c |= TTY_QUOTE; #endif } *clp->c_cf = 0; /* wipe out to avoid information disclosure */ if (++clp->c_cf == clp->c_ce) clp->c_cf = clp->c_cs; if (--clp->c_cc == 0) clp->c_cf = clp->c_cl = (u_char *)0; out: splx(s); return c; } /* * Copy clist to buffer. * Return number of bytes moved. */ int q_to_b(struct clist *clp, u_char *cp, int count) { int cc; u_char *p = cp; int s; s = spltty(); /* optimize this while loop */ while (count > 0 && clp->c_cc > 0) { cc = clp->c_cl - clp->c_cf; if (clp->c_cf >= clp->c_cl) cc = clp->c_ce - clp->c_cf; if (cc > count) cc = count; memcpy(p, clp->c_cf, cc); count -= cc; p += cc; clp->c_cc -= cc; clp->c_cf += cc; if (clp->c_cf == clp->c_ce) clp->c_cf = clp->c_cs; } if (clp->c_cc == 0) clp->c_cf = clp->c_cl = (u_char *)0; splx(s); return p - cp; } /* * Return count of contiguous characters in clist. * Stop counting if flag&character is non-null. */ int ndqb(struct clist *clp, int flag) { int count = 0; int i; int cc; int s; s = spltty(); if ((cc = clp->c_cc) == 0) goto out; if (flag == 0) { count = clp->c_cl - clp->c_cf; if (count <= 0) count = clp->c_ce - clp->c_cf; goto out; } i = clp->c_cf - clp->c_cs; if (flag & TTY_QUOTE) { while (cc-- > 0 && !(clp->c_cs[i++] & (flag & ~TTY_QUOTE) || isset(clp->c_cq, i))) { count++; if (i == clp->c_cn) break; } } else { while (cc-- > 0 && !(clp->c_cs[i++] & flag)) { count++; if (i == clp->c_cn) break; } } out: splx(s); return count; } /* * Flush count bytes from clist. */ void ndflush(struct clist *clp, int count) { int cc; int s; s = spltty(); if (count == clp->c_cc) { clp->c_cc = 0; clp->c_cf = clp->c_cl = (u_char *)0; goto out; } /* optimize this while loop */ while (count > 0 && clp->c_cc > 0) { cc = clp->c_cl - clp->c_cf; if (clp->c_cf >= clp->c_cl) cc = clp->c_ce - clp->c_cf; if (cc > count) cc = count; count -= cc; clp->c_cc -= cc; clp->c_cf += cc; if (clp->c_cf == clp->c_ce) clp->c_cf = clp->c_cs; } if (clp->c_cc == 0) clp->c_cf = clp->c_cl = (u_char *)0; out: splx(s); } /* * Put a character into the output queue. */ int putc(int c, struct clist *clp) { int i; int s; s = spltty(); if (clp->c_cc == clp->c_cn) goto out; if (clp->c_cc == 0) { if (!clp->c_cs) { #if defined(DIAGNOSTIC) || 1 printf("putc: required clalloc\n"); #endif if (clalloc(clp, clp->c_cn, 1)) { out: splx(s); return -1; } } clp->c_cf = clp->c_cl = clp->c_cs; } *clp->c_cl = c & 0xff; i = clp->c_cl - clp->c_cs; if (clp->c_cq) { #ifdef QBITS if (c & TTY_QUOTE) setbit(clp->c_cq, i); else clrbit(clp->c_cq, i); #else q = clp->c_cq + i; *q = (c & TTY_QUOTE) ? 1 : 0; #endif } clp->c_cc++; clp->c_cl++; if (clp->c_cl == clp->c_ce) clp->c_cl = clp->c_cs; splx(s); return 0; } #ifdef QBITS /* * optimized version of * * for (i = 0; i < len; i++) * clrbit(cp, off + len); */ static void clrbits(u_char *cp, unsigned int off, unsigned int len) { unsigned int sbi, ebi; u_char *scp, *ecp; unsigned int end; unsigned char mask; scp = cp + off / NBBY; sbi = off % NBBY; end = off + len + NBBY - 1; ecp = cp + end / NBBY - 1; ebi = end % NBBY + 1; if (scp >= ecp) { mask = ((1 << len) - 1) << sbi; *scp &= ~mask; } else { mask = (1 << sbi) - 1; *scp++ &= mask; mask = (1 << ebi) - 1; *ecp &= ~mask; while (scp < ecp) *scp++ = 0x00; } } #endif /* * Copy buffer to clist. * Return number of bytes not transferred. */ int b_to_q(const u_char *cp, int count, struct clist *clp) { int cc; const u_char *p = cp; int s; if (count <= 0) return 0; s = spltty(); if (clp->c_cc == clp->c_cn) goto out; if (clp->c_cc == 0) { if (!clp->c_cs) { #if defined(DIAGNOSTIC) || 1 printf("b_to_q: required clalloc\n"); #endif if (clalloc(clp, clp->c_cn, 1)) goto out; } clp->c_cf = clp->c_cl = clp->c_cs; } /* optimize this while loop */ while (count > 0 && clp->c_cc < clp->c_cn) { cc = clp->c_ce - clp->c_cl; if (clp->c_cf > clp->c_cl) cc = clp->c_cf - clp->c_cl; if (cc > count) cc = count; memcpy(clp->c_cl, p, cc); if (clp->c_cq) { #ifdef QBITS clrbits(clp->c_cq, clp->c_cl - clp->c_cs, cc); #else memset(clp->c_cl - clp->c_cs + clp->c_cq, 0, cc); #endif } p += cc; count -= cc; clp->c_cc += cc; clp->c_cl += cc; if (clp->c_cl == clp->c_ce) clp->c_cl = clp->c_cs; } out: splx(s); return count; } static int tty_global_cc; /* * Given a non-NULL pointer into the clist return the pointer * to the next character in the list or return NULL if no more chars. * * Callers must not allow getc's to happen between firstc's and getc's * so that the pointer becomes invalid. Note that interrupts are NOT * masked. */ u_char * nextc(struct clist *clp, u_char *cp, int *c) { if (clp->c_cf == cp) { /* * First time initialization. */ tty_global_cc = clp->c_cc; } if (tty_global_cc == 0 || cp == NULL) return NULL; if (--tty_global_cc == 0) return NULL; if (++cp == clp->c_ce) cp = clp->c_cs; *c = *cp & 0xff; if (clp->c_cq) { #ifdef QBITS if (isset(clp->c_cq, cp - clp->c_cs)) *c |= TTY_QUOTE; #else if (*(clp->c_cf - clp->c_cs + clp->c_cq)) *c |= TTY_QUOTE; #endif } return cp; } /* * Given a non-NULL pointer into the clist return the pointer * to the first character in the list or return NULL if no more chars. * * Callers must not allow getc's to happen between firstc's and getc's * so that the pointer becomes invalid. Note that interrupts are NOT * masked. * * *c is set to the NEXT character */ u_char * firstc(struct clist *clp, int *c) { u_char *cp; tty_global_cc = clp->c_cc; if (tty_global_cc == 0) return NULL; cp = clp->c_cf; *c = *cp & 0xff; if (clp->c_cq) { #ifdef QBITS if (isset(clp->c_cq, cp - clp->c_cs)) *c |= TTY_QUOTE; #else if (*(cp - clp->c_cs + clp->c_cq)) *c |= TTY_QUOTE; #endif } return clp->c_cf; } /* * Remove the last character in the clist and return it. */ int unputc(struct clist *clp) { unsigned int c = -1; int s; s = spltty(); if (clp->c_cc == 0) goto out; if (clp->c_cl == clp->c_cs) clp->c_cl = clp->c_ce - 1; else --clp->c_cl; clp->c_cc--; c = *clp->c_cl & 0xff; if (clp->c_cq) { #ifdef QBITS if (isset(clp->c_cq, clp->c_cl - clp->c_cs)) c |= TTY_QUOTE; #else if (*(clp->c_cf - clp->c_cs + clp->c_cq)) c |= TTY_QUOTE; #endif } if (clp->c_cc == 0) clp->c_cf = clp->c_cl = (u_char *)0; out: splx(s); return c; } /* * Put the chars in the from queue on the end of the to queue. */ void catq(struct clist *from, struct clist *to) { int c; while ((c = getc(from)) != -1) putc(c, to); } |
| 8 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 197 198 199 200 201 202 203 204 205 206 207 208 209 210 211 212 213 214 215 216 217 218 219 220 221 222 223 224 225 226 227 228 229 230 231 232 233 234 235 236 237 238 239 240 241 242 243 244 245 246 247 248 249 250 251 252 253 254 255 256 257 258 259 260 261 262 263 264 265 266 267 268 269 270 271 272 273 274 275 276 277 278 279 280 281 282 283 284 285 286 287 288 289 290 291 292 293 294 295 296 297 298 299 300 301 302 303 304 305 306 307 308 309 310 311 312 313 314 315 316 317 318 319 320 321 322 323 324 325 326 327 328 329 330 331 332 333 334 335 336 337 338 339 340 341 342 343 344 345 346 347 348 349 350 351 352 353 354 355 356 357 358 359 360 361 362 363 364 365 366 367 368 369 370 371 372 373 374 375 376 377 378 379 380 381 382 383 384 385 386 387 388 389 390 391 392 393 394 395 396 397 398 399 400 401 402 403 404 405 406 407 408 409 410 411 412 413 414 415 416 417 418 419 420 421 422 423 424 425 426 427 428 429 430 431 432 433 434 435 436 437 438 439 440 441 442 443 444 445 446 447 448 449 450 451 452 453 454 455 456 457 458 459 460 461 462 463 464 465 466 467 468 469 470 471 472 473 474 475 476 477 478 479 480 481 482 483 484 485 486 487 488 489 490 491 492 493 494 495 496 497 498 499 500 501 502 503 504 505 506 507 508 509 510 511 512 513 514 515 516 517 518 519 520 521 522 523 524 525 526 527 528 529 530 531 532 533 534 535 536 537 538 539 540 541 542 543 544 545 546 547 548 549 550 551 552 553 554 555 556 557 558 559 560 561 562 563 564 565 566 567 568 569 570 571 572 573 574 575 576 577 578 579 580 581 582 583 584 585 586 587 588 589 590 591 592 593 594 595 596 597 598 599 600 601 602 603 604 605 606 607 608 609 610 611 612 613 614 615 616 617 618 619 620 621 622 623 624 625 626 627 628 629 630 631 632 633 634 635 636 637 638 639 640 641 642 643 644 645 646 647 648 649 650 651 652 653 654 655 656 657 658 659 660 661 662 663 664 665 666 667 668 669 670 671 672 673 674 675 676 677 678 679 680 681 682 683 684 685 686 687 688 689 690 691 692 693 694 695 696 697 698 699 700 701 702 703 704 705 706 707 708 709 710 711 712 713 714 715 716 717 718 719 720 721 722 723 724 725 726 727 728 729 730 731 732 733 734 735 736 737 738 739 740 741 742 743 744 745 746 747 748 749 750 751 752 753 754 755 756 757 758 759 760 761 762 763 764 765 766 767 768 769 770 771 772 773 774 775 776 777 778 779 780 781 782 783 784 785 786 787 788 789 790 791 792 793 794 795 796 797 798 799 800 801 802 803 804 805 806 807 808 809 810 811 812 813 814 815 816 817 818 819 820 821 822 823 824 825 826 827 828 829 830 831 832 833 834 835 836 837 838 839 840 841 842 843 844 845 846 847 848 849 850 851 852 853 854 855 856 857 858 859 860 861 862 863 864 865 866 867 868 869 870 871 872 873 874 875 876 877 878 879 880 881 882 883 884 885 886 887 888 889 890 891 892 893 894 895 896 897 898 899 900 901 902 903 904 905 906 907 908 909 910 911 912 913 914 915 916 917 918 919 920 921 922 923 924 925 926 927 928 929 930 931 932 933 934 935 936 937 938 939 940 941 942 943 944 945 946 947 948 949 950 951 952 953 954 955 956 957 958 959 960 961 962 963 964 965 966 967 968 969 970 971 972 973 974 975 976 977 978 979 980 981 982 983 984 985 986 987 988 989 990 991 992 993 994 995 996 997 998 999 1000 1001 1002 1003 1004 1005 1006 1007 1008 1009 1010 1011 1012 1013 1014 1015 1016 1017 1018 1019 1020 1021 1022 1023 1024 1025 1026 1027 1028 1029 1030 1031 1032 1033 1034 1035 1036 1037 1038 1039 1040 1041 1042 1043 1044 1045 1046 1047 1048 1049 1050 1051 1052 1053 1054 1055 1056 1057 1058 1059 1060 1061 1062 1063 1064 1065 1066 1067 1068 1069 1070 1071 1072 1073 1074 1075 1076 1077 1078 1079 1080 1081 1082 1083 1084 1085 1086 1087 1088 1089 1090 1091 1092 1093 1094 1095 1096 1097 1098 1099 1100 1101 1102 1103 1104 1105 1106 1107 1108 1109 1110 1111 1112 1113 1114 1115 1116 1117 1118 1119 1120 1121 1122 1123 1124 1125 1126 1127 1128 1129 1130 1131 1132 1133 1134 1135 1136 1137 1138 1139 1140 1141 1142 1143 1144 1145 1146 1147 1148 1149 1150 1151 1152 1153 1154 1155 1156 1157 1158 1159 1160 1161 1162 1163 1164 1165 1166 1167 1168 1169 1170 1171 1172 1173 1174 1175 1176 1177 1178 1179 1180 1181 1182 1183 1184 1185 1186 1187 1188 1189 1190 1191 1192 1193 1194 1195 1196 1197 1198 1199 1200 1201 1202 1203 1204 1205 1206 1207 1208 1209 1210 1211 1212 1213 1214 1215 1216 1217 1218 1219 1220 1221 1222 1223 1224 1225 1226 1227 1228 1229 1230 1231 1232 1233 1234 1235 1236 1237 1238 1239 1240 1241 1242 1243 1244 1245 1246 1247 1248 1249 1250 1251 1252 1253 1254 1255 1256 1257 1258 1259 1260 1261 1262 1263 1264 1265 1266 1267 1268 1269 1270 1271 1272 1273 1274 1275 1276 1277 1278 1279 1280 1281 1282 1283 1284 1285 1286 1287 1288 1289 1290 1291 1292 1293 1294 1295 1296 1297 1298 1299 1300 1301 1302 1303 1304 | /* $NetBSD: if_axe.c,v 1.151 2022/08/20 14:08:59 riastradh Exp $ */ /* $OpenBSD: if_axe.c,v 1.137 2016/04/13 11:03:37 mpi Exp $ */ /* * Copyright (c) 2005, 2006, 2007 Jonathan Gray <jsg@openbsd.org> * * Permission to use, copy, modify, and distribute this software for any * purpose with or without fee is hereby granted, provided that the above * copyright notice and this permission notice appear in all copies. * * THE SOFTWARE IS PROVIDED "AS IS" AND THE AUTHOR DISCLAIMS ALL WARRANTIES * WITH REGARD TO THIS SOFTWARE INCLUDING ALL IMPLIED WARRANTIES OF * MERCHANTABILITY AND FITNESS. IN NO EVENT SHALL THE AUTHOR BE LIABLE FOR * ANY SPECIAL, DIRECT, INDIRECT, OR CONSEQUENTIAL DAMAGES OR ANY DAMAGES * WHATSOEVER RESULTING FROM LOSS OF USE, DATA OR PROFITS, WHETHER IN AN * ACTION OF CONTRACT, NEGLIGENCE OR OTHER TORTIOUS ACTION, ARISING OUT OF * OR IN CONNECTION WITH THE USE OR PERFORMANCE OF THIS SOFTWARE. */ /* * Copyright (c) 1997, 1998, 1999, 2000-2003 * Bill Paul <wpaul@windriver.com>. All rights reserved. * * Redistribution and use in source and binary forms, with or without * modification, are permitted provided that the following conditions * are met: * 1. Redistributions of source code must retain the above copyright * notice, this list of conditions and the following disclaimer. * 2. Redistributions in binary form must reproduce the above copyright * notice, this list of conditions and the following disclaimer in the * documentation and/or other materials provided with the distribution. * 3. All advertising materials mentioning features or use of this software * must display the following acknowledgement: * This product includes software developed by Bill Paul. * 4. Neither the name of the author nor the names of any co-contributors * may be used to endorse or promote products derived from this software * without specific prior written permission. * * THIS SOFTWARE IS PROVIDED BY Bill Paul AND CONTRIBUTORS ``AS IS'' AND * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE * ARE DISCLAIMED. IN NO EVENT SHALL Bill Paul OR THE VOICES IN HIS HEAD * BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR * CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF * SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS * INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN * CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) * ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF * THE POSSIBILITY OF SUCH DAMAGE. */ /* * ASIX Electronics AX88172/AX88178/AX88778 USB 2.0 ethernet driver. * Used in the LinkSys USB200M and various other adapters. * * Written by Bill Paul <wpaul@windriver.com> * Senior Engineer * Wind River Systems */ /* * The AX88172 provides USB ethernet supports at 10 and 100Mbps. * It uses an external PHY (reference designs use a RealTek chip), * and has a 64-bit multicast hash filter. There is some information * missing from the manual which one needs to know in order to make * the chip function: * * - You must set bit 7 in the RX control register, otherwise the * chip won't receive any packets. * - You must initialize all 3 IPG registers, or you won't be able * to send any packets. * * Note that this device appears to only support loading the station * address via autoload from the EEPROM (i.e. there's no way to manually * set it). * * (Adam Weinberger wanted me to name this driver if_gir.c.) */ /* * Ax88178 and Ax88772 support backported from the OpenBSD driver. * 2007/02/12, J.R. Oldroyd, fbsd@opal.com * * Manual here: * http://www.asix.com.tw/FrootAttach/datasheet/AX88178_datasheet_Rev10.pdf * http://www.asix.com.tw/FrootAttach/datasheet/AX88772_datasheet_Rev10.pdf */ #include <sys/cdefs.h> __KERNEL_RCSID(0, "$NetBSD: if_axe.c,v 1.151 2022/08/20 14:08:59 riastradh Exp $"); #ifdef _KERNEL_OPT #include "opt_usb.h" #include "opt_net_mpsafe.h" #endif #include <sys/param.h> #include <dev/usb/usbnet.h> #include <dev/usb/usbhist.h> #include <dev/usb/if_axereg.h> struct axe_type { struct usb_devno axe_dev; uint16_t axe_flags; }; struct axe_softc { struct usbnet axe_un; /* usbnet:un_flags values */ #define AX178 __BIT(0) /* AX88178 */ #define AX772 __BIT(1) /* AX88772 */ #define AX772A __BIT(2) /* AX88772A */ #define AX772B __BIT(3) /* AX88772B */ #define AXSTD_FRAME __BIT(12) #define AXCSUM_FRAME __BIT(13) uint8_t axe_ipgs[3]; uint8_t axe_phyaddrs[2]; uint16_t sc_pwrcfg; uint16_t sc_lenmask; }; #define AXE_IS_178_FAMILY(un) \ ((un)->un_flags & (AX178 | AX772 | AX772A | AX772B)) #define AXE_IS_772(un) \ ((un)->un_flags & (AX772 | AX772A | AX772B)) #define AXE_IS_172(un) (AXE_IS_178_FAMILY(un) == 0) #define AX_RXCSUM \ (IFCAP_CSUM_IPv4_Rx | \ IFCAP_CSUM_TCPv4_Rx | IFCAP_CSUM_UDPv4_Rx | \ IFCAP_CSUM_TCPv6_Rx | IFCAP_CSUM_UDPv6_Rx) #define AX_TXCSUM \ (IFCAP_CSUM_IPv4_Tx | \ IFCAP_CSUM_TCPv4_Tx | IFCAP_CSUM_UDPv4_Tx | \ IFCAP_CSUM_TCPv6_Tx | IFCAP_CSUM_UDPv6_Tx) /* * AXE_178_MAX_FRAME_BURST * max frame burst size for Ax88178 and Ax88772 * 0 2048 bytes * 1 4096 bytes * 2 8192 bytes * 3 16384 bytes * use the largest your system can handle without USB stalling. * * NB: 88772 parts appear to generate lots of input errors with * a 2K rx buffer and 8K is only slightly faster than 4K on an * EHCI port on a T42 so change at your own risk. */ #define AXE_178_MAX_FRAME_BURST 1 #ifdef USB_DEBUG #ifndef AXE_DEBUG #define axedebug 0 #else static int axedebug = 0; SYSCTL_SETUP(sysctl_hw_axe_setup, "sysctl hw.axe setup") { int err; const struct sysctlnode *rnode; const struct sysctlnode *cnode; err = sysctl_createv(clog, 0, NULL, &rnode, CTLFLAG_PERMANENT, CTLTYPE_NODE, "axe", SYSCTL_DESCR("axe global controls"), NULL, 0, NULL, 0, CTL_HW, CTL_CREATE, CTL_EOL); if (err) goto fail; /* control debugging printfs */ err = sysctl_createv(clog, 0, &rnode, &cnode, CTLFLAG_PERMANENT | CTLFLAG_READWRITE, CTLTYPE_INT, "debug", SYSCTL_DESCR("Enable debugging output"), NULL, 0, &axedebug, sizeof(axedebug), CTL_CREATE, CTL_EOL); if (err) goto fail; return; fail: aprint_error("%s: sysctl_createv failed (err = %d)\n", __func__, err); } #endif /* AXE_DEBUG */ #endif /* USB_DEBUG */ #define DPRINTF(FMT,A,B,C,D) USBHIST_LOGN(axedebug,1,FMT,A,B,C,D) #define DPRINTFN(N,FMT,A,B,C,D) USBHIST_LOGN(axedebug,N,FMT,A,B,C,D) #define AXEHIST_FUNC() USBHIST_FUNC() #define AXEHIST_CALLED(name) USBHIST_CALLED(axedebug) /* * Various supported device vendors/products. */ static const struct axe_type axe_devs[] = { { { USB_VENDOR_ABOCOM, USB_PRODUCT_ABOCOM_UFE2000 }, 0 }, { { USB_VENDOR_ACERCM, USB_PRODUCT_ACERCM_EP1427X2 }, 0 }, { { USB_VENDOR_APPLE, USB_PRODUCT_APPLE_ETHERNET }, AX772 }, { { USB_VENDOR_ASIX, USB_PRODUCT_ASIX_AX88172 }, 0 }, { { USB_VENDOR_ASIX, USB_PRODUCT_ASIX_AX88772 }, AX772 }, { { USB_VENDOR_ASIX, USB_PRODUCT_ASIX_AX88772A }, AX772 }, { { USB_VENDOR_ASIX, USB_PRODUCT_ASIX_AX88772B }, AX772B }, { { USB_VENDOR_ASIX, USB_PRODUCT_ASIX_AX88772B_1 }, AX772B }, { { USB_VENDOR_ASIX, USB_PRODUCT_ASIX_AX88178 }, AX178 }, { { USB_VENDOR_ATEN, USB_PRODUCT_ATEN_UC210T }, 0 }, { { USB_VENDOR_BELKIN, USB_PRODUCT_BELKIN_F5D5055 }, AX178 }, { { USB_VENDOR_BILLIONTON, USB_PRODUCT_BILLIONTON_USB2AR }, 0}, { { USB_VENDOR_CISCOLINKSYS, USB_PRODUCT_CISCOLINKSYS_USB200MV2 }, AX772A }, { { USB_VENDOR_COREGA, USB_PRODUCT_COREGA_FETHER_USB2_TX }, 0 }, { { USB_VENDOR_DLINK, USB_PRODUCT_DLINK_DUBE100 }, 0 }, { { USB_VENDOR_DLINK, USB_PRODUCT_DLINK_DUBE100B1 }, AX772 }, { { USB_VENDOR_DLINK2, USB_PRODUCT_DLINK2_DUBE100B1 }, AX772 }, { { USB_VENDOR_DLINK, USB_PRODUCT_DLINK_DUBE100C1 }, AX772B }, { { USB_VENDOR_GOODWAY, USB_PRODUCT_GOODWAY_GWUSB2E }, 0 }, { { USB_VENDOR_IODATA, USB_PRODUCT_IODATA_ETGUS2 }, AX178 }, { { USB_VENDOR_JVC, USB_PRODUCT_JVC_MP_PRX1 }, 0 }, { { USB_VENDOR_LENOVO, USB_PRODUCT_LENOVO_ETHERNET }, AX772B }, { { USB_VENDOR_LINKSYS, USB_PRODUCT_LINKSYS_HG20F9 }, AX772B }, { { USB_VENDOR_LINKSYS2, USB_PRODUCT_LINKSYS2_USB200M }, 0 }, { { USB_VENDOR_LINKSYS4, USB_PRODUCT_LINKSYS4_USB1000 }, AX178 }, { { USB_VENDOR_LOGITEC, USB_PRODUCT_LOGITEC_LAN_GTJU2 }, AX178 }, { { USB_VENDOR_MELCO, USB_PRODUCT_MELCO_LUAU2GT }, AX178 }, { { USB_VENDOR_MELCO, USB_PRODUCT_MELCO_LUAU2KTX }, 0 }, { { USB_VENDOR_MSI, USB_PRODUCT_MSI_AX88772A }, AX772 }, { { USB_VENDOR_NETGEAR, USB_PRODUCT_NETGEAR_FA120 }, 0 }, { { USB_VENDOR_OQO, USB_PRODUCT_OQO_ETHER01PLUS }, AX772 }, { { USB_VENDOR_PLANEX3, USB_PRODUCT_PLANEX3_GU1000T }, AX178 }, { { USB_VENDOR_SITECOM, USB_PRODUCT_SITECOM_LN029 }, 0 }, { { USB_VENDOR_SITECOMEU, USB_PRODUCT_SITECOMEU_LN028 }, AX178 }, { { USB_VENDOR_SITECOMEU, USB_PRODUCT_SITECOMEU_LN031 }, AX178 }, { { USB_VENDOR_SYSTEMTALKS, USB_PRODUCT_SYSTEMTALKS_SGCX2UL }, 0 }, }; #define axe_lookup(v, p) ((const struct axe_type *)usb_lookup(axe_devs, v, p)) static const struct ax88772b_mfb ax88772b_mfb_table[] = { { 0x8000, 0x8001, 2048 }, { 0x8100, 0x8147, 4096 }, { 0x8200, 0x81EB, 6144 }, { 0x8300, 0x83D7, 8192 }, { 0x8400, 0x851E, 16384 }, { 0x8500, 0x8666, 20480 }, { 0x8600, 0x87AE, 24576 }, { 0x8700, 0x8A3D, 32768 } }; static int axe_match(device_t, cfdata_t, void *); static void axe_attach(device_t, device_t, void *); CFATTACH_DECL_NEW(axe, sizeof(struct axe_softc), axe_match, axe_attach, usbnet_detach, usbnet_activate); static void axe_uno_stop(struct ifnet *, int); static void axe_uno_mcast(struct ifnet *); static int axe_uno_init(struct ifnet *); static int axe_uno_mii_read_reg(struct usbnet *, int, int, uint16_t *); static int axe_uno_mii_write_reg(struct usbnet *, int, int, uint16_t); static void axe_uno_mii_statchg(struct ifnet *); static void axe_uno_rx_loop(struct usbnet *, struct usbnet_chain *, uint32_t); static unsigned axe_uno_tx_prepare(struct usbnet *, struct mbuf *, struct usbnet_chain *); static void axe_ax88178_init(struct axe_softc *); static void axe_ax88772_init(struct axe_softc *); static void axe_ax88772a_init(struct axe_softc *); static void axe_ax88772b_init(struct axe_softc *); static const struct usbnet_ops axe_ops = { .uno_stop = axe_uno_stop, .uno_mcast = axe_uno_mcast, .uno_read_reg = axe_uno_mii_read_reg, .uno_write_reg = axe_uno_mii_write_reg, .uno_statchg = axe_uno_mii_statchg, .uno_tx_prepare = axe_uno_tx_prepare, .uno_rx_loop = axe_uno_rx_loop, .uno_init = axe_uno_init, }; static usbd_status axe_cmd(struct axe_softc *sc, int cmd, int index, int val, void *buf) { AXEHIST_FUNC(); AXEHIST_CALLED(); struct usbnet * const un = &sc->axe_un; usb_device_request_t req; usbd_status err; if (usbnet_isdying(un)) return -1; DPRINTFN(20, "cmd %#jx index %#jx val %#jx", cmd, index, val, 0); if (AXE_CMD_DIR(cmd)) req.bmRequestType = UT_WRITE_VENDOR_DEVICE; else req.bmRequestType = UT_READ_VENDOR_DEVICE; req.bRequest = AXE_CMD_CMD(cmd); USETW(req.wValue, val); USETW(req.wIndex, index); USETW(req.wLength, AXE_CMD_LEN(cmd)); err = usbd_do_request(un->un_udev, &req, buf); if (err) DPRINTF("cmd %jd err %jd", cmd, err, 0, 0); return err; } static int axe_uno_mii_read_reg(struct usbnet *un, int phy, int reg, uint16_t *val) { AXEHIST_FUNC(); AXEHIST_CALLED(); struct axe_softc * const sc = usbnet_softc(un); usbd_status err; uint16_t data; DPRINTFN(30, "phy %#jx reg %#jx\n", phy, reg, 0, 0); if (un->un_phyno != phy) { *val = 0; return EINVAL; } axe_cmd(sc, AXE_CMD_MII_OPMODE_SW, 0, 0, NULL); err = axe_cmd(sc, AXE_CMD_MII_READ_REG, reg, phy, &data); axe_cmd(sc, AXE_CMD_MII_OPMODE_HW, 0, 0, NULL); if (err) { device_printf(un->un_dev, "read PHY failed\n"); *val = 0; return EIO; } *val = le16toh(data); if (AXE_IS_772(un) && reg == MII_BMSR) { /* * BMSR of AX88772 indicates that it supports extended * capability but the extended status register is * reserved for embedded ethernet PHY. So clear the * extended capability bit of BMSR. */ *val &= ~BMSR_EXTCAP; } DPRINTFN(30, "phy %#jx reg %#jx val %#jx", phy, reg, *val, 0); return 0; } static int axe_uno_mii_write_reg(struct usbnet *un, int phy, int reg, uint16_t val) { struct axe_softc * const sc = usbnet_softc(un); usbd_status err; uint16_t aval; if (un->un_phyno != phy) return EINVAL; aval = htole16(val); axe_cmd(sc, AXE_CMD_MII_OPMODE_SW, 0, 0, NULL); err = axe_cmd(sc, AXE_CMD_MII_WRITE_REG, reg, phy, &aval); axe_cmd(sc, AXE_CMD_MII_OPMODE_HW, 0, 0, NULL); if (err) return EIO; return 0; } static void axe_uno_mii_statchg(struct ifnet *ifp) { AXEHIST_FUNC(); AXEHIST_CALLED(); struct usbnet * const un = ifp->if_softc; struct axe_softc * const sc = usbnet_softc(un); struct mii_data *mii = usbnet_mii(un); int val, err; if (usbnet_isdying(un)) return; val = 0; if (AXE_IS_172(un)) { if (mii->mii_media_active & IFM_FDX) val |= AXE_MEDIA_FULL_DUPLEX; } else { if (mii->mii_media_active & IFM_FDX) { val |= AXE_MEDIA_FULL_DUPLEX; if (mii->mii_media_active & IFM_ETH_TXPAUSE) val |= AXE_178_MEDIA_TXFLOW_CONTROL_EN; if (mii->mii_media_active & IFM_ETH_RXPAUSE) val |= AXE_178_MEDIA_RXFLOW_CONTROL_EN; } val |= AXE_178_MEDIA_RX_EN | AXE_178_MEDIA_MAGIC; if (un->un_flags & AX178) val |= AXE_178_MEDIA_ENCK; switch (IFM_SUBTYPE(mii->mii_media_active)) { case IFM_1000_T: val |= AXE_178_MEDIA_GMII | AXE_178_MEDIA_ENCK; usbnet_set_link(un, true); break; case IFM_100_TX: val |= AXE_178_MEDIA_100TX; usbnet_set_link(un, true); break; case IFM_10_T: usbnet_set_link(un, true); break; } } DPRINTF("val=%#jx", val, 0, 0, 0); err = axe_cmd(sc, AXE_CMD_WRITE_MEDIA, 0, val, NULL); if (err) device_printf(un->un_dev, "media change failed\n"); } static void axe_uno_mcast(struct ifnet *ifp) { AXEHIST_FUNC(); AXEHIST_CALLED(); struct usbnet * const un = ifp->if_softc; struct axe_softc * const sc = usbnet_softc(un); struct ethercom *ec = usbnet_ec(un); struct ether_multi *enm; struct ether_multistep step; uint16_t rxmode; uint32_t h = 0; uint8_t mchash[8] = { 0, 0, 0, 0, 0, 0, 0, 0 }; if (usbnet_isdying(un)) return; if (axe_cmd(sc, AXE_CMD_RXCTL_READ, 0, 0, &rxmode)) { device_printf(un->un_dev, "can't read rxmode"); return; } rxmode = le16toh(rxmode); rxmode &= ~(AXE_RXCMD_ALLMULTI | AXE_RXCMD_PROMISC | AXE_RXCMD_MULTICAST); ETHER_LOCK(ec); if (usbnet_ispromisc(un)) { ec->ec_flags |= ETHER_F_ALLMULTI; ETHER_UNLOCK(ec); /* run promisc. mode */ rxmode |= AXE_RXCMD_ALLMULTI; /* ??? */ rxmode |= AXE_RXCMD_PROMISC; goto update; } ec->ec_flags &= ~ETHER_F_ALLMULTI; ETHER_FIRST_MULTI(step, ec, enm); while (enm != NULL) { if (memcmp(enm->enm_addrlo, enm->enm_addrhi, ETHER_ADDR_LEN)) { ec->ec_flags |= ETHER_F_ALLMULTI; ETHER_UNLOCK(ec); /* accept all mcast frames */ rxmode |= AXE_RXCMD_ALLMULTI; goto update; } h = ether_crc32_be(enm->enm_addrlo, ETHER_ADDR_LEN); mchash[h >> 29] |= 1U << ((h >> 26) & 7); ETHER_NEXT_MULTI(step, enm); } ETHER_UNLOCK(ec); if (h != 0) rxmode |= AXE_RXCMD_MULTICAST; /* activate mcast hash filter */ axe_cmd(sc, AXE_CMD_WRITE_MCAST, 0, 0, mchash); update: axe_cmd(sc, AXE_CMD_RXCTL_WRITE, 0, rxmode, NULL); } static void axe_ax_init(struct usbnet *un) { struct axe_softc * const sc = usbnet_softc(un); int cmd = AXE_178_CMD_READ_NODEID; if (un->un_flags & AX178) { axe_ax88178_init(sc); } else if (un->un_flags & AX772) { axe_ax88772_init(sc); } else if (un->un_flags & AX772A) { axe_ax88772a_init(sc); } else if (un->un_flags & AX772B) { axe_ax88772b_init(sc); return; } else { cmd = AXE_172_CMD_READ_NODEID; } if (axe_cmd(sc, cmd, 0, 0, un->un_eaddr)) { aprint_error_dev(un->un_dev, "failed to read ethernet address\n"); } } static void axe_reset(struct usbnet *un) { if (usbnet_isdying(un)) return; /* * softnet_lock can be taken when NET_MPAFE is not defined when calling * if_addr_init -> if_init. This doesn't mix well with the * usbd_delay_ms calls in the init routines as things like nd6_slowtimo * can fire during the wait and attempt to take softnet_lock and then * block the softclk thread meaning the wait never ends. */ #ifndef NET_MPSAFE /* XXX What to reset? */ /* Wait a little while for the chip to get its brains in order. */ DELAY(1000); #else axe_ax_init(un); #endif } static int axe_get_phyno(struct axe_softc *sc, int sel) { int phyno; switch (AXE_PHY_TYPE(sc->axe_phyaddrs[sel])) { case PHY_TYPE_100_HOME: /* FALLTHROUGH */ case PHY_TYPE_GIG: phyno = AXE_PHY_NO(sc->axe_phyaddrs[sel]); break; case PHY_TYPE_SPECIAL: /* FALLTHROUGH */ case PHY_TYPE_RSVD: /* FALLTHROUGH */ case PHY_TYPE_NON_SUP: /* FALLTHROUGH */ default: phyno = -1; break; } return phyno; } #define AXE_GPIO_WRITE(x, y) do { \ axe_cmd(sc, AXE_CMD_WRITE_GPIO, 0, (x), NULL); \ usbd_delay_ms(sc->axe_un.un_udev, hztoms(y)); \ } while (0) static void axe_ax88178_init(struct axe_softc *sc) { AXEHIST_FUNC(); AXEHIST_CALLED(); struct usbnet * const un = &sc->axe_un; int gpio0, ledmode, phymode; uint16_t eeprom, val; axe_cmd(sc, AXE_CMD_SROM_WR_ENABLE, 0, 0, NULL); /* XXX magic */ if (axe_cmd(sc, AXE_CMD_SROM_READ, 0, 0x0017, &eeprom) != 0) eeprom = 0xffff; axe_cmd(sc, AXE_CMD_SROM_WR_DISABLE, 0, 0, NULL); eeprom = le16toh(eeprom); DPRINTF("EEPROM is %#jx", eeprom, 0, 0, 0); /* if EEPROM is invalid we have to use to GPIO0 */ if (eeprom == 0xffff) { phymode = AXE_PHY_MODE_MARVELL; gpio0 = 1; ledmode = 0; } else { phymode = eeprom & 0x7f; gpio0 = (eeprom & 0x80) ? 0 : 1; ledmode = eeprom >> 8; } DPRINTF("use gpio0: %jd, phymode %jd", gpio0, phymode, 0, 0); /* Program GPIOs depending on PHY hardware. */ switch (phymode) { case AXE_PHY_MODE_MARVELL: if (gpio0 == 1) { AXE_GPIO_WRITE(AXE_GPIO_RELOAD_EEPROM | AXE_GPIO0_EN, hz / 32); AXE_GPIO_WRITE(AXE_GPIO0_EN | AXE_GPIO2 | AXE_GPIO2_EN, hz / 32); AXE_GPIO_WRITE(AXE_GPIO0_EN | AXE_GPIO2_EN, hz / 4); AXE_GPIO_WRITE(AXE_GPIO0_EN | AXE_GPIO2 | AXE_GPIO2_EN, hz / 32); } else { AXE_GPIO_WRITE(AXE_GPIO_RELOAD_EEPROM | AXE_GPIO1 | AXE_GPIO1_EN, hz / 3); if (ledmode == 1) { AXE_GPIO_WRITE(AXE_GPIO1_EN, hz / 3); AXE_GPIO_WRITE(AXE_GPIO1 | AXE_GPIO1_EN, hz / 3); } else { AXE_GPIO_WRITE(AXE_GPIO1 | AXE_GPIO1_EN | AXE_GPIO2 | AXE_GPIO2_EN, hz / 32); AXE_GPIO_WRITE(AXE_GPIO1 | AXE_GPIO1_EN | AXE_GPIO2_EN, hz / 4); AXE_GPIO_WRITE(AXE_GPIO1 | AXE_GPIO1_EN | AXE_GPIO2 | AXE_GPIO2_EN, hz / 32); } } break; case AXE_PHY_MODE_CICADA: case AXE_PHY_MODE_CICADA_V2: case AXE_PHY_MODE_CICADA_V2_ASIX: if (gpio0 == 1) AXE_GPIO_WRITE(AXE_GPIO_RELOAD_EEPROM | AXE_GPIO0 | AXE_GPIO0_EN, hz / 32); else AXE_GPIO_WRITE(AXE_GPIO_RELOAD_EEPROM | AXE_GPIO1 | AXE_GPIO1_EN, hz / 32); break; case AXE_PHY_MODE_AGERE: AXE_GPIO_WRITE(AXE_GPIO_RELOAD_EEPROM | AXE_GPIO1 | AXE_GPIO1_EN, hz / 32); AXE_GPIO_WRITE(AXE_GPIO1 | AXE_GPIO1_EN | AXE_GPIO2 | AXE_GPIO2_EN, hz / 32); AXE_GPIO_WRITE(AXE_GPIO1 | AXE_GPIO1_EN | AXE_GPIO2_EN, hz / 4); AXE_GPIO_WRITE(AXE_GPIO1 | AXE_GPIO1_EN | AXE_GPIO2 | AXE_GPIO2_EN, hz / 32); break; case AXE_PHY_MODE_REALTEK_8211CL: case AXE_PHY_MODE_REALTEK_8211BN: case AXE_PHY_MODE_REALTEK_8251CL: val = gpio0 == 1 ? AXE_GPIO0 | AXE_GPIO0_EN : AXE_GPIO1 | AXE_GPIO1_EN; AXE_GPIO_WRITE(val, hz / 32); AXE_GPIO_WRITE(val | AXE_GPIO2 | AXE_GPIO2_EN, hz / 32); AXE_GPIO_WRITE(val | AXE_GPIO2_EN, hz / 4); AXE_GPIO_WRITE(val | AXE_GPIO2 | AXE_GPIO2_EN, hz / 32); if (phymode == AXE_PHY_MODE_REALTEK_8211CL) { axe_uno_mii_write_reg(un, un->un_phyno, 0x1F, 0x0005); axe_uno_mii_write_reg(un, un->un_phyno, 0x0C, 0x0000); axe_uno_mii_read_reg(un, un->un_phyno, 0x0001, &val); axe_uno_mii_write_reg(un, un->un_phyno, 0x01, val | 0x0080); axe_uno_mii_write_reg(un, un->un_phyno, 0x1F, 0x0000); } break; default: /* Unknown PHY model or no need to program GPIOs. */ break; } /* soft reset */ axe_cmd(sc, AXE_CMD_SW_RESET_REG, 0, AXE_SW_RESET_CLEAR, NULL); usbd_delay_ms(un->un_udev, 150); axe_cmd(sc, AXE_CMD_SW_RESET_REG, 0, AXE_SW_RESET_PRL | AXE_178_RESET_MAGIC, NULL); usbd_delay_ms(un->un_udev, 150); /* Enable MII/GMII/RGMII interface to work with external PHY. */ axe_cmd(sc, AXE_CMD_SW_PHY_SELECT, 0, 0, NULL); usbd_delay_ms(un->un_udev, 10); axe_cmd(sc, AXE_CMD_RXCTL_WRITE, 0, 0, NULL); } static void axe_ax88772_init(struct axe_softc *sc) { AXEHIST_FUNC(); AXEHIST_CALLED(); struct usbnet * const un = &sc->axe_un; axe_cmd(sc, AXE_CMD_WRITE_GPIO, 0, 0x00b0, NULL); usbd_delay_ms(un->un_udev, 40); if (un->un_phyno == AXE_772_PHY_NO_EPHY) { /* ask for the embedded PHY */ axe_cmd(sc, AXE_CMD_SW_PHY_SELECT, 0, AXE_SW_PHY_SELECT_EMBEDDED, NULL); usbd_delay_ms(un->un_udev, 10); /* power down and reset state, pin reset state */ axe_cmd(sc, AXE_CMD_SW_RESET_REG, 0, AXE_SW_RESET_CLEAR, NULL); usbd_delay_ms(un->un_udev, 60); /* power down/reset state, pin operating state */ axe_cmd(sc, AXE_CMD_SW_RESET_REG, 0, AXE_SW_RESET_IPPD | AXE_SW_RESET_PRL, NULL); usbd_delay_ms(un->un_udev, 150); /* power up, reset */ axe_cmd(sc, AXE_CMD_SW_RESET_REG, 0, AXE_SW_RESET_PRL, NULL); /* power up, operating */ axe_cmd(sc, AXE_CMD_SW_RESET_REG, 0, AXE_SW_RESET_IPRL | AXE_SW_RESET_PRL, NULL); } else { /* ask for external PHY */ axe_cmd(sc, AXE_CMD_SW_PHY_SELECT, 0, AXE_SW_PHY_SELECT_EXT, NULL); usbd_delay_ms(un->un_udev, 10); /* power down internal PHY */ axe_cmd(sc, AXE_CMD_SW_RESET_REG, 0, AXE_SW_RESET_IPPD | AXE_SW_RESET_PRL, NULL); } usbd_delay_ms(un->un_udev, 150); axe_cmd(sc, AXE_CMD_RXCTL_WRITE, 0, 0, NULL); } static void axe_ax88772_phywake(struct axe_softc *sc) { AXEHIST_FUNC(); AXEHIST_CALLED(); struct usbnet * const un = &sc->axe_un; if (un->un_phyno == AXE_772_PHY_NO_EPHY) { /* Manually select internal(embedded) PHY - MAC mode. */ axe_cmd(sc, AXE_CMD_SW_PHY_SELECT, 0, AXE_SW_PHY_SELECT_EMBEDDED, NULL); usbd_delay_ms(un->un_udev, hztoms(hz / 32)); } else { /* * Manually select external PHY - MAC mode. * Reverse MII/RMII is for AX88772A PHY mode. */ axe_cmd(sc, AXE_CMD_SW_PHY_SELECT, 0, AXE_SW_PHY_SELECT_SS_ENB | AXE_SW_PHY_SELECT_EXT | AXE_SW_PHY_SELECT_SS_MII, NULL); usbd_delay_ms(un->un_udev, hztoms(hz / 32)); } axe_cmd(sc, AXE_CMD_SW_RESET_REG, 0, AXE_SW_RESET_IPPD | AXE_SW_RESET_IPRL, NULL); /* T1 = min 500ns everywhere */ usbd_delay_ms(un->un_udev, 150); /* Take PHY out of power down. */ if (un->un_phyno == AXE_772_PHY_NO_EPHY) { axe_cmd(sc, AXE_CMD_SW_RESET_REG, 0, AXE_SW_RESET_IPRL, NULL); } else { axe_cmd(sc, AXE_CMD_SW_RESET_REG, 0, AXE_SW_RESET_PRTE, NULL); } /* 772 T2 is 60ms. 772A T2 is 160ms, 772B T2 is 600ms */ usbd_delay_ms(un->un_udev, 600); axe_cmd(sc, AXE_CMD_SW_RESET_REG, 0, AXE_SW_RESET_CLEAR, NULL); /* T3 = 500ns everywhere */ usbd_delay_ms(un->un_udev, hztoms(hz / 32)); axe_cmd(sc, AXE_CMD_SW_RESET_REG, 0, AXE_SW_RESET_IPRL, NULL); usbd_delay_ms(un->un_udev, hztoms(hz / 32)); } static void axe_ax88772a_init(struct axe_softc *sc) { AXEHIST_FUNC(); AXEHIST_CALLED(); /* Reload EEPROM. */ AXE_GPIO_WRITE(AXE_GPIO_RELOAD_EEPROM, hz / 32); axe_ax88772_phywake(sc); /* Stop MAC. */ axe_cmd(sc, AXE_CMD_RXCTL_WRITE, 0, 0, NULL); } static void axe_ax88772b_init(struct axe_softc *sc) { AXEHIST_FUNC(); AXEHIST_CALLED(); struct usbnet * const un = &sc->axe_un; uint16_t eeprom; int i; /* Reload EEPROM. */ AXE_GPIO_WRITE(AXE_GPIO_RELOAD_EEPROM , hz / 32); /* * Save PHY power saving configuration(high byte) and * clear EEPROM checksum value(low byte). */ if (axe_cmd(sc, AXE_CMD_SROM_READ, 0, AXE_EEPROM_772B_PHY_PWRCFG, &eeprom)) { aprint_error_dev(un->un_dev, "failed to read eeprom\n"); return; } sc->sc_pwrcfg = le16toh(eeprom) & 0xFF00; /* * Auto-loaded default station address from internal ROM is * 00:00:00:00:00:00 such that an explicit access to EEPROM * is required to get real station address. */ uint8_t *eaddr = un->un_eaddr; for (i = 0; i < ETHER_ADDR_LEN / 2; i++) { if (axe_cmd(sc, AXE_CMD_SROM_READ, 0, AXE_EEPROM_772B_NODE_ID + i, &eeprom)) { aprint_error_dev(un->un_dev, "failed to read eeprom\n"); eeprom = 0; } eeprom = le16toh(eeprom); *eaddr++ = (uint8_t)(eeprom & 0xFF); *eaddr++ = (uint8_t)((eeprom >> 8) & 0xFF); } /* Wakeup PHY. */ axe_ax88772_phywake(sc); /* Stop MAC. */ axe_cmd(sc, AXE_CMD_RXCTL_WRITE, 0, 0, NULL); } #undef AXE_GPIO_WRITE /* * Probe for a AX88172 chip. */ static int axe_match(device_t parent, cfdata_t match, void *aux) { struct usb_attach_arg *uaa = aux; return axe_lookup(uaa->uaa_vendor, uaa->uaa_product) != NULL ? UMATCH_VENDOR_PRODUCT : UMATCH_NONE; } /* * Attach the interface. Allocate softc structures, do ifmedia * setup and ethernet/BPF attach. */ static void axe_attach(device_t parent, device_t self, void *aux) { AXEHIST_FUNC(); AXEHIST_CALLED(); USBNET_MII_DECL_DEFAULT(unm); struct axe_softc *sc = device_private(self); struct usbnet * const un = &sc->axe_un; struct usb_attach_arg *uaa = aux; struct usbd_device *dev = uaa->uaa_device; usbd_status err; usb_interface_descriptor_t *id; usb_endpoint_descriptor_t *ed; char *devinfop; unsigned bufsz; int i; KASSERT((void *)sc == un); aprint_naive("\n"); aprint_normal("\n"); devinfop = usbd_devinfo_alloc(dev, 0); aprint_normal_dev(self, "%s\n", devinfop); usbd_devinfo_free(devinfop); un->un_dev = self; un->un_udev = dev; un->un_sc = sc; un->un_ops = &axe_ops; un->un_rx_xfer_flags = USBD_SHORT_XFER_OK; un->un_tx_xfer_flags = USBD_FORCE_SHORT_XFER; un->un_rx_list_cnt = AXE_RX_LIST_CNT; un->un_tx_list_cnt = AXE_TX_LIST_CNT; err = usbd_set_config_no(dev, AXE_CONFIG_NO, 1); if (err) { aprint_error_dev(self, "failed to set configuration" ", err=%s\n", usbd_errstr(err)); return; } un->un_flags = axe_lookup(uaa->uaa_vendor, uaa->uaa_product)->axe_flags; err = usbd_device2interface_handle(dev, AXE_IFACE_IDX, &un->un_iface); if (err) { aprint_error_dev(self, "getting interface handle failed\n"); return; } id = usbd_get_interface_descriptor(un->un_iface); /* decide on what our bufsize will be */ if (AXE_IS_172(un)) bufsz = AXE_172_BUFSZ; else bufsz = (un->un_udev->ud_speed == USB_SPEED_HIGH) ? AXE_178_MAX_BUFSZ : AXE_178_MIN_BUFSZ; un->un_rx_bufsz = un->un_tx_bufsz = bufsz; un->un_ed[USBNET_ENDPT_RX] = 0; un->un_ed[USBNET_ENDPT_TX] = 0; un->un_ed[USBNET_ENDPT_INTR] = 0; /* Find endpoints. */ for (i = 0; i < id->bNumEndpoints; i++) { ed = usbd_interface2endpoint_descriptor(un->un_iface, i); if (ed == NULL) { aprint_error_dev(self, "couldn't get ep %d\n", i); return; } const uint8_t xt = UE_GET_XFERTYPE(ed->bmAttributes); const uint8_t dir = UE_GET_DIR(ed->bEndpointAddress); if (dir == UE_DIR_IN && xt == UE_BULK && un->un_ed[USBNET_ENDPT_RX] == 0) { un->un_ed[USBNET_ENDPT_RX] = ed->bEndpointAddress; } else if (dir == UE_DIR_OUT && xt == UE_BULK && un->un_ed[USBNET_ENDPT_TX] == 0) { un->un_ed[USBNET_ENDPT_TX] = ed->bEndpointAddress; } else if (dir == UE_DIR_IN && xt == UE_INTERRUPT) { un->un_ed[USBNET_ENDPT_INTR] = ed->bEndpointAddress; } } /* Set these up now for axe_cmd(). */ usbnet_attach(un); /* We need the PHYID for init dance in some cases */ if (axe_cmd(sc, AXE_CMD_READ_PHYID, 0, 0, &sc->axe_phyaddrs)) { aprint_error_dev(self, "failed to read phyaddrs\n"); return; } DPRINTF(" phyaddrs[0]: %jx phyaddrs[1]: %jx", sc->axe_phyaddrs[0], sc->axe_phyaddrs[1], 0, 0); un->un_phyno = axe_get_phyno(sc, AXE_PHY_SEL_PRI); if (un->un_phyno == -1) un->un_phyno = axe_get_phyno(sc, AXE_PHY_SEL_SEC); if (un->un_phyno == -1) { DPRINTF(" no valid PHY address found, assuming PHY address 0", 0, 0, 0, 0); un->un_phyno = 0; } /* Initialize controller and get station address. */ axe_ax_init(un); /* * Fetch IPG values. */ if (un->un_flags & (AX772A | AX772B)) { /* Set IPG values. */ sc->axe_ipgs[0] = AXE_IPG0_DEFAULT; sc->axe_ipgs[1] = AXE_IPG1_DEFAULT; sc->axe_ipgs[2] = AXE_IPG2_DEFAULT; } else { if (axe_cmd(sc, AXE_CMD_READ_IPG012, 0, 0, sc->axe_ipgs)) { aprint_error_dev(self, "failed to read ipg\n"); return; } } if (!AXE_IS_172(un)) usbnet_ec(un)->ec_capabilities = ETHERCAP_VLAN_MTU; if (un->un_flags & AX772B) { struct ifnet *ifp = usbnet_ifp(un); ifp->if_capabilities = IFCAP_CSUM_IPv4_Rx | IFCAP_CSUM_TCPv4_Rx | IFCAP_CSUM_UDPv4_Rx | IFCAP_CSUM_TCPv6_Rx | IFCAP_CSUM_UDPv6_Rx; /* * Checksum offloading of AX88772B also works with VLAN * tagged frames but there is no way to take advantage * of the feature because vlan(4) assumes * IFCAP_VLAN_HWTAGGING is prerequisite condition to * support checksum offloading with VLAN. VLAN hardware * tagging support of AX88772B is very limited so it's * not possible to announce IFCAP_VLAN_HWTAGGING. */ } if (un->un_flags & (AX772A | AX772B | AX178)) unm.un_mii_flags = MIIF_DOPAUSE; usbnet_attach_ifp(un, IFF_SIMPLEX | IFF_BROADCAST | IFF_MULTICAST, 0, &unm); } static void axe_uno_rx_loop(struct usbnet * un, struct usbnet_chain *c, uint32_t total_len) { AXEHIST_FUNC(); AXEHIST_CALLED(); struct axe_softc * const sc = usbnet_softc(un); struct ifnet *ifp = usbnet_ifp(un); uint8_t *buf = c->unc_buf; do { u_int pktlen = 0; u_int rxlen = 0; int flags = 0; if ((un->un_flags & AXSTD_FRAME) != 0) { struct axe_sframe_hdr hdr; if (total_len < sizeof(hdr)) { if_statinc(ifp, if_ierrors); break; } memcpy(&hdr, buf, sizeof(hdr)); DPRINTFN(20, "total_len %#jx len %#jx ilen %#jx", total_len, (le16toh(hdr.len) & AXE_RH1M_RXLEN_MASK), (le16toh(hdr.ilen) & AXE_RH1M_RXLEN_MASK), 0); total_len -= sizeof(hdr); buf += sizeof(hdr); if (((le16toh(hdr.len) & AXE_RH1M_RXLEN_MASK) ^ (le16toh(hdr.ilen) & AXE_RH1M_RXLEN_MASK)) != AXE_RH1M_RXLEN_MASK) { if_statinc(ifp, if_ierrors); break; } rxlen = le16toh(hdr.len) & AXE_RH1M_RXLEN_MASK; if (total_len < rxlen) { pktlen = total_len; total_len = 0; } else { pktlen = rxlen; rxlen = roundup2(rxlen, 2); total_len -= rxlen; } } else if ((un->un_flags & AXCSUM_FRAME) != 0) { struct axe_csum_hdr csum_hdr; if (total_len < sizeof(csum_hdr)) { if_statinc(ifp, if_ierrors); break; } memcpy(&csum_hdr, buf, sizeof(csum_hdr)); csum_hdr.len = le16toh(csum_hdr.len); csum_hdr.ilen = le16toh(csum_hdr.ilen); csum_hdr.cstatus = le16toh(csum_hdr.cstatus); DPRINTFN(20, "total_len %#jx len %#jx ilen %#jx" " cstatus %#jx", total_len, csum_hdr.len, csum_hdr.ilen, csum_hdr.cstatus); if ((AXE_CSUM_RXBYTES(csum_hdr.len) ^ AXE_CSUM_RXBYTES(csum_hdr.ilen)) != sc->sc_lenmask) { /* we lost sync */ if_statinc(ifp, if_ierrors); DPRINTFN(20, "len %#jx ilen %#jx lenmask %#jx " "err", AXE_CSUM_RXBYTES(csum_hdr.len), AXE_CSUM_RXBYTES(csum_hdr.ilen), sc->sc_lenmask, 0); break; } /* * Get total transferred frame length including * checksum header. The length should be multiple * of 4. */ pktlen = AXE_CSUM_RXBYTES(csum_hdr.len); u_int len = sizeof(csum_hdr) + pktlen; len = (len + 3) & ~3; if (total_len < len) { DPRINTFN(20, "total_len %#jx < len %#jx", total_len, len, 0, 0); /* invalid length */ if_statinc(ifp, if_ierrors); break; } buf += sizeof(csum_hdr); const uint16_t cstatus = csum_hdr.cstatus; if (cstatus & AXE_CSUM_HDR_L3_TYPE_IPV4) { if (cstatus & AXE_CSUM_HDR_L4_CSUM_ERR) flags |= M_CSUM_TCP_UDP_BAD; if (cstatus & AXE_CSUM_HDR_L3_CSUM_ERR) flags |= M_CSUM_IPv4_BAD; const uint16_t l4type = cstatus & AXE_CSUM_HDR_L4_TYPE_MASK; if (l4type == AXE_CSUM_HDR_L4_TYPE_TCP) flags |= M_CSUM_TCPv4; if (l4type == AXE_CSUM_HDR_L4_TYPE_UDP) flags |= M_CSUM_UDPv4; } if (total_len < len) { pktlen = total_len; total_len = 0; } else { total_len -= len; rxlen = len - sizeof(csum_hdr); } DPRINTFN(20, "total_len %#jx len %#jx pktlen %#jx" " rxlen %#jx", total_len, len, pktlen, rxlen); } else { /* AX172 */ pktlen = rxlen = total_len; total_len = 0; } usbnet_enqueue(un, buf, pktlen, flags, 0, 0); buf += rxlen; } while (total_len > 0); DPRINTFN(10, "start rx", 0, 0, 0, 0); } static unsigned axe_uno_tx_prepare(struct usbnet *un, struct mbuf *m, struct usbnet_chain *c) { AXEHIST_FUNC(); AXEHIST_CALLED(); struct axe_sframe_hdr hdr, tlr; size_t hdr_len = 0, tlr_len = 0; int length, boundary; if (!AXE_IS_172(un)) { /* * Copy the mbuf data into a contiguous buffer, leaving two * bytes at the beginning to hold the frame length. */ boundary = (un->un_udev->ud_speed == USB_SPEED_HIGH) ? 512 : 64; hdr.len = htole16(m->m_pkthdr.len); hdr.ilen = ~hdr.len; hdr_len = sizeof(hdr); length = hdr_len + m->m_pkthdr.len; if ((length % boundary) == 0) { tlr.len = 0x0000; tlr.ilen = 0xffff; tlr_len = sizeof(tlr); } DPRINTFN(20, "length %jx m_pkthdr.len %jx hdrsize %#jx", length, m->m_pkthdr.len, sizeof(hdr), 0); } if ((unsigned)m->m_pkthdr.len > un->un_tx_bufsz - hdr_len - tlr_len) return 0; length = hdr_len + m->m_pkthdr.len + tlr_len; if (hdr_len) memcpy(c->unc_buf, &hdr, hdr_len); m_copydata(m, 0, m->m_pkthdr.len, c->unc_buf + hdr_len); if (tlr_len) memcpy(c->unc_buf + length - tlr_len, &tlr, tlr_len); return length; } static void axe_csum_cfg(struct axe_softc *sc) { struct usbnet * const un = &sc->axe_un; struct ifnet * const ifp = usbnet_ifp(un); uint16_t csum1, csum2; if ((un->un_flags & AX772B) != 0) { csum1 = 0; csum2 = 0; if ((ifp->if_capenable & IFCAP_CSUM_IPv4_Tx) != 0) csum1 |= AXE_TXCSUM_IP; if ((ifp->if_capenable & IFCAP_CSUM_TCPv4_Tx) != 0) csum1 |= AXE_TXCSUM_TCP; if ((ifp->if_capenable & IFCAP_CSUM_UDPv4_Tx) != 0) csum1 |= AXE_TXCSUM_UDP; if ((ifp->if_capenable & IFCAP_CSUM_TCPv6_Tx) != 0) csum1 |= AXE_TXCSUM_TCPV6; if ((ifp->if_capenable & IFCAP_CSUM_UDPv6_Tx) != 0) csum1 |= AXE_TXCSUM_UDPV6; axe_cmd(sc, AXE_772B_CMD_WRITE_TXCSUM, csum2, csum1, NULL); csum1 = 0; csum2 = 0; if ((ifp->if_capenable & IFCAP_CSUM_IPv4_Rx) != 0) csum1 |= AXE_RXCSUM_IP; if ((ifp->if_capenable & IFCAP_CSUM_TCPv4_Rx) != 0) csum1 |= AXE_RXCSUM_TCP; if ((ifp->if_capenable & IFCAP_CSUM_UDPv4_Rx) != 0) csum1 |= AXE_RXCSUM_UDP; if ((ifp->if_capenable & IFCAP_CSUM_TCPv6_Rx) != 0) csum1 |= AXE_RXCSUM_TCPV6; if ((ifp->if_capenable & IFCAP_CSUM_UDPv6_Rx) != 0) csum1 |= AXE_RXCSUM_UDPV6; axe_cmd(sc, AXE_772B_CMD_WRITE_RXCSUM, csum2, csum1, NULL); } } static int axe_uno_init(struct ifnet *ifp) { AXEHIST_FUNC(); AXEHIST_CALLED(); struct usbnet * const un = ifp->if_softc; struct axe_softc * const sc = usbnet_softc(un); int rxmode; /* Reset the ethernet interface. */ axe_reset(un); #if 0 ret = asix_write_gpio(dev, AX_GPIO_RSE | AX_GPIO_GPO_2 | AX_GPIO_GPO2EN, 5, in_pm); #endif /* Set MAC address and transmitter IPG values. */ if (AXE_IS_172(un)) { axe_cmd(sc, AXE_172_CMD_WRITE_NODEID, 0, 0, un->un_eaddr); axe_cmd(sc, AXE_172_CMD_WRITE_IPG0, 0, sc->axe_ipgs[0], NULL); axe_cmd(sc, AXE_172_CMD_WRITE_IPG1, 0, sc->axe_ipgs[1], NULL); axe_cmd(sc, AXE_172_CMD_WRITE_IPG2, 0, sc->axe_ipgs[2], NULL); } else { axe_cmd(sc, AXE_178_CMD_WRITE_NODEID, 0, 0, un->un_eaddr); axe_cmd(sc, AXE_178_CMD_WRITE_IPG012, sc->axe_ipgs[2], (sc->axe_ipgs[1] << 8) | (sc->axe_ipgs[0]), NULL); un->un_flags &= ~(AXSTD_FRAME | AXCSUM_FRAME); if ((un->un_flags & AX772B) != 0 && (ifp->if_capenable & AX_RXCSUM) != 0) { sc->sc_lenmask = AXE_CSUM_HDR_LEN_MASK; un->un_flags |= AXCSUM_FRAME; } else { sc->sc_lenmask = AXE_HDR_LEN_MASK; un->un_flags |= AXSTD_FRAME; } } /* Configure TX/RX checksum offloading. */ axe_csum_cfg(sc); if (un->un_flags & AX772B) { /* AX88772B uses different maximum frame burst configuration. */ axe_cmd(sc, AXE_772B_CMD_RXCTL_WRITE_CFG, ax88772b_mfb_table[AX88772B_MFB_16K].threshold, ax88772b_mfb_table[AX88772B_MFB_16K].byte_cnt, NULL); } /* Enable receiver, set RX mode */ rxmode = (AXE_RXCMD_BROADCAST | AXE_RXCMD_MULTICAST | AXE_RXCMD_ENABLE); if (AXE_IS_172(un)) rxmode |= AXE_172_RXCMD_UNICAST; else { if (un->un_flags & AX772B) { /* * Select RX header format type 1. Aligning IP * header on 4 byte boundary is not needed when * checksum offloading feature is not used * because we always copy the received frame in * RX handler. When RX checksum offloading is * active, aligning IP header is required to * reflect actual frame length including RX * header size. */ rxmode |= AXE_772B_RXCMD_HDR_TYPE_1; if (un->un_flags & AXCSUM_FRAME) rxmode |= AXE_772B_RXCMD_IPHDR_ALIGN; } else { /* * Default Rx buffer size is too small to get * maximum performance. */ #if 0 if (un->un_udev->ud_speed == USB_SPEED_HIGH) { /* Largest possible USB buffer size for AX88178 */ } #endif rxmode |= AXE_178_RXCMD_MFB_16384; } } DPRINTF("rxmode %#jx", rxmode, 0, 0, 0); axe_cmd(sc, AXE_CMD_RXCTL_WRITE, 0, rxmode, NULL); return 0; } static void axe_uno_stop(struct ifnet *ifp, int disable) { struct usbnet * const un = ifp->if_softc; axe_reset(un); } #ifdef _MODULE #include "ioconf.c" #endif USBNET_MODULE(axe) |
| 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 | /* $NetBSD: exec_elf32.c,v 1.143 2019/11/20 19:37:53 pgoyette Exp $ */ /* * Copyright (c) 1996 Christopher G. Demetriou * All rights reserved. * * Redistribution and use in source and binary forms, with or without * modification, are permitted provided that the following conditions * are met: * 1. Redistributions of source code must retain the above copyright * notice, this list of conditions and the following disclaimer. * 2. Redistributions in binary form must reproduce the above copyright * notice, this list of conditions and the following disclaimer in the * documentation and/or other materials provided with the distribution. * 3. All advertising materials mentioning features or use of this software * must display the following acknowledgement: * This product includes software developed by Christopher G. Demetriou * for the NetBSD Project. * 4. The name of the author may not be used to endorse or promote products * derived from this software without specific prior written permission * * THIS SOFTWARE IS PROVIDED BY THE AUTHOR ``AS IS'' AND ANY EXPRESS OR * IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES * OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. * IN NO EVENT SHALL THE AUTHOR BE LIABLE FOR ANY DIRECT, INDIRECT, * INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT * NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, * DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY * THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT * (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF * THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. */ #include <sys/cdefs.h> __KERNEL_RCSID(0, "$NetBSD: exec_elf32.c,v 1.143 2019/11/20 19:37:53 pgoyette Exp $"); #define ELFSIZE 32 #include "exec_elf.c" #include <sys/module.h> #define ELF32_AUXSIZE (ELF_AUX_ENTRIES * sizeof(Aux32Info) \ + MAXPATHLEN + ALIGN(1)) MODULE(MODULE_CLASS_EXEC, exec_elf32, NULL); static struct execsw exec_elf32_execsw[] = { { .es_hdrsz = sizeof (Elf32_Ehdr), .es_makecmds = exec_elf32_makecmds, .u = { .elf_probe_func = netbsd_elf32_probe, }, .es_emul = &emul_netbsd, .es_prio = EXECSW_PRIO_FIRST, .es_arglen = ELF32_AUXSIZE, .es_copyargs = elf32_copyargs, .es_setregs = NULL, .es_coredump = coredump_elf32, .es_setup_stack = exec_setup_stack, }, #if EXEC_ELF_NOTELESS { .es_hdrsz = sizeof (Elf32_Ehdr), .es_makecmds = exec_elf32_makecmds, .u { elf_probe_func = NULL, }, .es_emul = &emul_netbsd, .es_prio = EXECSW_PRIO_LAST, .es_arglen = ELF32_AUXSIZE, .es_copyargs = elf32_copyargs, .es_setregs = NULL, .es_coredump = coredump_elf32, .es_setup_stack = exec_setup_stack, }, #endif }; static int exec_elf32_modcmd(modcmd_t cmd, void *arg) { #if ARCH_ELFSIZE == 64 /* * If we are on a 64bit system, we don't want the 32bit execsw[] to be * added in the global array, because the exec_elf32 module only works * on 32bit systems. * * However, we need the exec_elf32 module, because it will make the 32bit * functions available for netbsd32 and linux32. * * Therefore, allow this module on 64bit systems, but make it dormant. */ (void)exec_elf32_execsw; /* unused */ switch (cmd) { case MODULE_CMD_INIT: case MODULE_CMD_FINI: return 0; default: return ENOTTY; } #else /* ARCH_ELFSIZE == 64 */ switch (cmd) { case MODULE_CMD_INIT: return exec_add(exec_elf32_execsw, __arraycount(exec_elf32_execsw)); case MODULE_CMD_FINI: return exec_remove(exec_elf32_execsw, __arraycount(exec_elf32_execsw)); default: return ENOTTY; } #endif /* ARCH_ELFSIZE == 64 */ } |
| 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 197 198 199 200 201 202 203 204 205 206 207 208 209 210 211 212 213 214 215 216 217 218 219 220 221 222 223 224 225 226 227 228 229 230 231 232 233 234 235 236 237 238 239 240 241 242 243 244 245 246 247 248 249 250 251 252 253 254 255 256 257 258 259 260 261 262 263 264 265 266 267 268 269 270 271 272 273 274 275 276 277 278 279 280 281 282 283 284 285 286 287 288 289 290 291 292 293 294 295 296 297 298 299 300 301 302 303 304 305 306 307 308 309 310 311 312 313 314 315 316 317 318 319 320 321 322 323 324 325 326 327 328 329 330 331 332 333 334 335 336 337 338 339 340 341 342 343 344 345 346 347 348 349 350 351 352 353 354 355 356 357 358 359 360 361 362 363 364 365 366 367 368 369 370 371 372 373 374 375 376 377 378 379 380 381 382 383 384 385 386 387 388 389 390 391 392 393 394 395 396 397 398 399 400 401 402 403 404 405 406 407 408 409 410 411 412 413 414 415 416 417 418 419 420 421 422 423 424 425 426 427 428 429 430 431 432 433 434 435 436 437 438 439 440 441 442 443 444 445 446 447 448 449 450 451 452 453 454 455 456 457 458 459 460 461 462 463 464 465 466 467 468 469 470 471 472 473 474 475 476 477 478 479 480 481 482 483 484 485 486 487 488 489 490 491 492 493 494 495 496 497 498 499 500 501 502 503 504 505 506 507 508 509 510 511 512 513 514 515 516 517 518 519 520 521 522 523 524 525 526 527 528 529 530 531 532 533 534 535 536 537 538 539 540 541 542 543 544 545 546 547 548 549 550 551 552 553 554 555 556 557 558 559 560 561 562 563 564 565 566 567 568 569 570 571 572 573 574 575 576 577 578 579 580 581 582 583 584 585 586 587 588 589 590 591 592 593 594 595 596 597 598 599 600 601 602 603 604 605 606 607 608 609 610 611 612 613 614 615 616 617 618 619 620 621 622 623 624 625 626 627 628 629 630 631 632 633 634 635 636 637 638 639 640 641 642 643 644 645 646 647 648 649 650 651 652 653 654 655 656 657 658 659 660 661 662 663 664 665 666 667 668 669 670 671 672 673 674 675 676 677 678 679 680 681 682 683 684 685 686 687 688 689 690 691 692 693 694 695 696 697 698 699 700 701 702 703 704 705 706 707 708 709 710 711 712 713 714 715 716 717 718 719 720 721 722 723 724 725 726 727 728 729 730 731 732 733 734 735 736 737 738 739 740 741 742 743 744 745 746 747 748 749 750 751 752 753 754 755 756 757 758 759 760 761 762 763 764 765 766 767 768 769 770 771 772 773 774 775 776 777 778 779 780 781 782 783 784 785 786 787 788 789 790 791 792 793 794 795 796 797 798 799 800 801 802 803 804 805 806 807 808 809 810 811 812 813 814 815 816 817 818 819 820 821 822 823 824 825 826 827 828 829 830 831 832 833 834 835 836 837 838 839 840 841 842 843 844 845 846 847 848 849 850 851 852 853 854 855 856 857 858 859 860 861 862 863 864 865 866 867 868 869 870 871 872 873 874 875 876 877 878 879 880 881 882 883 884 885 886 887 888 889 890 891 892 893 894 895 896 897 898 899 900 901 902 903 904 905 906 907 908 909 910 911 912 913 914 915 916 917 918 919 920 921 922 923 924 925 926 927 928 929 930 931 932 933 934 935 936 937 938 939 940 941 942 943 944 945 946 947 948 949 950 951 952 953 954 955 956 957 958 959 960 961 962 963 964 965 966 967 968 969 970 971 972 973 974 975 976 977 978 979 980 981 982 983 984 985 986 987 988 989 990 991 992 993 994 995 996 997 998 999 1000 1001 1002 1003 1004 1005 1006 1007 1008 1009 1010 1011 1012 1013 1014 1015 1016 1017 1018 1019 1020 1021 1022 1023 1024 1025 1026 1027 1028 1029 1030 1031 1032 1033 1034 1035 1036 1037 1038 1039 1040 1041 1042 1043 1044 1045 1046 1047 1048 1049 1050 1051 1052 1053 1054 1055 1056 1057 1058 1059 1060 1061 1062 1063 1064 1065 1066 1067 1068 1069 1070 1071 1072 1073 1074 1075 1076 1077 1078 1079 1080 1081 1082 1083 1084 1085 1086 1087 1088 1089 1090 1091 1092 1093 1094 1095 1096 1097 1098 1099 1100 1101 1102 1103 1104 1105 1106 1107 1108 1109 1110 1111 1112 1113 1114 1115 1116 1117 1118 1119 1120 1121 1122 1123 1124 1125 1126 1127 1128 1129 1130 1131 1132 1133 1134 1135 1136 1137 1138 1139 1140 1141 1142 1143 1144 1145 1146 1147 1148 1149 1150 1151 1152 1153 1154 1155 1156 1157 1158 1159 1160 1161 1162 1163 1164 1165 1166 1167 1168 1169 1170 1171 1172 1173 1174 1175 1176 1177 1178 1179 1180 1181 1182 1183 1184 1185 1186 1187 1188 1189 1190 1191 1192 1193 1194 1195 1196 1197 1198 1199 1200 1201 1202 1203 1204 1205 1206 1207 1208 1209 1210 1211 1212 1213 1214 1215 1216 1217 1218 1219 1220 1221 1222 1223 1224 1225 1226 1227 1228 1229 1230 1231 1232 1233 1234 1235 1236 1237 1238 1239 1240 1241 1242 1243 1244 1245 1246 1247 1248 1249 1250 1251 1252 1253 1254 1255 1256 1257 1258 1259 1260 1261 1262 1263 1264 1265 1266 1267 1268 1269 1270 1271 1272 1273 1274 1275 1276 1277 1278 1279 1280 1281 1282 1283 1284 1285 1286 1287 1288 1289 1290 1291 1292 1293 1294 1295 1296 1297 1298 1299 1300 1301 1302 1303 1304 1305 1306 1307 1308 1309 1310 1311 1312 1313 1314 1315 1316 1317 1318 1319 1320 1321 1322 1323 1324 1325 1326 1327 1328 1329 1330 1331 1332 1333 1334 1335 1336 1337 1338 1339 1340 1341 1342 1343 1344 1345 1346 1347 1348 1349 1350 1351 1352 1353 1354 1355 1356 1357 1358 1359 1360 1361 1362 1363 1364 1365 1366 1367 1368 1369 1370 1371 1372 1373 1374 1375 1376 1377 1378 1379 1380 1381 1382 1383 1384 1385 1386 1387 1388 1389 1390 1391 1392 1393 1394 1395 1396 1397 1398 1399 1400 1401 1402 1403 1404 1405 1406 1407 1408 1409 1410 1411 1412 1413 1414 1415 1416 1417 1418 1419 1420 1421 1422 1423 1424 1425 1426 1427 1428 1429 1430 1431 1432 1433 1434 1435 1436 1437 1438 1439 1440 1441 1442 1443 1444 1445 1446 1447 1448 1449 1450 1451 1452 1453 1454 1455 1456 1457 1458 1459 1460 1461 1462 1463 1464 1465 1466 1467 1468 1469 1470 1471 1472 1473 1474 1475 1476 1477 1478 1479 1480 1481 1482 1483 1484 1485 1486 1487 1488 1489 1490 1491 1492 1493 1494 1495 1496 1497 1498 1499 1500 1501 1502 1503 1504 1505 1506 1507 1508 1509 1510 1511 1512 1513 1514 1515 1516 1517 1518 1519 1520 1521 1522 1523 1524 1525 1526 1527 1528 1529 1530 1531 1532 1533 1534 1535 1536 1537 1538 1539 1540 1541 1542 1543 1544 1545 1546 1547 1548 | /* $NetBSD: lfs_accessors.h,v 1.51 2022/04/24 20:32:44 rillig Exp $ */ /* from NetBSD: lfs.h,v 1.165 2015/07/24 06:59:32 dholland Exp */ /* from NetBSD: dinode.h,v 1.25 2016/01/22 23:06:10 dholland Exp */ /* from NetBSD: dir.h,v 1.25 2015/09/01 06:16:03 dholland Exp */ /*- * Copyright (c) 1999, 2000, 2001, 2002, 2003 The NetBSD Foundation, Inc. * All rights reserved. * * This code is derived from software contributed to The NetBSD Foundation * by Konrad E. Schroder <perseant@hhhh.org>. * * Redistribution and use in source and binary forms, with or without * modification, are permitted provided that the following conditions * are met: * 1. Redistributions of source code must retain the above copyright * notice, this list of conditions and the following disclaimer. * 2. Redistributions in binary form must reproduce the above copyright * notice, this list of conditions and the following disclaimer in the * documentation and/or other materials provided with the distribution. * * THIS SOFTWARE IS PROVIDED BY THE NETBSD FOUNDATION, INC. AND CONTRIBUTORS * ``AS IS'' AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED * TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR * PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE FOUNDATION OR CONTRIBUTORS * BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR * CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF * SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS * INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN * CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) * ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE * POSSIBILITY OF SUCH DAMAGE. */ /*- * Copyright (c) 1991, 1993 * The Regents of the University of California. All rights reserved. * * Redistribution and use in source and binary forms, with or without * modification, are permitted provided that the following conditions * are met: * 1. Redistributions of source code must retain the above copyright * notice, this list of conditions and the following disclaimer. * 2. Redistributions in binary form must reproduce the above copyright * notice, this list of conditions and the following disclaimer in the * documentation and/or other materials provided with the distribution. * 3. Neither the name of the University nor the names of its contributors * may be used to endorse or promote products derived from this software * without specific prior written permission. * * THIS SOFTWARE IS PROVIDED BY THE REGENTS AND CONTRIBUTORS ``AS IS'' AND * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE * ARE DISCLAIMED. IN NO EVENT SHALL THE REGENTS OR CONTRIBUTORS BE LIABLE * FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL * DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS * OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) * HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT * LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY * OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF * SUCH DAMAGE. * * @(#)lfs.h 8.9 (Berkeley) 5/8/95 */ /* * Copyright (c) 2002 Networks Associates Technology, Inc. * All rights reserved. * * This software was developed for the FreeBSD Project by Marshall * Kirk McKusick and Network Associates Laboratories, the Security * Research Division of Network Associates, Inc. under DARPA/SPAWAR * contract N66001-01-C-8035 ("CBOSS"), as part of the DARPA CHATS * research program * * Copyright (c) 1982, 1989, 1993 * The Regents of the University of California. All rights reserved. * (c) UNIX System Laboratories, Inc. * All or some portions of this file are derived from material licensed * to the University of California by American Telephone and Telegraph * Co. or Unix System Laboratories, Inc. and are reproduced herein with * the permission of UNIX System Laboratories, Inc. * * Redistribution and use in source and binary forms, with or without * modification, are permitted provided that the following conditions * are met: * 1. Redistributions of source code must retain the above copyright * notice, this list of conditions and the following disclaimer. * 2. Redistributions in binary form must reproduce the above copyright * notice, this list of conditions and the following disclaimer in the * documentation and/or other materials provided with the distribution. * 3. Neither the name of the University nor the names of its contributors * may be used to endorse or promote products derived from this software * without specific prior written permission. * * THIS SOFTWARE IS PROVIDED BY THE REGENTS AND CONTRIBUTORS ``AS IS'' AND * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE * ARE DISCLAIMED. IN NO EVENT SHALL THE REGENTS OR CONTRIBUTORS BE LIABLE * FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL * DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS * OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) * HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT * LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY * OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF * SUCH DAMAGE. * * @(#)dinode.h 8.9 (Berkeley) 3/29/95 */ /* * Copyright (c) 1982, 1986, 1989, 1993 * The Regents of the University of California. All rights reserved. * (c) UNIX System Laboratories, Inc. * All or some portions of this file are derived from material licensed * to the University of California by American Telephone and Telegraph * Co. or Unix System Laboratories, Inc. and are reproduced herein with * the permission of UNIX System Laboratories, Inc. * * Redistribution and use in source and binary forms, with or without * modification, are permitted provided that the following conditions * are met: * 1. Redistributions of source code must retain the above copyright * notice, this list of conditions and the following disclaimer. * 2. Redistributions in binary form must reproduce the above copyright * notice, this list of conditions and the following disclaimer in the * documentation and/or other materials provided with the distribution. * 3. Neither the name of the University nor the names of its contributors * may be used to endorse or promote products derived from this software * without specific prior written permission. * * THIS SOFTWARE IS PROVIDED BY THE REGENTS AND CONTRIBUTORS ``AS IS'' AND * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE * ARE DISCLAIMED. IN NO EVENT SHALL THE REGENTS OR CONTRIBUTORS BE LIABLE * FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL * DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS * OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) * HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT * LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY * OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF * SUCH DAMAGE. * * @(#)dir.h 8.5 (Berkeley) 4/27/95 */ #ifndef _UFS_LFS_LFS_ACCESSORS_H_ #define _UFS_LFS_LFS_ACCESSORS_H_ #if defined(_KERNEL_OPT) #include "opt_lfs.h" #endif #include <sys/bswap.h> #include <ufs/lfs/lfs.h> #if !defined(_KERNEL) && !defined(_STANDALONE) #include <assert.h> #include <string.h> #define KASSERT assert #else #include <sys/systm.h> #endif /* * STRUCT_LFS is used by the libsa code to get accessors that work * with struct salfs instead of struct lfs, and by the cleaner to * get accessors that work with struct clfs. */ #ifndef STRUCT_LFS #define STRUCT_LFS struct lfs #endif /* * byte order */ /* * For now at least, the bootblocks shall not be endian-independent. * We can see later if it fits in the size budget. Also disable the * byteswapping if LFS_EI is off. * * Caution: these functions "know" that bswap16/32/64 are unsigned, * and if that changes will likely break silently. */ #if defined(_STANDALONE) || (defined(_KERNEL) && !defined(LFS_EI)) #define LFS_SWAP_int16_t(fs, val) (val) #define LFS_SWAP_int32_t(fs, val) (val) #define LFS_SWAP_int64_t(fs, val) (val) #define LFS_SWAP_uint16_t(fs, val) (val) #define LFS_SWAP_uint32_t(fs, val) (val) #define LFS_SWAP_uint64_t(fs, val) (val) #else #define LFS_SWAP_int16_t(fs, val) \ ((fs)->lfs_dobyteswap ? (int16_t)bswap16(val) : (val)) #define LFS_SWAP_int32_t(fs, val) \ ((fs)->lfs_dobyteswap ? (int32_t)bswap32(val) : (val)) #define LFS_SWAP_int64_t(fs, val) \ ((fs)->lfs_dobyteswap ? (int64_t)bswap64(val) : (val)) #define LFS_SWAP_uint16_t(fs, val) \ ((fs)->lfs_dobyteswap ? bswap16(val) : (val)) #define LFS_SWAP_uint32_t(fs, val) \ ((fs)->lfs_dobyteswap ? bswap32(val) : (val)) #define LFS_SWAP_uint64_t(fs, val) \ ((fs)->lfs_dobyteswap ? bswap64(val) : (val)) #endif /* * For handling directories we will need to know if the volume is * little-endian. */ #if BYTE_ORDER == LITTLE_ENDIAN #define LFS_LITTLE_ENDIAN_ONDISK(fs) (!(fs)->lfs_dobyteswap) #else #define LFS_LITTLE_ENDIAN_ONDISK(fs) ((fs)->lfs_dobyteswap) #endif /* * Suppress spurious warnings -- we use * * type *foo = &obj->member; * * in macros to verify that obj->member has the right type. When the * object is a packed structure with misaligned members, this causes * some compiles to squeal that taking the address might lead to * undefined behaviour later on -- which is helpful in general, not * relevant in this case, because we don't do anything with foo * afterward; we only declare it to get a type check and then we * discard it. */ #ifdef __GNUC__ #if defined(__clang__) #pragma clang diagnostic push #pragma clang diagnostic ignored "-Waddress-of-packed-member" #elif __GNUC_PREREQ__(9,0) #pragma GCC diagnostic push #pragma GCC diagnostic ignored "-Waddress-of-packed-member" #endif #endif /* * directories */ #define LFS_DIRHEADERSIZE(fs) \ ((fs)->lfs_is64 ? sizeof(struct lfs_dirheader64) : sizeof(struct lfs_dirheader32)) /* * The LFS_DIRSIZ macro gives the minimum record length which will hold * the directory entry. This requires the amount of space in struct lfs_direct * without the d_name field, plus enough space for the name with a terminating * null byte (dp->d_namlen+1), rounded up to a 4 byte boundary. */ #define LFS_DIRECTSIZ(fs, namlen) \ (LFS_DIRHEADERSIZE(fs) + (((namlen)+1 + 3) &~ 3)) /* * The size of the largest possible directory entry. This is * used by ulfs_dirhash to figure the size of an array, so we * need a single constant value true for both lfs32 and lfs64. */ #define LFS_MAXDIRENTRYSIZE \ (sizeof(struct lfs_dirheader64) + (((LFS_MAXNAMLEN+1)+1 + 3) & ~3)) #if (BYTE_ORDER == LITTLE_ENDIAN) #define LFS_OLDDIRSIZ(oldfmt, dp, needswap) \ (((oldfmt) && !(needswap)) ? \ LFS_DIRECTSIZ((dp)->d_type) : LFS_DIRECTSIZ((dp)->d_namlen)) #else #define LFS_OLDDIRSIZ(oldfmt, dp, needswap) \ (((oldfmt) && (needswap)) ? \ LFS_DIRECTSIZ((dp)->d_type) : LFS_DIRECTSIZ((dp)->d_namlen)) #endif #define LFS_DIRSIZ(fs, dp) LFS_DIRECTSIZ(fs, lfs_dir_getnamlen(fs, dp)) /* Constants for the first argument of LFS_OLDDIRSIZ */ #define LFS_OLDDIRFMT 1 #define LFS_NEWDIRFMT 0 #define LFS_NEXTDIR(fs, dp) \ ((LFS_DIRHEADER *)((char *)(dp) + lfs_dir_getreclen(fs, dp))) static __inline char * lfs_dir_nameptr(const STRUCT_LFS *fs, LFS_DIRHEADER *dh) { if (fs->lfs_is64) { return (char *)(&dh->u_64 + 1); } else { return (char *)(&dh->u_32 + 1); } } static __inline uint64_t lfs_dir_getino(const STRUCT_LFS *fs, const LFS_DIRHEADER *dh) { if (fs->lfs_is64) { return LFS_SWAP_uint64_t(fs, dh->u_64.dh_ino); } else { return LFS_SWAP_uint32_t(fs, dh->u_32.dh_ino); } } static __inline uint16_t lfs_dir_getreclen(const STRUCT_LFS *fs, const LFS_DIRHEADER *dh) { if (fs->lfs_is64) { return LFS_SWAP_uint16_t(fs, dh->u_64.dh_reclen); } else { return LFS_SWAP_uint16_t(fs, dh->u_32.dh_reclen); } } static __inline uint8_t lfs_dir_gettype(const STRUCT_LFS *fs, const LFS_DIRHEADER *dh) { if (fs->lfs_is64) { KASSERT(fs->lfs_hasolddirfmt == 0); return dh->u_64.dh_type; } else if (fs->lfs_hasolddirfmt) { return LFS_DT_UNKNOWN; } else { return dh->u_32.dh_type; } } static __inline uint8_t lfs_dir_getnamlen(const STRUCT_LFS *fs, const LFS_DIRHEADER *dh) { if (fs->lfs_is64) { KASSERT(fs->lfs_hasolddirfmt == 0); return dh->u_64.dh_namlen; } else if (fs->lfs_hasolddirfmt && LFS_LITTLE_ENDIAN_ONDISK(fs)) { /* low-order byte of old 16-bit namlen field */ return dh->u_32.dh_type; } else { return dh->u_32.dh_namlen; } } static __inline void lfs_dir_setino(STRUCT_LFS *fs, LFS_DIRHEADER *dh, uint64_t ino) { if (fs->lfs_is64) { dh->u_64.dh_ino = LFS_SWAP_uint64_t(fs, ino); } else { dh->u_32.dh_ino = LFS_SWAP_uint32_t(fs, ino); } } static __inline void lfs_dir_setreclen(STRUCT_LFS *fs, LFS_DIRHEADER *dh, uint16_t reclen) { if (fs->lfs_is64) { dh->u_64.dh_reclen = LFS_SWAP_uint16_t(fs, reclen); } else { dh->u_32.dh_reclen = LFS_SWAP_uint16_t(fs, reclen); } } static __inline void lfs_dir_settype(const STRUCT_LFS *fs, LFS_DIRHEADER *dh, uint8_t type) { if (fs->lfs_is64) { KASSERT(fs->lfs_hasolddirfmt == 0); dh->u_64.dh_type = type; } else if (fs->lfs_hasolddirfmt) { /* do nothing */ return; } else { dh->u_32.dh_type = type; } } static __inline void lfs_dir_setnamlen(const STRUCT_LFS *fs, LFS_DIRHEADER *dh, uint8_t namlen) { if (fs->lfs_is64) { KASSERT(fs->lfs_hasolddirfmt == 0); dh->u_64.dh_namlen = namlen; } else if (fs->lfs_hasolddirfmt && LFS_LITTLE_ENDIAN_ONDISK(fs)) { /* low-order byte of old 16-bit namlen field */ dh->u_32.dh_type = namlen; } else { dh->u_32.dh_namlen = namlen; } } static __inline void lfs_copydirname(STRUCT_LFS *fs, char *dest, const char *src, unsigned namlen, unsigned reclen) { unsigned spacelen; KASSERT(reclen > LFS_DIRHEADERSIZE(fs)); spacelen = reclen - LFS_DIRHEADERSIZE(fs); /* must always be at least 1 byte as a null terminator */ KASSERT(spacelen > namlen); memcpy(dest, src, namlen); memset(dest + namlen, '\0', spacelen - namlen); } static __inline LFS_DIRHEADER * lfs_dirtemplate_dotdot(STRUCT_LFS *fs, union lfs_dirtemplate *dt) { /* XXX blah, be nice to have a way to do this w/o casts */ if (fs->lfs_is64) { return (LFS_DIRHEADER *)&dt->u_64.dotdot_header; } else { return (LFS_DIRHEADER *)&dt->u_32.dotdot_header; } } static __inline char * lfs_dirtemplate_dotdotname(STRUCT_LFS *fs, union lfs_dirtemplate *dt) { if (fs->lfs_is64) { return dt->u_64.dotdot_name; } else { return dt->u_32.dotdot_name; } } /* * dinodes */ /* * Maximum length of a symlink that can be stored within the inode. */ #define LFS32_MAXSYMLINKLEN ((ULFS_NDADDR + ULFS_NIADDR) * sizeof(int32_t)) #define LFS64_MAXSYMLINKLEN ((ULFS_NDADDR + ULFS_NIADDR) * sizeof(int64_t)) #define LFS_MAXSYMLINKLEN(fs) \ ((fs)->lfs_is64 ? LFS64_MAXSYMLINKLEN : LFS32_MAXSYMLINKLEN) #define DINOSIZE(fs) ((fs)->lfs_is64 ? sizeof(struct lfs64_dinode) : sizeof(struct lfs32_dinode)) #define DINO_IN_BLOCK(fs, base, ix) \ ((union lfs_dinode *)((char *)(base) + DINOSIZE(fs) * (ix))) static __inline void lfs_copy_dinode(STRUCT_LFS *fs, union lfs_dinode *dst, const union lfs_dinode *src) { /* * We can do structure assignment of the structs, but not of * the whole union, as the union is the size of the (larger) * 64-bit struct and on a 32-bit fs the upper half of it might * be off the end of a buffer or otherwise invalid. */ if (fs->lfs_is64) { dst->u_64 = src->u_64; } else { dst->u_32 = src->u_32; } } #define LFS_DEF_DINO_ACCESSOR(type, type32, field) \ static __inline type \ lfs_dino_get##field(STRUCT_LFS *fs, union lfs_dinode *dip) \ { \ if (fs->lfs_is64) { \ return LFS_SWAP_##type(fs, dip->u_64.di_##field); \ } else { \ return LFS_SWAP_##type32(fs, dip->u_32.di_##field); \ } \ } \ static __inline void \ lfs_dino_set##field(STRUCT_LFS *fs, union lfs_dinode *dip, type val) \ { \ if (fs->lfs_is64) { \ type *p = &dip->u_64.di_##field; \ (void)p; \ dip->u_64.di_##field = LFS_SWAP_##type(fs, val); \ } else { \ type32 *p = &dip->u_32.di_##field; \ (void)p; \ dip->u_32.di_##field = LFS_SWAP_##type32(fs, val); \ } \ } \ LFS_DEF_DINO_ACCESSOR(uint16_t, uint16_t, mode) LFS_DEF_DINO_ACCESSOR(int16_t, int16_t, nlink) LFS_DEF_DINO_ACCESSOR(uint64_t, uint32_t, inumber) LFS_DEF_DINO_ACCESSOR(uint64_t, uint64_t, size) LFS_DEF_DINO_ACCESSOR(int64_t, int32_t, atime) LFS_DEF_DINO_ACCESSOR(int32_t, int32_t, atimensec) LFS_DEF_DINO_ACCESSOR(int64_t, int32_t, mtime) LFS_DEF_DINO_ACCESSOR(int32_t, int32_t, mtimensec) LFS_DEF_DINO_ACCESSOR(int64_t, int32_t, ctime) LFS_DEF_DINO_ACCESSOR(int32_t, int32_t, ctimensec) LFS_DEF_DINO_ACCESSOR(uint32_t, uint32_t, flags) LFS_DEF_DINO_ACCESSOR(uint64_t, uint32_t, blocks) LFS_DEF_DINO_ACCESSOR(int32_t, int32_t, gen) LFS_DEF_DINO_ACCESSOR(uint32_t, uint32_t, uid) LFS_DEF_DINO_ACCESSOR(uint32_t, uint32_t, gid) /* XXX this should be done differently (it's a fake field) */ LFS_DEF_DINO_ACCESSOR(int64_t, int32_t, rdev) static __inline daddr_t lfs_dino_getdb(STRUCT_LFS *fs, union lfs_dinode *dip, unsigned ix) { KASSERT(ix < ULFS_NDADDR); if (fs->lfs_is64) { return LFS_SWAP_int64_t(fs, dip->u_64.di_db[ix]); } else { /* note: this must sign-extend or UNWRITTEN gets trashed */ return (int32_t)LFS_SWAP_int32_t(fs, dip->u_32.di_db[ix]); } } static __inline daddr_t lfs_dino_getib(STRUCT_LFS *fs, union lfs_dinode *dip, unsigned ix) { KASSERT(ix < ULFS_NIADDR); if (fs->lfs_is64) { return LFS_SWAP_int64_t(fs, dip->u_64.di_ib[ix]); } else { /* note: this must sign-extend or UNWRITTEN gets trashed */ return (int32_t)LFS_SWAP_int32_t(fs, dip->u_32.di_ib[ix]); } } static __inline void lfs_dino_setdb(STRUCT_LFS *fs, union lfs_dinode *dip, unsigned ix, daddr_t val) { KASSERT(ix < ULFS_NDADDR); if (fs->lfs_is64) { dip->u_64.di_db[ix] = LFS_SWAP_int64_t(fs, val); } else { dip->u_32.di_db[ix] = LFS_SWAP_uint32_t(fs, val); } } static __inline void lfs_dino_setib(STRUCT_LFS *fs, union lfs_dinode *dip, unsigned ix, daddr_t val) { KASSERT(ix < ULFS_NIADDR); if (fs->lfs_is64) { dip->u_64.di_ib[ix] = LFS_SWAP_int64_t(fs, val); } else { dip->u_32.di_ib[ix] = LFS_SWAP_uint32_t(fs, val); } } /* birthtime is present only in the 64-bit inode */ static __inline void lfs_dino_setbirthtime(STRUCT_LFS *fs, union lfs_dinode *dip, const struct timespec *ts) { if (fs->lfs_is64) { dip->u_64.di_birthtime = ts->tv_sec; dip->u_64.di_birthnsec = ts->tv_nsec; } else { /* drop it on the floor */ } } /* * indirect blocks */ static __inline daddr_t lfs_iblock_get(STRUCT_LFS *fs, void *block, unsigned ix) { if (fs->lfs_is64) { // XXX re-enable these asserts after reorging this file //KASSERT(ix < lfs_sb_getbsize(fs) / sizeof(int64_t)); return (daddr_t)(((int64_t *)block)[ix]); } else { //KASSERT(ix < lfs_sb_getbsize(fs) / sizeof(int32_t)); /* must sign-extend or UNWRITTEN gets trashed */ return (daddr_t)(int64_t)(((int32_t *)block)[ix]); } } static __inline void lfs_iblock_set(STRUCT_LFS *fs, void *block, unsigned ix, daddr_t val) { if (fs->lfs_is64) { //KASSERT(ix < lfs_sb_getbsize(fs) / sizeof(int64_t)); ((int64_t *)block)[ix] = val; } else { //KASSERT(ix < lfs_sb_getbsize(fs) / sizeof(int32_t)); ((int32_t *)block)[ix] = val; } } /* * "struct buf" associated definitions */ # define LFS_LOCK_BUF(bp) do { \ if (((bp)->b_flags & B_LOCKED) == 0 && bp->b_iodone == NULL) { \ mutex_enter(&lfs_lock); \ ++locked_queue_count; \ locked_queue_bytes += bp->b_bufsize; \ mutex_exit(&lfs_lock); \ } \ (bp)->b_flags |= B_LOCKED; \ } while (0) # define LFS_UNLOCK_BUF(bp) do { \ if (((bp)->b_flags & B_LOCKED) != 0 && bp->b_iodone == NULL) { \ mutex_enter(&lfs_lock); \ --locked_queue_count; \ locked_queue_bytes -= bp->b_bufsize; \ if (locked_queue_count < LFS_WAIT_BUFS && \ locked_queue_bytes < LFS_WAIT_BYTES) \ cv_broadcast(&locked_queue_cv); \ mutex_exit(&lfs_lock); \ } \ (bp)->b_flags &= ~B_LOCKED; \ } while (0) /* * "struct inode" associated definitions */ #define LFS_SET_UINO(ip, states) do { \ if (((states) & IN_ACCESSED) && !((ip)->i_state & IN_ACCESSED)) \ lfs_sb_adduinodes((ip)->i_lfs, 1); \ if (((states) & IN_CLEANING) && !((ip)->i_state & IN_CLEANING)) \ lfs_sb_adduinodes((ip)->i_lfs, 1); \ if (((states) & IN_MODIFIED) && !((ip)->i_state & IN_MODIFIED)) \ lfs_sb_adduinodes((ip)->i_lfs, 1); \ (ip)->i_state |= (states); \ } while (0) #define LFS_CLR_UINO(ip, states) do { \ if (((states) & IN_ACCESSED) && ((ip)->i_state & IN_ACCESSED)) \ lfs_sb_subuinodes((ip)->i_lfs, 1); \ if (((states) & IN_CLEANING) && ((ip)->i_state & IN_CLEANING)) \ lfs_sb_subuinodes((ip)->i_lfs, 1); \ if (((states) & IN_MODIFIED) && ((ip)->i_state & IN_MODIFIED)) \ lfs_sb_subuinodes((ip)->i_lfs, 1); \ (ip)->i_state &= ~(states); \ if (lfs_sb_getuinodes((ip)->i_lfs) < 0) { \ panic("lfs_uinodes < 0"); \ } \ } while (0) #define LFS_ITIMES(ip, acc, mod, cre) \ while ((ip)->i_state & (IN_ACCESS | IN_CHANGE | IN_UPDATE | IN_MODIFY)) \ lfs_itimes(ip, acc, mod, cre) /* * On-disk and in-memory checkpoint segment usage structure. */ #define SEGUPB(fs) (lfs_sb_getsepb(fs)) #define SEGTABSIZE_SU(fs) \ ((lfs_sb_getnseg(fs) + SEGUPB(fs) - 1) / lfs_sb_getsepb(fs)) #ifdef _KERNEL # define SHARE_IFLOCK(F) \ do { \ rw_enter(&(F)->lfs_iflock, RW_READER); \ } while(0) # define UNSHARE_IFLOCK(F) \ do { \ rw_exit(&(F)->lfs_iflock); \ } while(0) #else /* ! _KERNEL */ # define SHARE_IFLOCK(F) # define UNSHARE_IFLOCK(F) #endif /* ! _KERNEL */ /* Read in the block with a specific segment usage entry from the ifile. */ #define LFS_SEGENTRY(SP, F, IN, BP) do { \ int _e; \ SHARE_IFLOCK(F); \ VTOI((F)->lfs_ivnode)->i_state |= IN_ACCESS; \ if ((_e = bread((F)->lfs_ivnode, \ ((IN) / lfs_sb_getsepb(F)) + lfs_sb_getcleansz(F), \ lfs_sb_getbsize(F), 0, &(BP))) != 0) \ panic("lfs: ifile read: segentry %llu: error %d\n", \ (unsigned long long)(IN), _e); \ if (lfs_sb_getversion(F) == 1) \ (SP) = (SEGUSE *)((SEGUSE_V1 *)(BP)->b_data + \ ((IN) & (lfs_sb_getsepb(F) - 1))); \ else \ (SP) = (SEGUSE *)(BP)->b_data + ((IN) % lfs_sb_getsepb(F)); \ UNSHARE_IFLOCK(F); \ } while (0) #define LFS_WRITESEGENTRY(SP, F, IN, BP) do { \ if ((SP)->su_nbytes == 0) \ (SP)->su_flags |= SEGUSE_EMPTY; \ else \ (SP)->su_flags &= ~SEGUSE_EMPTY; \ (F)->lfs_suflags[(F)->lfs_activesb][(IN)] = (SP)->su_flags; \ LFS_BWRITE_LOG(BP); \ } while (0) /* * FINFO (file info) entries. */ /* Size of an on-disk block pointer, e.g. in an indirect block. */ /* XXX: move to a more suitable location in this file */ #define LFS_BLKPTRSIZE(fs) ((fs)->lfs_is64 ? sizeof(int64_t) : sizeof(int32_t)) /* Size of an on-disk inode number. */ /* XXX: move to a more suitable location in this file */ #define LFS_INUMSIZE(fs) ((fs)->lfs_is64 ? sizeof(int64_t) : sizeof(int32_t)) /* size of a FINFO, without the block pointers */ #define FINFOSIZE(fs) ((fs)->lfs_is64 ? sizeof(FINFO64) : sizeof(FINFO32)) /* Full size of the provided FINFO record, including its block pointers. */ #define FINFO_FULLSIZE(fs, fip) \ (FINFOSIZE(fs) + lfs_fi_getnblocks(fs, fip) * LFS_BLKPTRSIZE(fs)) #define NEXT_FINFO(fs, fip) \ ((FINFO *)((char *)(fip) + FINFO_FULLSIZE(fs, fip))) #define LFS_DEF_FI_ACCESSOR(type, type32, field) \ static __inline type \ lfs_fi_get##field(STRUCT_LFS *fs, FINFO *fip) \ { \ if (fs->lfs_is64) { \ return fip->u_64.fi_##field; \ } else { \ return fip->u_32.fi_##field; \ } \ } \ static __inline void \ lfs_fi_set##field(STRUCT_LFS *fs, FINFO *fip, type val) \ { \ if (fs->lfs_is64) { \ type *p = &fip->u_64.fi_##field; \ (void)p; \ fip->u_64.fi_##field = val; \ } else { \ type32 *p = &fip->u_32.fi_##field; \ (void)p; \ fip->u_32.fi_##field = val; \ } \ } \ LFS_DEF_FI_ACCESSOR(uint32_t, uint32_t, nblocks) LFS_DEF_FI_ACCESSOR(uint32_t, uint32_t, version) LFS_DEF_FI_ACCESSOR(uint64_t, uint32_t, ino) LFS_DEF_FI_ACCESSOR(uint32_t, uint32_t, lastlength) static __inline daddr_t lfs_fi_getblock(STRUCT_LFS *fs, FINFO *fip, unsigned idx) { void *firstblock; firstblock = (char *)fip + FINFOSIZE(fs); KASSERT(idx < lfs_fi_getnblocks(fs, fip)); if (fs->lfs_is64) { return ((int64_t *)firstblock)[idx]; } else { return ((int32_t *)firstblock)[idx]; } } static __inline void lfs_fi_setblock(STRUCT_LFS *fs, FINFO *fip, unsigned idx, daddr_t blk) { void *firstblock; firstblock = (char *)fip + FINFOSIZE(fs); KASSERT(idx < lfs_fi_getnblocks(fs, fip)); if (fs->lfs_is64) { ((int64_t *)firstblock)[idx] = blk; } else { ((int32_t *)firstblock)[idx] = blk; } } /* * inode info entries (in the segment summary) */ #define IINFOSIZE(fs) ((fs)->lfs_is64 ? sizeof(IINFO64) : sizeof(IINFO32)) /* iinfos scroll backward from the end of the segment summary block */ #define SEGSUM_IINFOSTART(fs, buf) \ ((IINFO *)((char *)buf + lfs_sb_getsumsize(fs) - IINFOSIZE(fs))) #define NEXTLOWER_IINFO(fs, iip) \ ((IINFO *)((char *)(iip) - IINFOSIZE(fs))) #define NTH_IINFO(fs, buf, n) \ ((IINFO *)((char *)SEGSUM_IINFOSTART(fs, buf) - (n)*IINFOSIZE(fs))) static __inline uint64_t lfs_ii_getblock(STRUCT_LFS *fs, IINFO *iip) { if (fs->lfs_is64) { return iip->u_64.ii_block; } else { return iip->u_32.ii_block; } } static __inline void lfs_ii_setblock(STRUCT_LFS *fs, IINFO *iip, uint64_t block) { if (fs->lfs_is64) { iip->u_64.ii_block = block; } else { iip->u_32.ii_block = block; } } /* * Index file inode entries. */ #define IFILE_ENTRYSIZE(fs) \ ((fs)->lfs_is64 ? sizeof(IFILE64) : sizeof(IFILE32)) /* * LFSv1 compatibility code is not allowed to touch if_atime, since it * may not be mapped! */ /* Read in the block with a specific inode from the ifile. */ #define LFS_IENTRY(IP, F, IN, BP) do { \ int _e; \ SHARE_IFLOCK(F); \ VTOI((F)->lfs_ivnode)->i_state |= IN_ACCESS; \ if ((_e = bread((F)->lfs_ivnode, \ (IN) / lfs_sb_getifpb(F) + lfs_sb_getcleansz(F) + lfs_sb_getsegtabsz(F), \ lfs_sb_getbsize(F), 0, &(BP))) != 0) \ panic("lfs: ifile ino %d read %d", (int)(IN), _e); \ if ((F)->lfs_is64) { \ (IP) = (IFILE *)((IFILE64 *)(BP)->b_data + \ (IN) % lfs_sb_getifpb(F)); \ } else if (lfs_sb_getversion(F) > 1) { \ (IP) = (IFILE *)((IFILE32 *)(BP)->b_data + \ (IN) % lfs_sb_getifpb(F)); \ } else { \ (IP) = (IFILE *)((IFILE_V1 *)(BP)->b_data + \ (IN) % lfs_sb_getifpb(F)); \ } \ UNSHARE_IFLOCK(F); \ } while (0) #define LFS_IENTRY_NEXT(IP, F) do { \ if ((F)->lfs_is64) { \ (IP) = (IFILE *)((IFILE64 *)(IP) + 1); \ } else if (lfs_sb_getversion(F) > 1) { \ (IP) = (IFILE *)((IFILE32 *)(IP) + 1); \ } else { \ (IP) = (IFILE *)((IFILE_V1 *)(IP) + 1); \ } \ } while (0) #define LFS_DEF_IF_ACCESSOR(type, type32, field) \ static __inline type \ lfs_if_get##field(STRUCT_LFS *fs, IFILE *ifp) \ { \ if (fs->lfs_is64) { \ return ifp->u_64.if_##field; \ } else { \ return ifp->u_32.if_##field; \ } \ } \ static __inline void \ lfs_if_set##field(STRUCT_LFS *fs, IFILE *ifp, type val) \ { \ if (fs->lfs_is64) { \ type *p = &ifp->u_64.if_##field; \ (void)p; \ ifp->u_64.if_##field = val; \ } else { \ type32 *p = &ifp->u_32.if_##field; \ (void)p; \ ifp->u_32.if_##field = val; \ } \ } \ LFS_DEF_IF_ACCESSOR(uint32_t, uint32_t, version) LFS_DEF_IF_ACCESSOR(int64_t, int32_t, daddr) LFS_DEF_IF_ACCESSOR(uint64_t, uint32_t, nextfree) LFS_DEF_IF_ACCESSOR(uint64_t, uint32_t, atime_sec) LFS_DEF_IF_ACCESSOR(uint32_t, uint32_t, atime_nsec) /* * Cleaner information structure. This resides in the ifile and is used * to pass information from the kernel to the cleaner. */ #define CLEANSIZE_SU(fs) \ ((((fs)->lfs_is64 ? sizeof(CLEANERINFO64) : sizeof(CLEANERINFO32)) + \ lfs_sb_getbsize(fs) - 1) >> lfs_sb_getbshift(fs)) #define LFS_DEF_CI_ACCESSOR(type, type32, field) \ static __inline type \ lfs_ci_get##field(STRUCT_LFS *fs, CLEANERINFO *cip) \ { \ if (fs->lfs_is64) { \ return cip->u_64.field; \ } else { \ return cip->u_32.field; \ } \ } \ static __inline void \ lfs_ci_set##field(STRUCT_LFS *fs, CLEANERINFO *cip, type val) \ { \ if (fs->lfs_is64) { \ type *p = &cip->u_64.field; \ (void)p; \ cip->u_64.field = val; \ } else { \ type32 *p = &cip->u_32.field; \ (void)p; \ cip->u_32.field = val; \ } \ } \ LFS_DEF_CI_ACCESSOR(uint32_t, uint32_t, clean) LFS_DEF_CI_ACCESSOR(uint32_t, uint32_t, dirty) LFS_DEF_CI_ACCESSOR(int64_t, int32_t, bfree) LFS_DEF_CI_ACCESSOR(int64_t, int32_t, avail) LFS_DEF_CI_ACCESSOR(uint64_t, uint32_t, free_head) LFS_DEF_CI_ACCESSOR(uint64_t, uint32_t, free_tail) LFS_DEF_CI_ACCESSOR(uint32_t, uint32_t, flags) static __inline void lfs_ci_shiftcleantodirty(STRUCT_LFS *fs, CLEANERINFO *cip, unsigned num) { lfs_ci_setclean(fs, cip, lfs_ci_getclean(fs, cip) - num); lfs_ci_setdirty(fs, cip, lfs_ci_getdirty(fs, cip) + num); } static __inline void lfs_ci_shiftdirtytoclean(STRUCT_LFS *fs, CLEANERINFO *cip, unsigned num) { lfs_ci_setdirty(fs, cip, lfs_ci_getdirty(fs, cip) - num); lfs_ci_setclean(fs, cip, lfs_ci_getclean(fs, cip) + num); } /* Read in the block with the cleaner info from the ifile. */ #define LFS_CLEANERINFO(CP, F, BP) do { \ int _e; \ SHARE_IFLOCK(F); \ VTOI((F)->lfs_ivnode)->i_state |= IN_ACCESS; \ _e = bread((F)->lfs_ivnode, \ (daddr_t)0, lfs_sb_getbsize(F), 0, &(BP)); \ if (_e) \ panic("lfs: ifile read: cleanerinfo: error %d\n", _e); \ (CP) = (CLEANERINFO *)(BP)->b_data; \ UNSHARE_IFLOCK(F); \ } while (0) /* * Synchronize the Ifile cleaner info with current avail and bfree. */ #define LFS_SYNC_CLEANERINFO(cip, fs, bp, w) do { \ mutex_enter(&lfs_lock); \ if ((w) || lfs_ci_getbfree(fs, cip) != lfs_sb_getbfree(fs) || \ lfs_ci_getavail(fs, cip) != lfs_sb_getavail(fs) - fs->lfs_ravail - \ fs->lfs_favail) { \ lfs_ci_setbfree(fs, cip, lfs_sb_getbfree(fs)); \ lfs_ci_setavail(fs, cip, lfs_sb_getavail(fs) - fs->lfs_ravail - \ fs->lfs_favail); \ if (((bp)->b_flags & B_GATHERED) == 0) { \ fs->lfs_flags |= LFS_IFDIRTY; \ } \ mutex_exit(&lfs_lock); \ (void) LFS_BWRITE_LOG(bp); /* Ifile */ \ } else { \ mutex_exit(&lfs_lock); \ brelse(bp, 0); \ } \ } while (0) /* * Get the head of the inode free list. * Always called with the segment lock held. */ #define LFS_GET_HEADFREE(FS, CIP, BP, FREEP) do { \ if (lfs_sb_getversion(FS) > 1) { \ LFS_CLEANERINFO((CIP), (FS), (BP)); \ lfs_sb_setfreehd(FS, lfs_ci_getfree_head(FS, CIP)); \ brelse(BP, 0); \ } \ *(FREEP) = lfs_sb_getfreehd(FS); \ } while (0) #define LFS_PUT_HEADFREE(FS, CIP, BP, VAL) do { \ lfs_sb_setfreehd(FS, VAL); \ if (lfs_sb_getversion(FS) > 1) { \ LFS_CLEANERINFO((CIP), (FS), (BP)); \ lfs_ci_setfree_head(FS, CIP, VAL); \ LFS_BWRITE_LOG(BP); \ mutex_enter(&lfs_lock); \ (FS)->lfs_flags |= LFS_IFDIRTY; \ mutex_exit(&lfs_lock); \ } \ } while (0) #define LFS_GET_TAILFREE(FS, CIP, BP, FREEP) do { \ LFS_CLEANERINFO((CIP), (FS), (BP)); \ *(FREEP) = lfs_ci_getfree_tail(FS, CIP); \ brelse(BP, 0); \ } while (0) #define LFS_PUT_TAILFREE(FS, CIP, BP, VAL) do { \ LFS_CLEANERINFO((CIP), (FS), (BP)); \ lfs_ci_setfree_tail(FS, CIP, VAL); \ LFS_BWRITE_LOG(BP); \ mutex_enter(&lfs_lock); \ (FS)->lfs_flags |= LFS_IFDIRTY; \ mutex_exit(&lfs_lock); \ } while (0) /* * On-disk segment summary information */ #define SEGSUM_SIZE(fs) \ (fs->lfs_is64 ? sizeof(SEGSUM64) : \ lfs_sb_getversion(fs) > 1 ? sizeof(SEGSUM32) : sizeof(SEGSUM_V1)) /* * The SEGSUM structure is followed by FINFO structures. Get the pointer * to the first FINFO. * * XXX this can't be a macro yet; this file needs to be resorted. */ #if 0 static __inline FINFO * segsum_finfobase(STRUCT_LFS *fs, SEGSUM *ssp) { return (FINFO *)((char *)ssp + SEGSUM_SIZE(fs)); } #else #define SEGSUM_FINFOBASE(fs, ssp) \ ((FINFO *)((char *)(ssp) + SEGSUM_SIZE(fs))); #endif #define LFS_DEF_SS_ACCESSOR(type, type32, field) \ static __inline type \ lfs_ss_get##field(STRUCT_LFS *fs, SEGSUM *ssp) \ { \ if (fs->lfs_is64) { \ return ssp->u_64.ss_##field; \ } else { \ return ssp->u_32.ss_##field; \ } \ } \ static __inline void \ lfs_ss_set##field(STRUCT_LFS *fs, SEGSUM *ssp, type val) \ { \ if (fs->lfs_is64) { \ type *p = &ssp->u_64.ss_##field; \ (void)p; \ ssp->u_64.ss_##field = val; \ } else { \ type32 *p = &ssp->u_32.ss_##field; \ (void)p; \ ssp->u_32.ss_##field = val; \ } \ } \ LFS_DEF_SS_ACCESSOR(uint32_t, uint32_t, sumsum) LFS_DEF_SS_ACCESSOR(uint32_t, uint32_t, datasum) LFS_DEF_SS_ACCESSOR(uint32_t, uint32_t, magic) LFS_DEF_SS_ACCESSOR(uint32_t, uint32_t, ident) LFS_DEF_SS_ACCESSOR(int64_t, int32_t, next) LFS_DEF_SS_ACCESSOR(uint16_t, uint16_t, nfinfo) LFS_DEF_SS_ACCESSOR(uint16_t, uint16_t, ninos) LFS_DEF_SS_ACCESSOR(uint16_t, uint16_t, flags) LFS_DEF_SS_ACCESSOR(uint64_t, uint32_t, reclino) LFS_DEF_SS_ACCESSOR(uint64_t, uint64_t, serial) LFS_DEF_SS_ACCESSOR(uint64_t, uint64_t, create) static __inline size_t lfs_ss_getsumstart(STRUCT_LFS *fs) { /* These are actually all the same. */ if (fs->lfs_is64) { return offsetof(SEGSUM64, ss_datasum); } else /* if (lfs_sb_getversion(fs) > 1) */ { return offsetof(SEGSUM32, ss_datasum); } /* else { return offsetof(SEGSUM_V1, ss_datasum); } */ /* * XXX ^^^ until this file is resorted lfs_sb_getversion isn't * defined yet. */ } static __inline uint32_t lfs_ss_getocreate(STRUCT_LFS *fs, SEGSUM *ssp) { KASSERT(fs->lfs_is64 == 0); /* XXX need to resort this file before we can do this */ //KASSERT(lfs_sb_getversion(fs) == 1); return ssp->u_v1.ss_create; } static __inline void lfs_ss_setocreate(STRUCT_LFS *fs, SEGSUM *ssp, uint32_t val) { KASSERT(fs->lfs_is64 == 0); /* XXX need to resort this file before we can do this */ //KASSERT(lfs_sb_getversion(fs) == 1); ssp->u_v1.ss_create = val; } /* * Super block. */ /* * Generate accessors for the on-disk superblock fields with cpp. */ #define LFS_DEF_SB_ACCESSOR_FULL(type, type32, field) \ static __inline type \ lfs_sb_get##field(STRUCT_LFS *fs) \ { \ if (fs->lfs_is64) { \ return fs->lfs_dlfs_u.u_64.dlfs_##field; \ } else { \ return fs->lfs_dlfs_u.u_32.dlfs_##field; \ } \ } \ static __inline void \ lfs_sb_set##field(STRUCT_LFS *fs, type val) \ { \ if (fs->lfs_is64) { \ fs->lfs_dlfs_u.u_64.dlfs_##field = val; \ } else { \ fs->lfs_dlfs_u.u_32.dlfs_##field = val; \ } \ } \ static __inline void \ lfs_sb_add##field(STRUCT_LFS *fs, type val) \ { \ if (fs->lfs_is64) { \ type *p64 = &fs->lfs_dlfs_u.u_64.dlfs_##field; \ *p64 += val; \ } else { \ type32 *p32 = &fs->lfs_dlfs_u.u_32.dlfs_##field; \ *p32 += val; \ } \ } \ static __inline void \ lfs_sb_sub##field(STRUCT_LFS *fs, type val) \ { \ if (fs->lfs_is64) { \ type *p64 = &fs->lfs_dlfs_u.u_64.dlfs_##field; \ *p64 -= val; \ } else { \ type32 *p32 = &fs->lfs_dlfs_u.u_32.dlfs_##field; \ *p32 -= val; \ } \ } #define LFS_DEF_SB_ACCESSOR(t, f) LFS_DEF_SB_ACCESSOR_FULL(t, t, f) #define LFS_DEF_SB_ACCESSOR_32ONLY(type, field, val64) \ static __inline type \ lfs_sb_get##field(STRUCT_LFS *fs) \ { \ if (fs->lfs_is64) { \ return val64; \ } else { \ return fs->lfs_dlfs_u.u_32.dlfs_##field; \ } \ } LFS_DEF_SB_ACCESSOR(uint32_t, version) LFS_DEF_SB_ACCESSOR_FULL(uint64_t, uint32_t, size) LFS_DEF_SB_ACCESSOR(uint32_t, ssize) LFS_DEF_SB_ACCESSOR_FULL(uint64_t, uint32_t, dsize) LFS_DEF_SB_ACCESSOR(uint32_t, bsize) LFS_DEF_SB_ACCESSOR(uint32_t, fsize) LFS_DEF_SB_ACCESSOR(uint32_t, frag) LFS_DEF_SB_ACCESSOR_FULL(uint64_t, uint32_t, freehd) LFS_DEF_SB_ACCESSOR_FULL(int64_t, int32_t, bfree) LFS_DEF_SB_ACCESSOR_FULL(uint64_t, uint32_t, nfiles) LFS_DEF_SB_ACCESSOR_FULL(int64_t, int32_t, avail) LFS_DEF_SB_ACCESSOR(int32_t, uinodes) LFS_DEF_SB_ACCESSOR_FULL(int64_t, int32_t, idaddr) LFS_DEF_SB_ACCESSOR_32ONLY(uint32_t, ifile, LFS_IFILE_INUM) LFS_DEF_SB_ACCESSOR_FULL(int64_t, int32_t, lastseg) LFS_DEF_SB_ACCESSOR_FULL(int64_t, int32_t, nextseg) LFS_DEF_SB_ACCESSOR_FULL(int64_t, int32_t, curseg) LFS_DEF_SB_ACCESSOR_FULL(int64_t, int32_t, offset) LFS_DEF_SB_ACCESSOR_FULL(int64_t, int32_t, lastpseg) LFS_DEF_SB_ACCESSOR(uint32_t, inopf) LFS_DEF_SB_ACCESSOR(uint32_t, minfree) LFS_DEF_SB_ACCESSOR(uint64_t, maxfilesize) LFS_DEF_SB_ACCESSOR(uint32_t, fsbpseg) LFS_DEF_SB_ACCESSOR(uint32_t, inopb) LFS_DEF_SB_ACCESSOR(uint32_t, ifpb) LFS_DEF_SB_ACCESSOR(uint32_t, sepb) LFS_DEF_SB_ACCESSOR(uint32_t, nindir) LFS_DEF_SB_ACCESSOR(uint32_t, nseg) LFS_DEF_SB_ACCESSOR(uint32_t, nspf) LFS_DEF_SB_ACCESSOR(uint32_t, cleansz) LFS_DEF_SB_ACCESSOR(uint32_t, segtabsz) LFS_DEF_SB_ACCESSOR_32ONLY(uint32_t, segmask, 0) LFS_DEF_SB_ACCESSOR_32ONLY(uint32_t, segshift, 0) LFS_DEF_SB_ACCESSOR(uint64_t, bmask) LFS_DEF_SB_ACCESSOR(uint32_t, bshift) LFS_DEF_SB_ACCESSOR(uint64_t, ffmask) LFS_DEF_SB_ACCESSOR(uint32_t, ffshift) LFS_DEF_SB_ACCESSOR(uint64_t, fbmask) LFS_DEF_SB_ACCESSOR(uint32_t, fbshift) LFS_DEF_SB_ACCESSOR(uint32_t, blktodb) LFS_DEF_SB_ACCESSOR(uint32_t, fsbtodb) LFS_DEF_SB_ACCESSOR(uint32_t, sushift) LFS_DEF_SB_ACCESSOR(int32_t, maxsymlinklen) LFS_DEF_SB_ACCESSOR(uint32_t, cksum) LFS_DEF_SB_ACCESSOR(uint16_t, pflags) LFS_DEF_SB_ACCESSOR(uint32_t, nclean) LFS_DEF_SB_ACCESSOR(int32_t, dmeta) LFS_DEF_SB_ACCESSOR(uint32_t, minfreeseg) LFS_DEF_SB_ACCESSOR(uint32_t, sumsize) LFS_DEF_SB_ACCESSOR(uint64_t, serial) LFS_DEF_SB_ACCESSOR(uint32_t, ibsize) LFS_DEF_SB_ACCESSOR_FULL(int64_t, int32_t, s0addr) LFS_DEF_SB_ACCESSOR(uint64_t, tstamp) LFS_DEF_SB_ACCESSOR(uint32_t, inodefmt) LFS_DEF_SB_ACCESSOR(uint32_t, interleave) LFS_DEF_SB_ACCESSOR(uint32_t, ident) LFS_DEF_SB_ACCESSOR(uint32_t, resvseg) /* special-case accessors */ /* * the v1 otstamp field lives in what's now dlfs_inopf */ #define lfs_sb_getotstamp(fs) lfs_sb_getinopf(fs) #define lfs_sb_setotstamp(fs, val) lfs_sb_setinopf(fs, val) /* * lfs_sboffs is an array */ static __inline int32_t lfs_sb_getsboff(STRUCT_LFS *fs, unsigned n) { #ifdef KASSERT /* ugh */ KASSERT(n < LFS_MAXNUMSB); #endif if (fs->lfs_is64) { return fs->lfs_dlfs_u.u_64.dlfs_sboffs[n]; } else { return fs->lfs_dlfs_u.u_32.dlfs_sboffs[n]; } } static __inline void lfs_sb_setsboff(STRUCT_LFS *fs, unsigned n, int32_t val) { #ifdef KASSERT /* ugh */ KASSERT(n < LFS_MAXNUMSB); #endif if (fs->lfs_is64) { fs->lfs_dlfs_u.u_64.dlfs_sboffs[n] = val; } else { fs->lfs_dlfs_u.u_32.dlfs_sboffs[n] = val; } } /* * lfs_fsmnt is a string */ static __inline const char * lfs_sb_getfsmnt(STRUCT_LFS *fs) { if (fs->lfs_is64) { return (const char *)fs->lfs_dlfs_u.u_64.dlfs_fsmnt; } else { return (const char *)fs->lfs_dlfs_u.u_32.dlfs_fsmnt; } } static __inline void lfs_sb_setfsmnt(STRUCT_LFS *fs, const char *str) { if (fs->lfs_is64) { (void)strncpy((char *)fs->lfs_dlfs_u.u_64.dlfs_fsmnt, str, sizeof(fs->lfs_dlfs_u.u_64.dlfs_fsmnt)); } else { (void)strncpy((char *)fs->lfs_dlfs_u.u_32.dlfs_fsmnt, str, sizeof(fs->lfs_dlfs_u.u_32.dlfs_fsmnt)); } } /* Highest addressable fsb */ #define LFS_MAX_DADDR(fs) \ ((fs)->lfs_is64 ? 0x7fffffffffffffff : 0x7fffffff) /* LFS_NINDIR is the number of indirects in a file system block. */ #define LFS_NINDIR(fs) (lfs_sb_getnindir(fs)) /* LFS_INOPB is the number of inodes in a secondary storage block. */ #define LFS_INOPB(fs) (lfs_sb_getinopb(fs)) /* LFS_INOPF is the number of inodes in a fragment. */ #define LFS_INOPF(fs) (lfs_sb_getinopf(fs)) #define lfs_blkoff(fs, loc) ((int)((loc) & lfs_sb_getbmask(fs))) #define lfs_fragoff(fs, loc) /* calculates (loc % fs->lfs_fsize) */ \ ((int)((loc) & lfs_sb_getffmask(fs))) /* XXX: lowercase these as they're no longer macros */ /* Frags to diskblocks */ static __inline uint64_t LFS_FSBTODB(STRUCT_LFS *fs, uint64_t b) { #if defined(_KERNEL) return b << (lfs_sb_getffshift(fs) - DEV_BSHIFT); #else return b << lfs_sb_getfsbtodb(fs); #endif } /* Diskblocks to frags */ static __inline uint64_t LFS_DBTOFSB(STRUCT_LFS *fs, uint64_t b) { #if defined(_KERNEL) return b >> (lfs_sb_getffshift(fs) - DEV_BSHIFT); #else return b >> lfs_sb_getfsbtodb(fs); #endif } #define lfs_lblkno(fs, loc) ((loc) >> lfs_sb_getbshift(fs)) #define lfs_lblktosize(fs, blk) ((blk) << lfs_sb_getbshift(fs)) /* Frags to bytes */ static __inline uint64_t lfs_fsbtob(STRUCT_LFS *fs, uint64_t b) { return b << lfs_sb_getffshift(fs); } /* Bytes to frags */ static __inline uint64_t lfs_btofsb(STRUCT_LFS *fs, uint64_t b) { return b >> lfs_sb_getffshift(fs); } #define lfs_numfrags(fs, loc) /* calculates (loc / fs->lfs_fsize) */ \ ((loc) >> lfs_sb_getffshift(fs)) #define lfs_blkroundup(fs, size)/* calculates roundup(size, lfs_sb_getbsize(fs)) */ \ ((off_t)(((size) + lfs_sb_getbmask(fs)) & (~lfs_sb_getbmask(fs)))) #define lfs_fragroundup(fs, size)/* calculates roundup(size, fs->lfs_fsize) */ \ ((off_t)(((size) + lfs_sb_getffmask(fs)) & (~lfs_sb_getffmask(fs)))) #define lfs_fragstoblks(fs, frags)/* calculates (frags / fs->fs_frag) */ \ ((frags) >> lfs_sb_getfbshift(fs)) #define lfs_blkstofrags(fs, blks)/* calculates (blks * fs->fs_frag) */ \ ((blks) << lfs_sb_getfbshift(fs)) #define lfs_fragnum(fs, fsb) /* calculates (fsb % fs->lfs_frag) */ \ ((fsb) & ((fs)->lfs_frag - 1)) #define lfs_blknum(fs, fsb) /* calculates rounddown(fsb, fs->lfs_frag) */ \ ((fsb) &~ ((fs)->lfs_frag - 1)) #define lfs_dblksize(fs, dp, lbn) \ (((lbn) >= ULFS_NDADDR || lfs_dino_getsize(fs, dp) >= ((lbn) + 1) << lfs_sb_getbshift(fs)) \ ? lfs_sb_getbsize(fs) \ : (lfs_fragroundup(fs, lfs_blkoff(fs, lfs_dino_getsize(fs, dp))))) #define lfs_segsize(fs) (lfs_sb_getversion(fs) == 1 ? \ lfs_lblktosize((fs), lfs_sb_getssize(fs)) : \ lfs_sb_getssize(fs)) /* XXX segtod produces a result in frags despite the 'd' */ #define lfs_segtod(fs, seg) (lfs_btofsb(fs, lfs_segsize(fs)) * (seg)) #define lfs_dtosn(fs, daddr) /* block address to segment number */ \ ((uint32_t)(((daddr) - lfs_sb_gets0addr(fs)) / lfs_segtod((fs), 1))) #define lfs_sntod(fs, sn) /* segment number to disk address */ \ ((daddr_t)(lfs_segtod((fs), (sn)) + lfs_sb_gets0addr(fs))) /* XXX, blah. make this appear only if struct inode is defined */ #ifdef _UFS_LFS_LFS_INODE_H_ static __inline uint32_t lfs_blksize(STRUCT_LFS *fs, struct inode *ip, uint64_t lbn) { if (lbn >= ULFS_NDADDR || lfs_dino_getsize(fs, ip->i_din) >= (lbn + 1) << lfs_sb_getbshift(fs)) { return lfs_sb_getbsize(fs); } else { return lfs_fragroundup(fs, lfs_blkoff(fs, lfs_dino_getsize(fs, ip->i_din))); } } #endif /* * union lfs_blocks */ static __inline void lfs_blocks_fromvoid(STRUCT_LFS *fs, union lfs_blocks *bp, void *p) { if (fs->lfs_is64) { bp->b64 = p; } else { bp->b32 = p; } } static __inline void lfs_blocks_fromfinfo(STRUCT_LFS *fs, union lfs_blocks *bp, FINFO *fip) { void *firstblock; firstblock = (char *)fip + FINFOSIZE(fs); if (fs->lfs_is64) { bp->b64 = (int64_t *)firstblock; } else { bp->b32 = (int32_t *)firstblock; } } static __inline daddr_t lfs_blocks_get(STRUCT_LFS *fs, union lfs_blocks *bp, unsigned idx) { if (fs->lfs_is64) { return bp->b64[idx]; } else { return bp->b32[idx]; } } static __inline void lfs_blocks_set(STRUCT_LFS *fs, union lfs_blocks *bp, unsigned idx, daddr_t val) { if (fs->lfs_is64) { bp->b64[idx] = val; } else { bp->b32[idx] = val; } } static __inline void lfs_blocks_inc(STRUCT_LFS *fs, union lfs_blocks *bp) { if (fs->lfs_is64) { bp->b64++; } else { bp->b32++; } } static __inline int lfs_blocks_eq(STRUCT_LFS *fs, union lfs_blocks *bp1, union lfs_blocks *bp2) { if (fs->lfs_is64) { return bp1->b64 == bp2->b64; } else { return bp1->b32 == bp2->b32; } } static __inline int lfs_blocks_sub(STRUCT_LFS *fs, union lfs_blocks *bp1, union lfs_blocks *bp2) { /* (remember that the pointers are typed) */ if (fs->lfs_is64) { return bp1->b64 - bp2->b64; } else { return bp1->b32 - bp2->b32; } } /* * struct segment */ /* * Macros for determining free space on the disk, with the variable metadata * of segment summaries and inode blocks taken into account. */ /* * Estimate number of clean blocks not available for writing because * they will contain metadata or overhead. This is calculated as * * E = ((C * M / D) * D + (0) * (T - D)) / T * or more simply * E = (C * M) / T * * where * C is the clean space, * D is the dirty space, * M is the dirty metadata, and * T = C + D is the total space on disk. * * This approximates the old formula of E = C * M / D when D is close to T, * but avoids falsely reporting "disk full" when the sample size (D) is small. */ #define LFS_EST_CMETA(F) (( \ (lfs_sb_getdmeta(F) * (int64_t)lfs_sb_getnclean(F)) / \ (lfs_sb_getnseg(F)))) /* Estimate total size of the disk not including metadata */ #define LFS_EST_NONMETA(F) (lfs_sb_getdsize(F) - lfs_sb_getdmeta(F) - LFS_EST_CMETA(F)) /* Estimate number of blocks actually available for writing */ #define LFS_EST_BFREE(F) (lfs_sb_getbfree(F) > LFS_EST_CMETA(F) ? \ lfs_sb_getbfree(F) - LFS_EST_CMETA(F) : 0) /* Amount of non-meta space not available to mortal man */ #define LFS_EST_RSVD(F) ((LFS_EST_NONMETA(F) * \ (uint64_t)lfs_sb_getminfree(F)) / \ 100) /* Can credential C write BB blocks? XXX: kauth_cred_geteuid is abusive */ #define ISSPACE(F, BB, C) \ ((((C) == NOCRED || kauth_cred_geteuid(C) == 0) && \ LFS_EST_BFREE(F) >= (BB)) || \ (kauth_cred_geteuid(C) != 0 && IS_FREESPACE(F, BB))) /* Can an ordinary user write BB blocks */ #define IS_FREESPACE(F, BB) \ (LFS_EST_BFREE(F) >= (BB) + LFS_EST_RSVD(F)) /* * The minimum number of blocks to create a new inode. This is: * directory direct block (1) + ULFS_NIADDR indirect blocks + inode block (1) + * ifile direct block (1) + ULFS_NIADDR indirect blocks = 3 + 2 * ULFS_NIADDR blocks. */ #define LFS_NRESERVE(F) (lfs_btofsb((F), (2 * ULFS_NIADDR + 3) << lfs_sb_getbshift(F))) /* * Suppress spurious clang warnings */ #ifdef __GNUC__ #if defined(__clang__) #pragma clang diagnostic pop #elif __GNUC_PREREQ__(9,0) #pragma GCC diagnostic pop #endif #endif #endif /* _UFS_LFS_LFS_ACCESSORS_H_ */ |
| 2 1 1 8 1 1 1 1 1 207 207 1 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 197 198 199 200 201 202 203 204 205 206 207 208 209 210 211 212 213 214 215 216 217 218 219 220 221 222 223 224 225 226 227 228 229 230 231 232 233 234 235 236 237 238 239 240 241 242 243 244 245 246 247 248 249 250 251 252 253 254 255 256 257 258 259 260 261 262 263 264 265 266 267 268 269 270 271 272 273 274 275 276 277 278 279 280 281 282 283 284 285 286 287 288 289 290 291 292 293 294 295 296 297 298 299 300 301 302 303 304 305 306 307 308 309 310 311 312 313 314 315 316 317 318 319 320 321 322 323 324 325 326 327 328 329 330 331 332 333 334 335 336 337 338 339 340 341 342 343 344 345 346 347 348 349 350 351 352 353 354 355 356 357 358 359 360 361 362 363 364 365 366 367 368 369 370 371 372 373 374 375 376 377 378 379 380 381 382 383 384 385 386 387 388 389 390 391 392 393 394 395 396 397 398 399 400 401 402 403 404 405 406 407 408 409 410 411 412 413 414 415 416 417 418 419 420 421 422 423 424 425 426 427 428 429 430 431 432 433 434 435 436 437 438 439 440 441 442 443 444 445 446 447 448 449 450 451 452 453 454 455 456 457 458 459 460 461 462 463 464 465 466 467 468 469 470 471 472 473 474 475 476 477 478 479 480 481 482 483 484 485 486 487 488 489 490 491 492 493 494 495 496 497 498 499 500 501 502 503 504 505 506 507 508 509 510 511 512 513 514 515 516 517 518 519 520 521 522 523 524 525 526 527 528 529 530 531 532 533 534 535 536 537 538 539 540 541 542 543 544 545 546 547 548 549 550 551 552 553 554 555 556 557 558 559 560 561 562 563 564 565 566 567 568 569 570 571 572 573 | /* $NetBSD: sysmon_wdog.c,v 1.30 2021/12/31 11:05:41 riastradh Exp $ */ /*- * Copyright (c) 2000 Zembu Labs, Inc. * All rights reserved. * * Author: Jason R. Thorpe <thorpej@zembu.com> * * Redistribution and use in source and binary forms, with or without * modification, are permitted provided that the following conditions * are met: * 1. Redistributions of source code must retain the above copyright * notice, this list of conditions and the following disclaimer. * 2. Redistributions in binary form must reproduce the above copyright * notice, this list of conditions and the following disclaimer in the * documentation and/or other materials provided with the distribution. * 3. All advertising materials mentioning features or use of this software * must display the following acknowledgement: * This product includes software developed by Zembu Labs, Inc. * 4. Neither the name of Zembu Labs nor the names of its employees may * be used to endorse or promote products derived from this software * without specific prior written permission. * * THIS SOFTWARE IS PROVIDED BY ZEMBU LABS, INC. ``AS IS'' AND ANY EXPRESS * OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WAR- * RANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DIS- * CLAIMED. IN NO EVENT SHALL ZEMBU LABS BE LIABLE FOR ANY DIRECT, INDIRECT, * INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT * NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, * DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY * THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT * (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF * THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. */ /* * Watchdog timer framework for sysmon. Hardware (and software) * watchdog timers can register themselves here to provide a * watchdog function, which provides an abstract interface to the * user. */ #include <sys/cdefs.h> __KERNEL_RCSID(0, "$NetBSD: sysmon_wdog.c,v 1.30 2021/12/31 11:05:41 riastradh Exp $"); #include <sys/param.h> #include <sys/conf.h> #include <sys/errno.h> #include <sys/fcntl.h> #include <sys/condvar.h> #include <sys/mutex.h> #include <sys/callout.h> #include <sys/kernel.h> #include <sys/systm.h> #include <sys/proc.h> #include <sys/module.h> #include <sys/once.h> #include <dev/sysmon/sysmonvar.h> static LIST_HEAD(, sysmon_wdog) sysmon_wdog_list = LIST_HEAD_INITIALIZER(&sysmon_wdog_list); static int sysmon_wdog_count; static kmutex_t sysmon_wdog_list_mtx, sysmon_wdog_mtx; static kcondvar_t sysmon_wdog_cv; static struct sysmon_wdog *sysmon_armed_wdog; static callout_t sysmon_wdog_callout; static void *sysmon_wdog_sdhook; static void *sysmon_wdog_cphook; struct sysmon_wdog *sysmon_wdog_find(const char *); void sysmon_wdog_release(struct sysmon_wdog *); int sysmon_wdog_setmode(struct sysmon_wdog *, int, u_int); void sysmon_wdog_ktickle(void *); void sysmon_wdog_critpoll(void *); void sysmon_wdog_shutdown(void *); void sysmon_wdog_ref(struct sysmon_wdog *); static struct sysmon_opvec sysmon_wdog_opvec = { sysmonopen_wdog, sysmonclose_wdog, sysmonioctl_wdog, NULL, NULL, NULL }; MODULE(MODULE_CLASS_DRIVER, sysmon_wdog, "sysmon"); ONCE_DECL(once_wdog); static int wdog_preinit(void) { mutex_init(&sysmon_wdog_list_mtx, MUTEX_DEFAULT, IPL_NONE); mutex_init(&sysmon_wdog_mtx, MUTEX_DEFAULT, IPL_SOFTCLOCK); cv_init(&sysmon_wdog_cv, "wdogref"); callout_init(&sysmon_wdog_callout, 0); return 0; } int sysmon_wdog_init(void) { int error; (void)RUN_ONCE(&once_wdog, wdog_preinit); sysmon_wdog_sdhook = shutdownhook_establish(sysmon_wdog_shutdown, NULL); if (sysmon_wdog_sdhook == NULL) printf("WARNING: unable to register watchdog shutdown hook\n"); sysmon_wdog_cphook = critpollhook_establish(sysmon_wdog_critpoll, NULL); if (sysmon_wdog_cphook == NULL) printf("WARNING: unable to register watchdog critpoll hook\n"); error = sysmon_attach_minor(SYSMON_MINOR_WDOG, &sysmon_wdog_opvec); return error; } int sysmon_wdog_fini(void) { int error; if ( ! LIST_EMPTY(&sysmon_wdog_list)) return EBUSY; error = sysmon_attach_minor(SYSMON_MINOR_WDOG, NULL); if (error == 0) { callout_destroy(&sysmon_wdog_callout); critpollhook_disestablish(sysmon_wdog_cphook); shutdownhook_disestablish(sysmon_wdog_sdhook); cv_destroy(&sysmon_wdog_cv); mutex_destroy(&sysmon_wdog_mtx); mutex_destroy(&sysmon_wdog_list_mtx); } return error; } /* * sysmonopen_wdog: * * Open the system monitor device. */ int sysmonopen_wdog(dev_t dev, int flag, int mode, struct lwp *l) { return 0; } /* * sysmonclose_wdog: * * Close the system monitor device. */ int sysmonclose_wdog(dev_t dev, int flag, int mode, struct lwp *l) { struct sysmon_wdog *smw; int error = 0; /* * If this is the last close, and there is a watchdog * running in UTICKLE mode, we need to disable it, * otherwise the system will reset in short order. * * XXX Maybe we should just go into KTICKLE mode? */ mutex_enter(&sysmon_wdog_mtx); if ((smw = sysmon_armed_wdog) != NULL) { if ((smw->smw_mode & WDOG_MODE_MASK) == WDOG_MODE_UTICKLE) { error = sysmon_wdog_setmode(smw, WDOG_MODE_DISARMED, smw->smw_period); if (error) { printf("WARNING: UNABLE TO DISARM " "WATCHDOG %s ON CLOSE!\n", smw->smw_name); /* * ...we will probably reboot soon. */ } } } mutex_exit(&sysmon_wdog_mtx); return error; } /* * sysmonioctl_wdog: * * Perform a watchdog control request. */ int sysmonioctl_wdog(dev_t dev, u_long cmd, void *data, int flag, struct lwp *l) { struct sysmon_wdog *smw; int error = 0; switch (cmd) { case WDOGIOC_GMODE: { struct wdog_mode *wm = (void *) data; wm->wm_name[sizeof(wm->wm_name) - 1] = '\0'; smw = sysmon_wdog_find(wm->wm_name); if (smw == NULL) { error = ESRCH; break; } wm->wm_mode = smw->smw_mode; wm->wm_period = smw->smw_period; sysmon_wdog_release(smw); break; } case WDOGIOC_SMODE: { struct wdog_mode *wm = (void *) data; if ((flag & FWRITE) == 0) { error = EPERM; break; } wm->wm_name[sizeof(wm->wm_name) - 1] = '\0'; smw = sysmon_wdog_find(wm->wm_name); if (smw == NULL) { error = ESRCH; break; } if (wm->wm_mode & ~(WDOG_MODE_MASK|WDOG_FEATURE_MASK)) error = EINVAL; else { mutex_enter(&sysmon_wdog_mtx); error = sysmon_wdog_setmode(smw, wm->wm_mode, wm->wm_period); mutex_exit(&sysmon_wdog_mtx); } sysmon_wdog_release(smw); break; } case WDOGIOC_WHICH: { struct wdog_mode *wm = (void *) data; mutex_enter(&sysmon_wdog_mtx); if ((smw = sysmon_armed_wdog) != NULL) { strcpy(wm->wm_name, smw->smw_name); wm->wm_mode = smw->smw_mode; wm->wm_period = smw->smw_period; } else error = ESRCH; mutex_exit(&sysmon_wdog_mtx); break; } case WDOGIOC_TICKLE: if ((flag & FWRITE) == 0) { error = EPERM; break; } mutex_enter(&sysmon_wdog_mtx); if ((smw = sysmon_armed_wdog) != NULL) { error = (*smw->smw_tickle)(smw); if (error == 0) smw->smw_tickler = l->l_proc->p_pid; } else error = ESRCH; mutex_exit(&sysmon_wdog_mtx); break; case WDOGIOC_GTICKLER: if ((smw = sysmon_armed_wdog) != NULL) *(pid_t *)data = smw->smw_tickler; else error = ESRCH; break; case WDOGIOC_GWDOGS: { struct wdog_conf *wc = (void *) data; char *cp; int i; mutex_enter(&sysmon_wdog_list_mtx); if (wc->wc_names == NULL) wc->wc_count = sysmon_wdog_count; else { for (i = 0, cp = wc->wc_names, smw = LIST_FIRST(&sysmon_wdog_list); i < sysmon_wdog_count && smw != NULL && error == 0; i++, cp += WDOG_NAMESIZE, smw = LIST_NEXT(smw, smw_list)) error = copyout(smw->smw_name, cp, strlen(smw->smw_name) + 1); wc->wc_count = i; } mutex_exit(&sysmon_wdog_list_mtx); break; } default: error = ENOTTY; } return error; } /* * sysmon_wdog_register: * * Register a watchdog device. */ int sysmon_wdog_register(struct sysmon_wdog *smw) { struct sysmon_wdog *lsmw; int error = 0; (void)RUN_ONCE(&once_wdog, wdog_preinit); mutex_enter(&sysmon_wdog_list_mtx); LIST_FOREACH(lsmw, &sysmon_wdog_list, smw_list) { if (strcmp(lsmw->smw_name, smw->smw_name) == 0) { error = EEXIST; goto out; } } smw->smw_mode = WDOG_MODE_DISARMED; smw->smw_tickler = (pid_t) -1; smw->smw_refcnt = 0; sysmon_wdog_count++; LIST_INSERT_HEAD(&sysmon_wdog_list, smw, smw_list); out: mutex_exit(&sysmon_wdog_list_mtx); return error; } /* * sysmon_wdog_unregister: * * Unregister a watchdog device. */ int sysmon_wdog_unregister(struct sysmon_wdog *smw) { int rc = 0; mutex_enter(&sysmon_wdog_list_mtx); while (smw->smw_refcnt > 0 && rc == 0) { aprint_debug("%s: %d users remain\n", smw->smw_name, smw->smw_refcnt); rc = cv_wait_sig(&sysmon_wdog_cv, &sysmon_wdog_list_mtx); } if (rc == 0) { sysmon_wdog_count--; LIST_REMOVE(smw, smw_list); } mutex_exit(&sysmon_wdog_list_mtx); return rc; } /* * sysmon_wdog_critpoll: * * Perform critical operations during long polling periods */ void sysmon_wdog_critpoll(void *arg) { struct sysmon_wdog *smw = sysmon_armed_wdog; if (smw == NULL) return; if ((smw->smw_mode & WDOG_MODE_MASK) == WDOG_MODE_KTICKLE) { if ((*smw->smw_tickle)(smw) != 0) { printf("WARNING: KERNEL TICKLE OF WATCHDOG %s " "FAILED!\n", smw->smw_name); } } } /* * sysmon_wdog_find: * * Find a watchdog device. We increase the reference * count on a match. */ struct sysmon_wdog * sysmon_wdog_find(const char *name) { struct sysmon_wdog *smw; mutex_enter(&sysmon_wdog_list_mtx); LIST_FOREACH(smw, &sysmon_wdog_list, smw_list) { if (strcmp(smw->smw_name, name) == 0) break; } if (smw != NULL) smw->smw_refcnt++; mutex_exit(&sysmon_wdog_list_mtx); return smw; } /* * sysmon_wdog_release: * * Release a watchdog device. */ void sysmon_wdog_release(struct sysmon_wdog *smw) { mutex_enter(&sysmon_wdog_list_mtx); KASSERT(smw->smw_refcnt != 0); smw->smw_refcnt--; cv_signal(&sysmon_wdog_cv); mutex_exit(&sysmon_wdog_list_mtx); } void sysmon_wdog_ref(struct sysmon_wdog *smw) { mutex_enter(&sysmon_wdog_list_mtx); smw->smw_refcnt++; mutex_exit(&sysmon_wdog_list_mtx); } /* * sysmon_wdog_setmode: * * Set the mode of a watchdog device. */ int sysmon_wdog_setmode(struct sysmon_wdog *smw, int mode, u_int period) { u_int operiod = smw->smw_period; int omode = smw->smw_mode; int error = 0; smw->smw_period = period; smw->smw_mode = mode; switch (mode & WDOG_MODE_MASK) { case WDOG_MODE_DISARMED: if (smw != sysmon_armed_wdog) { error = EINVAL; goto out; } break; case WDOG_MODE_KTICKLE: case WDOG_MODE_UTICKLE: case WDOG_MODE_ETICKLE: if (sysmon_armed_wdog != NULL) { error = EBUSY; goto out; } break; default: error = EINVAL; goto out; } error = (*smw->smw_setmode)(smw); out: if (error) { smw->smw_period = operiod; smw->smw_mode = omode; } else { if ((mode & WDOG_MODE_MASK) == WDOG_MODE_DISARMED) { sysmon_armed_wdog = NULL; smw->smw_tickler = (pid_t) -1; sysmon_wdog_release(smw); if ((omode & WDOG_MODE_MASK) == WDOG_MODE_KTICKLE) callout_stop(&sysmon_wdog_callout); } else { sysmon_armed_wdog = smw; sysmon_wdog_ref(smw); if ((mode & WDOG_MODE_MASK) == WDOG_MODE_KTICKLE) { callout_reset(&sysmon_wdog_callout, WDOG_PERIOD_TO_TICKS(smw->smw_period) / 2, sysmon_wdog_ktickle, NULL); } } } return error; } /* * sysmon_wdog_ktickle: * * Kernel watchdog tickle routine. */ void sysmon_wdog_ktickle(void *arg) { struct sysmon_wdog *smw; mutex_enter(&sysmon_wdog_mtx); if ((smw = sysmon_armed_wdog) != NULL) { if ((*smw->smw_tickle)(smw) != 0) { printf("WARNING: KERNEL TICKLE OF WATCHDOG %s " "FAILED!\n", smw->smw_name); /* * ...we will probably reboot soon. */ } callout_reset(&sysmon_wdog_callout, WDOG_PERIOD_TO_TICKS(smw->smw_period) / 2, sysmon_wdog_ktickle, NULL); } mutex_exit(&sysmon_wdog_mtx); } /* * sysmon_wdog_shutdown: * * Perform shutdown-time operations. */ void sysmon_wdog_shutdown(void *arg) { struct sysmon_wdog *smw; /* * XXX Locking here? I don't think it's necessary. */ if ((smw = sysmon_armed_wdog) != NULL) { if (sysmon_wdog_setmode(smw, WDOG_MODE_DISARMED, smw->smw_period)) printf("WARNING: FAILED TO SHUTDOWN WATCHDOG %s!\n", smw->smw_name); } } static int sysmon_wdog_modcmd(modcmd_t cmd, void *arg) { int ret; switch (cmd) { case MODULE_CMD_INIT: ret = sysmon_wdog_init(); break; case MODULE_CMD_FINI: ret = sysmon_wdog_fini(); break; case MODULE_CMD_STAT: default: ret = ENOTTY; } return ret; } |
| 27 8 633 631 1 634 634 38 2 2 2 3 6 15 2 7 1 1 9 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 197 198 199 200 201 202 203 204 205 206 207 208 209 210 211 212 213 214 215 216 217 218 219 220 221 222 223 224 225 226 227 228 229 230 231 232 233 234 235 236 237 238 239 240 241 242 243 244 245 246 247 248 249 250 251 252 253 254 255 256 257 258 259 260 261 262 263 264 265 266 267 268 269 270 271 272 273 274 275 276 277 278 279 280 281 282 283 284 285 286 287 288 289 290 291 292 293 294 295 296 297 298 299 300 301 302 303 304 305 306 307 308 309 310 311 312 313 314 315 316 317 318 319 320 321 322 323 324 325 326 327 328 329 330 331 332 333 334 335 336 337 338 339 340 341 342 343 344 345 346 347 348 349 350 351 352 353 354 355 356 357 358 359 360 361 362 363 364 365 366 367 368 369 370 371 372 373 374 375 376 377 378 379 380 381 382 383 384 385 386 387 388 389 390 391 392 393 394 395 396 397 398 399 400 401 402 403 404 405 406 407 408 409 410 411 412 413 414 415 416 417 418 419 420 421 422 423 424 425 426 427 428 429 430 431 432 433 434 435 436 437 438 439 440 441 442 443 444 445 446 447 448 449 450 451 452 453 454 455 456 457 458 459 460 461 462 463 464 465 466 467 468 469 470 471 472 473 474 475 476 477 478 479 480 481 482 483 484 485 486 487 488 489 490 491 492 493 494 495 496 497 498 499 500 501 502 503 504 505 506 507 508 509 510 511 512 513 514 515 516 517 518 519 520 521 522 523 524 525 526 527 528 529 530 531 532 533 534 535 536 537 538 539 540 541 542 543 544 545 546 547 548 549 550 551 552 553 554 555 556 557 558 559 560 561 562 563 564 565 566 567 568 569 570 571 572 573 574 575 576 577 578 579 580 581 582 583 584 585 586 587 588 589 590 591 592 593 594 595 596 597 598 599 600 601 602 603 604 605 606 607 608 609 610 611 612 613 614 615 616 617 618 619 620 621 622 623 624 625 626 627 628 629 630 631 632 633 634 635 636 637 638 639 640 641 642 643 644 645 646 647 648 649 650 651 652 653 654 655 656 657 658 659 660 661 662 663 664 665 666 667 668 669 670 671 672 673 674 675 676 677 678 679 680 681 682 683 684 685 686 687 688 689 690 691 692 693 694 695 696 697 698 699 700 701 702 703 704 705 706 707 708 709 710 711 712 713 714 715 716 717 718 719 720 721 722 723 724 725 726 727 728 729 730 731 732 733 734 735 736 737 | /* $NetBSD: subr_disk.c,v 1.134 2022/03/28 12:33:59 riastradh Exp $ */ /*- * Copyright (c) 1996, 1997, 1999, 2000, 2009 The NetBSD Foundation, Inc. * All rights reserved. * * This code is derived from software contributed to The NetBSD Foundation * by Jason R. Thorpe of the Numerical Aerospace Simulation Facility, * NASA Ames Research Center. * * Redistribution and use in source and binary forms, with or without * modification, are permitted provided that the following conditions * are met: * 1. Redistributions of source code must retain the above copyright * notice, this list of conditions and the following disclaimer. * 2. Redistributions in binary form must reproduce the above copyright * notice, this list of conditions and the following disclaimer in the * documentation and/or other materials provided with the distribution. * * THIS SOFTWARE IS PROVIDED BY THE NETBSD FOUNDATION, INC. AND CONTRIBUTORS * ``AS IS'' AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED * TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR * PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE FOUNDATION OR CONTRIBUTORS * BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR * CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF * SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS * INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN * CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) * ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE * POSSIBILITY OF SUCH DAMAGE. */ /* * Copyright (c) 1982, 1986, 1988, 1993 * The Regents of the University of California. All rights reserved. * (c) UNIX System Laboratories, Inc. * All or some portions of this file are derived from material licensed * to the University of California by American Telephone and Telegraph * Co. or Unix System Laboratories, Inc. and are reproduced herein with * the permission of UNIX System Laboratories, Inc. * * Redistribution and use in source and binary forms, with or without * modification, are permitted provided that the following conditions * are met: * 1. Redistributions of source code must retain the above copyright * notice, this list of conditions and the following disclaimer. * 2. Redistributions in binary form must reproduce the above copyright * notice, this list of conditions and the following disclaimer in the * documentation and/or other materials provided with the distribution. * 3. Neither the name of the University nor the names of its contributors * may be used to endorse or promote products derived from this software * without specific prior written permission. * * THIS SOFTWARE IS PROVIDED BY THE REGENTS AND CONTRIBUTORS ``AS IS'' AND * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE * ARE DISCLAIMED. IN NO EVENT SHALL THE REGENTS OR CONTRIBUTORS BE LIABLE * FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL * DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS * OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) * HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT * LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY * OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF * SUCH DAMAGE. * * @(#)ufs_disksubr.c 8.5 (Berkeley) 1/21/94 */ #include <sys/cdefs.h> __KERNEL_RCSID(0, "$NetBSD: subr_disk.c,v 1.134 2022/03/28 12:33:59 riastradh Exp $"); #include <sys/param.h> #include <sys/kernel.h> #include <sys/kmem.h> #include <sys/buf.h> #include <sys/fcntl.h> #include <sys/syslog.h> #include <sys/disklabel.h> #include <sys/disk.h> #include <sys/sysctl.h> #include <lib/libkern/libkern.h> /* * Disk error is the preface to plaintive error messages * about failing disk transfers. It prints messages of the form hp0g: hard error reading fsbn 12345 of 12344-12347 (hp0 bn %d cn %d tn %d sn %d) * if the offset of the error in the transfer and a disk label * are both available. blkdone should be -1 if the position of the error * is unknown; the disklabel pointer may be null from drivers that have not * been converted to use them. The message is printed with printf * if pri is LOG_PRINTF, otherwise it uses log at the specified priority. * The message should be completed (with at least a newline) with printf * or addlog, respectively. There is no trailing space. */ #ifndef PRIdaddr #define PRIdaddr PRId64 #endif void diskerr(const struct buf *bp, const char *dname, const char *what, int pri, int blkdone, const struct disklabel *lp) { int unit = DISKUNIT(bp->b_dev), part = DISKPART(bp->b_dev); void (*pr)(const char *, ...) __printflike(1, 2); char partname = 'a' + part; daddr_t sn; if (/*CONSTCOND*/0) /* Compiler will error this if the format is wrong... */ printf("%" PRIdaddr, bp->b_blkno); if (pri != LOG_PRINTF) { static const char fmt[] = ""; log(pri, fmt); pr = addlog; } else pr = printf; (*pr)("%s%d%c: %s %sing fsbn ", dname, unit, partname, what, bp->b_flags & B_READ ? "read" : "writ"); sn = bp->b_blkno; if (bp->b_bcount <= DEV_BSIZE) (*pr)("%" PRIdaddr, sn); else { if (blkdone >= 0) { sn += blkdone; (*pr)("%" PRIdaddr " of ", sn); } (*pr)("%" PRIdaddr "-%" PRIdaddr "", bp->b_blkno, bp->b_blkno + (bp->b_bcount - 1) / DEV_BSIZE); } if (lp && (blkdone >= 0 || bp->b_bcount <= lp->d_secsize)) { sn += lp->d_partitions[part].p_offset; (*pr)(" (%s%d bn %" PRIdaddr "; cn %" PRIdaddr "", dname, unit, sn, sn / lp->d_secpercyl); sn %= lp->d_secpercyl; (*pr)(" tn %" PRIdaddr " sn %" PRIdaddr ")", sn / lp->d_nsectors, sn % lp->d_nsectors); } } /* * Searches the iostatlist for the disk corresponding to the * name provided. */ struct disk * disk_find(const char *name) { struct io_stats *stat; stat = iostat_find(name); if ((stat != NULL) && (stat->io_type == IOSTAT_DISK)) return stat->io_parent; return (NULL); } void disk_init(struct disk *diskp, const char *name, const struct dkdriver *driver) { u_int blocksize = DEV_BSIZE; /* * Initialize the wedge-related locks and other fields. */ mutex_init(&diskp->dk_rawlock, MUTEX_DEFAULT, IPL_NONE); mutex_init(&diskp->dk_openlock, MUTEX_DEFAULT, IPL_NONE); LIST_INIT(&diskp->dk_wedges); diskp->dk_nwedges = 0; diskp->dk_labelsector = LABELSECTOR; diskp->dk_blkshift = DK_BSIZE2BLKSHIFT(blocksize); diskp->dk_byteshift = DK_BSIZE2BYTESHIFT(blocksize); diskp->dk_name = name; diskp->dk_driver = driver; } /* * Rename a disk. */ void disk_rename(struct disk *diskp, const char *name) { diskp->dk_name = name; iostat_rename(diskp->dk_stats, diskp->dk_name); } /* * Attach a disk. */ void disk_attach(struct disk *diskp) { /* * Allocate and initialize the disklabel structures. */ diskp->dk_label = kmem_zalloc(sizeof(struct disklabel), KM_SLEEP); diskp->dk_cpulabel = kmem_zalloc(sizeof(struct cpu_disklabel), KM_SLEEP); /* * Set up the stats collection. */ diskp->dk_stats = iostat_alloc(IOSTAT_DISK, diskp, diskp->dk_name); } int disk_begindetach(struct disk *dk, int (*lastclose)(device_t), device_t self, int flags) { int rc; rc = 0; mutex_enter(&dk->dk_openlock); if (dk->dk_openmask == 0) ; /* nothing to do */ else if ((flags & DETACH_FORCE) == 0) rc = EBUSY; else if (lastclose != NULL) rc = (*lastclose)(self); mutex_exit(&dk->dk_openlock); return rc; } /* * Detach a disk. */ void disk_detach(struct disk *diskp) { /* * Remove from the drivelist. */ iostat_free(diskp->dk_stats); /* * Release the disk-info dictionary. */ if (diskp->dk_info) { prop_object_release(diskp->dk_info); diskp->dk_info = NULL; } /* * Free the space used by the disklabel structures. */ kmem_free(diskp->dk_label, sizeof(*diskp->dk_label)); kmem_free(diskp->dk_cpulabel, sizeof(*diskp->dk_cpulabel)); } void disk_destroy(struct disk *diskp) { mutex_destroy(&diskp->dk_openlock); mutex_destroy(&diskp->dk_rawlock); } /* * Mark the disk as having work queued for metrics collection. */ void disk_wait(struct disk *diskp) { iostat_wait(diskp->dk_stats); } /* * Mark the disk as busy for metrics collection. */ void disk_busy(struct disk *diskp) { iostat_busy(diskp->dk_stats); } /* * Finished disk operations, gather metrics. */ void disk_unbusy(struct disk *diskp, long bcount, int read) { iostat_unbusy(diskp->dk_stats, bcount, read); } /* * Return true if disk has an I/O operation in flight. */ bool disk_isbusy(struct disk *diskp) { return iostat_isbusy(diskp->dk_stats); } /* * Bounds checking against the media size, used for the raw partition. * secsize, mediasize and b_blkno must all be the same units. * Possibly this has to be DEV_BSIZE (512). */ int bounds_check_with_mediasize(struct buf *bp, int secsize, uint64_t mediasize) { int64_t sz; if (bp->b_blkno < 0) { /* Reject negative offsets immediately. */ bp->b_error = EINVAL; return 0; } sz = howmany((int64_t)bp->b_bcount, secsize); /* * bp->b_bcount is a 32-bit value, and we rejected a negative * bp->b_blkno already, so "bp->b_blkno + sz" cannot overflow. */ if (bp->b_blkno + sz > mediasize) { sz = mediasize - bp->b_blkno; if (sz == 0) { /* If exactly at end of disk, return EOF. */ bp->b_resid = bp->b_bcount; return 0; } if (sz < 0) { /* If past end of disk, return EINVAL. */ bp->b_error = EINVAL; return 0; } /* Otherwise, truncate request. */ bp->b_bcount = sz * secsize; } return 1; } /* * Determine the size of the transfer, and make sure it is * within the boundaries of the partition. Adjust transfer * if needed, and signal errors or early completion. */ int bounds_check_with_label(struct disk *dk, struct buf *bp, int wlabel) { struct disklabel *lp = dk->dk_label; struct partition *p = lp->d_partitions + DISKPART(bp->b_dev); uint64_t p_size, p_offset, labelsector; int64_t sz; if (bp->b_blkno < 0) { /* Reject negative offsets immediately. */ bp->b_error = EINVAL; return -1; } /* Protect against division by zero. XXX: Should never happen?!?! */ if ((lp->d_secsize / DEV_BSIZE) == 0 || lp->d_secpercyl == 0) { bp->b_error = EINVAL; return -1; } p_size = (uint64_t)p->p_size << dk->dk_blkshift; p_offset = (uint64_t)p->p_offset << dk->dk_blkshift; #if RAW_PART == 3 labelsector = lp->d_partitions[2].p_offset; #else labelsector = lp->d_partitions[RAW_PART].p_offset; #endif labelsector = (labelsector + dk->dk_labelsector) << dk->dk_blkshift; sz = howmany((int64_t)bp->b_bcount, DEV_BSIZE); /* * bp->b_bcount is a 32-bit value, and we rejected a negative * bp->b_blkno already, so "bp->b_blkno + sz" cannot overflow. */ if (bp->b_blkno + sz > p_size) { sz = p_size - bp->b_blkno; if (sz == 0) { /* If exactly at end of disk, return EOF. */ bp->b_resid = bp->b_bcount; return 0; } if (sz < 0) { /* If past end of disk, return EINVAL. */ bp->b_error = EINVAL; return -1; } /* Otherwise, truncate request. */ bp->b_bcount = sz << DEV_BSHIFT; } /* Overwriting disk label? */ if (bp->b_blkno + p_offset <= labelsector && bp->b_blkno + p_offset + sz > labelsector && (bp->b_flags & B_READ) == 0 && !wlabel) { bp->b_error = EROFS; return -1; } /* calculate cylinder for disksort to order transfers with */ bp->b_cylinder = (bp->b_blkno + p->p_offset) / (lp->d_secsize / DEV_BSIZE) / lp->d_secpercyl; return 1; } int disk_read_sectors(void (*strat)(struct buf *), const struct disklabel *lp, struct buf *bp, unsigned int sector, int count) { if ((lp->d_secsize / DEV_BSIZE) == 0 || lp->d_secpercyl == 0) return EINVAL; bp->b_blkno = btodb((off_t)sector * lp->d_secsize); bp->b_bcount = count * lp->d_secsize; bp->b_flags = (bp->b_flags & ~B_WRITE) | B_READ; bp->b_oflags &= ~BO_DONE; bp->b_cylinder = sector / lp->d_secpercyl; (*strat)(bp); return biowait(bp); } const char * convertdisklabel(struct disklabel *lp, void (*strat)(struct buf *), struct buf *bp, uint32_t secperunit) { struct partition rp, *altp, *p; int geom_ok; const char *str; memset(&rp, 0, sizeof(rp)); rp.p_size = secperunit; rp.p_fstype = FS_UNUSED; /* If we can seek to d_secperunit - 1, believe the disk geometry. */ if (secperunit != 0 && disk_read_sectors(strat, lp, bp, secperunit - 1, 1) == 0) geom_ok = 1; else geom_ok = 0; #if 0 printf("%s: secperunit (%" PRIu32 ") %s\n", __func__, secperunit, geom_ok ? "ok" : "not ok"); #endif p = &lp->d_partitions[RAW_PART]; if (RAW_PART == 'c' - 'a') altp = &lp->d_partitions['d' - 'a']; else altp = &lp->d_partitions['c' - 'a']; if (lp->d_npartitions > RAW_PART && p->p_offset == 0 && p->p_size != 0) return NULL; /* already a raw partition */ else if (lp->d_npartitions > MAX('c', 'd') - 'a' && altp->p_offset == 0 && altp->p_size != 0) { /* alternate partition ('c' or 'd') is suitable for raw slot, * swap with 'd' or 'c'. */ rp = *p; *p = *altp; *altp = rp; return NULL; } else if (lp->d_npartitions <= RAW_PART && lp->d_npartitions > 'c' - 'a') { /* No raw partition is present, but the alternate is present. * Copy alternate to raw partition. */ lp->d_npartitions = RAW_PART + 1; *p = *altp; return NULL; } else if (!geom_ok) str = "no raw partition and disk reports bad geometry"; else if (lp->d_npartitions <= RAW_PART) { memset(&lp->d_partitions[lp->d_npartitions], 0, sizeof(struct partition) * (RAW_PART - lp->d_npartitions)); *p = rp; lp->d_npartitions = RAW_PART + 1; return NULL; } else if (lp->d_npartitions < MAXPARTITIONS) { memmove(p + 1, p, sizeof(struct partition) * (lp->d_npartitions - RAW_PART)); *p = rp; lp->d_npartitions++; return NULL; } else str = "no raw partition and partition table is full"; #ifdef DIAGNOSTIC printf("Bad partition: %s\n", str); printf("type = %u, subtype = %u, typename = %s\n", lp->d_type, lp->d_subtype, lp->d_typename); printf("secsize = %u, nsectors = %u, ntracks = %u\n", lp->d_secsize, lp->d_nsectors, lp->d_ntracks); printf("ncylinders = %u, secpercyl = %u, secperunit = %u\n", lp->d_ncylinders, lp->d_secpercyl, lp->d_secperunit); printf("npartitions = %u\n", lp->d_npartitions); for (size_t i = 0; i < MIN(lp->d_npartitions, MAXPARTITIONS); i++) { p = &lp->d_partitions[i]; printf("\t%c: offset = %u size = %u fstype = %u\n", (char)(i + 'a'), p->p_offset, p->p_size, p->p_fstype); } #endif return str; } /* * disk_ioctl -- * Generic disk ioctl handling. */ int disk_ioctl(struct disk *dk, dev_t dev, u_long cmd, void *data, int flag, struct lwp *l) { struct dkwedge_info *dkw; struct partinfo *pi; struct partition *dp; #ifdef __HAVE_OLD_DISKLABEL struct disklabel newlabel; #endif switch (cmd) { case DIOCGDISKINFO: if (dk->dk_info == NULL) return ENOTSUP; return prop_dictionary_copyout_ioctl(data, cmd, dk->dk_info); case DIOCGSECTORSIZE: *(u_int *)data = dk->dk_geom.dg_secsize; return 0; case DIOCGMEDIASIZE: *(off_t *)data = (off_t)dk->dk_geom.dg_secsize * dk->dk_geom.dg_secperunit; return 0; default: break; } if (dev == NODEV) return EPASSTHROUGH; /* The following should be moved to dk_ioctl */ switch (cmd) { case DIOCGDINFO: if (dk->dk_label == NULL) return EBUSY; memcpy(data, dk->dk_label, sizeof (*dk->dk_label)); return 0; #ifdef __HAVE_OLD_DISKLABEL case ODIOCGDINFO: if (dk->dk_label == NULL) return EBUSY; memcpy(&newlabel, dk->dk_label, sizeof(newlabel)); if (newlabel.d_npartitions > OLDMAXPARTITIONS) return ENOTTY; memcpy(data, &newlabel, sizeof(struct olddisklabel)); return 0; #endif case DIOCGPARTINFO: pi = data; memset(pi, 0, sizeof(*pi)); pi->pi_secsize = dk->dk_geom.dg_secsize; pi->pi_bsize = MAX(BLKDEV_IOSIZE, pi->pi_secsize); if (DISKPART(dev) == RAW_PART) { pi->pi_size = dk->dk_geom.dg_secperunit; return 0; } if (dk->dk_label == NULL) return EBUSY; dp = &dk->dk_label->d_partitions[DISKPART(dev)]; pi->pi_offset = dp->p_offset; pi->pi_size = dp->p_size; pi->pi_fstype = dp->p_fstype; pi->pi_frag = dp->p_frag; pi->pi_fsize = dp->p_fsize; pi->pi_cpg = dp->p_cpg; /* * dholland 20130616: XXX this logic should not be * here. It is here because the old buffer cache * demands that all accesses to the same blocks need * to be the same size; but it only works for FFS and * nowadays I think it'll fail silently if the size * info in the disklabel is wrong. (Or missing.) The * buffer cache needs to be smarter; or failing that * we need a reliable way here to get the right block * size; or a reliable way to guarantee that (a) the * fs is not mounted when we get here and (b) any * buffers generated here will get purged when the fs * does get mounted. */ if (dp->p_fstype == FS_BSDFFS && dp->p_frag != 0 && dp->p_fsize != 0) pi->pi_bsize = dp->p_frag * dp->p_fsize; return 0; case DIOCAWEDGE: if ((flag & FWRITE) == 0) return EBADF; dkw = data; strlcpy(dkw->dkw_parent, dk->dk_name, sizeof(dkw->dkw_parent)); return dkwedge_add(dkw); case DIOCDWEDGE: if ((flag & FWRITE) == 0) return EBADF; dkw = data; strlcpy(dkw->dkw_parent, dk->dk_name, sizeof(dkw->dkw_parent)); return dkwedge_del(dkw); case DIOCLWEDGES: return dkwedge_list(dk, data, l); case DIOCMWEDGES: if ((flag & FWRITE) == 0) return EBADF; dkwedge_discover(dk); return 0; case DIOCRMWEDGES: if ((flag & FWRITE) == 0) return EBADF; dkwedge_delall(dk); return 0; default: return EPASSTHROUGH; } } void disk_set_info(device_t dev, struct disk *dk, const char *type) { struct disk_geom *dg = &dk->dk_geom; if (dg->dg_secsize == 0) { #ifdef DIAGNOSTIC printf("%s: fixing 0 sector size\n", dk->dk_name); #endif dg->dg_secsize = DEV_BSIZE; } dk->dk_blkshift = DK_BSIZE2BLKSHIFT(dg->dg_secsize); dk->dk_byteshift = DK_BSIZE2BYTESHIFT(dg->dg_secsize); if (dg->dg_secperunit == 0) { #ifdef DIAGNOSTIC if (dg->dg_ncylinders == 0) { printf("%s: secperunit and ncylinders are zero\n", dk->dk_name); } if (dg->dg_nsectors == 0 || dg->dg_ntracks == 0) { printf("%s: secperunit and (sectors or tracks) " "are zero\n", dk->dk_name); } #endif dg->dg_secperunit = (int64_t) dg->dg_nsectors * dg->dg_ntracks * dg->dg_ncylinders; } if (dg->dg_ncylinders == 0) { if (dg->dg_ntracks && dg->dg_nsectors) dg->dg_ncylinders = dg->dg_secperunit / (dg->dg_ntracks * dg->dg_nsectors); } prop_dictionary_t disk_info, odisk_info, geom; disk_info = prop_dictionary_create(); geom = prop_dictionary_create(); prop_dictionary_set_uint64(geom, "sectors-per-unit", dg->dg_secperunit); prop_dictionary_set_uint32(geom, "sector-size", dg->dg_secsize); if (dg->dg_nsectors) prop_dictionary_set_uint16(geom, "sectors-per-track", dg->dg_nsectors); if (dg->dg_ntracks) prop_dictionary_set_uint16(geom, "tracks-per-cylinder", dg->dg_ntracks); if (dg->dg_ncylinders) prop_dictionary_set_uint64(geom, "cylinders-per-unit", dg->dg_ncylinders); prop_dictionary_set(disk_info, "geometry", geom); if (type) prop_dictionary_set_string_nocopy(disk_info, "type", type); prop_object_release(geom); odisk_info = dk->dk_info; dk->dk_info = disk_info; if (dev) prop_dictionary_set(device_properties(dev), "disk-info", disk_info); /* * Don't release disk_info here; we keep a reference to it. * disk_detach() will release it when we go away. */ if (odisk_info) prop_object_release(odisk_info); } int disklabel_dev_unit(dev_t dev) { return DISKUNIT(dev); } |
| 1 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 197 198 199 200 201 202 203 204 205 206 207 208 209 210 211 212 213 214 215 216 217 218 219 220 221 222 223 224 225 226 227 228 229 230 231 232 233 234 235 236 237 238 239 240 241 242 243 244 245 246 247 248 249 250 251 252 253 254 255 256 257 258 259 260 261 262 263 264 265 266 267 268 269 270 271 272 273 274 275 276 277 278 279 280 281 282 283 284 285 286 287 288 289 290 291 292 293 294 295 296 297 298 299 300 301 302 303 304 305 306 307 308 309 310 311 312 313 314 315 316 317 318 319 320 321 322 323 324 325 326 327 328 329 330 331 332 333 334 335 336 337 338 339 340 341 342 343 344 345 346 347 348 349 350 351 352 353 354 355 356 357 358 359 360 361 362 363 364 365 366 367 368 369 370 371 372 373 374 375 376 377 378 379 380 381 382 383 384 385 386 387 388 389 390 391 392 393 394 395 396 397 398 399 400 401 402 403 404 405 406 407 408 409 410 411 412 413 414 415 416 417 418 419 420 421 422 423 424 425 426 427 428 429 430 431 432 433 434 435 436 437 438 439 440 441 442 443 444 445 446 447 448 449 450 451 452 453 454 455 456 457 458 459 460 461 462 463 464 465 466 467 468 469 470 471 472 473 474 475 476 477 478 479 480 481 482 483 484 485 486 487 488 489 490 491 492 493 494 495 496 497 498 499 500 501 502 503 504 505 506 507 508 509 510 511 512 513 514 515 516 517 518 519 520 521 522 523 524 525 526 527 528 529 530 531 532 533 534 535 536 537 538 539 540 541 542 543 544 545 546 547 548 549 550 551 552 553 554 555 556 557 558 559 560 561 562 563 564 565 566 567 568 569 570 571 572 573 574 575 576 577 578 579 580 581 582 583 584 585 586 587 588 589 590 591 592 593 594 595 596 597 598 599 600 601 602 603 604 605 606 607 608 609 610 611 612 613 614 615 616 617 618 619 620 621 622 623 624 625 626 627 628 629 630 631 632 633 634 635 636 637 638 639 640 641 642 643 644 645 646 647 648 649 650 651 652 653 654 655 656 657 658 659 660 661 662 663 664 665 666 667 668 669 670 671 672 673 674 675 676 677 678 679 680 681 682 683 684 685 686 687 688 689 690 691 692 693 694 695 696 697 698 699 700 701 702 703 704 705 706 707 708 709 710 711 712 713 714 715 716 717 718 719 720 721 722 723 724 725 726 727 728 729 730 731 732 733 734 735 736 737 738 739 740 741 742 743 744 745 746 747 748 749 750 751 752 753 754 755 756 757 758 759 760 761 762 763 764 765 766 767 768 769 770 771 772 773 774 775 776 777 778 779 780 781 782 783 784 785 786 787 788 789 790 791 792 793 794 795 796 797 798 799 800 801 802 803 804 805 806 807 808 809 810 811 812 813 814 815 816 817 818 819 820 821 822 823 824 825 826 827 828 829 830 831 832 833 834 835 836 837 838 839 840 841 842 843 844 845 846 847 848 849 850 851 852 853 854 855 856 857 858 859 860 861 862 863 864 865 866 867 868 869 870 871 872 873 874 875 876 877 878 879 880 881 882 883 884 885 886 887 888 889 890 891 892 893 894 895 896 897 898 899 900 901 902 903 904 905 906 907 908 909 910 911 912 913 914 915 916 917 918 919 920 921 922 923 924 925 926 927 928 929 930 931 932 933 934 935 936 937 938 939 940 941 942 943 944 945 946 947 948 949 950 951 952 953 954 955 956 957 958 959 960 961 962 963 964 965 966 967 968 969 970 971 972 973 974 975 976 977 978 979 980 981 982 983 984 985 986 987 988 989 990 991 992 993 994 995 996 997 998 999 1000 1001 1002 1003 1004 1005 1006 1007 1008 1009 1010 1011 1012 1013 1014 1015 1016 1017 1018 1019 1020 1021 1022 1023 1024 1025 1026 1027 1028 1029 1030 1031 1032 1033 1034 1035 1036 1037 1038 1039 1040 1041 1042 1043 1044 1045 1046 1047 1048 1049 1050 1051 1052 1053 1054 1055 1056 1057 1058 1059 1060 1061 1062 1063 1064 1065 1066 1067 1068 1069 1070 1071 1072 1073 1074 1075 1076 1077 1078 1079 1080 1081 1082 1083 1084 1085 1086 1087 1088 1089 1090 1091 1092 1093 1094 1095 1096 1097 1098 1099 1100 1101 1102 1103 1104 1105 1106 1107 1108 1109 1110 1111 1112 1113 1114 1115 1116 1117 1118 1119 1120 1121 1122 1123 1124 1125 1126 1127 1128 1129 1130 1131 1132 1133 1134 1135 1136 1137 1138 1139 1140 1141 1142 1143 1144 1145 1146 1147 1148 1149 1150 1151 1152 1153 1154 1155 1156 1157 1158 1159 1160 1161 1162 1163 1164 1165 1166 1167 1168 1169 1170 1171 1172 1173 1174 1175 1176 1177 1178 1179 1180 1181 1182 1183 1184 1185 1186 1187 1188 1189 1190 1191 1192 1193 1194 1195 1196 1197 1198 1199 1200 1201 1202 1203 1204 1205 1206 1207 1208 1209 1210 1211 1212 1213 1214 1215 1216 1217 1218 1219 1220 1221 1222 1223 1224 1225 1226 1227 1228 1229 1230 1231 1232 1233 1234 1235 1236 1237 1238 1239 1240 1241 1242 1243 1244 1245 1246 1247 1248 1249 1250 1251 1252 1253 1254 1255 1256 1257 1258 1259 1260 1261 1262 1263 1264 1265 1266 1267 1268 1269 1270 1271 1272 1273 1274 1275 1276 1277 1278 1279 1280 1281 1282 1283 1284 1285 1286 1287 1288 1289 1290 1291 1292 1293 1294 1295 1296 1297 1298 1299 1300 1301 1302 1303 1304 1305 1306 1307 1308 1309 1310 1311 1312 1313 1314 1315 1316 1317 1318 1319 1320 1321 1322 1323 1324 1325 1326 1327 1328 1329 1330 1331 1332 1333 1334 1335 1336 1337 1338 1339 1340 1341 1342 1343 1344 1345 1346 1347 1348 1349 1350 1351 1352 1353 1354 1355 1356 1357 1358 1359 1360 1361 1362 1363 1364 1365 1366 1367 1368 1369 1370 1371 1372 1373 1374 1375 1376 1377 1378 1379 1380 1381 1382 1383 1384 1385 1386 1387 1388 1389 1390 1391 1392 1393 1394 1395 1396 1397 1398 1399 1400 1401 1402 1403 1404 1405 1406 1407 1408 1409 1410 1411 1412 1413 1414 1415 1416 1417 1418 1419 1420 1421 1422 1423 1424 1425 1426 1427 1428 1429 1430 1431 1432 1433 1434 1435 1436 1437 1438 1439 1440 1441 1442 1443 1444 1445 1446 1447 1448 1449 1450 1451 1452 1453 1454 1455 1456 1457 1458 1459 1460 1461 1462 1463 1464 1465 1466 1467 1468 1469 1470 1471 1472 1473 1474 1475 1476 1477 1478 1479 1480 1481 1482 1483 1484 1485 1486 1487 1488 1489 1490 1491 1492 1493 1494 1495 1496 1497 1498 1499 1500 1501 1502 1503 1504 1505 1506 1507 1508 1509 1510 1511 1512 1513 1514 1515 1516 1517 1518 1519 1520 1521 1522 1523 1524 1525 1526 1527 1528 1529 1530 1531 1532 1533 1534 1535 1536 1537 1538 1539 1540 1541 1542 1543 1544 1545 1546 1547 1548 1549 1550 1551 1552 1553 1554 1555 1556 1557 1558 1559 1560 1561 1562 1563 1564 1565 1566 1567 1568 1569 1570 1571 1572 1573 1574 1575 1576 1577 1578 1579 1580 1581 1582 1583 1584 1585 1586 1587 1588 1589 1590 1591 1592 1593 1594 1595 1596 1597 1598 1599 1600 1601 1602 1603 1604 1605 1606 1607 1608 1609 1610 1611 1612 1613 1614 1615 1616 1617 1618 1619 1620 1621 1622 1623 1624 1625 1626 1627 1628 1629 1630 1631 1632 1633 1634 1635 1636 1637 1638 1639 1640 1641 1642 1643 1644 1645 1646 1647 1648 1649 | /* $NetBSD: hdaudio.c,v 1.18 2022/04/07 19:33:37 andvar Exp $ */ /* * Copyright (c) 2009 Precedence Technologies Ltd <support@precedence.co.uk> * Copyright (c) 2009 Jared D. McNeill <jmcneill@invisible.ca> * All rights reserved. * * This code is derived from software contributed to The NetBSD Foundation * by Precedence Technologies Ltd * * Redistribution and use in source and binary forms, with or without * modification, are permitted provided that the following conditions * are met: * 1. Redistributions of source code must retain the above copyright * notice, this list of conditions and the following disclaimer. * 2. The name of the author may not be used to endorse or promote products * derived from this software without specific prior written permission. * * THIS SOFTWARE IS PROVIDED BY THE AUTHOR ``AS IS'' AND ANY EXPRESS OR * IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES * OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. * IN NO EVENT SHALL THE AUTHOR BE LIABLE FOR ANY DIRECT, INDIRECT, * INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, * BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; * LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED * AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, * OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY * OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF * SUCH DAMAGE. */ #include <sys/cdefs.h> __KERNEL_RCSID(0, "$NetBSD: hdaudio.c,v 1.18 2022/04/07 19:33:37 andvar Exp $"); #include <sys/types.h> #include <sys/param.h> #include <sys/systm.h> #include <sys/device.h> #include <sys/conf.h> #include <sys/bus.h> #include <sys/kmem.h> #include <sys/module.h> #include "hdaudiovar.h" #include "hdaudioreg.h" #include "hdaudioio.h" #include "hdaudio_verbose.h" #include "hdaudiodevs.h" /* #define HDAUDIO_DEBUG */ #define HDAUDIO_RESET_TIMEOUT 5000 #define HDAUDIO_CORB_TIMEOUT 1000 #define HDAUDIO_RIRB_TIMEOUT 5000 #define HDAUDIO_CODEC_DELAY 1000 /* spec calls for 250 */ dev_type_open(hdaudioopen); dev_type_close(hdaudioclose); dev_type_ioctl(hdaudioioctl); const struct cdevsw hdaudio_cdevsw = { .d_open = hdaudioopen, .d_close = hdaudioclose, .d_read = noread, .d_write = nowrite, .d_ioctl = hdaudioioctl, .d_stop = nostop, .d_tty = notty, .d_poll = nopoll, .d_mmap = nommap, .d_kqfilter = nokqfilter, .d_discard = nodiscard, .d_flag = D_OTHER }; extern struct cfdriver hdaudio_cd; #define HDAUDIOUNIT(x) minor((x)) static void hdaudio_stream_init(struct hdaudio_softc *sc, int nis, int nos, int nbidir) { int i, cnt = 0; for (i = 0; i < nis && cnt < HDAUDIO_MAX_STREAMS; i++) { sc->sc_stream[cnt].st_host = sc; sc->sc_stream[cnt].st_enable = true; sc->sc_stream[cnt].st_shift = cnt; sc->sc_stream[cnt++].st_type = HDAUDIO_STREAM_ISS; } for (i = 0; i < nos && cnt < HDAUDIO_MAX_STREAMS; i++) { sc->sc_stream[cnt].st_host = sc; sc->sc_stream[cnt].st_enable = true; sc->sc_stream[cnt].st_shift = cnt; sc->sc_stream[cnt++].st_type = HDAUDIO_STREAM_OSS; } for (i = 0; i < nbidir && cnt < HDAUDIO_MAX_STREAMS; i++) { sc->sc_stream[cnt].st_host = sc; sc->sc_stream[cnt].st_enable = true; sc->sc_stream[cnt].st_shift = cnt; sc->sc_stream[cnt++].st_type = HDAUDIO_STREAM_BSS; } for (i = 0; i < cnt; i++) hdaudio_stream_stop(&sc->sc_stream[i]); sc->sc_stream_mask = 0; } static void hdaudio_codec_init(struct hdaudio_softc *sc) { int i; for (i = 0; i < HDAUDIO_MAX_CODECS; i++) { sc->sc_codec[i].co_addr = i; sc->sc_codec[i].co_host = sc; } } static void hdaudio_init(struct hdaudio_softc *sc) { const uint8_t vmaj = hda_read1(sc, HDAUDIO_MMIO_VMAJ); const uint8_t vmin = hda_read1(sc, HDAUDIO_MMIO_VMIN); const uint16_t gcap = hda_read2(sc, HDAUDIO_MMIO_GCAP); const int nis = HDAUDIO_GCAP_ISS(gcap); const int nos = HDAUDIO_GCAP_OSS(gcap); const int nbidir = HDAUDIO_GCAP_BSS(gcap); const int nsdo = HDAUDIO_GCAP_NSDO(gcap); const int addr64 = HDAUDIO_GCAP_64OK(gcap); hda_print(sc, "HDA ver. %d.%d, OSS %d, ISS %d, BSS %d, SDO %d%s\n", vmaj, vmin, nos, nis, nbidir, nsdo, addr64 ? ", 64-bit" : ""); /* Initialize codecs and streams */ hdaudio_codec_init(sc); hdaudio_stream_init(sc, nis, nos, nbidir); } static int hdaudio_codec_probe(struct hdaudio_softc *sc) { uint16_t statests; int codecid; statests = hda_read2(sc, HDAUDIO_MMIO_STATESTS); for (codecid = 0; codecid < HDAUDIO_MAX_CODECS; codecid++) if (statests & (1 << codecid)) sc->sc_codec[codecid].co_valid = true; hda_write2(sc, HDAUDIO_MMIO_STATESTS, statests); return statests; } int hdaudio_dma_alloc(struct hdaudio_softc *sc, struct hdaudio_dma *dma, int flags) { int err; KASSERT(dma->dma_size > 0); err = bus_dmamem_alloc(sc->sc_dmat, dma->dma_size, 128, 0, dma->dma_segs, sizeof(dma->dma_segs) / sizeof(dma->dma_segs[0]), &dma->dma_nsegs, BUS_DMA_WAITOK); if (err) return err; err = bus_dmamem_map(sc->sc_dmat, dma->dma_segs, dma->dma_nsegs, dma->dma_size, &dma->dma_addr, BUS_DMA_WAITOK | flags); if (err) goto free; err = bus_dmamap_create(sc->sc_dmat, dma->dma_size, dma->dma_nsegs, dma->dma_size, 0, BUS_DMA_WAITOK, &dma->dma_map); if (err) goto unmap; err = bus_dmamap_load(sc->sc_dmat, dma->dma_map, dma->dma_addr, dma->dma_size, NULL, BUS_DMA_WAITOK | flags); if (err) goto destroy; memset(dma->dma_addr, 0, dma->dma_size); bus_dmamap_sync(sc->sc_dmat, dma->dma_map, 0, dma->dma_size, BUS_DMASYNC_PREWRITE); dma->dma_valid = true; return 0; destroy: bus_dmamap_destroy(sc->sc_dmat, dma->dma_map); unmap: bus_dmamem_unmap(sc->sc_dmat, dma->dma_addr, dma->dma_size); free: bus_dmamem_free(sc->sc_dmat, dma->dma_segs, dma->dma_nsegs); dma->dma_valid = false; return err; } void hdaudio_dma_free(struct hdaudio_softc *sc, struct hdaudio_dma *dma) { if (dma->dma_valid == false) return; bus_dmamap_unload(sc->sc_dmat, dma->dma_map); bus_dmamap_destroy(sc->sc_dmat, dma->dma_map); bus_dmamem_unmap(sc->sc_dmat, dma->dma_addr, dma->dma_size); bus_dmamem_free(sc->sc_dmat, dma->dma_segs, dma->dma_nsegs); dma->dma_valid = false; } static void hdaudio_corb_enqueue(struct hdaudio_softc *sc, int addr, int nid, uint32_t control, uint32_t param) { uint32_t *corb = DMA_KERNADDR(&sc->sc_corb); uint32_t verb; uint16_t corbrp; int wp; /* Build command */ verb = (addr << 28) | (nid << 20) | (control << 8) | param; /* Fetch and update write pointer */ corbrp = hda_read2(sc, HDAUDIO_MMIO_CORBWP); wp = (corbrp & 0xff) + 1; if (wp >= (sc->sc_corb.dma_size / sizeof(*corb))) wp = 0; /* Enqueue command */ bus_dmamap_sync(sc->sc_dmat, sc->sc_corb.dma_map, 0, sc->sc_corb.dma_size, BUS_DMASYNC_POSTWRITE); corb[wp] = verb; bus_dmamap_sync(sc->sc_dmat, sc->sc_corb.dma_map, 0, sc->sc_corb.dma_size, BUS_DMASYNC_PREWRITE); /* Commit updated write pointer */ hda_write2(sc, HDAUDIO_MMIO_CORBWP, wp); } static void hdaudio_rirb_unsol(struct hdaudio_softc *sc, struct rirb_entry *entry) { struct hdaudio_codec *co; struct hdaudio_function_group *fg; uint8_t codecid = RIRB_CODEC_ID(entry); unsigned int i; if (codecid >= HDAUDIO_MAX_CODECS) { hda_error(sc, "unsol: codec id 0x%02x out of range\n", codecid); return; } co = &sc->sc_codec[codecid]; if (sc->sc_codec[codecid].co_valid == false) { hda_error(sc, "unsol: codec id 0x%02x not valid\n", codecid); return; } for (i = 0; i < co->co_nfg; i++) { fg = &co->co_fg[i]; if (fg->fg_device && fg->fg_unsol) fg->fg_unsol(fg->fg_device, entry->resp); } } static uint32_t hdaudio_rirb_dequeue(struct hdaudio_softc *sc, bool unsol) { uint16_t rirbwp; uint64_t *rirb = DMA_KERNADDR(&sc->sc_rirb); struct rirb_entry entry; int retry; for (;;) { retry = HDAUDIO_RIRB_TIMEOUT; rirbwp = hda_read2(sc, HDAUDIO_MMIO_RIRBWP); while (--retry > 0 && (rirbwp & 0xff) == sc->sc_rirbrp) { if (unsol) { /* don't wait for more unsol events */ hda_trace(sc, "unsol: rirb empty\n"); return 0xffffffff; } hda_delay(10); rirbwp = hda_read2(sc, HDAUDIO_MMIO_RIRBWP); } if (retry == 0) { hda_error(sc, "RIRB timeout\n"); return 0xffffffff; } sc->sc_rirbrp++; if (sc->sc_rirbrp >= (sc->sc_rirb.dma_size / sizeof(*rirb))) sc->sc_rirbrp = 0; bus_dmamap_sync(sc->sc_dmat, sc->sc_rirb.dma_map, 0, sc->sc_rirb.dma_size, BUS_DMASYNC_POSTREAD); entry = *(struct rirb_entry *)&rirb[sc->sc_rirbrp]; bus_dmamap_sync(sc->sc_dmat, sc->sc_rirb.dma_map, 0, sc->sc_rirb.dma_size, BUS_DMASYNC_PREREAD); hda_trace(sc, "%s: response %08X %08X\n", unsol ? "unsol" : "cmd ", entry.resp, entry.resp_ex); if (RIRB_UNSOL(&entry)) { hdaudio_rirb_unsol(sc, &entry); continue; } return entry.resp; } } uint32_t hdaudio_command(struct hdaudio_codec *co, int nid, uint32_t control, uint32_t param) { uint32_t result; struct hdaudio_softc *sc = co->co_host; mutex_enter(&sc->sc_corb_mtx); result = hdaudio_command_unlocked(co, nid, control, param); mutex_exit(&sc->sc_corb_mtx); return result; } uint32_t hdaudio_command_unlocked(struct hdaudio_codec *co, int nid, uint32_t control, uint32_t param) { struct hdaudio_softc *sc = co->co_host; uint32_t result; hda_trace(sc, "cmd : request %08X %08X (%02X)\n", control, param, nid); hdaudio_corb_enqueue(sc, co->co_addr, nid, control, param); result = hdaudio_rirb_dequeue(sc, false); /* Clear response interrupt status */ hda_write1(sc, HDAUDIO_MMIO_RIRBSTS, hda_read1(sc, HDAUDIO_MMIO_RIRBSTS)); return result; } static int hdaudio_corb_setsize(struct hdaudio_softc *sc) { uint8_t corbsize; bus_size_t bufsize = 0; /* * The size of the CORB is programmable to 2, 16, or 256 entries * by using the CORBSIZE register. Choose a size based on the * controller capabilities, preferring a larger size when possible. */ corbsize = hda_read1(sc, HDAUDIO_MMIO_CORBSIZE); corbsize &= ~0x3; if ((corbsize >> 4) & 0x4) { corbsize |= 0x2; bufsize = 1024; } else if ((corbsize >> 4) & 0x2) { corbsize |= 0x1; bufsize = 64; } else if ((corbsize >> 4) & 0x1) { corbsize |= 0x0; bufsize = 8; } else { hda_error(sc, "couldn't configure CORB size\n"); return ENXIO; } #if defined(HDAUDIO_DEBUG) hda_print(sc, "using %d byte CORB (cap %X)\n", (int)bufsize, corbsize >> 4); #endif sc->sc_corb.dma_size = bufsize; sc->sc_corb.dma_sizereg = corbsize; return 0; } static int hdaudio_corb_config(struct hdaudio_softc *sc) { uint32_t corbubase, corblbase; uint16_t corbrp; int retry = HDAUDIO_CORB_TIMEOUT; /* Program command buffer base address and size */ corblbase = (uint32_t)DMA_DMAADDR(&sc->sc_corb); corbubase = (uint32_t)(((uint64_t)DMA_DMAADDR(&sc->sc_corb)) >> 32); hda_write4(sc, HDAUDIO_MMIO_CORBLBASE, corblbase); hda_write4(sc, HDAUDIO_MMIO_CORBUBASE, corbubase); hda_write1(sc, HDAUDIO_MMIO_CORBSIZE, sc->sc_corb.dma_sizereg); /* Clear the read and write pointers */ hda_write2(sc, HDAUDIO_MMIO_CORBRP, HDAUDIO_CORBRP_RP_RESET); hda_write2(sc, HDAUDIO_MMIO_CORBRP, 0); do { hda_delay(10); corbrp = hda_read2(sc, HDAUDIO_MMIO_CORBRP); } while (--retry > 0 && (corbrp & HDAUDIO_CORBRP_RP_RESET) != 0); if (retry == 0) { hda_error(sc, "timeout resetting CORB\n"); return ETIME; } hda_write2(sc, HDAUDIO_MMIO_CORBWP, 0); return 0; } static int hdaudio_corb_stop(struct hdaudio_softc *sc) { uint8_t corbctl; int retry = HDAUDIO_CORB_TIMEOUT; /* Stop the CORB if necessary */ corbctl = hda_read1(sc, HDAUDIO_MMIO_CORBCTL); if (corbctl & HDAUDIO_CORBCTL_RUN) { corbctl &= ~HDAUDIO_CORBCTL_RUN; hda_write1(sc, HDAUDIO_MMIO_CORBCTL, corbctl); do { hda_delay(10); corbctl = hda_read1(sc, HDAUDIO_MMIO_CORBCTL); } while (--retry > 0 && (corbctl & HDAUDIO_CORBCTL_RUN) != 0); if (retry == 0) { hda_error(sc, "timeout stopping CORB\n"); return ETIME; } } return 0; } static int hdaudio_corb_start(struct hdaudio_softc *sc) { uint8_t corbctl; int retry = HDAUDIO_CORB_TIMEOUT; /* Start the CORB if necessary */ corbctl = hda_read1(sc, HDAUDIO_MMIO_CORBCTL); if ((corbctl & HDAUDIO_CORBCTL_RUN) == 0) { corbctl |= HDAUDIO_CORBCTL_RUN; hda_write1(sc, HDAUDIO_MMIO_CORBCTL, corbctl); do { hda_delay(10); corbctl = hda_read1(sc, HDAUDIO_MMIO_CORBCTL); } while (--retry > 0 && (corbctl & HDAUDIO_CORBCTL_RUN) == 0); if (retry == 0) { hda_error(sc, "timeout starting CORB\n"); return ETIME; } } return 0; } static int hdaudio_rirb_stop(struct hdaudio_softc *sc) { uint8_t rirbctl; int retry = HDAUDIO_RIRB_TIMEOUT; /* Stop the RIRB if necessary */ rirbctl = hda_read1(sc, HDAUDIO_MMIO_RIRBCTL); if (rirbctl & (HDAUDIO_RIRBCTL_RUN|HDAUDIO_RIRBCTL_ROI_EN)) { rirbctl &= ~HDAUDIO_RIRBCTL_RUN; rirbctl &= ~HDAUDIO_RIRBCTL_ROI_EN; hda_write1(sc, HDAUDIO_MMIO_RIRBCTL, rirbctl); do { hda_delay(10); rirbctl = hda_read1(sc, HDAUDIO_MMIO_RIRBCTL); } while (--retry > 0 && (rirbctl & HDAUDIO_RIRBCTL_RUN) != 0); if (retry == 0) { hda_error(sc, "timeout stopping RIRB\n"); return ETIME; } } return 0; } static int hdaudio_rirb_start(struct hdaudio_softc *sc) { uint8_t rirbctl; int retry = HDAUDIO_RIRB_TIMEOUT; /* Set the RIRB interrupt count */ hda_write2(sc, HDAUDIO_MMIO_RINTCNT, 1); /* Start the RIRB */ rirbctl = hda_read1(sc, HDAUDIO_MMIO_RIRBCTL); rirbctl |= HDAUDIO_RIRBCTL_RUN; rirbctl |= HDAUDIO_RIRBCTL_INT_EN; hda_write1(sc, HDAUDIO_MMIO_RIRBCTL, rirbctl); do { hda_delay(10); rirbctl = hda_read1(sc, HDAUDIO_MMIO_RIRBCTL); } while (--retry > 0 && (rirbctl & HDAUDIO_RIRBCTL_RUN) == 0); if (retry == 0) { hda_error(sc, "timeout starting RIRB\n"); return ETIME; } return 0; } static int hdaudio_rirb_setsize(struct hdaudio_softc *sc) { uint8_t rirbsize; bus_size_t bufsize = 0; /* * The size of the RIRB is programmable to 2, 16, or 256 entries * by using the RIRBSIZE register. Choose a size based on the * controller capabilities, preferring a larger size when possible. */ rirbsize = hda_read1(sc, HDAUDIO_MMIO_RIRBSIZE); rirbsize &= ~0x3; if ((rirbsize >> 4) & 0x4) { rirbsize |= 0x2; bufsize = 2048; } else if ((rirbsize >> 4) & 0x2) { rirbsize |= 0x1; bufsize = 128; } else if ((rirbsize >> 4) & 0x1) { rirbsize |= 0x0; bufsize = 16; } else { hda_error(sc, "couldn't configure RIRB size\n"); return ENXIO; } #if defined(HDAUDIO_DEBUG) hda_print(sc, "using %d byte RIRB (cap %X)\n", (int)bufsize, rirbsize >> 4); #endif sc->sc_rirb.dma_size = bufsize; sc->sc_rirb.dma_sizereg = rirbsize; return 0; } static int hdaudio_rirb_config(struct hdaudio_softc *sc) { uint32_t rirbubase, rirblbase; /* Program command buffer base address and size */ rirblbase = (uint32_t)DMA_DMAADDR(&sc->sc_rirb); rirbubase = (uint32_t)(((uint64_t)DMA_DMAADDR(&sc->sc_rirb)) >> 32); hda_write4(sc, HDAUDIO_MMIO_RIRBLBASE, rirblbase); hda_write4(sc, HDAUDIO_MMIO_RIRBUBASE, rirbubase); hda_write1(sc, HDAUDIO_MMIO_RIRBSIZE, sc->sc_rirb.dma_sizereg); /* Clear the write pointer */ hda_write2(sc, HDAUDIO_MMIO_RIRBWP, HDAUDIO_RIRBWP_WP_RESET); sc->sc_rirbrp = 0; return 0; } static int hdaudio_reset(struct hdaudio_softc *sc) { int retry = HDAUDIO_RESET_TIMEOUT; uint32_t gctl; int err; if ((err = hdaudio_rirb_stop(sc)) != 0) { hda_error(sc, "couldn't reset because RIRB is busy\n"); return err; } if ((err = hdaudio_corb_stop(sc)) != 0) { hda_error(sc, "couldn't reset because CORB is busy\n"); return err; } /* Disable wake events */ hda_write2(sc, HDAUDIO_MMIO_WAKEEN, 0); /* Disable interrupts */ hda_write4(sc, HDAUDIO_MMIO_INTCTL, 0); /* Clear state change status register */ hda_write2(sc, HDAUDIO_MMIO_STATESTS, hda_read2(sc, HDAUDIO_MMIO_STATESTS)); hda_write1(sc, HDAUDIO_MMIO_RIRBSTS, hda_read1(sc, HDAUDIO_MMIO_RIRBSTS)); /* Put the controller into reset state */ gctl = hda_read4(sc, HDAUDIO_MMIO_GCTL); gctl &= ~HDAUDIO_GCTL_CRST; hda_write4(sc, HDAUDIO_MMIO_GCTL, gctl); do { hda_delay(10); gctl = hda_read4(sc, HDAUDIO_MMIO_GCTL); } while (--retry > 0 && (gctl & HDAUDIO_GCTL_CRST) != 0); if (retry == 0) { hda_error(sc, "timeout entering reset state\n"); return ETIME; } hda_delay(1000); /* Now the controller is in reset state, so bring it out */ retry = HDAUDIO_RESET_TIMEOUT; hda_write4(sc, HDAUDIO_MMIO_GCTL, gctl | HDAUDIO_GCTL_CRST); do { hda_delay(10); gctl = hda_read4(sc, HDAUDIO_MMIO_GCTL); } while (--retry > 0 && (gctl & HDAUDIO_GCTL_CRST) == 0); if (retry == 0) { hda_error(sc, "timeout leaving reset state\n"); return ETIME; } hda_delay(2000); /* Accept unsolicited responses */ hda_write4(sc, HDAUDIO_MMIO_GCTL, gctl | HDAUDIO_GCTL_UNSOL_EN); return 0; } static void hdaudio_intr_enable(struct hdaudio_softc *sc) { hda_write4(sc, HDAUDIO_MMIO_INTSTS, hda_read4(sc, HDAUDIO_MMIO_INTSTS)); hda_write4(sc, HDAUDIO_MMIO_INTCTL, HDAUDIO_INTCTL_GIE | HDAUDIO_INTCTL_CIE); } static void hdaudio_intr_disable(struct hdaudio_softc *sc) { hda_write4(sc, HDAUDIO_MMIO_INTCTL, 0); } static int hdaudio_config_print(void *opaque, const char *pnp) { prop_dictionary_t dict = opaque; uint8_t fgtype, nid; uint16_t vendor, product; const char *type = "unknown"; prop_dictionary_get_uint8(dict, "function-group-type", &fgtype); prop_dictionary_get_uint8(dict, "node-id", &nid); prop_dictionary_get_uint16(dict, "vendor-id", &vendor); prop_dictionary_get_uint16(dict, "product-id", &product); if (pnp) { if (fgtype == HDAUDIO_GROUP_TYPE_AFG) type = "hdafg"; else if (fgtype == HDAUDIO_GROUP_TYPE_VSM_FG) type = "hdvsmfg"; aprint_normal("%s at %s", type, pnp); } aprint_debug(" vendor 0x%04X product 0x%04X nid 0x%02X", vendor, product, nid); return UNCONF; } static void hdaudio_attach_fg(struct hdaudio_function_group *fg, prop_array_t config) { struct hdaudio_codec *co = fg->fg_codec; struct hdaudio_softc *sc = co->co_host; prop_dictionary_t args = prop_dictionary_create(); uint64_t fgptr = (vaddr_t)fg; int locs[1]; prop_dictionary_set_uint8(args, "function-group-type", fg->fg_type); prop_dictionary_set_uint64(args, "function-group", fgptr); prop_dictionary_set_uint8(args, "node-id", fg->fg_nid); prop_dictionary_set_uint16(args, "vendor-id", fg->fg_vendor); prop_dictionary_set_uint16(args, "product-id", fg->fg_product); if (config) prop_dictionary_set(args, "pin-config", config); locs[0] = fg->fg_nid; fg->fg_device = config_found(sc->sc_dev, args, hdaudio_config_print, CFARGS(.submatch = config_stdsubmatch, .locators = locs)); prop_object_release(args); } static void hdaudio_codec_attach(struct hdaudio_codec *co) { struct hdaudio_softc *sc = co->co_host; struct hdaudio_function_group *fg; uint32_t vid, snc, fgrp; int starting_node, num_nodes, nid; if (co->co_valid == false) return; vid = hdaudio_command(co, 0, CORB_GET_PARAMETER, COP_VENDOR_ID); snc = hdaudio_command(co, 0, CORB_GET_PARAMETER, COP_SUBORDINATE_NODE_COUNT); /* make sure the vendor and product IDs are valid */ if (vid == 0xffffffff || vid == 0x00000000) return; #ifdef HDAUDIO_DEBUG uint32_t rid = hdaudio_command(co, 0, CORB_GET_PARAMETER, COP_REVISION_ID); hda_print(sc, "Codec%02X: %04X:%04X HDA %d.%d rev %d stepping %d\n", co->co_addr, vid >> 16, vid & 0xffff, (rid >> 20) & 0xf, (rid >> 16) & 0xf, (rid >> 8) & 0xff, rid & 0xff); #endif starting_node = (snc >> 16) & 0xff; num_nodes = snc & 0xff; /* * If the total number of nodes is 0, there's nothing we can do. * This shouldn't happen, so complain about it. */ if (num_nodes == 0) { hda_error(sc, "Codec%02X: No subordinate nodes found (%08x)\n", co->co_addr, snc); return; } co->co_nfg = num_nodes; co->co_fg = kmem_zalloc(co->co_nfg * sizeof(*co->co_fg), KM_SLEEP); for (nid = starting_node; nid < starting_node + num_nodes; nid++) { fg = &co->co_fg[nid - starting_node]; fg->fg_codec = co; fg->fg_nid = nid; fg->fg_vendor = vid >> 16; fg->fg_product = vid & 0xffff; fgrp = hdaudio_command(co, nid, CORB_GET_PARAMETER, COP_FUNCTION_GROUP_TYPE); switch (fgrp & 0xff) { case 0x01: /* Audio Function Group */ fg->fg_type = HDAUDIO_GROUP_TYPE_AFG; break; case 0x02: /* Vendor Specific Modem Function Group */ fg->fg_type = HDAUDIO_GROUP_TYPE_VSM_FG; break; default: /* Function group type not supported */ fg->fg_type = HDAUDIO_GROUP_TYPE_UNKNOWN; break; } hdaudio_attach_fg(fg, NULL); } } int hdaudio_stream_tag(struct hdaudio_stream *st) { int ret = 0; switch (st->st_type) { case HDAUDIO_STREAM_ISS: ret = 1; break; case HDAUDIO_STREAM_OSS: ret = 2; break; case HDAUDIO_STREAM_BSS: ret = 3; break; } return ret; } int hdaudio_attach(device_t dev, struct hdaudio_softc *sc) { int err, i; KASSERT(sc->sc_memvalid == true); sc->sc_dev = dev; mutex_init(&sc->sc_corb_mtx, MUTEX_DEFAULT, IPL_AUDIO); mutex_init(&sc->sc_stream_mtx, MUTEX_DEFAULT, IPL_AUDIO); /* * Put the controller into a known state by entering and leaving * CRST as necessary. */ if ((err = hdaudio_reset(sc)) != 0) goto fail; /* * From the spec: * * Must wait 250us after reading CRST as a 1 before assuming that * codecs have all made status change requests and have been * registered by the controller. * * In reality, we need to wait longer than this. */ hda_delay(HDAUDIO_CODEC_DELAY); /* * Read device capabilities */ hdaudio_init(sc); /* * Detect codecs */ if (hdaudio_codec_probe(sc) == 0) { hda_error(sc, "no codecs found\n"); err = ENODEV; goto fail; } /* * Ensure that the device is in a known state */ hda_write2(sc, HDAUDIO_MMIO_STATESTS, HDAUDIO_STATESTS_SDIWAKE); hda_write1(sc, HDAUDIO_MMIO_RIRBSTS, HDAUDIO_RIRBSTS_RIRBOIS | HDAUDIO_RIRBSTS_RINTFL); hda_write4(sc, HDAUDIO_MMIO_INTSTS, hda_read4(sc, HDAUDIO_MMIO_INTSTS)); hda_write4(sc, HDAUDIO_MMIO_DPLBASE, 0); hda_write4(sc, HDAUDIO_MMIO_DPUBASE, 0); /* * Initialize the CORB. First negotiate a command buffer size, * then allocate and configure it. */ if ((err = hdaudio_corb_setsize(sc)) != 0) goto fail; if ((err = hdaudio_dma_alloc(sc, &sc->sc_corb, BUS_DMA_WRITE)) != 0) goto fail; if ((err = hdaudio_corb_config(sc)) != 0) goto fail; /* * Initialize the RIRB. */ if ((err = hdaudio_rirb_setsize(sc)) != 0) goto fail; if ((err = hdaudio_dma_alloc(sc, &sc->sc_rirb, BUS_DMA_READ)) != 0) goto fail; if ((err = hdaudio_rirb_config(sc)) != 0) goto fail; /* * Start the CORB and RIRB */ if ((err = hdaudio_corb_start(sc)) != 0) goto fail; if ((err = hdaudio_rirb_start(sc)) != 0) goto fail; /* * Identify and attach discovered codecs */ for (i = 0; i < HDAUDIO_MAX_CODECS; i++) hdaudio_codec_attach(&sc->sc_codec[i]); /* * Enable interrupts */ hdaudio_intr_enable(sc); fail: if (err) hda_error(sc, "device driver failed to attach\n"); return err; } int hdaudio_detach(struct hdaudio_softc *sc, int flags) { int error; /* Disable interrupts */ hdaudio_intr_disable(sc); error = config_detach_children(sc->sc_dev, flags); if (error != 0) { hdaudio_intr_enable(sc); return error; } mutex_destroy(&sc->sc_corb_mtx); mutex_destroy(&sc->sc_stream_mtx); hdaudio_dma_free(sc, &sc->sc_corb); hdaudio_dma_free(sc, &sc->sc_rirb); return 0; } bool hdaudio_resume(struct hdaudio_softc *sc) { if (hdaudio_reset(sc) != 0) return false; hda_delay(HDAUDIO_CODEC_DELAY); /* * Ensure that the device is in a known state */ hda_write2(sc, HDAUDIO_MMIO_STATESTS, HDAUDIO_STATESTS_SDIWAKE); hda_write1(sc, HDAUDIO_MMIO_RIRBSTS, HDAUDIO_RIRBSTS_RIRBOIS | HDAUDIO_RIRBSTS_RINTFL); hda_write4(sc, HDAUDIO_MMIO_INTSTS, hda_read4(sc, HDAUDIO_MMIO_INTSTS)); hda_write4(sc, HDAUDIO_MMIO_DPLBASE, 0); hda_write4(sc, HDAUDIO_MMIO_DPUBASE, 0); if (hdaudio_corb_config(sc) != 0) return false; if (hdaudio_rirb_config(sc) != 0) return false; if (hdaudio_corb_start(sc) != 0) return false; if (hdaudio_rirb_start(sc) != 0) return false; hdaudio_intr_enable(sc); return true; } int hdaudio_rescan(struct hdaudio_softc *sc, const char *ifattr, const int *locs) { struct hdaudio_codec *co; struct hdaudio_function_group *fg; unsigned int codec; for (codec = 0; codec < HDAUDIO_MAX_CODECS; codec++) { co = &sc->sc_codec[codec]; fg = co->co_fg; if (!co->co_valid || fg == NULL) continue; if (fg->fg_device) continue; hdaudio_attach_fg(fg, NULL); } return 0; } void hdaudio_childdet(struct hdaudio_softc *sc, device_t child) { struct hdaudio_codec *co; struct hdaudio_function_group *fg; unsigned int codec; for (codec = 0; codec < HDAUDIO_MAX_CODECS; codec++) { co = &sc->sc_codec[codec]; fg = co->co_fg; if (!co->co_valid || fg == NULL) continue; if (fg->fg_device == child) fg->fg_device = NULL; } } int hdaudio_intr(struct hdaudio_softc *sc) { struct hdaudio_stream *st; uint32_t intsts, stream_mask; int streamid = 0; uint8_t rirbsts; intsts = hda_read4(sc, HDAUDIO_MMIO_INTSTS); if (!(intsts & HDAUDIO_INTSTS_GIS)) return 0; if (intsts & HDAUDIO_INTSTS_CIS) { rirbsts = hda_read1(sc, HDAUDIO_MMIO_RIRBSTS); if (rirbsts & HDAUDIO_RIRBSTS_RINTFL) { mutex_enter(&sc->sc_corb_mtx); hdaudio_rirb_dequeue(sc, true); mutex_exit(&sc->sc_corb_mtx); } if (rirbsts & (HDAUDIO_RIRBSTS_RIRBOIS|HDAUDIO_RIRBSTS_RINTFL)) hda_write1(sc, HDAUDIO_MMIO_RIRBSTS, rirbsts); hda_write4(sc, HDAUDIO_MMIO_INTSTS, HDAUDIO_INTSTS_CIS); } if (intsts & HDAUDIO_INTSTS_SIS_MASK) { mutex_enter(&sc->sc_stream_mtx); stream_mask = intsts & sc->sc_stream_mask; while (streamid < HDAUDIO_MAX_STREAMS && stream_mask != 0) { st = &sc->sc_stream[streamid++]; if ((stream_mask & 1) != 0 && st->st_intr) { st->st_intr(st); } stream_mask >>= 1; } mutex_exit(&sc->sc_stream_mtx); hda_write4(sc, HDAUDIO_MMIO_INTSTS, HDAUDIO_INTSTS_SIS_MASK); } return 1; } struct hdaudio_stream * hdaudio_stream_establish(struct hdaudio_softc *sc, enum hdaudio_stream_type type, int (*intr)(struct hdaudio_stream *), void *cookie) { struct hdaudio_stream *st; struct hdaudio_dma dma; int i, err; dma.dma_size = sizeof(struct hdaudio_bdl_entry) * HDAUDIO_BDL_MAX; dma.dma_sizereg = 0; err = hdaudio_dma_alloc(sc, &dma, BUS_DMA_COHERENT | BUS_DMA_NOCACHE); if (err) return NULL; mutex_enter(&sc->sc_stream_mtx); for (i = 0; i < HDAUDIO_MAX_STREAMS; i++) { st = &sc->sc_stream[i]; if (st->st_enable == false) break; if (st->st_type != type) continue; if (sc->sc_stream_mask & (1 << i)) continue; /* Allocate stream */ st->st_bdl = dma; st->st_intr = intr; st->st_cookie = cookie; sc->sc_stream_mask |= (1 << i); mutex_exit(&sc->sc_stream_mtx); return st; } mutex_exit(&sc->sc_stream_mtx); /* No streams of requested type available */ hdaudio_dma_free(sc, &dma); return NULL; } void hdaudio_stream_disestablish(struct hdaudio_stream *st) { struct hdaudio_softc *sc = st->st_host; struct hdaudio_dma dma; KASSERT(sc->sc_stream_mask & (1 << st->st_shift)); mutex_enter(&sc->sc_stream_mtx); sc->sc_stream_mask &= ~(1 << st->st_shift); st->st_intr = NULL; st->st_cookie = NULL; dma = st->st_bdl; st->st_bdl.dma_valid = false; mutex_exit(&sc->sc_stream_mtx); /* Can't bus_dmamem_unmap while holding a mutex. */ hdaudio_dma_free(sc, &dma); } /* * Convert most of audio_params_t to stream fmt descriptor; noticeably missing * is the # channels bits, as this is encoded differently in codec and * stream descriptors. * * TODO: validate that the stream and selected codecs can handle the fmt */ uint16_t hdaudio_stream_param(struct hdaudio_stream *st, const audio_params_t *param) { uint16_t fmt = 0; switch (param->encoding) { case AUDIO_ENCODING_AC3: fmt |= HDAUDIO_FMT_TYPE_NONPCM; break; default: fmt |= HDAUDIO_FMT_TYPE_PCM; break; } switch (param->sample_rate) { case 8000: fmt |= HDAUDIO_FMT_BASE_48 | HDAUDIO_FMT_MULT(1) | HDAUDIO_FMT_DIV(6); break; case 11025: fmt |= HDAUDIO_FMT_BASE_44 | HDAUDIO_FMT_MULT(1) | HDAUDIO_FMT_DIV(4); break; case 16000: fmt |= HDAUDIO_FMT_BASE_48 | HDAUDIO_FMT_MULT(1) | HDAUDIO_FMT_DIV(3); break; case 22050: fmt |= HDAUDIO_FMT_BASE_44 | HDAUDIO_FMT_MULT(1) | HDAUDIO_FMT_DIV(2); break; case 32000: fmt |= HDAUDIO_FMT_BASE_48 | HDAUDIO_FMT_MULT(2) | HDAUDIO_FMT_DIV(3); break; case 44100: fmt |= HDAUDIO_FMT_BASE_44 | HDAUDIO_FMT_MULT(1); break; case 48000: fmt |= HDAUDIO_FMT_BASE_48 | HDAUDIO_FMT_MULT(1); break; case 88200: fmt |= HDAUDIO_FMT_BASE_44 | HDAUDIO_FMT_MULT(2); break; case 96000: fmt |= HDAUDIO_FMT_BASE_48 | HDAUDIO_FMT_MULT(2); break; case 176400: fmt |= HDAUDIO_FMT_BASE_44 | HDAUDIO_FMT_MULT(4); break; case 192000: fmt |= HDAUDIO_FMT_BASE_48 | HDAUDIO_FMT_MULT(4); break; default: return 0; } if (param->precision == 16 && param->validbits == 8) fmt |= HDAUDIO_FMT_BITS_8_16; else if (param->precision == 16 && param->validbits == 16) fmt |= HDAUDIO_FMT_BITS_16_16; else if (param->precision == 32 && param->validbits == 20) fmt |= HDAUDIO_FMT_BITS_20_32; else if (param->precision == 32 && param->validbits == 24) fmt |= HDAUDIO_FMT_BITS_24_32; else if (param->precision == 32 && param->validbits == 32) fmt |= HDAUDIO_FMT_BITS_32_32; else return 0; return fmt; } void hdaudio_stream_reset(struct hdaudio_stream *st) { struct hdaudio_softc *sc = st->st_host; int snum = st->st_shift; int retry; uint8_t ctl0; ctl0 = hda_read1(sc, HDAUDIO_SD_CTL0(snum)); ctl0 |= HDAUDIO_CTL_SRST; hda_write1(sc, HDAUDIO_SD_CTL0(snum), ctl0); retry = HDAUDIO_RESET_TIMEOUT; do { ctl0 = hda_read1(sc, HDAUDIO_SD_CTL0(snum)); if (ctl0 & HDAUDIO_CTL_SRST) break; hda_delay(10); } while (--retry > 0); ctl0 &= ~HDAUDIO_CTL_SRST; hda_write1(sc, HDAUDIO_SD_CTL0(snum), ctl0); retry = HDAUDIO_RESET_TIMEOUT; do { ctl0 = hda_read1(sc, HDAUDIO_SD_CTL0(snum)); if (!(ctl0 & HDAUDIO_CTL_SRST)) break; hda_delay(10); } while (--retry > 0); if (retry == 0) { hda_error(sc, "timeout leaving stream reset state\n"); return; } } void hdaudio_stream_start(struct hdaudio_stream *st, int blksize, bus_size_t dmasize, const audio_params_t *params) { struct hdaudio_softc *sc = st->st_host; struct hdaudio_bdl_entry *bdl; uint64_t dmaaddr; uint32_t intctl; uint16_t fmt; uint8_t ctl0, ctl2; int cnt, snum = st->st_shift; KASSERT(sc->sc_stream_mask & (1 << st->st_shift)); KASSERT(st->st_data.dma_valid == true); KASSERT(st->st_bdl.dma_valid == true); hdaudio_stream_stop(st); hdaudio_stream_reset(st); /* * Configure buffer descriptor list */ dmaaddr = DMA_DMAADDR(&st->st_data); bdl = DMA_KERNADDR(&st->st_bdl); for (cnt = 0; cnt < HDAUDIO_BDL_MAX; cnt++) { bdl[cnt].address_lo = (uint32_t)dmaaddr; bdl[cnt].address_hi = dmaaddr >> 32; bdl[cnt].length = blksize; bdl[cnt].flags = HDAUDIO_BDL_ENTRY_IOC; dmaaddr += blksize; if (dmaaddr >= DMA_DMAADDR(&st->st_data) + dmasize) { cnt++; break; } } /* * Program buffer descriptor list */ dmaaddr = DMA_DMAADDR(&st->st_bdl); hda_write4(sc, HDAUDIO_SD_BDPL(snum), (uint32_t)dmaaddr); hda_write4(sc, HDAUDIO_SD_BDPU(snum), (uint32_t)(dmaaddr >> 32)); hda_write2(sc, HDAUDIO_SD_LVI(snum), (cnt - 1) & 0xff); /* * Program cyclic buffer length */ hda_write4(sc, HDAUDIO_SD_CBL(snum), dmasize); /* * Program stream number (tag). Although controller hardware is * capable of transmitting any stream number (0-15), by convention * stream 0 is reserved as unused by software, so that converters * whose stream numbers have been reset to 0 do not unintentionally * decode data not intended for them. */ ctl2 = hda_read1(sc, HDAUDIO_SD_CTL2(snum)); ctl2 &= ~0xf0; ctl2 |= hdaudio_stream_tag(st) << 4; hda_write1(sc, HDAUDIO_SD_CTL2(snum), ctl2); /* * Program stream format */ fmt = hdaudio_stream_param(st, params) | HDAUDIO_FMT_CHAN(params->channels); hda_write2(sc, HDAUDIO_SD_FMT(snum), fmt); /* * Switch on interrupts for this stream */ intctl = hda_read4(sc, HDAUDIO_MMIO_INTCTL); intctl |= (1 << st->st_shift); hda_write4(sc, HDAUDIO_MMIO_INTCTL, intctl); /* * Start running the stream */ ctl0 = hda_read1(sc, HDAUDIO_SD_CTL0(snum)); ctl0 |= HDAUDIO_CTL_DEIE | HDAUDIO_CTL_FEIE | HDAUDIO_CTL_IOCE | HDAUDIO_CTL_RUN; hda_write1(sc, HDAUDIO_SD_CTL0(snum), ctl0); } void hdaudio_stream_stop(struct hdaudio_stream *st) { struct hdaudio_softc *sc = st->st_host; uint32_t intctl; uint8_t ctl0; int snum = st->st_shift; /* * Stop running the stream */ ctl0 = hda_read1(sc, HDAUDIO_SD_CTL0(snum)); ctl0 &= ~(HDAUDIO_CTL_DEIE | HDAUDIO_CTL_FEIE | HDAUDIO_CTL_IOCE | HDAUDIO_CTL_RUN); hda_write1(sc, HDAUDIO_SD_CTL0(snum), ctl0); /* * Switch off interrupts for this stream */ intctl = hda_read4(sc, HDAUDIO_MMIO_INTCTL); intctl &= ~(1 << st->st_shift); hda_write4(sc, HDAUDIO_MMIO_INTCTL, intctl); } /* * /dev/hdaudioN interface */ static const char * hdaudioioctl_fgrp_to_cstr(enum function_group_type type) { switch (type) { case HDAUDIO_GROUP_TYPE_AFG: return "afg"; case HDAUDIO_GROUP_TYPE_VSM_FG: return "vsmfg"; default: return "unknown"; } } static struct hdaudio_function_group * hdaudioioctl_fgrp_lookup(struct hdaudio_softc *sc, int codecid, int nid) { struct hdaudio_codec *co; struct hdaudio_function_group *fg = NULL; int i; if (codecid < 0 || codecid >= HDAUDIO_MAX_CODECS) return NULL; co = &sc->sc_codec[codecid]; if (co->co_valid == false) return NULL; for (i = 0; i < co->co_nfg; i++) if (co->co_fg[i].fg_nid == nid) { fg = &co->co_fg[i]; break; } return fg; } static int hdaudioioctl_fgrp_info(struct hdaudio_softc *sc, prop_dictionary_t request, prop_dictionary_t response) { struct hdaudio_codec *co; struct hdaudio_function_group *fg; prop_array_t array; prop_dictionary_t dict; int codecid, fgid; array = prop_array_create(); if (array == NULL) return ENOMEM; for (codecid = 0; codecid < HDAUDIO_MAX_CODECS; codecid++) { co = &sc->sc_codec[codecid]; if (co->co_valid == false) continue; for (fgid = 0; fgid < co->co_nfg; fgid++) { fg = &co->co_fg[fgid]; dict = prop_dictionary_create(); if (dict == NULL) return ENOMEM; prop_dictionary_set_string_nocopy(dict, "type", hdaudioioctl_fgrp_to_cstr(fg->fg_type)); prop_dictionary_set_int16(dict, "nid", fg->fg_nid); prop_dictionary_set_int16(dict, "codecid", codecid); prop_dictionary_set_uint16(dict, "vendor-id", fg->fg_vendor); prop_dictionary_set_uint16(dict, "product-id", fg->fg_product); prop_dictionary_set_uint32(dict, "subsystem-id", sc->sc_subsystem); if (fg->fg_device) prop_dictionary_set_string(dict, "device", device_xname(fg->fg_device)); else prop_dictionary_set_string_nocopy(dict, "device", "<none>"); prop_array_add(array, dict); } } prop_dictionary_set(response, "function-group-info", array); return 0; } static int hdaudioioctl_fgrp_getconfig(struct hdaudio_softc *sc, prop_dictionary_t request, prop_dictionary_t response) { struct hdaudio_function_group *fg; prop_dictionary_t dict; prop_array_t array; uint32_t nodecnt, wcap, config; int16_t codecid, nid, i; int startnode, endnode; if (!prop_dictionary_get_int16(request, "codecid", &codecid) || !prop_dictionary_get_int16(request, "nid", &nid)) return EINVAL; fg = hdaudioioctl_fgrp_lookup(sc, codecid, nid); if (fg == NULL) return ENODEV; array = prop_array_create(); if (array == NULL) return ENOMEM; nodecnt = hdaudio_command(fg->fg_codec, fg->fg_nid, CORB_GET_PARAMETER, COP_SUBORDINATE_NODE_COUNT); startnode = COP_NODECNT_STARTNODE(nodecnt); endnode = startnode + COP_NODECNT_NUMNODES(nodecnt); for (i = startnode; i < endnode; i++) { wcap = hdaudio_command(fg->fg_codec, i, CORB_GET_PARAMETER, COP_AUDIO_WIDGET_CAPABILITIES); if (COP_AWCAP_TYPE(wcap) != COP_AWCAP_TYPE_PIN_COMPLEX) continue; config = hdaudio_command(fg->fg_codec, i, CORB_GET_CONFIGURATION_DEFAULT, 0); dict = prop_dictionary_create(); if (dict == NULL) return ENOMEM; prop_dictionary_set_int16(dict, "nid", i); prop_dictionary_set_uint32(dict, "config", config); prop_array_add(array, dict); } prop_dictionary_set(response, "pin-config", array); return 0; } static int hdaudioioctl_fgrp_setconfig(struct hdaudio_softc *sc, prop_dictionary_t request, prop_dictionary_t response) { struct hdaudio_function_group *fg; prop_array_t config; int16_t codecid, nid; int err; if (!prop_dictionary_get_int16(request, "codecid", &codecid) || !prop_dictionary_get_int16(request, "nid", &nid)) return EINVAL; fg = hdaudioioctl_fgrp_lookup(sc, codecid, nid); if (fg == NULL) return ENODEV; if (fg->fg_device) { err = config_detach(fg->fg_device, 0); if (err) return err; fg->fg_device = NULL; } /* "pin-config" may be NULL, this means "use BIOS configuration" */ config = prop_dictionary_get(request, "pin-config"); if (config && prop_object_type(config) != PROP_TYPE_ARRAY) { prop_object_release(config); return EINVAL; } hdaudio_attach_fg(fg, config); if (config) prop_object_release(config); return 0; } static int hdaudio_dispatch_fgrp_ioctl(struct hdaudio_softc *sc, u_long cmd, prop_dictionary_t request, prop_dictionary_t response) { struct hdaudio_function_group *fg; int (*infocb)(void *, prop_dictionary_t, prop_dictionary_t); prop_dictionary_t fgrp_dict; uint64_t info_fn; int16_t codecid, nid; void *fgrp_sc; bool rv; int err; if (!prop_dictionary_get_int16(request, "codecid", &codecid) || !prop_dictionary_get_int16(request, "nid", &nid)) return EINVAL; fg = hdaudioioctl_fgrp_lookup(sc, codecid, nid); if (fg == NULL) return ENODEV; if (fg->fg_device == NULL) return ENXIO; fgrp_sc = device_private(fg->fg_device); fgrp_dict = device_properties(fg->fg_device); switch (fg->fg_type) { case HDAUDIO_GROUP_TYPE_AFG: switch (cmd) { case HDAUDIO_FGRP_CODEC_INFO: rv = prop_dictionary_get_uint64(fgrp_dict, "codecinfo-callback", &info_fn); if (!rv) return ENXIO; infocb = (void *)(uintptr_t)info_fn; err = infocb(fgrp_sc, request, response); break; case HDAUDIO_FGRP_WIDGET_INFO: rv = prop_dictionary_get_uint64(fgrp_dict, "widgetinfo-callback", &info_fn); if (!rv) return ENXIO; infocb = (void *)(uintptr_t)info_fn; err = infocb(fgrp_sc, request, response); break; default: err = EINVAL; break; } break; default: err = EINVAL; break; } return err; } int hdaudioopen(dev_t dev, int flag, int mode, struct lwp *l) { device_t self; self = device_lookup(&hdaudio_cd, HDAUDIOUNIT(dev)); if (self == NULL) return ENXIO; return 0; } int hdaudioclose(dev_t dev, int flag, int mode, struct lwp *l) { return 0; } int hdaudioioctl(dev_t dev, u_long cmd, void *addr, int flag, struct lwp *l) { struct hdaudio_softc *sc; struct plistref *pref = addr; prop_dictionary_t request, response; int err; sc = device_lookup_private(&hdaudio_cd, HDAUDIOUNIT(dev)); if (sc == NULL) return ENXIO; response = prop_dictionary_create(); if (response == NULL) return ENOMEM; err = prop_dictionary_copyin_ioctl(pref, cmd, &request); if (err) { prop_object_release(response); return err; } switch (cmd) { case HDAUDIO_FGRP_INFO: err = hdaudioioctl_fgrp_info(sc, request, response); break; case HDAUDIO_FGRP_GETCONFIG: err = hdaudioioctl_fgrp_getconfig(sc, request, response); break; case HDAUDIO_FGRP_SETCONFIG: err = hdaudioioctl_fgrp_setconfig(sc, request, response); break; case HDAUDIO_FGRP_CODEC_INFO: case HDAUDIO_FGRP_WIDGET_INFO: err = hdaudio_dispatch_fgrp_ioctl(sc, cmd, request, response); break; default: err = EINVAL; break; } if (!err) err = prop_dictionary_copyout_ioctl(pref, cmd, response); if (response) prop_object_release(response); prop_object_release(request); return err; } MODULE(MODULE_CLASS_DRIVER, hdaudio, "audio"); #ifdef _MODULE static const struct cfiattrdata hdaudiobuscf_iattrdata = { "hdaudiobus", 1, { { "nid", "-1", -1 }, } }; static const struct cfiattrdata * const hdaudio_attrs[] = { &hdaudiobuscf_iattrdata, NULL }; CFDRIVER_DECL(hdaudio, DV_AUDIODEV, hdaudio_attrs); #endif static int hdaudio_modcmd(modcmd_t cmd, void *opaque) { int error = 0; #ifdef _MODULE int bmaj = -1, cmaj = -1; #endif switch (cmd) { case MODULE_CMD_INIT: #ifdef _MODULE error = devsw_attach("hdaudio", NULL, &bmaj, &hdaudio_cdevsw, &cmaj); if (error) break; error = config_cfdriver_attach(&hdaudio_cd); if (error) devsw_detach(NULL, &hdaudio_cdevsw); #endif break; case MODULE_CMD_FINI: #ifdef _MODULE error = config_cfdriver_detach(&hdaudio_cd); if (error) break; devsw_detach(NULL, &hdaudio_cdevsw); #endif break; default: error = ENOTTY; break; } return error; } DEV_VERBOSE_DEFINE(hdaudio); |
| 2823 2617 2617 2030 1931 2460 1351 147 2612 2476 1354 2793 2796 2797 2792 2794 2794 873 2645 2795 1401 2657 70 2600 649 1341 1341 85 623 48 416 939 1146 1146 66 1339 1339 1340 1 28 1901 60 1661 1662 1663 1661 1663 1392 1301 1053 1053 1662 1661 1160 1159 146 1063 176 2 2 1063 1060 1061 725 338 340 340 2469 2467 1804 1540 1541 1505 1541 2815 196 2798 2797 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 197 198 199 200 201 202 203 204 205 206 207 208 209 210 211 212 213 214 215 216 217 218 219 220 221 222 223 224 225 226 227 228 229 230 231 232 233 234 235 236 237 238 239 240 241 242 243 244 245 246 247 248 249 250 251 252 253 254 255 256 257 258 259 260 261 262 263 264 265 266 267 268 269 270 271 272 273 274 275 276 277 278 279 280 281 282 283 284 285 286 287 288 289 290 291 292 293 294 295 296 297 298 299 300 301 302 303 304 305 306 307 308 309 310 311 312 313 314 315 316 317 318 319 320 321 322 323 324 325 326 327 328 329 330 331 332 333 334 335 336 337 338 339 340 341 342 343 344 345 346 347 348 349 350 351 352 353 354 355 356 357 358 359 360 361 362 363 364 365 366 367 368 369 370 371 372 373 374 375 376 377 378 379 380 381 382 383 384 385 386 387 388 389 390 391 392 393 394 395 396 397 398 399 400 401 402 403 404 405 406 407 408 409 410 411 412 413 414 415 416 417 418 419 420 421 422 423 424 425 426 427 428 429 430 431 432 433 434 435 436 437 438 439 440 441 442 443 444 445 446 447 448 449 450 451 452 453 454 455 456 457 458 459 460 461 462 463 464 465 466 467 468 469 470 471 472 473 474 475 476 477 478 479 480 481 482 483 484 485 486 487 488 489 490 491 492 493 494 495 496 497 498 499 500 501 502 503 504 505 506 507 508 509 510 511 512 513 514 515 516 517 518 519 520 521 522 523 524 525 526 527 528 529 530 531 532 533 534 535 536 537 538 539 540 541 542 543 544 545 546 547 548 549 550 551 552 553 554 555 556 557 558 559 560 561 562 563 564 565 566 567 568 569 570 571 572 573 574 575 576 577 578 579 580 581 582 583 584 585 586 587 588 589 590 591 592 593 594 595 596 597 598 599 600 601 602 603 604 605 606 607 608 609 610 611 612 613 614 615 616 617 618 619 620 621 622 623 624 625 626 627 628 629 630 631 632 633 634 635 636 637 638 639 640 641 642 643 644 645 646 647 648 649 650 651 652 653 654 655 656 657 658 659 660 661 662 663 664 665 666 667 668 669 670 671 672 673 674 675 676 677 678 679 680 681 682 683 684 685 686 687 688 689 690 691 692 693 694 695 696 697 698 699 700 701 702 703 704 705 706 707 708 709 710 711 712 713 714 715 716 717 718 719 720 721 722 723 724 725 726 727 728 729 730 731 732 733 734 735 736 737 738 739 740 741 742 743 744 745 746 747 748 749 750 751 752 753 754 755 756 757 758 759 760 761 762 763 764 765 766 767 768 769 770 771 772 773 774 775 776 777 778 779 780 781 782 783 784 785 786 787 788 789 790 791 792 793 794 795 796 797 798 799 800 801 802 803 804 805 806 807 808 809 810 811 812 813 814 815 816 817 818 819 820 821 822 823 824 825 826 827 828 829 830 831 832 833 834 835 836 837 838 839 840 841 842 843 844 845 846 847 848 849 850 851 852 853 854 855 856 857 858 859 860 861 862 863 864 865 866 867 868 869 870 871 872 873 874 875 876 877 878 879 880 881 882 883 884 885 886 887 888 889 890 891 892 893 894 895 896 897 898 899 900 901 902 903 904 905 906 907 908 909 910 911 912 913 914 915 916 917 918 919 920 921 922 923 924 925 926 927 928 929 930 931 932 933 934 935 936 937 938 939 940 941 942 943 944 945 946 947 948 949 950 951 952 953 954 955 956 957 958 959 960 961 962 963 964 965 966 967 968 969 970 971 972 973 974 975 976 977 978 979 980 981 982 983 984 985 986 987 988 989 990 991 992 993 994 995 996 997 998 999 1000 1001 1002 1003 1004 1005 1006 1007 1008 1009 1010 1011 1012 1013 1014 1015 1016 1017 1018 1019 1020 1021 1022 1023 1024 1025 1026 1027 1028 1029 1030 1031 1032 1033 1034 1035 1036 1037 1038 1039 1040 1041 1042 1043 1044 1045 1046 1047 1048 1049 1050 1051 1052 1053 1054 1055 1056 1057 1058 1059 1060 1061 1062 1063 1064 1065 1066 1067 1068 1069 1070 1071 1072 1073 1074 1075 1076 1077 1078 1079 1080 1081 1082 1083 1084 1085 1086 1087 1088 1089 1090 1091 1092 1093 1094 1095 1096 1097 1098 1099 1100 1101 1102 1103 1104 1105 1106 1107 1108 1109 1110 1111 1112 1113 1114 1115 1116 1117 1118 1119 1120 1121 1122 1123 1124 1125 1126 1127 1128 1129 1130 1131 1132 1133 1134 1135 1136 1137 1138 1139 1140 1141 1142 1143 1144 1145 1146 1147 1148 1149 1150 1151 1152 1153 1154 1155 1156 1157 1158 1159 1160 1161 1162 1163 1164 1165 1166 1167 1168 1169 1170 1171 1172 1173 1174 1175 1176 1177 1178 1179 1180 1181 1182 1183 1184 1185 1186 1187 1188 1189 1190 1191 1192 1193 1194 1195 1196 1197 1198 1199 1200 1201 1202 1203 1204 1205 1206 1207 1208 1209 1210 1211 1212 1213 1214 1215 1216 1217 1218 1219 1220 1221 1222 1223 1224 1225 1226 1227 1228 1229 1230 1231 1232 1233 1234 1235 1236 1237 1238 1239 1240 1241 1242 1243 1244 1245 | /* $NetBSD: kern_runq.c,v 1.69 2020/05/23 21:24:41 ad Exp $ */ /*- * Copyright (c) 2019, 2020 The NetBSD Foundation, Inc. * All rights reserved. * * This code is derived from software contributed to The NetBSD Foundation * by Andrew Doran. * * Redistribution and use in source and binary forms, with or without * modification, are permitted provided that the following conditions * are met: * 1. Redistributions of source code must retain the above copyright * notice, this list of conditions and the following disclaimer. * 2. Redistributions in binary form must reproduce the above copyright * notice, this list of conditions and the following disclaimer in the * documentation and/or other materials provided with the distribution. * * THIS SOFTWARE IS PROVIDED BY THE NETBSD FOUNDATION, INC. AND CONTRIBUTORS * ``AS IS'' AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED * TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR * PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE FOUNDATION OR CONTRIBUTORS * BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR * CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF * SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS * INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN * CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) * ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE * POSSIBILITY OF SUCH DAMAGE. */ /* * Copyright (c) 2007, 2008 Mindaugas Rasiukevicius <rmind at NetBSD org> * All rights reserved. * * Redistribution and use in source and binary forms, with or without * modification, are permitted provided that the following conditions * are met: * 1. Redistributions of source code must retain the above copyright * notice, this list of conditions and the following disclaimer. * 2. Redistributions in binary form must reproduce the above copyright * notice, this list of conditions and the following disclaimer in the * documentation and/or other materials provided with the distribution. * * THIS SOFTWARE IS PROVIDED BY THE AUTHOR AND CONTRIBUTORS ``AS IS'' AND * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE * ARE DISCLAIMED. IN NO EVENT SHALL THE AUTHOR OR CONTRIBUTORS BE LIABLE * FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL * DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS * OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) * HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT * LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY * OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF * SUCH DAMAGE. */ #include <sys/cdefs.h> __KERNEL_RCSID(0, "$NetBSD: kern_runq.c,v 1.69 2020/05/23 21:24:41 ad Exp $"); #include "opt_dtrace.h" #include <sys/param.h> #include <sys/kernel.h> #include <sys/bitops.h> #include <sys/cpu.h> #include <sys/idle.h> #include <sys/intr.h> #include <sys/kmem.h> #include <sys/lwp.h> #include <sys/mutex.h> #include <sys/proc.h> #include <sys/pset.h> #include <sys/sched.h> #include <sys/syscallargs.h> #include <sys/sysctl.h> #include <sys/systm.h> #include <sys/types.h> #include <sys/evcnt.h> #include <sys/atomic.h> /* * Bits per map. */ #define BITMAP_BITS (32) #define BITMAP_SHIFT (5) #define BITMAP_MSB (0x80000000U) #define BITMAP_MASK (BITMAP_BITS - 1) const int schedppq = 1; static void *sched_getrq(struct schedstate_percpu *, const pri_t); #ifdef MULTIPROCESSOR static lwp_t * sched_catchlwp(struct cpu_info *); #endif /* * Preemption control. */ #ifdef __HAVE_PREEMPTION # ifdef DEBUG int sched_kpreempt_pri = 0; # else int sched_kpreempt_pri = PRI_USER_RT; # endif #else int sched_kpreempt_pri = 1000; #endif /* * Migration and balancing. */ static u_int cacheht_time; /* Cache hotness time */ static u_int min_catch; /* Minimal LWP count for catching */ static u_int skim_interval; /* Rate limit for stealing LWPs */ #ifdef KDTRACE_HOOKS struct lwp *curthread; #endif void runq_init(void) { /* Pulling from remote packages, LWP must not have run for 10ms. */ cacheht_time = 10; /* Minimal count of LWPs for catching */ min_catch = 1; /* Steal from other CPUs at most every 10ms. */ skim_interval = 10; } void sched_cpuattach(struct cpu_info *ci) { struct schedstate_percpu *spc; size_t size; void *p; u_int i; spc = &ci->ci_schedstate; spc->spc_nextpkg = ci; if (spc->spc_lwplock == NULL) { spc->spc_lwplock = mutex_obj_alloc(MUTEX_DEFAULT, IPL_SCHED); } if (ci == lwp0.l_cpu) { /* Initialize the scheduler structure of the primary LWP */ lwp0.l_mutex = spc->spc_lwplock; } if (spc->spc_mutex != NULL) { /* Already initialized. */ return; } /* Allocate the run queue */ size = roundup2(sizeof(spc->spc_queue[0]) * PRI_COUNT, coherency_unit) + coherency_unit; p = kmem_alloc(size, KM_SLEEP); spc->spc_queue = (void *)roundup2((uintptr_t)p, coherency_unit); /* Initialize run queues */ spc->spc_mutex = mutex_obj_alloc(MUTEX_DEFAULT, IPL_SCHED); for (i = 0; i < PRI_COUNT; i++) TAILQ_INIT(&spc->spc_queue[i]); } /* * Control of the runqueue. */ static inline void * sched_getrq(struct schedstate_percpu *spc, const pri_t prio) { KASSERT(prio < PRI_COUNT); return &spc->spc_queue[prio]; } /* * Put an LWP onto a run queue. The LWP must be locked by spc_mutex for * l_cpu. */ void sched_enqueue(struct lwp *l) { struct schedstate_percpu *spc; TAILQ_HEAD(, lwp) *q_head; const pri_t eprio = lwp_eprio(l); struct cpu_info *ci; ci = l->l_cpu; spc = &ci->ci_schedstate; KASSERT(lwp_locked(l, l->l_cpu->ci_schedstate.spc_mutex)); /* Enqueue the thread */ q_head = sched_getrq(spc, eprio); if (TAILQ_EMPTY(q_head)) { u_int i; uint32_t q; /* Mark bit */ i = eprio >> BITMAP_SHIFT; q = BITMAP_MSB >> (eprio & BITMAP_MASK); KASSERT((spc->spc_bitmap[i] & q) == 0); spc->spc_bitmap[i] |= q; } /* * Determine run queue position according to POSIX. XXX Explicitly * lowering a thread's priority with pthread_setschedparam() is not * handled. */ if ((l->l_pflag & LP_PREEMPTING) != 0) { switch (l->l_class) { case SCHED_OTHER: TAILQ_INSERT_TAIL(q_head, l, l_runq); break; case SCHED_FIFO: TAILQ_INSERT_HEAD(q_head, l, l_runq); break; case SCHED_RR: if (getticks() - l->l_rticks >= sched_rrticks) { TAILQ_INSERT_TAIL(q_head, l, l_runq); } else { TAILQ_INSERT_HEAD(q_head, l, l_runq); } break; default: /* SCHED_OTHER */ panic("sched_enqueue: LWP %p has class %d\n", l, l->l_class); } } else { TAILQ_INSERT_TAIL(q_head, l, l_runq); } spc->spc_flags &= ~SPCF_IDLE; spc->spc_count++; if ((l->l_pflag & LP_BOUND) == 0) { atomic_store_relaxed(&spc->spc_mcount, atomic_load_relaxed(&spc->spc_mcount) + 1); } /* * Update the value of highest priority in the runqueue, * if priority of this thread is higher. */ if (eprio > spc->spc_maxpriority) spc->spc_maxpriority = eprio; sched_newts(l); } /* * Remove and LWP from the run queue it's on. The LWP must be in state * LSRUN. */ void sched_dequeue(struct lwp *l) { TAILQ_HEAD(, lwp) *q_head; struct schedstate_percpu *spc; const pri_t eprio = lwp_eprio(l); spc = &l->l_cpu->ci_schedstate; KASSERT(lwp_locked(l, spc->spc_mutex)); KASSERT(eprio <= spc->spc_maxpriority); KASSERT(spc->spc_bitmap[eprio >> BITMAP_SHIFT] != 0); KASSERT(spc->spc_count > 0); if (spc->spc_migrating == l) spc->spc_migrating = NULL; spc->spc_count--; if ((l->l_pflag & LP_BOUND) == 0) { atomic_store_relaxed(&spc->spc_mcount, atomic_load_relaxed(&spc->spc_mcount) - 1); } q_head = sched_getrq(spc, eprio); TAILQ_REMOVE(q_head, l, l_runq); if (TAILQ_EMPTY(q_head)) { u_int i; uint32_t q; /* Unmark bit */ i = eprio >> BITMAP_SHIFT; q = BITMAP_MSB >> (eprio & BITMAP_MASK); KASSERT((spc->spc_bitmap[i] & q) != 0); spc->spc_bitmap[i] &= ~q; /* * Update the value of highest priority in the runqueue, in a * case it was a last thread in the queue of highest priority. */ if (eprio != spc->spc_maxpriority) return; do { if (spc->spc_bitmap[i] != 0) { q = ffs(spc->spc_bitmap[i]); spc->spc_maxpriority = (i << BITMAP_SHIFT) + (BITMAP_BITS - q); return; } } while (i--); /* If not found - set the lowest value */ spc->spc_maxpriority = 0; } } /* * Cause a preemption on the given CPU, if the priority "pri" is higher * priority than the running LWP. If "unlock" is specified, and ideally it * will be for concurrency reasons, spc_mutex will be dropped before return. */ void sched_resched_cpu(struct cpu_info *ci, pri_t pri, bool unlock) { struct schedstate_percpu *spc; u_int o, n, f; lwp_t *l; spc = &ci->ci_schedstate; KASSERT(mutex_owned(spc->spc_mutex)); /* * If the priority level we're evaluating wouldn't cause a new LWP * to be run on the CPU, then we have nothing to do. */ if (pri <= spc->spc_curpriority || !mp_online) { if (__predict_true(unlock)) { spc_unlock(ci); } return; } /* * Figure out what kind of preemption we should do. */ l = ci->ci_onproc; if ((l->l_flag & LW_IDLE) != 0) { f = RESCHED_IDLE | RESCHED_UPREEMPT; } else if (pri >= sched_kpreempt_pri && (l->l_pflag & LP_INTR) == 0) { /* We can't currently preempt softints - should be able to. */ #ifdef __HAVE_PREEMPTION f = RESCHED_KPREEMPT; #else /* Leave door open for test: set kpreempt_pri with sysctl. */ f = RESCHED_UPREEMPT; #endif /* * l_dopreempt must be set with the CPU locked to sync with * mi_switch(). It must also be set with an atomic to sync * with kpreempt(). */ atomic_or_uint(&l->l_dopreempt, DOPREEMPT_ACTIVE); } else { f = RESCHED_UPREEMPT; } if (ci != curcpu()) { f |= RESCHED_REMOTE; } /* * Things can start as soon as ci_want_resched is touched: x86 has * an instruction that monitors the memory cell it's in. Drop the * schedstate lock in advance, otherwise the remote CPU can awaken * and immediately block on the lock. */ if (__predict_true(unlock)) { spc_unlock(ci); } /* * The caller almost always has a second scheduler lock held: either * the running LWP lock (spc_lwplock), or a sleep queue lock. That * keeps preemption disabled, which among other things ensures all * LWPs involved won't be freed while we're here (see lwp_dtor()). */ KASSERT(kpreempt_disabled()); for (o = 0;; o = n) { n = atomic_cas_uint(&ci->ci_want_resched, o, o | f); if (__predict_true(o == n)) { /* * We're the first to set a resched on the CPU. Try * to avoid causing a needless trip through trap() * to handle an AST fault, if it's known the LWP * will either block or go through userret() soon. */ if (l != curlwp || cpu_intr_p()) { cpu_need_resched(ci, l, f); } break; } if (__predict_true( (n & (RESCHED_KPREEMPT|RESCHED_UPREEMPT)) >= (f & (RESCHED_KPREEMPT|RESCHED_UPREEMPT)))) { /* Already in progress, nothing to do. */ break; } } } /* * Cause a preemption on the given CPU, if the priority of LWP "l" in state * LSRUN, is higher priority than the running LWP. If "unlock" is * specified, and ideally it will be for concurrency reasons, spc_mutex will * be dropped before return. */ void sched_resched_lwp(struct lwp *l, bool unlock) { struct cpu_info *ci = l->l_cpu; KASSERT(lwp_locked(l, ci->ci_schedstate.spc_mutex)); KASSERT(l->l_stat == LSRUN); sched_resched_cpu(ci, lwp_eprio(l), unlock); } /* * Migration and balancing. */ #ifdef MULTIPROCESSOR /* * Estimate if LWP is cache-hot. */ static inline bool lwp_cache_hot(const struct lwp *l) { /* Leave new LWPs in peace, determination has already been made. */ if (l->l_stat == LSIDL) return true; if (__predict_false(l->l_slptime != 0 || l->l_rticks == 0)) return false; return (getticks() - l->l_rticks < mstohz(cacheht_time)); } /* * Check if LWP can migrate to the chosen CPU. */ static inline bool sched_migratable(const struct lwp *l, struct cpu_info *ci) { const struct schedstate_percpu *spc = &ci->ci_schedstate; KASSERT(lwp_locked(__UNCONST(l), NULL)); /* Is CPU offline? */ if (__predict_false(spc->spc_flags & SPCF_OFFLINE)) return false; /* Is affinity set? */ if (__predict_false(l->l_affinity)) return kcpuset_isset(l->l_affinity, cpu_index(ci)); /* Is there a processor-set? */ return (spc->spc_psid == l->l_psid); } /* * A small helper to do round robin through CPU packages. */ static struct cpu_info * sched_nextpkg(void) { struct schedstate_percpu *spc = &curcpu()->ci_schedstate; spc->spc_nextpkg = spc->spc_nextpkg->ci_sibling[CPUREL_PACKAGE1ST]; return spc->spc_nextpkg; } /* * Find a CPU to run LWP "l". Look for the CPU with the lowest priority * thread. In case of equal priority, prefer first class CPUs, and amongst * the remainder choose the CPU with the fewest runqueue entries. * * Begin the search in the CPU package which "pivot" is a member of. */ static struct cpu_info * __noinline sched_bestcpu(struct lwp *l, struct cpu_info *pivot) { struct cpu_info *bestci, *curci, *outer; struct schedstate_percpu *bestspc, *curspc; pri_t bestpri, curpri; /* * If this fails (it shouldn't), run on the given CPU. This also * gives us a weak preference for "pivot" to begin with. */ bestci = pivot; bestspc = &bestci->ci_schedstate; if (sched_migratable(l, bestci)) { bestpri = MAX(bestspc->spc_curpriority, bestspc->spc_maxpriority); } else { /* Invalidate the priority. */ bestpri = PRI_COUNT; } /* In the outer loop scroll through all CPU packages. */ pivot = pivot->ci_package1st; outer = pivot; do { /* In the inner loop scroll through all CPUs in package. */ curci = outer; do { if (!sched_migratable(l, curci)) { continue; } curspc = &curci->ci_schedstate; /* If this CPU is idle and 1st class, we're done. */ if ((curspc->spc_flags & (SPCF_IDLE | SPCF_1STCLASS)) == (SPCF_IDLE | SPCF_1STCLASS)) { return curci; } curpri = MAX(curspc->spc_curpriority, curspc->spc_maxpriority); if (curpri > bestpri) { continue; } if (curpri == bestpri) { /* Prefer first class CPUs over others. */ if ((curspc->spc_flags & SPCF_1STCLASS) == 0 && (bestspc->spc_flags & SPCF_1STCLASS) != 0) { continue; } /* * Pick the least busy CPU. Make sure this is not * <=, otherwise it defeats the above preference. */ if (bestspc->spc_count < curspc->spc_count) { continue; } } bestpri = curpri; bestci = curci; bestspc = curspc; } while (curci = curci->ci_sibling[CPUREL_PACKAGE], curci != outer); } while (outer = outer->ci_sibling[CPUREL_PACKAGE1ST], outer != pivot); return bestci; } /* * Estimate the migration of LWP to the other CPU. * Take and return the CPU, if migration is needed. */ struct cpu_info * sched_takecpu(struct lwp *l) { struct schedstate_percpu *spc, *tspc; struct cpu_info *ci, *curci, *tci; pri_t eprio; int flags; KASSERT(lwp_locked(l, NULL)); /* If thread is strictly bound, do not estimate other CPUs */ ci = l->l_cpu; if (l->l_pflag & LP_BOUND) return ci; spc = &ci->ci_schedstate; eprio = lwp_eprio(l); /* * Handle new LWPs. For vfork() with a timeshared child, make it * run on the same CPU as the parent if no other LWPs in queue. * Otherwise scatter far and wide - try for an even distribution * across all CPU packages and CPUs. */ if (l->l_stat == LSIDL) { if (curlwp->l_vforkwaiting && l->l_class == SCHED_OTHER) { if (sched_migratable(l, curlwp->l_cpu) && eprio > curlwp->l_cpu->ci_schedstate.spc_maxpriority) { return curlwp->l_cpu; } } else { return sched_bestcpu(l, sched_nextpkg()); } flags = SPCF_IDLE; } else { flags = SPCF_IDLE | SPCF_1STCLASS; } /* * Try to send the LWP back to the first CPU in the same core if * idle. This keeps LWPs clustered in the run queues of 1st class * CPUs. This implies stickiness. If we didn't find a home for * a vfork() child above, try to use any SMT sibling to help out. */ tci = ci; do { tspc = &tci->ci_schedstate; if ((tspc->spc_flags & flags) == flags && sched_migratable(l, tci)) { return tci; } tci = tci->ci_sibling[CPUREL_CORE]; } while (tci != ci); /* * Otherwise the LWP is "sticky", i.e. generally preferring to stay * on the same CPU. */ if (sched_migratable(l, ci) && (eprio > spc->spc_curpriority || (lwp_cache_hot(l) && l->l_class == SCHED_OTHER))) { return ci; } /* * If the current CPU core is idle, run there and avoid the * expensive scan of CPUs below. */ curci = curcpu(); tci = curci; do { tspc = &tci->ci_schedstate; if ((tspc->spc_flags & flags) == flags && sched_migratable(l, tci)) { return tci; } tci = tci->ci_sibling[CPUREL_CORE]; } while (tci != curci); /* * Didn't find a new home above - happens infrequently. Start the * search in last CPU package that the LWP ran in, but expand to * include the whole system if needed. */ return sched_bestcpu(l, l->l_cpu); } /* * Tries to catch an LWP from the runqueue of other CPU. */ static struct lwp * sched_catchlwp(struct cpu_info *ci) { struct cpu_info *curci = curcpu(); struct schedstate_percpu *spc, *curspc; TAILQ_HEAD(, lwp) *q_head; struct lwp *l; bool gentle; curspc = &curci->ci_schedstate; spc = &ci->ci_schedstate; /* * Be more aggressive if this CPU is first class, and the other * is not. */ gentle = ((curspc->spc_flags & SPCF_1STCLASS) == 0 || (spc->spc_flags & SPCF_1STCLASS) != 0); if (atomic_load_relaxed(&spc->spc_mcount) < (gentle ? min_catch : 1) || curspc->spc_psid != spc->spc_psid) { spc_unlock(ci); return NULL; } /* Take the highest priority thread */ q_head = sched_getrq(spc, spc->spc_maxpriority); l = TAILQ_FIRST(q_head); for (;;) { /* Check the first and next result from the queue */ if (l == NULL) { break; } KASSERTMSG(l->l_stat == LSRUN, "%s l %p (%s) l_stat %d", ci->ci_data.cpu_name, l, (l->l_name ? l->l_name : l->l_proc->p_comm), l->l_stat); /* Look for threads, whose are allowed to migrate */ if ((l->l_pflag & LP_BOUND) || (gentle && lwp_cache_hot(l)) || !sched_migratable(l, curci)) { l = TAILQ_NEXT(l, l_runq); /* XXX Gap: could walk down priority list. */ continue; } /* Grab the thread, and move to the local run queue */ sched_dequeue(l); l->l_cpu = curci; lwp_unlock_to(l, curspc->spc_mutex); sched_enqueue(l); return l; } spc_unlock(ci); return l; } /* * Called from sched_idle() to handle migration. Return the CPU that we * pushed the LWP to (may be NULL). */ static struct cpu_info * sched_idle_migrate(void) { struct cpu_info *ci = curcpu(), *tci = NULL; struct schedstate_percpu *spc, *tspc; bool dlock = false; spc = &ci->ci_schedstate; spc_lock(ci); for (;;) { struct lwp *l; l = spc->spc_migrating; if (l == NULL) break; /* * If second attempt, and target CPU has changed, * drop the old lock. */ if (dlock == true && tci != l->l_target_cpu) { KASSERT(tci != NULL); spc_unlock(tci); dlock = false; } /* * Nothing to do if destination has changed to the * local CPU, or migration was done by other CPU. */ tci = l->l_target_cpu; if (tci == NULL || tci == ci) { spc->spc_migrating = NULL; l->l_target_cpu = NULL; break; } tspc = &tci->ci_schedstate; /* * Double-lock the runqueues. * We do that only once. */ if (dlock == false) { dlock = true; if (ci < tci) { spc_lock(tci); } else if (!mutex_tryenter(tspc->spc_mutex)) { spc_unlock(ci); spc_lock(tci); spc_lock(ci); /* Check the situation again.. */ continue; } } /* Migrate the thread */ KASSERT(l->l_stat == LSRUN); spc->spc_migrating = NULL; l->l_target_cpu = NULL; sched_dequeue(l); l->l_cpu = tci; lwp_setlock(l, tspc->spc_mutex); sched_enqueue(l); sched_resched_lwp(l, true); /* tci now unlocked */ spc_unlock(ci); return tci; } if (dlock == true) { KASSERT(tci != NULL); spc_unlock(tci); } spc_unlock(ci); return NULL; } /* * Try to steal an LWP from "tci". */ static bool sched_steal(struct cpu_info *ci, struct cpu_info *tci) { struct schedstate_percpu *spc, *tspc; lwp_t *l; spc = &ci->ci_schedstate; tspc = &tci->ci_schedstate; if (atomic_load_relaxed(&tspc->spc_mcount) != 0 && spc->spc_psid == tspc->spc_psid) { spc_dlock(ci, tci); l = sched_catchlwp(tci); spc_unlock(ci); if (l != NULL) { return true; } } return false; } /* * Called from each CPU's idle loop. */ void sched_idle(void) { struct cpu_info *ci, *inner, *outer, *first, *tci, *mci; struct schedstate_percpu *spc, *tspc; struct lwp *l; ci = curcpu(); spc = &ci->ci_schedstate; tci = NULL; mci = NULL; /* * Handle LWP migrations off this CPU to another. If there a is * migration to do then remember the CPU the LWP was sent to, and * don't steal the LWP back from that CPU below. */ if (spc->spc_migrating != NULL) { mci = sched_idle_migrate(); } /* If this CPU is offline, or we have an LWP to run, we're done. */ if ((spc->spc_flags & SPCF_OFFLINE) != 0 || spc->spc_count != 0) { return; } /* Deal with SMT. */ if (ci->ci_nsibling[CPUREL_CORE] > 1) { /* Try to help our siblings out. */ tci = ci->ci_sibling[CPUREL_CORE]; while (tci != ci) { if (tci != mci && sched_steal(ci, tci)) { return; } tci = tci->ci_sibling[CPUREL_CORE]; } /* * If not the first SMT in the core, and in the default * processor set, the search ends here. */ if ((spc->spc_flags & SPCF_1STCLASS) == 0 && spc->spc_psid == PS_NONE) { return; } } /* * Find something to run, unless this CPU exceeded the rate limit. * Start looking on the current package to maximise L2/L3 cache * locality. Then expand to looking at the rest of the system. * * XXX Should probably look at 2nd class CPUs first, but they will * shed jobs via preempt() anyway. */ if (spc->spc_nextskim > getticks()) { return; } spc->spc_nextskim = getticks() + mstohz(skim_interval); /* In the outer loop scroll through all CPU packages, starting here. */ first = ci->ci_package1st; outer = first; do { /* In the inner loop scroll through all CPUs in package. */ inner = outer; do { /* Don't hit the locks unless needed. */ tspc = &inner->ci_schedstate; if (ci == inner || ci == mci || spc->spc_psid != tspc->spc_psid || atomic_load_relaxed(&tspc->spc_mcount) < min_catch) { continue; } spc_dlock(ci, inner); l = sched_catchlwp(inner); spc_unlock(ci); if (l != NULL) { /* Got it! */ return; } } while (inner = inner->ci_sibling[CPUREL_PACKAGE], inner != outer); } while (outer = outer->ci_sibling[CPUREL_PACKAGE1ST], outer != first); } /* * Called from mi_switch() when an LWP has been preempted / has yielded. * The LWP is presently in the CPU's run queue. Here we look for a better * CPU to teleport the LWP to; there may not be one. */ void sched_preempted(struct lwp *l) { const int flags = SPCF_IDLE | SPCF_1STCLASS; struct schedstate_percpu *tspc; struct cpu_info *ci, *tci; ci = l->l_cpu; tspc = &ci->ci_schedstate; KASSERT(tspc->spc_count >= 1); /* * Try to select another CPU if: * * - there is no migration pending already * - and this LWP is running on a 2nd class CPU * - or this LWP is a child of vfork() that has just done execve() */ if (l->l_target_cpu != NULL || ((tspc->spc_flags & SPCF_1STCLASS) != 0 && (l->l_pflag & LP_TELEPORT) == 0)) { return; } /* * Fast path: if the first SMT in the core is idle, send it back * there, because the cache is shared (cheap) and we want all LWPs * to be clustered on 1st class CPUs (either running there or on * their runqueues). */ tci = ci->ci_sibling[CPUREL_CORE]; while (tci != ci) { tspc = &tci->ci_schedstate; if ((tspc->spc_flags & flags) == flags && sched_migratable(l, tci)) { l->l_target_cpu = tci; l->l_pflag &= ~LP_TELEPORT; return; } tci = tci->ci_sibling[CPUREL_CORE]; } if ((l->l_pflag & LP_TELEPORT) != 0) { /* * A child of vfork(): now that the parent is released, * scatter far and wide, to match the LSIDL distribution * done in sched_takecpu(). */ l->l_pflag &= ~LP_TELEPORT; tci = sched_bestcpu(l, sched_nextpkg()); if (tci != ci) { l->l_target_cpu = tci; } } else { /* * Try to find a better CPU to take it, but don't move to * another 2nd class CPU, and don't move to a non-idle CPU, * because that would prevent SMT being used to maximise * throughput. * * Search in the current CPU package in order to try and * keep L2/L3 cache locality, but expand to include the * whole system if needed. */ tci = sched_bestcpu(l, l->l_cpu); if (tci != ci && (tci->ci_schedstate.spc_flags & flags) == flags) { l->l_target_cpu = tci; } } } /* * Called during execve() by a child of vfork(). Does two things: * * - If the parent has been awoken and put back on curcpu then give the * CPU back to the parent. * * - If curlwp is not on a 1st class CPU then find somewhere else to run, * since it dodged the distribution in sched_takecpu() when first set * runnable. */ void sched_vforkexec(struct lwp *l, bool samecpu) { KASSERT(l == curlwp); if ((samecpu && ncpu > 1) || (l->l_cpu->ci_schedstate.spc_flags & SPCF_1STCLASS) == 0) { l->l_pflag |= LP_TELEPORT; preempt(); } } #else /* * stubs for !MULTIPROCESSOR */ struct cpu_info * sched_takecpu(struct lwp *l) { return l->l_cpu; } void sched_idle(void) { } void sched_preempted(struct lwp *l) { } void sched_vforkexec(struct lwp *l, bool samecpu) { KASSERT(l == curlwp); } #endif /* MULTIPROCESSOR */ /* * Scheduling statistics and balancing. */ void sched_lwp_stats(struct lwp *l) { int batch; KASSERT(lwp_locked(l, NULL)); /* Update sleep time */ if (l->l_stat == LSSLEEP || l->l_stat == LSSTOP || l->l_stat == LSSUSPENDED) l->l_slptime++; /* * Set that thread is more CPU-bound, if sum of run time exceeds the * sum of sleep time. Check if thread is CPU-bound a first time. */ batch = (l->l_rticksum > l->l_slpticksum); if (batch != 0) { if ((l->l_flag & LW_BATCH) == 0) batch = 0; l->l_flag |= LW_BATCH; } else l->l_flag &= ~LW_BATCH; /* Reset the time sums */ l->l_slpticksum = 0; l->l_rticksum = 0; /* Scheduler-specific hook */ sched_pstats_hook(l, batch); #ifdef KDTRACE_HOOKS curthread = l; #endif } /* * Scheduler mill. */ struct lwp * sched_nextlwp(void) { struct cpu_info *ci = curcpu(); struct schedstate_percpu *spc; TAILQ_HEAD(, lwp) *q_head; struct lwp *l; /* Update the last run time on switch */ l = curlwp; l->l_rticksum += (getticks() - l->l_rticks); /* Return to idle LWP if there is a migrating thread */ spc = &ci->ci_schedstate; if (__predict_false(spc->spc_migrating != NULL)) return NULL; /* Return to idle LWP if there is no runnable job */ if (__predict_false(spc->spc_count == 0)) return NULL; /* Take the highest priority thread */ KASSERT(spc->spc_bitmap[spc->spc_maxpriority >> BITMAP_SHIFT]); q_head = sched_getrq(spc, spc->spc_maxpriority); l = TAILQ_FIRST(q_head); KASSERT(l != NULL); sched_oncpu(l); l->l_rticks = getticks(); return l; } /* * sched_curcpu_runnable_p: return if curcpu() should exit the idle loop. */ bool sched_curcpu_runnable_p(void) { const struct cpu_info *ci; const struct schedstate_percpu *spc; bool rv; kpreempt_disable(); ci = curcpu(); spc = &ci->ci_schedstate; rv = (spc->spc_count != 0); #ifndef __HAVE_FAST_SOFTINTS rv |= (ci->ci_data.cpu_softints != 0); #endif kpreempt_enable(); return rv; } /* * Sysctl nodes and initialization. */ SYSCTL_SETUP(sysctl_sched_setup, "sysctl sched setup") { const struct sysctlnode *node = NULL; sysctl_createv(clog, 0, NULL, &node, CTLFLAG_PERMANENT, CTLTYPE_NODE, "sched", SYSCTL_DESCR("Scheduler options"), NULL, 0, NULL, 0, CTL_KERN, CTL_CREATE, CTL_EOL); if (node == NULL) return; sysctl_createv(clog, 0, &node, NULL, CTLFLAG_PERMANENT | CTLFLAG_READWRITE, CTLTYPE_INT, "cacheht_time", SYSCTL_DESCR("Cache hotness time (in ms)"), NULL, 0, &cacheht_time, 0, CTL_CREATE, CTL_EOL); sysctl_createv(clog, 0, &node, NULL, CTLFLAG_PERMANENT | CTLFLAG_READWRITE, CTLTYPE_INT, "skim_interval", SYSCTL_DESCR("Rate limit for stealing from other CPUs (in ms)"), NULL, 0, &skim_interval, 0, CTL_CREATE, CTL_EOL); sysctl_createv(clog, 0, &node, NULL, CTLFLAG_PERMANENT | CTLFLAG_READWRITE, CTLTYPE_INT, "min_catch", SYSCTL_DESCR("Minimal count of threads for catching"), NULL, 0, &min_catch, 0, CTL_CREATE, CTL_EOL); sysctl_createv(clog, 0, &node, NULL, CTLFLAG_PERMANENT | CTLFLAG_READWRITE, CTLTYPE_INT, "timesoftints", SYSCTL_DESCR("Track CPU time for soft interrupts"), NULL, 0, &softint_timing, 0, CTL_CREATE, CTL_EOL); sysctl_createv(clog, 0, &node, NULL, CTLFLAG_PERMANENT | CTLFLAG_READWRITE, CTLTYPE_INT, "kpreempt_pri", SYSCTL_DESCR("Minimum priority to trigger kernel preemption"), NULL, 0, &sched_kpreempt_pri, 0, CTL_CREATE, CTL_EOL); } /* * Debugging. */ #ifdef DDB void sched_print_runqueue(void (*pr)(const char *, ...)) { struct cpu_info *ci, *tci; struct schedstate_percpu *spc; struct lwp *l; struct proc *p; CPU_INFO_ITERATOR cii; for (CPU_INFO_FOREACH(cii, ci)) { int i; spc = &ci->ci_schedstate; (*pr)("Run-queue (CPU = %u):\n", ci->ci_index); (*pr)(" pid.lid = %d.%d, r_count = %u, " "maxpri = %d, mlwp = %p\n", #ifdef MULTIPROCESSOR ci->ci_curlwp->l_proc->p_pid, ci->ci_curlwp->l_lid, #else curlwp->l_proc->p_pid, curlwp->l_lid, #endif spc->spc_count, spc->spc_maxpriority, spc->spc_migrating); i = (PRI_COUNT >> BITMAP_SHIFT) - 1; do { uint32_t q; q = spc->spc_bitmap[i]; (*pr)(" bitmap[%d] => [ %d (0x%x) ]\n", i, ffs(q), q); } while (i--); } (*pr)(" %5s %4s %4s %10s %3s %18s %4s %4s %s\n", "LID", "PRI", "EPRI", "FL", "ST", "LWP", "CPU", "TCI", "LRTICKS"); PROCLIST_FOREACH(p, &allproc) { (*pr)(" /- %d (%s)\n", (int)p->p_pid, p->p_comm); LIST_FOREACH(l, &p->p_lwps, l_sibling) { ci = l->l_cpu; tci = l->l_target_cpu; (*pr)(" | %5d %4u %4u 0x%8.8x %3s %18p %4u %4d %u\n", (int)l->l_lid, l->l_priority, lwp_eprio(l), l->l_flag, l->l_stat == LSRUN ? "RQ" : (l->l_stat == LSSLEEP ? "SQ" : "-"), l, ci->ci_index, (tci ? tci->ci_index : -1), (u_int)(getticks() - l->l_rticks)); } } } #endif |
| 1 1 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 197 198 199 200 201 202 203 204 205 206 207 208 209 210 211 212 213 214 215 216 217 218 219 220 221 222 223 224 225 226 227 228 229 230 231 232 233 234 235 236 237 238 239 240 241 242 243 244 245 246 247 248 249 250 251 252 253 254 255 256 257 258 259 260 261 262 263 264 265 266 267 268 269 270 271 272 273 274 275 276 277 278 279 280 281 282 283 284 | /* $NetBSD: bthub.c,v 1.26 2021/08/07 16:19:09 thorpej Exp $ */ /*- * Copyright (c) 2006 Itronix Inc. * All rights reserved. * * Written by Iain Hibbert for Itronix Inc. * * Redistribution and use in source and binary forms, with or without * modification, are permitted provided that the following conditions * are met: * 1. Redistributions of source code must retain the above copyright * notice, this list of conditions and the following disclaimer. * 2. Redistributions in binary form must reproduce the above copyright * notice, this list of conditions and the following disclaimer in the * documentation and/or other materials provided with the distribution. * 3. The name of Itronix Inc. may not be used to endorse * or promote products derived from this software without specific * prior written permission. * * THIS SOFTWARE IS PROVIDED BY ITRONIX INC. ``AS IS'' AND * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED * TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR * PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL ITRONIX INC. BE LIABLE FOR ANY * DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES * (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; * LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND * ON ANY THEORY OF LIABILITY, WHETHER IN * CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) * ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE * POSSIBILITY OF SUCH DAMAGE. */ #include <sys/cdefs.h> __KERNEL_RCSID(0, "$NetBSD: bthub.c,v 1.26 2021/08/07 16:19:09 thorpej Exp $"); #include <sys/param.h> #include <sys/conf.h> #include <sys/device.h> #include <sys/fcntl.h> #include <sys/kernel.h> #include <sys/queue.h> #include <sys/malloc.h> #include <sys/mbuf.h> #include <sys/proc.h> #include <sys/systm.h> #include <prop/proplib.h> #include <netbt/bluetooth.h> #include <dev/bluetooth/btdev.h> #include "ioconf.h" /***************************************************************************** * * Bluetooth Device Hub */ /* autoconf(9) glue */ static int bthub_match(device_t, cfdata_t, void *); static void bthub_attach(device_t, device_t, void *); static int bthub_detach(device_t, int); CFATTACH_DECL_NEW(bthub, 0, bthub_match, bthub_attach, bthub_detach, NULL); /* control file */ dev_type_ioctl(bthubioctl); const struct cdevsw bthub_cdevsw = { .d_open = nullopen, .d_close = nullclose, .d_read = noread, .d_write = nowrite, .d_ioctl = bthubioctl, .d_stop = nostop, .d_tty = notty, .d_poll = nopoll, .d_mmap = nommap, .d_kqfilter = nokqfilter, .d_discard = nodiscard, .d_flag = D_OTHER, }; /* bthub functions */ static int bthub_print(void *, const char *); static int bthub_pioctl(dev_t, unsigned long, prop_dictionary_t, int, struct lwp *); /***************************************************************************** * * bthub autoconf(9) routines * * A Hub is attached to each Bluetooth Controller as it is enabled */ static int bthub_match(device_t self, cfdata_t cfdata, void *arg) { return 1; } static void bthub_attach(device_t parent, device_t self, void *aux) { bdaddr_t *addr = aux; prop_dictionary_t dict; prop_object_t obj; dict = device_properties(self); obj = prop_data_create_copy(addr, sizeof(*addr)); prop_dictionary_set(dict, BTDEVladdr, obj); prop_object_release(obj); aprint_verbose(" %s %2.2x:%2.2x:%2.2x:%2.2x:%2.2x:%2.2x", BTDEVladdr, addr->b[5], addr->b[4], addr->b[3], addr->b[2], addr->b[1], addr->b[0]); aprint_normal("\n"); if (!pmf_device_register(self, NULL, NULL)) { /* * XXX this should not be allowed to happen, but * avoiding it needs a pretty big rearrangement of * device attachments. */ aprint_error_dev(self, "couldn't establish power handler\n"); } } static int bthub_detach(device_t self, int flags) { pmf_device_deregister(self); return config_detach_children(self, flags); } /***************************************************************************** * * bthub access functions to control device */ int bthubioctl(dev_t devno, unsigned long cmd, void *data, int flag, struct lwp *l) { prop_dictionary_t dict; int err; switch(cmd) { case BTDEV_ATTACH: case BTDEV_DETACH: /* load dictionary */ err = prop_dictionary_copyin_ioctl(data, cmd, &dict); if (err == 0) { err = bthub_pioctl(devno, cmd, dict, flag, l); prop_object_release(dict); } break; default: err = EPASSTHROUGH; break; } return err; } static int bthub_pioctl(dev_t devno, unsigned long cmd, prop_dictionary_t dict, int flag, struct lwp *l) { prop_data_t laddr, raddr; prop_string_t service; prop_dictionary_t prop; prop_object_t obj; device_t dev, self; deviter_t di; int unit; /* validate local address */ laddr = prop_dictionary_get(dict, BTDEVladdr); if (prop_data_size(laddr) != sizeof(bdaddr_t)) return EINVAL; /* locate the relevant bthub */ for (unit = 0 ; ; unit++) { if (unit == bthub_cd.cd_ndevs) return ENXIO; self = device_lookup(&bthub_cd, unit); if (self == NULL) continue; prop = device_properties(self); obj = prop_dictionary_get(prop, BTDEVladdr); if (prop_data_equals(laddr, obj)) break; } /* validate remote address */ raddr = prop_dictionary_get(dict, BTDEVraddr); if (prop_data_size(raddr) != sizeof(bdaddr_t) || bdaddr_any(prop_data_value(raddr))) return EINVAL; /* validate service name */ service = prop_dictionary_get(dict, BTDEVservice); if (prop_object_type(service) != PROP_TYPE_STRING) return EINVAL; /* locate matching child device, if any */ deviter_init(&di, 0); while ((dev = deviter_next(&di)) != NULL) { if (device_parent(dev) != self) continue; prop = device_properties(dev); obj = prop_dictionary_get(prop, BTDEVraddr); if (!prop_object_equals(raddr, obj)) continue; obj = prop_dictionary_get(prop, BTDEVservice); if (!prop_object_equals(service, obj)) continue; break; } deviter_release(&di); switch (cmd) { case BTDEV_ATTACH: /* attach BTDEV */ if (dev != NULL) return EADDRINUSE; dev = config_found(self, dict, bthub_print, CFARGS_NONE); if (dev == NULL) return ENXIO; prop = device_properties(dev); prop_dictionary_set(prop, BTDEVladdr, laddr); prop_dictionary_set(prop, BTDEVraddr, raddr); prop_dictionary_set(prop, BTDEVservice, service); break; case BTDEV_DETACH: /* detach BTDEV */ if (dev == NULL) return ENXIO; config_detach(dev, DETACH_FORCE); break; } return 0; } static int bthub_print(void *aux, const char *pnp) { prop_dictionary_t dict = aux; prop_object_t obj; const bdaddr_t *raddr; if (pnp != NULL) { obj = prop_dictionary_get(dict, BTDEVtype); aprint_normal("%s: %s '%s',", pnp, BTDEVtype, prop_string_value(obj)); } obj = prop_dictionary_get(dict, BTDEVraddr); raddr = prop_data_value(obj); aprint_verbose(" %s %2.2x:%2.2x:%2.2x:%2.2x:%2.2x:%2.2x", BTDEVraddr, raddr->b[5], raddr->b[4], raddr->b[3], raddr->b[2], raddr->b[1], raddr->b[0]); return UNCONF; } |
| 11 30 11 9 34 6 16 16 16 11 5 5 5 1 4 15 15 15 2 13 15 15 12 12 26 118 4 114 93 93 16 7 7 5 5 5 5 5 1 4 30 30 30 30 1 24 3 13 15 12 3 1 27 16 2 2 12 25 26 26 26 14 16 16 5 5 5 1 1 4 4 5 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 197 198 199 200 201 202 203 204 205 206 207 208 209 210 211 212 213 214 215 216 217 218 219 220 221 222 223 224 225 226 227 228 229 230 231 232 233 234 235 236 237 238 239 240 241 242 243 244 245 246 247 248 249 250 251 252 253 254 255 256 257 258 259 260 261 262 263 264 265 266 267 268 269 270 271 272 273 274 275 276 277 278 279 280 281 282 283 284 285 286 287 288 289 290 291 292 293 294 295 296 297 298 299 300 301 302 303 304 305 306 307 308 309 310 311 312 313 314 315 316 317 318 319 320 321 322 323 324 325 326 327 328 329 330 331 332 333 334 335 336 337 338 339 340 341 342 343 344 345 346 347 348 349 350 351 352 353 354 355 356 357 358 359 360 361 362 363 364 365 366 367 368 369 370 371 372 373 374 375 376 377 378 379 380 381 382 383 384 385 386 387 388 389 390 391 392 393 394 395 396 397 398 399 400 401 402 403 404 405 406 407 408 409 410 411 412 413 414 415 416 417 418 419 420 421 422 423 424 425 426 427 428 429 430 431 432 433 434 435 436 437 438 439 440 441 442 443 444 445 446 447 448 449 450 451 452 453 454 455 456 457 458 459 460 461 462 463 464 465 466 467 468 469 470 471 472 473 474 475 476 477 478 479 480 481 482 483 484 485 486 487 488 489 490 491 492 493 494 495 496 497 498 499 500 501 502 503 504 505 506 507 508 509 510 511 512 513 514 515 516 517 518 519 520 521 522 523 524 525 526 527 528 529 530 531 532 533 534 535 536 537 538 539 540 541 542 543 544 545 546 547 548 549 550 551 552 553 554 555 556 557 558 559 560 561 562 563 564 565 566 567 568 569 570 571 572 573 574 575 576 577 578 579 580 581 582 583 584 585 586 587 588 589 590 591 592 593 594 595 596 597 598 599 600 601 602 603 604 605 606 607 608 609 610 611 612 613 614 615 616 617 618 619 620 621 622 623 624 625 626 627 628 629 630 631 632 633 634 635 636 637 638 639 640 641 642 643 644 645 646 647 648 649 650 651 652 653 654 655 656 657 658 659 660 661 662 663 664 665 666 667 668 669 670 671 672 673 674 675 676 677 678 679 680 681 682 683 684 685 686 687 688 689 690 691 692 693 694 695 696 697 698 699 700 701 702 703 704 705 706 707 708 709 710 711 712 713 714 715 716 717 718 719 720 721 722 723 724 725 726 727 728 729 730 731 732 733 734 735 736 737 738 739 740 741 742 743 744 745 746 747 748 749 750 751 752 753 754 755 756 757 758 759 760 761 762 763 764 765 766 767 768 769 770 771 772 773 774 775 776 777 778 779 780 781 782 783 784 785 786 787 788 789 790 791 792 793 794 795 796 797 798 799 800 801 802 803 804 805 806 807 808 809 810 811 812 813 814 815 816 817 818 819 820 821 822 823 824 825 826 827 828 829 830 831 832 833 834 835 836 837 838 839 840 841 842 843 844 845 846 847 848 849 850 851 852 853 854 855 856 857 858 859 860 861 862 863 864 865 866 867 868 869 870 871 872 873 874 875 876 877 878 879 880 881 882 883 884 885 886 887 888 889 890 891 892 893 894 895 896 897 898 899 900 901 902 903 904 905 906 907 908 909 910 911 912 913 914 915 916 917 918 919 920 921 922 923 924 925 926 927 928 929 930 931 932 933 934 935 936 937 938 939 940 941 942 943 944 945 946 947 948 949 950 951 952 953 954 955 956 957 958 959 960 961 962 963 964 965 966 967 968 969 970 971 972 973 974 975 976 977 978 979 980 981 982 983 984 985 986 987 988 989 990 991 992 993 994 995 996 997 998 999 1000 1001 1002 1003 1004 1005 1006 1007 1008 1009 1010 1011 1012 1013 1014 1015 1016 1017 1018 1019 1020 1021 1022 1023 1024 1025 1026 1027 1028 1029 1030 1031 1032 1033 1034 1035 1036 1037 1038 1039 1040 1041 1042 1043 1044 1045 1046 1047 1048 1049 1050 1051 1052 1053 1054 1055 1056 1057 1058 1059 1060 1061 1062 1063 1064 1065 1066 1067 1068 1069 1070 1071 1072 1073 1074 1075 1076 1077 1078 1079 1080 1081 1082 1083 1084 1085 1086 1087 1088 1089 1090 1091 1092 1093 1094 1095 1096 1097 1098 1099 1100 1101 1102 1103 1104 1105 1106 1107 1108 1109 1110 1111 1112 1113 1114 1115 1116 1117 1118 1119 1120 1121 1122 1123 1124 1125 1126 1127 1128 1129 1130 1131 1132 1133 1134 1135 1136 1137 1138 1139 1140 1141 1142 1143 1144 1145 1146 1147 1148 1149 1150 1151 1152 1153 1154 1155 1156 1157 1158 1159 1160 1161 1162 1163 1164 1165 1166 1167 1168 1169 1170 1171 1172 1173 1174 1175 1176 1177 1178 1179 1180 1181 1182 1183 1184 1185 1186 1187 1188 1189 1190 1191 1192 1193 1194 1195 1196 1197 1198 1199 1200 1201 1202 1203 1204 1205 1206 1207 1208 1209 1210 1211 1212 1213 1214 1215 1216 1217 1218 1219 1220 1221 1222 1223 1224 1225 1226 1227 1228 1229 1230 1231 1232 1233 1234 1235 1236 1237 1238 1239 1240 1241 1242 1243 1244 1245 1246 1247 1248 1249 1250 1251 1252 1253 1254 1255 1256 1257 1258 1259 1260 1261 1262 1263 1264 1265 1266 1267 1268 1269 1270 1271 1272 1273 1274 1275 1276 1277 1278 1279 1280 1281 1282 1283 1284 1285 1286 1287 1288 1289 1290 1291 1292 1293 1294 1295 1296 1297 1298 1299 1300 1301 1302 1303 1304 1305 1306 1307 1308 1309 1310 1311 1312 1313 1314 1315 1316 1317 1318 1319 1320 1321 1322 1323 1324 1325 1326 1327 1328 1329 1330 1331 1332 1333 1334 1335 1336 1337 1338 1339 1340 1341 1342 1343 1344 1345 1346 1347 1348 1349 1350 1351 1352 1353 1354 1355 1356 1357 1358 1359 1360 1361 1362 1363 1364 1365 1366 1367 1368 1369 1370 1371 1372 1373 1374 1375 1376 1377 1378 1379 1380 1381 1382 1383 1384 1385 1386 1387 1388 1389 1390 1391 1392 1393 1394 1395 1396 1397 1398 1399 1400 1401 1402 1403 1404 1405 1406 1407 1408 1409 | /* $NetBSD: uvm_aobj.c,v 1.156 2022/05/31 08:43:16 andvar Exp $ */ /* * Copyright (c) 1998 Chuck Silvers, Charles D. Cranor and * Washington University. * All rights reserved. * * Redistribution and use in source and binary forms, with or without * modification, are permitted provided that the following conditions * are met: * 1. Redistributions of source code must retain the above copyright * notice, this list of conditions and the following disclaimer. * 2. Redistributions in binary form must reproduce the above copyright * notice, this list of conditions and the following disclaimer in the * documentation and/or other materials provided with the distribution. * * THIS SOFTWARE IS PROVIDED BY THE AUTHOR ``AS IS'' AND ANY EXPRESS OR * IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES * OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. * IN NO EVENT SHALL THE AUTHOR BE LIABLE FOR ANY DIRECT, INDIRECT, * INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT * NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, * DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY * THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT * (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF * THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. * * from: Id: uvm_aobj.c,v 1.1.2.5 1998/02/06 05:14:38 chs Exp */ /* * uvm_aobj.c: anonymous memory uvm_object pager * * author: Chuck Silvers <chuq@chuq.com> * started: Jan-1998 * * - design mostly from Chuck Cranor */ #include <sys/cdefs.h> __KERNEL_RCSID(0, "$NetBSD: uvm_aobj.c,v 1.156 2022/05/31 08:43:16 andvar Exp $"); #ifdef _KERNEL_OPT #include "opt_uvmhist.h" #endif #include <sys/param.h> #include <sys/systm.h> #include <sys/kernel.h> #include <sys/kmem.h> #include <sys/pool.h> #include <sys/atomic.h> #include <uvm/uvm.h> #include <uvm/uvm_page_array.h> /* * An anonymous UVM object (aobj) manages anonymous-memory. In addition to * keeping the list of resident pages, it may also keep a list of allocated * swap blocks. Depending on the size of the object, this list is either * stored in an array (small objects) or in a hash table (large objects). * * Lock order * * uao_list_lock -> * uvm_object::vmobjlock */ /* * Note: for hash tables, we break the address space of the aobj into blocks * of UAO_SWHASH_CLUSTER_SIZE pages, which shall be a power of two. */ #define UAO_SWHASH_CLUSTER_SHIFT 4 #define UAO_SWHASH_CLUSTER_SIZE (1 << UAO_SWHASH_CLUSTER_SHIFT) /* Get the "tag" for this page index. */ #define UAO_SWHASH_ELT_TAG(idx) ((idx) >> UAO_SWHASH_CLUSTER_SHIFT) #define UAO_SWHASH_ELT_PAGESLOT_IDX(idx) \ ((idx) & (UAO_SWHASH_CLUSTER_SIZE - 1)) /* Given an ELT and a page index, find the swap slot. */ #define UAO_SWHASH_ELT_PAGESLOT(elt, idx) \ ((elt)->slots[UAO_SWHASH_ELT_PAGESLOT_IDX(idx)]) /* Given an ELT, return its pageidx base. */ #define UAO_SWHASH_ELT_PAGEIDX_BASE(ELT) \ ((elt)->tag << UAO_SWHASH_CLUSTER_SHIFT) /* The hash function. */ #define UAO_SWHASH_HASH(aobj, idx) \ (&(aobj)->u_swhash[(((idx) >> UAO_SWHASH_CLUSTER_SHIFT) \ & (aobj)->u_swhashmask)]) /* * The threshold which determines whether we will use an array or a * hash table to store the list of allocated swap blocks. */ #define UAO_SWHASH_THRESHOLD (UAO_SWHASH_CLUSTER_SIZE * 4) #define UAO_USES_SWHASH(aobj) \ ((aobj)->u_pages > UAO_SWHASH_THRESHOLD) /* The number of buckets in a hash, with an upper bound. */ #define UAO_SWHASH_MAXBUCKETS 256 #define UAO_SWHASH_BUCKETS(aobj) \ (MIN((aobj)->u_pages >> UAO_SWHASH_CLUSTER_SHIFT, UAO_SWHASH_MAXBUCKETS)) /* * uao_swhash_elt: when a hash table is being used, this structure defines * the format of an entry in the bucket list. */ struct uao_swhash_elt { LIST_ENTRY(uao_swhash_elt) list; /* the hash list */ voff_t tag; /* our 'tag' */ int count; /* our number of active slots */ int slots[UAO_SWHASH_CLUSTER_SIZE]; /* the slots */ }; /* * uao_swhash: the swap hash table structure */ LIST_HEAD(uao_swhash, uao_swhash_elt); /* * uao_swhash_elt_pool: pool of uao_swhash_elt structures. * Note: pages for this pool must not come from a pageable kernel map. */ static struct pool uao_swhash_elt_pool __cacheline_aligned; /* * uvm_aobj: the actual anon-backed uvm_object * * => the uvm_object is at the top of the structure, this allows * (struct uvm_aobj *) == (struct uvm_object *) * => only one of u_swslots and u_swhash is used in any given aobj */ struct uvm_aobj { struct uvm_object u_obj; /* has: lock, pgops, #pages, #refs */ pgoff_t u_pages; /* number of pages in entire object */ int u_flags; /* the flags (see uvm_aobj.h) */ int *u_swslots; /* array of offset->swapslot mappings */ /* * hashtable of offset->swapslot mappings * (u_swhash is an array of bucket heads) */ struct uao_swhash *u_swhash; u_long u_swhashmask; /* mask for hashtable */ LIST_ENTRY(uvm_aobj) u_list; /* global list of aobjs */ int u_freelist; /* freelist to allocate pages from */ }; static void uao_free(struct uvm_aobj *); static int uao_get(struct uvm_object *, voff_t, struct vm_page **, int *, int, vm_prot_t, int, int); static int uao_put(struct uvm_object *, voff_t, voff_t, int); #if defined(VMSWAP) static struct uao_swhash_elt *uao_find_swhash_elt (struct uvm_aobj *, int, bool); static bool uao_pagein(struct uvm_aobj *, int, int); static bool uao_pagein_page(struct uvm_aobj *, int); #endif /* defined(VMSWAP) */ static struct vm_page *uao_pagealloc(struct uvm_object *, voff_t, int); /* * aobj_pager * * note that some functions (e.g. put) are handled elsewhere */ const struct uvm_pagerops aobj_pager = { .pgo_reference = uao_reference, .pgo_detach = uao_detach, .pgo_get = uao_get, .pgo_put = uao_put, }; /* * uao_list: global list of active aobjs, locked by uao_list_lock */ static LIST_HEAD(aobjlist, uvm_aobj) uao_list __cacheline_aligned; static kmutex_t uao_list_lock __cacheline_aligned; /* * hash table/array related functions */ #if defined(VMSWAP) /* * uao_find_swhash_elt: find (or create) a hash table entry for a page * offset. * * => the object should be locked by the caller */ static struct uao_swhash_elt * uao_find_swhash_elt(struct uvm_aobj *aobj, int pageidx, bool create) { struct uao_swhash *swhash; struct uao_swhash_elt *elt; voff_t page_tag; swhash = UAO_SWHASH_HASH(aobj, pageidx); page_tag = UAO_SWHASH_ELT_TAG(pageidx); /* * now search the bucket for the requested tag */ LIST_FOREACH(elt, swhash, list) { if (elt->tag == page_tag) { return elt; } } if (!create) { return NULL; } /* * allocate a new entry for the bucket and init/insert it in */ elt = pool_get(&uao_swhash_elt_pool, PR_NOWAIT); if (elt == NULL) { return NULL; } LIST_INSERT_HEAD(swhash, elt, list); elt->tag = page_tag; elt->count = 0; memset(elt->slots, 0, sizeof(elt->slots)); return elt; } /* * uao_find_swslot: find the swap slot number for an aobj/pageidx * * => object must be locked by caller */ int uao_find_swslot(struct uvm_object *uobj, int pageidx) { struct uvm_aobj *aobj = (struct uvm_aobj *)uobj; struct uao_swhash_elt *elt; KASSERT(UVM_OBJ_IS_AOBJ(uobj)); /* * if noswap flag is set, then we never return a slot */ if (aobj->u_flags & UAO_FLAG_NOSWAP) return 0; /* * if hashing, look in hash table. */ if (UAO_USES_SWHASH(aobj)) { elt = uao_find_swhash_elt(aobj, pageidx, false); return elt ? UAO_SWHASH_ELT_PAGESLOT(elt, pageidx) : 0; } /* * otherwise, look in the array */ return aobj->u_swslots[pageidx]; } /* * uao_set_swslot: set the swap slot for a page in an aobj. * * => setting a slot to zero frees the slot * => object must be locked by caller * => we return the old slot number, or -1 if we failed to allocate * memory to record the new slot number */ int uao_set_swslot(struct uvm_object *uobj, int pageidx, int slot) { struct uvm_aobj *aobj = (struct uvm_aobj *)uobj; struct uao_swhash_elt *elt; int oldslot; UVMHIST_FUNC(__func__); UVMHIST_CALLARGS(pdhist, "aobj %#jx pageidx %jd slot %jd", (uintptr_t)aobj, pageidx, slot, 0); KASSERT(rw_write_held(uobj->vmobjlock) || uobj->uo_refs == 0); KASSERT(UVM_OBJ_IS_AOBJ(uobj)); /* * if noswap flag is set, then we can't set a non-zero slot. */ if (aobj->u_flags & UAO_FLAG_NOSWAP) { KASSERTMSG(slot == 0, "uao_set_swslot: no swap object"); return 0; } /* * are we using a hash table? if so, add it in the hash. */ if (UAO_USES_SWHASH(aobj)) { /* * Avoid allocating an entry just to free it again if * the page had not swap slot in the first place, and * we are freeing. */ elt = uao_find_swhash_elt(aobj, pageidx, slot != 0); if (elt == NULL) { return slot ? -1 : 0; } oldslot = UAO_SWHASH_ELT_PAGESLOT(elt, pageidx); UAO_SWHASH_ELT_PAGESLOT(elt, pageidx) = slot; /* * now adjust the elt's reference counter and free it if we've * dropped it to zero. */ if (slot) { if (oldslot == 0) elt->count++; } else { if (oldslot) elt->count--; if (elt->count == 0) { LIST_REMOVE(elt, list); pool_put(&uao_swhash_elt_pool, elt); } } } else { /* we are using an array */ oldslot = aobj->u_swslots[pageidx]; aobj->u_swslots[pageidx] = slot; } return oldslot; } #endif /* defined(VMSWAP) */ /* * end of hash/array functions */ /* * uao_free: free all resources held by an aobj, and then free the aobj * * => the aobj should be dead */ static void uao_free(struct uvm_aobj *aobj) { struct uvm_object *uobj = &aobj->u_obj; KASSERT(UVM_OBJ_IS_AOBJ(uobj)); KASSERT(rw_write_held(uobj->vmobjlock)); uao_dropswap_range(uobj, 0, 0); rw_exit(uobj->vmobjlock); #if defined(VMSWAP) if (UAO_USES_SWHASH(aobj)) { /* * free the hash table itself. */ hashdone(aobj->u_swhash, HASH_LIST, aobj->u_swhashmask); } else { /* * free the array itself. */ kmem_free(aobj->u_swslots, aobj->u_pages * sizeof(int)); } #endif /* defined(VMSWAP) */ /* * finally free the aobj itself */ uvm_obj_destroy(uobj, true); kmem_free(aobj, sizeof(struct uvm_aobj)); } /* * pager functions */ /* * uao_create: create an aobj of the given size and return its uvm_object. * * => for normal use, flags are always zero * => for the kernel object, the flags are: * UAO_FLAG_KERNOBJ - allocate the kernel object (can only happen once) * UAO_FLAG_KERNSWAP - enable swapping of kernel object (" ") */ struct uvm_object * uao_create(voff_t size, int flags) { static struct uvm_aobj kernel_object_store; static krwlock_t bootstrap_kernel_object_lock; static int kobj_alloced __diagused = 0; pgoff_t pages = round_page((uint64_t)size) >> PAGE_SHIFT; struct uvm_aobj *aobj; int refs; /* * Allocate a new aobj, unless kernel object is requested. */ if (flags & UAO_FLAG_KERNOBJ) { KASSERT(!kobj_alloced); aobj = &kernel_object_store; aobj->u_pages = pages; aobj->u_flags = UAO_FLAG_NOSWAP; refs = UVM_OBJ_KERN; kobj_alloced = UAO_FLAG_KERNOBJ; } else if (flags & UAO_FLAG_KERNSWAP) { KASSERT(kobj_alloced == UAO_FLAG_KERNOBJ); aobj = &kernel_object_store; kobj_alloced = UAO_FLAG_KERNSWAP; refs = 0xdeadbeaf; /* XXX: gcc */ } else { aobj = kmem_alloc(sizeof(struct uvm_aobj), KM_SLEEP); aobj->u_pages = pages; aobj->u_flags = 0; refs = 1; } /* * no freelist by default */ aobj->u_freelist = VM_NFREELIST; /* * allocate hash/array if necessary * * note: in the KERNSWAP case no need to worry about locking since * we are still booting we should be the only thread around. */ const int kernswap = (flags & UAO_FLAG_KERNSWAP) != 0; if (flags == 0 || kernswap) { #if defined(VMSWAP) /* allocate hash table or array depending on object size */ if (UAO_USES_SWHASH(aobj)) { aobj->u_swhash = hashinit(UAO_SWHASH_BUCKETS(aobj), HASH_LIST, true, &aobj->u_swhashmask); } else { aobj->u_swslots = kmem_zalloc(pages * sizeof(int), KM_SLEEP); } #endif /* defined(VMSWAP) */ /* * Replace kernel_object's temporary static lock with * a regular rw_obj. We cannot use uvm_obj_setlock() * because that would try to free the old lock. */ if (kernswap) { aobj->u_obj.vmobjlock = rw_obj_alloc(); rw_destroy(&bootstrap_kernel_object_lock); } if (flags) { aobj->u_flags &= ~UAO_FLAG_NOSWAP; /* clear noswap */ return &aobj->u_obj; } } /* * Initialise UVM object. */ const bool kernobj = (flags & UAO_FLAG_KERNOBJ) != 0; uvm_obj_init(&aobj->u_obj, &aobj_pager, !kernobj, refs); if (__predict_false(kernobj)) { /* Use a temporary static lock for kernel_object. */ rw_init(&bootstrap_kernel_object_lock); uvm_obj_setlock(&aobj->u_obj, &bootstrap_kernel_object_lock); } /* * now that aobj is ready, add it to the global list */ mutex_enter(&uao_list_lock); LIST_INSERT_HEAD(&uao_list, aobj, u_list); mutex_exit(&uao_list_lock); return(&aobj->u_obj); } /* * uao_set_pgfl: allocate pages only from the specified freelist. * * => must be called before any pages are allocated for the object. * => reset by setting it to VM_NFREELIST, meaning any freelist. */ void uao_set_pgfl(struct uvm_object *uobj, int freelist) { struct uvm_aobj *aobj = (struct uvm_aobj *)uobj; KASSERTMSG((0 <= freelist), "invalid freelist %d", freelist); KASSERTMSG((freelist <= VM_NFREELIST), "invalid freelist %d", freelist); aobj->u_freelist = freelist; } /* * uao_pagealloc: allocate a page for aobj. */ static inline struct vm_page * uao_pagealloc(struct uvm_object *uobj, voff_t offset, int flags) { struct uvm_aobj *aobj = (struct uvm_aobj *)uobj; if (__predict_true(aobj->u_freelist == VM_NFREELIST)) return uvm_pagealloc(uobj, offset, NULL, flags); else return uvm_pagealloc_strat(uobj, offset, NULL, flags, UVM_PGA_STRAT_ONLY, aobj->u_freelist); } /* * uao_init: set up aobj pager subsystem * * => called at boot time from uvm_pager_init() */ void uao_init(void) { static int uao_initialized; if (uao_initialized) return; uao_initialized = true; LIST_INIT(&uao_list); mutex_init(&uao_list_lock, MUTEX_DEFAULT, IPL_NONE); pool_init(&uao_swhash_elt_pool, sizeof(struct uao_swhash_elt), 0, 0, 0, "uaoeltpl", NULL, IPL_VM); } /* * uao_reference: hold a reference to an anonymous UVM object. */ void uao_reference(struct uvm_object *uobj) { /* Kernel object is persistent. */ if (UVM_OBJ_IS_KERN_OBJECT(uobj)) { return; } atomic_inc_uint(&uobj->uo_refs); } /* * uao_detach: drop a reference to an anonymous UVM object. */ void uao_detach(struct uvm_object *uobj) { struct uvm_aobj *aobj = (struct uvm_aobj *)uobj; struct uvm_page_array a; struct vm_page *pg; UVMHIST_FUNC(__func__); UVMHIST_CALLED(maphist); /* * Detaching from kernel object is a NOP. */ if (UVM_OBJ_IS_KERN_OBJECT(uobj)) return; /* * Drop the reference. If it was the last one, destroy the object. */ KASSERT(uobj->uo_refs > 0); UVMHIST_LOG(maphist," (uobj=%#jx) ref=%jd", (uintptr_t)uobj, uobj->uo_refs, 0, 0); #ifndef __HAVE_ATOMIC_AS_MEMBAR membar_release(); #endif if (atomic_dec_uint_nv(&uobj->uo_refs) > 0) { UVMHIST_LOG(maphist, "<- done (rc>0)", 0,0,0,0); return; } #ifndef __HAVE_ATOMIC_AS_MEMBAR membar_acquire(); #endif /* * Remove the aobj from the global list. */ mutex_enter(&uao_list_lock); LIST_REMOVE(aobj, u_list); mutex_exit(&uao_list_lock); /* * Free all the pages left in the aobj. For each page, when the * page is no longer busy (and thus after any disk I/O that it is * involved in is complete), release any swap resources and free * the page itself. */ uvm_page_array_init(&a, uobj, 0); rw_enter(uobj->vmobjlock, RW_WRITER); while ((pg = uvm_page_array_fill_and_peek(&a, 0, 0)) != NULL) { uvm_page_array_advance(&a); pmap_page_protect(pg, VM_PROT_NONE); if (pg->flags & PG_BUSY) { uvm_pagewait(pg, uobj->vmobjlock, "uao_det"); uvm_page_array_clear(&a); rw_enter(uobj->vmobjlock, RW_WRITER); continue; } uao_dropswap(&aobj->u_obj, pg->offset >> PAGE_SHIFT); uvm_pagefree(pg); } uvm_page_array_fini(&a); /* * Finally, free the anonymous UVM object itself. */ uao_free(aobj); } /* * uao_put: flush pages out of a uvm object * * => object should be locked by caller. we may _unlock_ the object * if (and only if) we need to clean a page (PGO_CLEANIT). * XXXJRT Currently, however, we don't. In the case of cleaning * XXXJRT a page, we simply just deactivate it. Should probably * XXXJRT handle this better, in the future (although "flushing" * XXXJRT anonymous memory isn't terribly important). * => if PGO_CLEANIT is not set, then we will neither unlock the object * or block. * => if PGO_ALLPAGE is set, then all pages in the object are valid targets * for flushing. * => we return 0 unless we encountered some sort of I/O error * XXXJRT currently never happens, as we never directly initiate * XXXJRT I/O */ static int uao_put(struct uvm_object *uobj, voff_t start, voff_t stop, int flags) { struct uvm_aobj *aobj = (struct uvm_aobj *)uobj; struct uvm_page_array a; struct vm_page *pg; voff_t curoff; UVMHIST_FUNC(__func__); UVMHIST_CALLED(maphist); KASSERT(UVM_OBJ_IS_AOBJ(uobj)); KASSERT(rw_write_held(uobj->vmobjlock)); if (flags & PGO_ALLPAGES) { start = 0; stop = aobj->u_pages << PAGE_SHIFT; } else { start = trunc_page(start); if (stop == 0) { stop = aobj->u_pages << PAGE_SHIFT; } else { stop = round_page(stop); } if (stop > (uint64_t)(aobj->u_pages << PAGE_SHIFT)) { printf("uao_put: strange, got an out of range " "flush %#jx > %#jx (fixed)\n", (uintmax_t)stop, (uintmax_t)(aobj->u_pages << PAGE_SHIFT)); stop = aobj->u_pages << PAGE_SHIFT; } } UVMHIST_LOG(maphist, " flush start=%#jx, stop=%#jx, flags=%#jx", start, stop, flags, 0); /* * Don't need to do any work here if we're not freeing * or deactivating pages. */ if ((flags & (PGO_DEACTIVATE|PGO_FREE)) == 0) { rw_exit(uobj->vmobjlock); return 0; } /* locked: uobj */ uvm_page_array_init(&a, uobj, 0); curoff = start; while ((pg = uvm_page_array_fill_and_peek(&a, curoff, 0)) != NULL) { if (pg->offset >= stop) { break; } /* * wait and try again if the page is busy. */ if (pg->flags & PG_BUSY) { uvm_pagewait(pg, uobj->vmobjlock, "uao_put"); uvm_page_array_clear(&a); rw_enter(uobj->vmobjlock, RW_WRITER); continue; } uvm_page_array_advance(&a); curoff = pg->offset + PAGE_SIZE; switch (flags & (PGO_CLEANIT|PGO_FREE|PGO_DEACTIVATE)) { /* * XXX In these first 3 cases, we always just * XXX deactivate the page. We may want to * XXX handle the different cases more specifically * XXX in the future. */ case PGO_CLEANIT|PGO_FREE: case PGO_CLEANIT|PGO_DEACTIVATE: case PGO_DEACTIVATE: deactivate_it: uvm_pagelock(pg); uvm_pagedeactivate(pg); uvm_pageunlock(pg); break; case PGO_FREE: /* * If there are multiple references to * the object, just deactivate the page. */ if (uobj->uo_refs > 1) goto deactivate_it; /* * free the swap slot and the page. */ pmap_page_protect(pg, VM_PROT_NONE); /* * freeing swapslot here is not strictly necessary. * however, leaving it here doesn't save much * because we need to update swap accounting anyway. */ uao_dropswap(uobj, pg->offset >> PAGE_SHIFT); uvm_pagefree(pg); break; default: panic("%s: impossible", __func__); } } rw_exit(uobj->vmobjlock); uvm_page_array_fini(&a); return 0; } /* * uao_get: fetch me a page * * we have three cases: * 1: page is resident -> just return the page. * 2: page is zero-fill -> allocate a new page and zero it. * 3: page is swapped out -> fetch the page from swap. * * case 1 can be handled with PGO_LOCKED, cases 2 and 3 cannot. * so, if the "center" page hits case 2/3 then we will need to return EBUSY. * * => prefer map unlocked (not required) * => object must be locked! we will _unlock_ it before starting any I/O. * => flags: PGO_LOCKED: fault data structures are locked * => NOTE: offset is the offset of pps[0], _NOT_ pps[centeridx] * => NOTE: caller must check for released pages!! */ static int uao_get(struct uvm_object *uobj, voff_t offset, struct vm_page **pps, int *npagesp, int centeridx, vm_prot_t access_type, int advice, int flags) { voff_t current_offset; struct vm_page *ptmp; int lcv, gotpages, maxpages, swslot, pageidx; bool overwrite = ((flags & PGO_OVERWRITE) != 0); struct uvm_page_array a; UVMHIST_FUNC(__func__); UVMHIST_CALLARGS(pdhist, "aobj=%#jx offset=%jd, flags=%#jx", (uintptr_t)uobj, offset, flags,0); /* * the object must be locked. it can only be a read lock when * processing a read fault with PGO_LOCKED. */ KASSERT(UVM_OBJ_IS_AOBJ(uobj)); KASSERT(rw_lock_held(uobj->vmobjlock)); KASSERT(rw_write_held(uobj->vmobjlock) || ((flags & PGO_LOCKED) != 0 && (access_type & VM_PROT_WRITE) == 0)); /* * get number of pages */ maxpages = *npagesp; /* * step 1: handled the case where fault data structures are locked. */ if (flags & PGO_LOCKED) { /* * step 1a: get pages that are already resident. only do * this if the data structures are locked (i.e. the first * time through). */ uvm_page_array_init(&a, uobj, 0); gotpages = 0; /* # of pages we got so far */ for (lcv = 0; lcv < maxpages; lcv++) { ptmp = uvm_page_array_fill_and_peek(&a, offset + (lcv << PAGE_SHIFT), maxpages); if (ptmp == NULL) { break; } KASSERT(ptmp->offset >= offset); lcv = (ptmp->offset - offset) >> PAGE_SHIFT; if (lcv >= maxpages) { break; } uvm_page_array_advance(&a); /* * to be useful must get a non-busy page */ if ((ptmp->flags & PG_BUSY) != 0) { continue; } /* * useful page: plug it in our result array */ KASSERT(uvm_pagegetdirty(ptmp) != UVM_PAGE_STATUS_CLEAN); pps[lcv] = ptmp; gotpages++; } uvm_page_array_fini(&a); /* * step 1b: now we've either done everything needed or we * to unlock and do some waiting or I/O. */ UVMHIST_LOG(pdhist, "<- done (done=%jd)", (pps[centeridx] != NULL), 0,0,0); *npagesp = gotpages; return pps[centeridx] != NULL ? 0 : EBUSY; } /* * step 2: get non-resident or busy pages. * object is locked. data structures are unlocked. */ if ((flags & PGO_SYNCIO) == 0) { goto done; } uvm_page_array_init(&a, uobj, 0); for (lcv = 0, current_offset = offset ; lcv < maxpages ;) { /* * we have yet to locate the current page (pps[lcv]). we * first look for a page that is already at the current offset. * if we find a page, we check to see if it is busy or * released. if that is the case, then we sleep on the page * until it is no longer busy or released and repeat the lookup. * if the page we found is neither busy nor released, then we * busy it (so we own it) and plug it into pps[lcv]. we are * ready to move on to the next page. */ ptmp = uvm_page_array_fill_and_peek(&a, current_offset, maxpages - lcv); if (ptmp != NULL && ptmp->offset == current_offset) { /* page is there, see if we need to wait on it */ if ((ptmp->flags & PG_BUSY) != 0) { UVMHIST_LOG(pdhist, "sleeping, ptmp->flags %#jx\n", ptmp->flags,0,0,0); uvm_pagewait(ptmp, uobj->vmobjlock, "uao_get"); rw_enter(uobj->vmobjlock, RW_WRITER); uvm_page_array_clear(&a); continue; } /* * if we get here then the page is resident and * unbusy. we busy it now (so we own it). if * overwriting, mark the page dirty up front as * it will be zapped via an unmanaged mapping. */ KASSERT(uvm_pagegetdirty(ptmp) != UVM_PAGE_STATUS_CLEAN); if (overwrite) { uvm_pagemarkdirty(ptmp, UVM_PAGE_STATUS_DIRTY); } /* we own it, caller must un-busy */ ptmp->flags |= PG_BUSY; UVM_PAGE_OWN(ptmp, "uao_get2"); pps[lcv++] = ptmp; current_offset += PAGE_SIZE; uvm_page_array_advance(&a); continue; } else { KASSERT(ptmp == NULL || ptmp->offset > current_offset); } /* * not resident. allocate a new busy/fake/clean page in the * object. if it's in swap we need to do I/O to fill in the * data, otherwise the page needs to be cleared: if it's not * destined to be overwritten, then zero it here and now. */ pageidx = current_offset >> PAGE_SHIFT; swslot = uao_find_swslot(uobj, pageidx); ptmp = uao_pagealloc(uobj, current_offset, swslot != 0 || overwrite ? 0 : UVM_PGA_ZERO); /* out of RAM? */ if (ptmp == NULL) { rw_exit(uobj->vmobjlock); UVMHIST_LOG(pdhist, "sleeping, ptmp == NULL",0,0,0,0); uvm_wait("uao_getpage"); rw_enter(uobj->vmobjlock, RW_WRITER); uvm_page_array_clear(&a); continue; } /* * if swslot == 0, page hasn't existed before and is zeroed. * otherwise we have a "fake/busy/clean" page that we just * allocated. do the needed "i/o", reading from swap. */ if (swslot != 0) { #if defined(VMSWAP) int error; UVMHIST_LOG(pdhist, "pagein from swslot %jd", swslot, 0,0,0); /* * page in the swapped-out page. * unlock object for i/o, relock when done. */ uvm_page_array_clear(&a); rw_exit(uobj->vmobjlock); error = uvm_swap_get(ptmp, swslot, PGO_SYNCIO); rw_enter(uobj->vmobjlock, RW_WRITER); /* * I/O done. check for errors. */ if (error != 0) { UVMHIST_LOG(pdhist, "<- done (error=%jd)", error,0,0,0); /* * remove the swap slot from the aobj * and mark the aobj as having no real slot. * don't free the swap slot, thus preventing * it from being used again. */ swslot = uao_set_swslot(uobj, pageidx, SWSLOT_BAD); if (swslot > 0) { uvm_swap_markbad(swslot, 1); } uvm_pagefree(ptmp); rw_exit(uobj->vmobjlock); UVMHIST_LOG(pdhist, "<- done (error)", error,lcv,0,0); if (lcv != 0) { uvm_page_unbusy(pps, lcv); } memset(pps, 0, maxpages * sizeof(pps[0])); uvm_page_array_fini(&a); return error; } #else /* defined(VMSWAP) */ panic("%s: pagein", __func__); #endif /* defined(VMSWAP) */ } /* * note that we will allow the page being writably-mapped * (!PG_RDONLY) regardless of access_type. if overwrite, * the page can be modified through an unmanaged mapping * so mark it dirty up front. */ if (overwrite) { uvm_pagemarkdirty(ptmp, UVM_PAGE_STATUS_DIRTY); } else { uvm_pagemarkdirty(ptmp, UVM_PAGE_STATUS_UNKNOWN); } /* * we got the page! clear the fake flag (indicates valid * data now in page) and plug into our result array. note * that page is still busy. * * it is the callers job to: * => check if the page is released * => unbusy the page * => activate the page */ KASSERT(uvm_pagegetdirty(ptmp) != UVM_PAGE_STATUS_CLEAN); KASSERT((ptmp->flags & PG_FAKE) != 0); KASSERT(ptmp->offset == current_offset); ptmp->flags &= ~PG_FAKE; pps[lcv++] = ptmp; current_offset += PAGE_SIZE; } uvm_page_array_fini(&a); /* * finally, unlock object and return. */ done: rw_exit(uobj->vmobjlock); UVMHIST_LOG(pdhist, "<- done (OK)",0,0,0,0); return 0; } #if defined(VMSWAP) /* * uao_dropswap: release any swap resources from this aobj page. * * => aobj must be locked or have a reference count of 0. */ void uao_dropswap(struct uvm_object *uobj, int pageidx) { int slot; KASSERT(UVM_OBJ_IS_AOBJ(uobj)); slot = uao_set_swslot(uobj, pageidx, 0); if (slot) { uvm_swap_free(slot, 1); } } /* * page in every page in every aobj that is paged-out to a range of swslots. * * => nothing should be locked. * => returns true if pagein was aborted due to lack of memory. */ bool uao_swap_off(int startslot, int endslot) { struct uvm_aobj *aobj; /* * Walk the list of all anonymous UVM objects. Grab the first. */ mutex_enter(&uao_list_lock); if ((aobj = LIST_FIRST(&uao_list)) == NULL) { mutex_exit(&uao_list_lock); return false; } uao_reference(&aobj->u_obj); do { struct uvm_aobj *nextaobj; bool rv; /* * Prefetch the next object and immediately hold a reference * on it, so neither the current nor the next entry could * disappear while we are iterating. */ if ((nextaobj = LIST_NEXT(aobj, u_list)) != NULL) { uao_reference(&nextaobj->u_obj); } mutex_exit(&uao_list_lock); /* * Page in all pages in the swap slot range. */ rw_enter(aobj->u_obj.vmobjlock, RW_WRITER); rv = uao_pagein(aobj, startslot, endslot); rw_exit(aobj->u_obj.vmobjlock); /* Drop the reference of the current object. */ uao_detach(&aobj->u_obj); if (rv) { if (nextaobj) { uao_detach(&nextaobj->u_obj); } return rv; } aobj = nextaobj; mutex_enter(&uao_list_lock); } while (aobj); mutex_exit(&uao_list_lock); return false; } /* * page in any pages from aobj in the given range. * * => aobj must be locked and is returned locked. * => returns true if pagein was aborted due to lack of memory. */ static bool uao_pagein(struct uvm_aobj *aobj, int startslot, int endslot) { bool rv; if (UAO_USES_SWHASH(aobj)) { struct uao_swhash_elt *elt; int buck; restart: for (buck = aobj->u_swhashmask; buck >= 0; buck--) { for (elt = LIST_FIRST(&aobj->u_swhash[buck]); elt != NULL; elt = LIST_NEXT(elt, list)) { int i; for (i = 0; i < UAO_SWHASH_CLUSTER_SIZE; i++) { int slot = elt->slots[i]; /* * if the slot isn't in range, skip it. */ if (slot < startslot || slot >= endslot) { continue; } /* * process the page, * the start over on this object * since the swhash elt * may have been freed. */ rv = uao_pagein_page(aobj, UAO_SWHASH_ELT_PAGEIDX_BASE(elt) + i); if (rv) { return rv; } goto restart; } } } } else { int i; for (i = 0; i < aobj->u_pages; i++) { int slot = aobj->u_swslots[i]; /* * if the slot isn't in range, skip it */ if (slot < startslot || slot >= endslot) { continue; } /* * process the page. */ rv = uao_pagein_page(aobj, i); if (rv) { return rv; } } } return false; } /* * uao_pagein_page: page in a single page from an anonymous UVM object. * * => Returns true if pagein was aborted due to lack of memory. * => Object must be locked and is returned locked. */ static bool uao_pagein_page(struct uvm_aobj *aobj, int pageidx) { struct uvm_object *uobj = &aobj->u_obj; struct vm_page *pg; int rv, npages; pg = NULL; npages = 1; KASSERT(rw_write_held(uobj->vmobjlock)); rv = uao_get(uobj, (voff_t)pageidx << PAGE_SHIFT, &pg, &npages, 0, VM_PROT_READ | VM_PROT_WRITE, 0, PGO_SYNCIO); /* * relock and finish up. */ rw_enter(uobj->vmobjlock, RW_WRITER); switch (rv) { case 0: break; case EIO: case ERESTART: /* * nothing more to do on errors. * ERESTART can only mean that the anon was freed, * so again there's nothing to do. */ return false; default: return true; } /* * ok, we've got the page now. * mark it as dirty, clear its swslot and un-busy it. */ uao_dropswap(&aobj->u_obj, pageidx); /* * make sure it's on a page queue. */ uvm_pagelock(pg); uvm_pageenqueue(pg); uvm_pagewakeup(pg); uvm_pageunlock(pg); pg->flags &= ~(PG_BUSY|PG_FAKE); uvm_pagemarkdirty(pg, UVM_PAGE_STATUS_DIRTY); UVM_PAGE_OWN(pg, NULL); return false; } /* * uao_dropswap_range: drop swapslots in the range. * * => aobj must be locked and is returned locked. * => start is inclusive. end is exclusive. */ void uao_dropswap_range(struct uvm_object *uobj, voff_t start, voff_t end) { struct uvm_aobj *aobj = (struct uvm_aobj *)uobj; int swpgonlydelta = 0; KASSERT(UVM_OBJ_IS_AOBJ(uobj)); KASSERT(rw_write_held(uobj->vmobjlock)); if (end == 0) { end = INT64_MAX; } if (UAO_USES_SWHASH(aobj)) { int i, hashbuckets = aobj->u_swhashmask + 1; voff_t taghi; voff_t taglo; taglo = UAO_SWHASH_ELT_TAG(start); taghi = UAO_SWHASH_ELT_TAG(end); for (i = 0; i < hashbuckets; i++) { struct uao_swhash_elt *elt, *next; for (elt = LIST_FIRST(&aobj->u_swhash[i]); elt != NULL; elt = next) { int startidx, endidx; int j; next = LIST_NEXT(elt, list); if (elt->tag < taglo || taghi < elt->tag) { continue; } if (elt->tag == taglo) { startidx = UAO_SWHASH_ELT_PAGESLOT_IDX(start); } else { startidx = 0; } if (elt->tag == taghi) { endidx = UAO_SWHASH_ELT_PAGESLOT_IDX(end); } else { endidx = UAO_SWHASH_CLUSTER_SIZE; } for (j = startidx; j < endidx; j++) { int slot = elt->slots[j]; KASSERT(uvm_pagelookup(&aobj->u_obj, (UAO_SWHASH_ELT_PAGEIDX_BASE(elt) + j) << PAGE_SHIFT) == NULL); if (slot > 0) { uvm_swap_free(slot, 1); swpgonlydelta++; KASSERT(elt->count > 0); elt->slots[j] = 0; elt->count--; } } if (elt->count == 0) { LIST_REMOVE(elt, list); pool_put(&uao_swhash_elt_pool, elt); } } } } else { int i; if (aobj->u_pages < end) { end = aobj->u_pages; } for (i = start; i < end; i++) { int slot = aobj->u_swslots[i]; if (slot > 0) { uvm_swap_free(slot, 1); swpgonlydelta++; } } } /* * adjust the counter of pages only in swap for all * the swap slots we've freed. */ if (swpgonlydelta > 0) { KASSERT(uvmexp.swpgonly >= swpgonlydelta); atomic_add_int(&uvmexp.swpgonly, -swpgonlydelta); } } #endif /* defined(VMSWAP) */ |
| 8 8 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 197 198 199 200 201 202 203 204 205 206 207 208 209 210 211 212 213 214 215 216 217 218 219 220 221 222 223 224 225 226 227 228 229 230 231 232 233 234 235 236 237 238 239 240 241 242 243 244 245 246 247 248 249 250 251 252 253 254 255 256 257 258 259 260 261 262 263 264 265 266 267 268 269 270 271 272 273 274 275 276 277 278 279 280 281 282 283 284 285 286 287 288 289 290 291 292 293 294 295 296 297 298 299 300 301 302 303 304 305 306 307 308 309 310 311 312 313 314 315 316 317 318 319 320 321 322 323 324 325 326 327 328 329 330 331 332 333 334 335 336 337 338 339 340 341 342 343 344 345 346 347 348 349 350 351 352 353 354 355 356 357 358 359 360 361 362 363 364 365 366 367 368 369 370 371 372 373 374 375 376 377 378 379 380 381 382 383 384 385 386 387 388 389 390 391 392 393 394 395 396 397 398 399 400 401 402 403 404 405 406 407 408 409 410 411 412 413 414 415 416 417 418 419 420 421 422 423 424 425 426 427 428 429 430 431 432 433 434 435 436 437 438 439 440 441 442 443 444 445 446 447 448 449 450 451 452 453 454 455 456 457 458 459 460 461 462 463 464 465 466 467 468 469 470 471 472 473 474 475 476 477 478 479 480 481 482 483 484 485 486 487 488 489 490 491 492 493 494 495 496 497 498 499 500 501 502 503 504 505 506 507 508 509 510 511 512 513 514 515 516 517 518 519 520 521 522 523 524 525 526 527 528 529 530 531 532 533 534 535 536 537 538 539 540 541 542 543 544 545 546 547 548 549 550 551 552 553 554 555 556 557 558 559 560 561 562 563 564 565 566 567 568 569 570 571 572 573 574 575 576 577 578 579 580 581 582 583 584 585 586 587 588 589 590 591 592 593 594 595 596 597 598 599 600 601 602 603 604 605 606 607 608 609 610 611 612 613 614 615 616 617 618 619 620 621 622 623 624 625 626 627 628 629 630 631 632 633 634 635 636 637 638 639 640 641 642 643 644 645 646 647 648 649 650 651 652 653 654 655 656 657 658 659 660 661 662 663 664 665 666 667 668 669 670 671 672 673 674 675 676 677 678 679 680 681 682 683 684 685 686 687 688 689 690 691 692 693 694 695 696 697 698 699 700 701 702 703 704 705 706 707 708 709 710 711 712 713 714 715 716 717 718 719 720 721 722 723 724 725 726 727 728 729 730 731 732 733 734 735 736 737 738 739 740 741 742 743 744 745 746 747 748 749 750 751 752 753 754 755 756 757 758 759 760 761 762 763 764 765 766 767 768 769 770 771 772 773 774 775 776 777 778 779 780 781 782 783 784 785 786 787 788 789 790 791 792 793 794 795 796 797 798 799 800 801 802 803 804 805 806 807 808 809 810 811 812 813 814 815 816 817 818 819 820 821 822 823 824 825 826 827 828 829 830 831 832 833 834 835 836 837 838 839 840 841 842 843 844 845 846 847 848 849 850 851 852 853 854 855 856 857 858 859 860 861 862 863 864 865 866 867 868 869 870 871 872 873 874 875 876 877 878 879 880 881 882 883 884 885 886 887 888 889 890 891 892 893 894 895 896 897 898 899 900 901 902 903 904 905 906 907 908 909 910 911 912 913 914 915 916 917 918 919 920 921 922 923 924 925 926 927 928 929 930 931 932 933 934 935 936 937 938 939 940 941 942 943 944 945 946 947 948 949 950 951 952 953 954 955 956 957 958 959 960 961 962 963 964 965 966 967 968 969 970 971 972 973 974 975 976 977 978 979 980 981 982 983 984 985 986 987 988 989 990 991 992 993 994 995 996 997 998 999 1000 1001 1002 1003 1004 1005 1006 1007 1008 1009 1010 1011 1012 1013 1014 1015 1016 1017 1018 1019 1020 1021 1022 1023 1024 1025 1026 1027 1028 1029 1030 1031 1032 1033 1034 1035 1036 1037 1038 1039 1040 1041 1042 1043 1044 1045 1046 1047 1048 1049 1050 1051 1052 1053 1054 1055 1056 1057 1058 1059 1060 1061 1062 1063 1064 1065 1066 1067 1068 1069 1070 1071 1072 | /* $NetBSD: if_urndis.c,v 1.47 2022/03/03 05:56:58 riastradh Exp $ */ /* $OpenBSD: if_urndis.c,v 1.31 2011/07/03 15:47:17 matthew Exp $ */ /* * Copyright (c) 2010 Jonathan Armani <armani@openbsd.org> * Copyright (c) 2010 Fabien Romano <fabien@openbsd.org> * Copyright (c) 2010 Michael Knudsen <mk@openbsd.org> * All rights reserved. * * Permission to use, copy, modify, and distribute this software for any * purpose with or without fee is hereby granted, provided that the above * copyright notice and this permission notice appear in all copies. * * THE SOFTWARE IS PROVIDED "AS IS" AND THE AUTHOR DISCLAIMS ALL WARRANTIES * WITH REGARD TO THIS SOFTWARE INCLUDING ALL IMPLIED WARRANTIES OF * MERCHANTABILITY AND FITNESS. IN NO EVENT SHALL THE AUTHOR BE LIABLE FOR * ANY SPECIAL, DIRECT, INDIRECT, OR CONSEQUENTIAL DAMAGES OR ANY DAMAGES * WHATSOEVER RESULTING FROM LOSS OF USE, DATA OR PROFITS, WHETHER IN AN * ACTION OF CONTRACT, NEGLIGENCE OR OTHER TORTIOUS ACTION, ARISING OUT OF * OR IN CONNECTION WITH THE USE OR PERFORMANCE OF THIS SOFTWARE. */ #include <sys/cdefs.h> __KERNEL_RCSID(0, "$NetBSD: if_urndis.c,v 1.47 2022/03/03 05:56:58 riastradh Exp $"); #ifdef _KERNEL_OPT #include "opt_usb.h" #endif #include <sys/param.h> #include <sys/kmem.h> #include <dev/usb/usbnet.h> #include <dev/usb/usbdevs.h> #include <dev/usb/usbcdc.h> #include <dev/ic/rndisreg.h> #define RNDIS_RX_LIST_CNT 1 #define RNDIS_TX_LIST_CNT 1 #define RNDIS_BUFSZ 1562 struct urndis_softc { struct usbnet sc_un; int sc_ifaceno_ctl; /* RNDIS device info */ uint32_t sc_filter; uint32_t sc_maxppt; uint32_t sc_maxtsz; uint32_t sc_palign; }; #ifdef URNDIS_DEBUG #define DPRINTF(x) do { printf x; } while (0) #else #define DPRINTF(x) #endif #define DEVNAME(un) (device_xname(un->un_dev)) #define URNDIS_RESPONSE_LEN 0x400 #if 0 static void urndis_watchdog(struct ifnet *); #endif static int urndis_uno_init(struct ifnet *); static void urndis_uno_rx_loop(struct usbnet *, struct usbnet_chain *, uint32_t); static unsigned urndis_uno_tx_prepare(struct usbnet *, struct mbuf *, struct usbnet_chain *); static uint32_t urndis_ctrl_handle_init(struct usbnet *, const struct rndis_comp_hdr *); static uint32_t urndis_ctrl_handle_query(struct usbnet *, const struct rndis_comp_hdr *, void **, size_t *); static uint32_t urndis_ctrl_handle_reset(struct usbnet *, const struct rndis_comp_hdr *); static uint32_t urndis_ctrl_handle_status(struct usbnet *, const struct rndis_comp_hdr *); static uint32_t urndis_ctrl_set(struct usbnet *, uint32_t, void *, size_t); static int urndis_match(device_t, cfdata_t, void *); static void urndis_attach(device_t, device_t, void *); static const struct usbnet_ops urndis_ops = { .uno_init = urndis_uno_init, .uno_tx_prepare = urndis_uno_tx_prepare, .uno_rx_loop = urndis_uno_rx_loop, }; CFATTACH_DECL_NEW(urndis, sizeof(struct urndis_softc), urndis_match, urndis_attach, usbnet_detach, usbnet_activate); /* * Supported devices that we can't match by class IDs. */ static const struct usb_devno urndis_devs[] = { { USB_VENDOR_HTC, USB_PRODUCT_HTC_ANDROID }, { USB_VENDOR_SAMSUNG, USB_PRODUCT_SAMSUNG_ANDROID2 }, { USB_VENDOR_SAMSUNG, USB_PRODUCT_SAMSUNG_ANDROID }, }; static usbd_status urndis_ctrl_msg(struct usbnet *un, uint8_t rt, uint8_t r, uint16_t index, uint16_t value, void *buf, size_t buflen) { usb_device_request_t req; req.bmRequestType = rt; req.bRequest = r; USETW(req.wValue, value); USETW(req.wIndex, index); USETW(req.wLength, buflen); return usbd_do_request(un->un_udev, &req, buf); } static usbd_status urndis_ctrl_send(struct usbnet *un, void *buf, size_t len) { struct urndis_softc *sc = usbnet_softc(un); usbd_status err; if (usbnet_isdying(un)) return(0); err = urndis_ctrl_msg(un, UT_WRITE_CLASS_INTERFACE, UR_GET_STATUS, sc->sc_ifaceno_ctl, 0, buf, len); if (err != USBD_NORMAL_COMPLETION) printf("%s: %s\n", DEVNAME(un), usbd_errstr(err)); return err; } static struct rndis_comp_hdr * urndis_ctrl_recv(struct usbnet *un) { struct urndis_softc *sc = usbnet_softc(un); struct rndis_comp_hdr *hdr; char *buf; usbd_status err; if (usbnet_isdying(un)) return(0); buf = kmem_alloc(URNDIS_RESPONSE_LEN, KM_SLEEP); err = urndis_ctrl_msg(un, UT_READ_CLASS_INTERFACE, UR_CLEAR_FEATURE, sc->sc_ifaceno_ctl, 0, buf, URNDIS_RESPONSE_LEN); if (err != USBD_NORMAL_COMPLETION && err != USBD_SHORT_XFER) { printf("%s: %s\n", DEVNAME(un), usbd_errstr(err)); kmem_free(buf, URNDIS_RESPONSE_LEN); return NULL; } hdr = (struct rndis_comp_hdr *)buf; DPRINTF(("%s: urndis_ctrl_recv: type %#x len %u\n", DEVNAME(un), le32toh(hdr->rm_type), le32toh(hdr->rm_len))); if (le32toh(hdr->rm_len) > URNDIS_RESPONSE_LEN) { printf("%s: ctrl message error: wrong size %u > %u\n", DEVNAME(un), le32toh(hdr->rm_len), URNDIS_RESPONSE_LEN); kmem_free(buf, URNDIS_RESPONSE_LEN); return NULL; } return hdr; } static uint32_t urndis_ctrl_handle(struct usbnet *un, struct rndis_comp_hdr *hdr, void **buf, size_t *bufsz) { uint32_t rval; DPRINTF(("%s: urndis_ctrl_handle\n", DEVNAME(un))); if (buf && bufsz) { *buf = NULL; *bufsz = 0; } switch (le32toh(hdr->rm_type)) { case REMOTE_NDIS_INITIALIZE_CMPLT: rval = urndis_ctrl_handle_init(un, hdr); break; case REMOTE_NDIS_QUERY_CMPLT: rval = urndis_ctrl_handle_query(un, hdr, buf, bufsz); break; case REMOTE_NDIS_RESET_CMPLT: rval = urndis_ctrl_handle_reset(un, hdr); break; case REMOTE_NDIS_KEEPALIVE_CMPLT: case REMOTE_NDIS_SET_CMPLT: rval = le32toh(hdr->rm_status); break; case REMOTE_NDIS_INDICATE_STATUS_MSG: rval = urndis_ctrl_handle_status(un, hdr); break; default: printf("%s: ctrl message error: unknown event %#x\n", DEVNAME(un), le32toh(hdr->rm_type)); rval = RNDIS_STATUS_FAILURE; } kmem_free(hdr, URNDIS_RESPONSE_LEN); return rval; } static uint32_t urndis_ctrl_handle_init(struct usbnet *un, const struct rndis_comp_hdr *hdr) { struct urndis_softc *sc = usbnet_softc(un); const struct rndis_init_comp *msg; msg = (const struct rndis_init_comp *) hdr; DPRINTF(("%s: urndis_ctrl_handle_init: len %u rid %u status %#x " "ver_major %u ver_minor %u devflags %#x medium %#x pktmaxcnt %u " "pktmaxsz %u align %u aflistoffset %u aflistsz %u\n", DEVNAME(un), le32toh(msg->rm_len), le32toh(msg->rm_rid), le32toh(msg->rm_status), le32toh(msg->rm_ver_major), le32toh(msg->rm_ver_minor), le32toh(msg->rm_devflags), le32toh(msg->rm_medium), le32toh(msg->rm_pktmaxcnt), le32toh(msg->rm_pktmaxsz), le32toh(msg->rm_align), le32toh(msg->rm_aflistoffset), le32toh(msg->rm_aflistsz))); if (le32toh(msg->rm_status) != RNDIS_STATUS_SUCCESS) { printf("%s: init failed %#x\n", DEVNAME(un), le32toh(msg->rm_status)); return le32toh(msg->rm_status); } if (le32toh(msg->rm_devflags) != RNDIS_DF_CONNECTIONLESS) { printf("%s: wrong device type (current type: %#x)\n", DEVNAME(un), le32toh(msg->rm_devflags)); return RNDIS_STATUS_FAILURE; } if (le32toh(msg->rm_medium) != RNDIS_MEDIUM_802_3) { printf("%s: medium not 802.3 (current medium: %#x)\n", DEVNAME(un), le32toh(msg->rm_medium)); return RNDIS_STATUS_FAILURE; } if (le32toh(msg->rm_ver_major) != RNDIS_MAJOR_VERSION || le32toh(msg->rm_ver_minor) != RNDIS_MINOR_VERSION) { printf("%s: version not %u.%u (current version: %u.%u)\n", DEVNAME(un), RNDIS_MAJOR_VERSION, RNDIS_MINOR_VERSION, le32toh(msg->rm_ver_major), le32toh(msg->rm_ver_minor)); return RNDIS_STATUS_FAILURE; } sc->sc_maxppt = le32toh(msg->rm_pktmaxcnt); sc->sc_maxtsz = le32toh(msg->rm_pktmaxsz); sc->sc_palign = 1U << le32toh(msg->rm_align); return le32toh(msg->rm_status); } static uint32_t urndis_ctrl_handle_query(struct usbnet *un, const struct rndis_comp_hdr *hdr, void **buf, size_t *bufsz) { const struct rndis_query_comp *msg; msg = (const struct rndis_query_comp *) hdr; DPRINTF(("%s: urndis_ctrl_handle_query: len %u rid %u status %#x " "buflen %u bufoff %u\n", DEVNAME(un), le32toh(msg->rm_len), le32toh(msg->rm_rid), le32toh(msg->rm_status), le32toh(msg->rm_infobuflen), le32toh(msg->rm_infobufoffset))); if (buf && bufsz) { *buf = NULL; *bufsz = 0; } if (le32toh(msg->rm_status) != RNDIS_STATUS_SUCCESS) { printf("%s: query failed %#x\n", DEVNAME(un), le32toh(msg->rm_status)); return le32toh(msg->rm_status); } if (le32toh(msg->rm_infobuflen) + le32toh(msg->rm_infobufoffset) + RNDIS_HEADER_OFFSET > le32toh(msg->rm_len)) { printf("%s: ctrl message error: invalid query info " "len/offset/end_position(%u/%u/%u) -> " "go out of buffer limit %u\n", DEVNAME(un), le32toh(msg->rm_infobuflen), le32toh(msg->rm_infobufoffset), le32toh(msg->rm_infobuflen) + le32toh(msg->rm_infobufoffset) + (uint32_t)RNDIS_HEADER_OFFSET, le32toh(msg->rm_len)); return RNDIS_STATUS_FAILURE; } if (buf && bufsz) { const char *p; *buf = kmem_alloc(le32toh(msg->rm_infobuflen), KM_SLEEP); *bufsz = le32toh(msg->rm_infobuflen); p = (const char *)&msg->rm_rid; p += le32toh(msg->rm_infobufoffset); memcpy(*buf, p, le32toh(msg->rm_infobuflen)); } return le32toh(msg->rm_status); } static uint32_t urndis_ctrl_handle_reset(struct usbnet *un, const struct rndis_comp_hdr *hdr) { struct urndis_softc *sc = usbnet_softc(un); const struct rndis_reset_comp *msg; uint32_t rval; msg = (const struct rndis_reset_comp *) hdr; rval = le32toh(msg->rm_status); DPRINTF(("%s: urndis_ctrl_handle_reset: len %u status %#x " "adrreset %u\n", DEVNAME(un), le32toh(msg->rm_len), rval, le32toh(msg->rm_adrreset))); if (rval != RNDIS_STATUS_SUCCESS) { printf("%s: reset failed %#x\n", DEVNAME(un), rval); return rval; } if (le32toh(msg->rm_adrreset) != 0) { uint32_t filter; filter = htole32(sc->sc_filter); rval = urndis_ctrl_set(un, OID_GEN_CURRENT_PACKET_FILTER, &filter, sizeof(filter)); if (rval != RNDIS_STATUS_SUCCESS) { printf("%s: unable to reset data filters\n", DEVNAME(un)); return rval; } } return rval; } static uint32_t urndis_ctrl_handle_status(struct usbnet *un, const struct rndis_comp_hdr *hdr) { const struct rndis_status_msg *msg; uint32_t rval; msg = (const struct rndis_status_msg *)hdr; rval = le32toh(msg->rm_status); DPRINTF(("%s: urndis_ctrl_handle_status: len %u status %#x " "stbuflen %u\n", DEVNAME(un), le32toh(msg->rm_len), rval, le32toh(msg->rm_stbuflen))); switch (rval) { case RNDIS_STATUS_MEDIA_CONNECT: case RNDIS_STATUS_MEDIA_DISCONNECT: case RNDIS_STATUS_OFFLOAD_CURRENT_CONFIG: rval = RNDIS_STATUS_SUCCESS; break; default: printf("%s: status %#x\n", DEVNAME(un), rval); } return rval; } static uint32_t urndis_ctrl_init(struct usbnet *un) { struct rndis_init_req *msg; uint32_t rval; struct rndis_comp_hdr *hdr; msg = kmem_alloc(sizeof(*msg), KM_SLEEP); msg->rm_type = htole32(REMOTE_NDIS_INITIALIZE_MSG); msg->rm_len = htole32(sizeof(*msg)); msg->rm_rid = htole32(0); msg->rm_ver_major = htole32(RNDIS_MAJOR_VERSION); msg->rm_ver_minor = htole32(RNDIS_MINOR_VERSION); msg->rm_max_xfersz = htole32(RNDIS_BUFSZ); DPRINTF(("%s: urndis_ctrl_init send: type %u len %u rid %u ver_major %u " "ver_minor %u max_xfersz %u\n", DEVNAME(un), le32toh(msg->rm_type), le32toh(msg->rm_len), le32toh(msg->rm_rid), le32toh(msg->rm_ver_major), le32toh(msg->rm_ver_minor), le32toh(msg->rm_max_xfersz))); rval = urndis_ctrl_send(un, msg, sizeof(*msg)); kmem_free(msg, sizeof(*msg)); if (rval != RNDIS_STATUS_SUCCESS) { printf("%s: init failed\n", DEVNAME(un)); return rval; } if ((hdr = urndis_ctrl_recv(un)) == NULL) { printf("%s: unable to get init response\n", DEVNAME(un)); return RNDIS_STATUS_FAILURE; } rval = urndis_ctrl_handle(un, hdr, NULL, NULL); return rval; } #if 0 static uint32_t urndis_ctrl_halt(struct usbnet *un) { struct rndis_halt_req *msg; uint32_t rval; msg = kmem_alloc(sizeof(*msg), KM_SLEEP); msg->rm_type = htole32(REMOTE_NDIS_HALT_MSG); msg->rm_len = htole32(sizeof(*msg)); msg->rm_rid = 0; DPRINTF(("%s: urndis_ctrl_halt send: type %u len %u rid %u\n", DEVNAME(un), le32toh(msg->rm_type), le32toh(msg->rm_len), le32toh(msg->rm_rid))); rval = urndis_ctrl_send(un, msg, sizeof(*msg)); kmem_free(msg, sizeof(*msg)); if (rval != RNDIS_STATUS_SUCCESS) printf("%s: halt failed\n", DEVNAME(un)); return rval; } #endif static uint32_t urndis_ctrl_query(struct usbnet *un, uint32_t oid, void *qbuf, size_t qlen, void **rbuf, size_t *rbufsz) { struct rndis_query_req *msg; uint32_t rval; struct rndis_comp_hdr *hdr; msg = kmem_alloc(sizeof(*msg) + qlen, KM_SLEEP); msg->rm_type = htole32(REMOTE_NDIS_QUERY_MSG); msg->rm_len = htole32(sizeof(*msg) + qlen); msg->rm_rid = 0; /* XXX */ msg->rm_oid = htole32(oid); msg->rm_infobuflen = htole32(qlen); if (qlen != 0) { msg->rm_infobufoffset = htole32(20); memcpy((char*)msg + 20, qbuf, qlen); } else msg->rm_infobufoffset = 0; msg->rm_devicevchdl = 0; DPRINTF(("%s: urndis_ctrl_query send: type %u len %u rid %u oid %#x " "infobuflen %u infobufoffset %u devicevchdl %u\n", DEVNAME(un), le32toh(msg->rm_type), le32toh(msg->rm_len), le32toh(msg->rm_rid), le32toh(msg->rm_oid), le32toh(msg->rm_infobuflen), le32toh(msg->rm_infobufoffset), le32toh(msg->rm_devicevchdl))); rval = urndis_ctrl_send(un, msg, sizeof(*msg)); kmem_free(msg, sizeof(*msg) + qlen); if (rval != RNDIS_STATUS_SUCCESS) { printf("%s: query failed\n", DEVNAME(un)); return rval; } if ((hdr = urndis_ctrl_recv(un)) == NULL) { printf("%s: unable to get query response\n", DEVNAME(un)); return RNDIS_STATUS_FAILURE; } rval = urndis_ctrl_handle(un, hdr, rbuf, rbufsz); return rval; } static uint32_t urndis_ctrl_set(struct usbnet *un, uint32_t oid, void *buf, size_t len) { struct rndis_set_req *msg; uint32_t rval; struct rndis_comp_hdr *hdr; msg = kmem_alloc(sizeof(*msg) + len, KM_SLEEP); msg->rm_type = htole32(REMOTE_NDIS_SET_MSG); msg->rm_len = htole32(sizeof(*msg) + len); msg->rm_rid = 0; /* XXX */ msg->rm_oid = htole32(oid); msg->rm_infobuflen = htole32(len); if (len != 0) { msg->rm_infobufoffset = htole32(20); memcpy((char*)msg + 20, buf, len); } else msg->rm_infobufoffset = 0; msg->rm_devicevchdl = 0; DPRINTF(("%s: urndis_ctrl_set send: type %u len %u rid %u oid %#x " "infobuflen %u infobufoffset %u devicevchdl %u\n", DEVNAME(un), le32toh(msg->rm_type), le32toh(msg->rm_len), le32toh(msg->rm_rid), le32toh(msg->rm_oid), le32toh(msg->rm_infobuflen), le32toh(msg->rm_infobufoffset), le32toh(msg->rm_devicevchdl))); rval = urndis_ctrl_send(un, msg, sizeof(*msg)); kmem_free(msg, sizeof(*msg) + len); if (rval != RNDIS_STATUS_SUCCESS) { printf("%s: set failed\n", DEVNAME(un)); return rval; } if ((hdr = urndis_ctrl_recv(un)) == NULL) { printf("%s: unable to get set response\n", DEVNAME(un)); return RNDIS_STATUS_FAILURE; } rval = urndis_ctrl_handle(un, hdr, NULL, NULL); if (rval != RNDIS_STATUS_SUCCESS) printf("%s: set failed %#x\n", DEVNAME(un), rval); return rval; } #if 0 static uint32_t urndis_ctrl_set_param(struct urndis_softc *un, const char *name, uint32_t type, void *buf, size_t len) { struct rndis_set_parameter *param; uint32_t rval; size_t namelen, tlen; if (name) namelen = strlen(name); else namelen = 0; tlen = sizeof(*param) + len + namelen; param = kmem_alloc(tlen, KM_SLEEP); param->rm_namelen = htole32(namelen); param->rm_valuelen = htole32(len); param->rm_type = htole32(type); if (namelen != 0) { param->rm_nameoffset = htole32(20); memcpy(param + 20, name, namelen); } else param->rm_nameoffset = 0; if (len != 0) { param->rm_valueoffset = htole32(20 + namelen); memcpy(param + 20 + namelen, buf, len); } else param->rm_valueoffset = 0; DPRINTF(("%s: urndis_ctrl_set_param send: nameoffset %u namelen %u " "type %#x valueoffset %u valuelen %u\n", DEVNAME(un), le32toh(param->rm_nameoffset), le32toh(param->rm_namelen), le32toh(param->rm_type), le32toh(param->rm_valueoffset), le32toh(param->rm_valuelen))); rval = urndis_ctrl_set(un, OID_GEN_RNDIS_CONFIG_PARAMETER, param, tlen); kmem_free(param, tlen); if (rval != RNDIS_STATUS_SUCCESS) printf("%s: set param failed %#x\n", DEVNAME(un), rval); return rval; } /* XXX : adrreset, get it from response */ static uint32_t urndis_ctrl_reset(struct usbnet *un) { struct rndis_reset_req *reset; uint32_t rval; struct rndis_comp_hdr *hdr; reset = kmem_alloc(sizeof(*reset), KM_SLEEP); reset->rm_type = htole32(REMOTE_NDIS_RESET_MSG); reset->rm_len = htole32(sizeof(*reset)); reset->rm_rid = 0; /* XXX rm_rid == reserved ... remove ? */ DPRINTF(("%s: urndis_ctrl_reset send: type %u len %u rid %u\n", DEVNAME(un), le32toh(reset->rm_type), le32toh(reset->rm_len), le32toh(reset->rm_rid))); rval = urndis_ctrl_send(un, reset, sizeof(*reset)); kmem_free(reset, sizeof(*reset)); if (rval != RNDIS_STATUS_SUCCESS) { printf("%s: reset failed\n", DEVNAME(un)); return rval; } if ((hdr = urndis_ctrl_recv(un)) == NULL) { printf("%s: unable to get reset response\n", DEVNAME(un)); return RNDIS_STATUS_FAILURE; } rval = urndis_ctrl_handle(un, hdr, NULL, NULL); return rval; } static uint32_t urndis_ctrl_keepalive(struct usbnet *un) { struct rndis_keepalive_req *keep; uint32_t rval; struct rndis_comp_hdr *hdr; keep = kmem_alloc(sizeof(*keep), KM_SLEEP); keep->rm_type = htole32(REMOTE_NDIS_KEEPALIVE_MSG); keep->rm_len = htole32(sizeof(*keep)); keep->rm_rid = 0; /* XXX rm_rid == reserved ... remove ? */ DPRINTF(("%s: urndis_ctrl_keepalive: type %u len %u rid %u\n", DEVNAME(un), le32toh(keep->rm_type), le32toh(keep->rm_len), le32toh(keep->rm_rid))); rval = urndis_ctrl_send(un, keep, sizeof(*keep)); kmem_free(keep, sizeof(*keep)); if (rval != RNDIS_STATUS_SUCCESS) { printf("%s: keepalive failed\n", DEVNAME(un)); return rval; } if ((hdr = urndis_ctrl_recv(un)) == NULL) { printf("%s: unable to get keepalive response\n", DEVNAME(un)); return RNDIS_STATUS_FAILURE; } rval = urndis_ctrl_handle(un, hdr, NULL, NULL); if (rval != RNDIS_STATUS_SUCCESS) { printf("%s: keepalive failed %#x\n", DEVNAME(un), rval); urndis_ctrl_reset(un); } return rval; } #endif static unsigned urndis_uno_tx_prepare(struct usbnet *un, struct mbuf *m, struct usbnet_chain *c) { struct rndis_packet_msg *msg; if ((unsigned)m->m_pkthdr.len > un->un_tx_bufsz - sizeof(*msg)) return 0; msg = (struct rndis_packet_msg *)c->unc_buf; memset(msg, 0, sizeof(*msg)); msg->rm_type = htole32(REMOTE_NDIS_PACKET_MSG); msg->rm_len = htole32(sizeof(*msg) + m->m_pkthdr.len); msg->rm_dataoffset = htole32(RNDIS_DATA_OFFSET); msg->rm_datalen = htole32(m->m_pkthdr.len); m_copydata(m, 0, m->m_pkthdr.len, ((char*)msg + RNDIS_DATA_OFFSET + RNDIS_HEADER_OFFSET)); DPRINTF(("%s: %s type %#x len %u data(off %u len %u)\n", __func__, DEVNAME(un), le32toh(msg->rm_type), le32toh(msg->rm_len), le32toh(msg->rm_dataoffset), le32toh(msg->rm_datalen))); return le32toh(msg->rm_len); } static void urndis_uno_rx_loop(struct usbnet * un, struct usbnet_chain *c, uint32_t total_len) { struct rndis_packet_msg *msg; struct ifnet *ifp = usbnet_ifp(un); int offset; offset = 0; while (total_len > 1) { msg = (struct rndis_packet_msg *)((char*)c->unc_buf + offset); DPRINTF(("%s: %s buffer size left %u\n", DEVNAME(un), __func__, total_len)); if (total_len < sizeof(*msg)) { printf("%s: urndis_decap invalid buffer total_len %u < " "minimum header %zu\n", DEVNAME(un), total_len, sizeof(*msg)); return; } DPRINTF(("%s: urndis_decap total_len %u data(off:%u len:%u) " "oobdata(off:%u len:%u nb:%u) perpacket(off:%u len:%u)\n", DEVNAME(un), le32toh(msg->rm_len), le32toh(msg->rm_dataoffset), le32toh(msg->rm_datalen), le32toh(msg->rm_oobdataoffset), le32toh(msg->rm_oobdatalen), le32toh(msg->rm_oobdataelements), le32toh(msg->rm_pktinfooffset), le32toh(msg->rm_pktinfooffset))); if (le32toh(msg->rm_type) != REMOTE_NDIS_PACKET_MSG) { printf("%s: urndis_decap invalid type %#x != %#x\n", DEVNAME(un), le32toh(msg->rm_type), REMOTE_NDIS_PACKET_MSG); return; } if (le32toh(msg->rm_len) < sizeof(*msg)) { printf("%s: urndis_decap invalid msg len %u < %zu\n", DEVNAME(un), le32toh(msg->rm_len), sizeof(*msg)); return; } if (le32toh(msg->rm_len) > total_len) { printf("%s: urndis_decap invalid msg len %u > buffer " "total_len %u\n", DEVNAME(un), le32toh(msg->rm_len), total_len); return; } if (le32toh(msg->rm_dataoffset) + le32toh(msg->rm_datalen) + RNDIS_HEADER_OFFSET > le32toh(msg->rm_len)) { printf("%s: urndis_decap invalid data " "len/offset/end_position(%u/%u/%u) -> " "go out of receive buffer limit %u\n", DEVNAME(un), le32toh(msg->rm_datalen), le32toh(msg->rm_dataoffset), le32toh(msg->rm_dataoffset) + le32toh(msg->rm_datalen) + (uint32_t)RNDIS_HEADER_OFFSET, le32toh(msg->rm_len)); return; } if (le32toh(msg->rm_datalen) < sizeof(struct ether_header)) { if_statinc(ifp, if_ierrors); printf("%s: urndis_decap invalid ethernet size " "%d < %zu\n", DEVNAME(un), le32toh(msg->rm_datalen), sizeof(struct ether_header)); return; } usbnet_enqueue(un, ((char*)&msg->rm_dataoffset + le32toh(msg->rm_dataoffset)), le32toh(msg->rm_datalen), 0, 0, 0); offset += le32toh(msg->rm_len); total_len -= le32toh(msg->rm_len); } } #if 0 static void urndis_watchdog(struct ifnet *ifp) { struct urndis_softc *sc = usbnet_softc(un); if (un->un_dying) return; if_statinc(ifp, if_oerrors); printf("%s: watchdog timeout\n", DEVNAME(un)); urndis_ctrl_keepalive(un); } #endif static int urndis_uno_init(struct ifnet *ifp) { struct usbnet *un = ifp->if_softc; KASSERT(IFNET_LOCKED(ifp)); if (urndis_ctrl_init(un) != RNDIS_STATUS_SUCCESS) return EIO; return 0; } static int urndis_match(device_t parent, cfdata_t match, void *aux) { struct usbif_attach_arg *uiaa = aux; usb_interface_descriptor_t *id; if (!uiaa->uiaa_iface) return UMATCH_NONE; id = usbd_get_interface_descriptor(uiaa->uiaa_iface); if (id == NULL) return UMATCH_NONE; if (id->bInterfaceClass == UICLASS_WIRELESS && id->bInterfaceSubClass == UISUBCLASS_RF && id->bInterfaceProtocol == UIPROTO_RNDIS) return UMATCH_IFACECLASS_IFACESUBCLASS_IFACEPROTO; return usb_lookup(urndis_devs, uiaa->uiaa_vendor, uiaa->uiaa_product) != NULL ? UMATCH_VENDOR_PRODUCT : UMATCH_NONE; } static void urndis_attach(device_t parent, device_t self, void *aux) { struct urndis_softc *sc = device_private(self); struct usbnet * const un = &sc->sc_un; struct usbif_attach_arg *uiaa = aux; struct usbd_device *dev = uiaa->uiaa_device; usb_interface_descriptor_t *id; usb_endpoint_descriptor_t *ed; usb_config_descriptor_t *cd; struct usbd_interface *iface_ctl; const usb_cdc_union_descriptor_t *ud; const usb_cdc_header_descriptor_t *desc; usbd_desc_iter_t iter; int if_ctl, if_data; int i, j, altcnt; void *buf; size_t bufsz; uint32_t filter; char *devinfop; KASSERT((void *)sc == un); aprint_naive("\n"); aprint_normal("\n"); devinfop = usbd_devinfo_alloc(dev, 0); aprint_normal_dev(self, "%s\n", devinfop); usbd_devinfo_free(devinfop); un->un_dev = self; un->un_udev = dev; un->un_sc = sc; un->un_ops = &urndis_ops; un->un_rx_xfer_flags = USBD_SHORT_XFER_OK; un->un_tx_xfer_flags = USBD_FORCE_SHORT_XFER; un->un_rx_list_cnt = RNDIS_RX_LIST_CNT; un->un_tx_list_cnt = RNDIS_TX_LIST_CNT; un->un_rx_bufsz = RNDIS_BUFSZ; un->un_tx_bufsz = RNDIS_BUFSZ; iface_ctl = uiaa->uiaa_iface; un->un_iface = uiaa->uiaa_iface; id = usbd_get_interface_descriptor(iface_ctl); if_ctl = id->bInterfaceNumber; sc->sc_ifaceno_ctl = if_ctl; if_data = -1; usb_desc_iter_init(un->un_udev, &iter); while ((desc = (const void *)usb_desc_iter_next(&iter)) != NULL) { if (desc->bDescriptorType != UDESC_CS_INTERFACE) { continue; } switch (desc->bDescriptorSubtype) { case UDESCSUB_CDC_UNION: /* XXX bail out when found first? */ ud = (const usb_cdc_union_descriptor_t *)desc; if (if_data == -1) if_data = ud->bSlaveInterface[0]; break; } } if (if_data == -1) { DPRINTF(("urndis_attach: no union interface\n")); un->un_iface = iface_ctl; } else { DPRINTF(("urndis_attach: union interface: ctl %u, data %u\n", if_ctl, if_data)); for (i = 0; i < uiaa->uiaa_nifaces; i++) { if (uiaa->uiaa_ifaces[i] != NULL) { id = usbd_get_interface_descriptor( uiaa->uiaa_ifaces[i]); if (id != NULL && id->bInterfaceNumber == if_data) { un->un_iface = uiaa->uiaa_ifaces[i]; uiaa->uiaa_ifaces[i] = NULL; } } } } if (un->un_iface == NULL) { aprint_error("%s: no data interface\n", DEVNAME(un)); return; } id = usbd_get_interface_descriptor(un->un_iface); cd = usbd_get_config_descriptor(un->un_udev); altcnt = usbd_get_no_alts(cd, id->bInterfaceNumber); for (j = 0; j < altcnt; j++) { if (usbd_set_interface(un->un_iface, j)) { aprint_error("%s: interface alternate setting %u " "failed\n", DEVNAME(un), j); return; } /* Find endpoints. */ id = usbd_get_interface_descriptor(un->un_iface); un->un_ed[USBNET_ENDPT_RX] = un->un_ed[USBNET_ENDPT_TX] = 0; for (i = 0; i < id->bNumEndpoints; i++) { ed = usbd_interface2endpoint_descriptor( un->un_iface, i); if (!ed) { aprint_error("%s: no descriptor for bulk " "endpoint %u\n", DEVNAME(un), i); return; } if (UE_GET_DIR(ed->bEndpointAddress) == UE_DIR_IN && UE_GET_XFERTYPE(ed->bmAttributes) == UE_BULK) { un->un_ed[USBNET_ENDPT_RX] = ed->bEndpointAddress; } else if ( UE_GET_DIR(ed->bEndpointAddress) == UE_DIR_OUT && UE_GET_XFERTYPE(ed->bmAttributes) == UE_BULK) { un->un_ed[USBNET_ENDPT_TX] = ed->bEndpointAddress; } } if (un->un_ed[USBNET_ENDPT_RX] != 0 && un->un_ed[USBNET_ENDPT_TX] != 0) { DPRINTF(("%s: in=%#x, out=%#x\n", DEVNAME(un), un->un_ed[USBNET_ENDPT_RX], un->un_ed[USBNET_ENDPT_TX])); break; } } if (un->un_ed[USBNET_ENDPT_RX] == 0) aprint_error("%s: could not find data bulk in\n", DEVNAME(un)); if (un->un_ed[USBNET_ENDPT_TX] == 0) aprint_error("%s: could not find data bulk out\n",DEVNAME(un)); if (un->un_ed[USBNET_ENDPT_RX] == 0 || un->un_ed[USBNET_ENDPT_TX] == 0) return; #if 0 ifp->if_watchdog = urndis_watchdog; #endif usbnet_attach(un); if (urndis_ctrl_init(un) != RNDIS_STATUS_SUCCESS) { aprint_error("%s: unable to initialize hardware\n", DEVNAME(un)); return; } if (urndis_ctrl_query(un, OID_802_3_PERMANENT_ADDRESS, NULL, 0, &buf, &bufsz) != RNDIS_STATUS_SUCCESS) { aprint_error("%s: unable to get hardware address\n", DEVNAME(un)); return; } if (bufsz == ETHER_ADDR_LEN) { memcpy(un->un_eaddr, buf, ETHER_ADDR_LEN); kmem_free(buf, bufsz); } else { aprint_error("%s: invalid address\n", DEVNAME(un)); if (buf && bufsz) kmem_free(buf, bufsz); return; } /* Initialize packet filter */ sc->sc_filter = RNDIS_PACKET_TYPE_BROADCAST; sc->sc_filter |= RNDIS_PACKET_TYPE_ALL_MULTICAST; filter = htole32(sc->sc_filter); if (urndis_ctrl_set(un, OID_GEN_CURRENT_PACKET_FILTER, &filter, sizeof(filter)) != RNDIS_STATUS_SUCCESS) { aprint_error("%s: unable to set data filters\n", DEVNAME(un)); return; } usbnet_attach_ifp(un, IFF_SIMPLEX | IFF_BROADCAST | IFF_MULTICAST, 0, NULL); } #ifdef _MODULE #include "ioconf.c" #endif USBNET_MODULE(urndis) |
| 4 3 2 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 | /* $NetBSD: strnlen.c,v 1.2 2014/01/09 11:25:11 apb Exp $ */ /*- * Copyright (c) 2009 David Schultz <das@FreeBSD.org> * All rights reserved. * * Redistribution and use in source and binary forms, with or without * modification, are permitted provided that the following conditions * are met: * 1. Redistributions of source code must retain the above copyright * notice, this list of conditions and the following disclaimer. * 2. Redistributions in binary form must reproduce the above copyright * notice, this list of conditions and the following disclaimer in the * documentation and/or other materials provided with the distribution. * * THIS SOFTWARE IS PROVIDED BY THE AUTHOR AND CONTRIBUTORS ``AS IS'' AND * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE * ARE DISCLAIMED. IN NO EVENT SHALL THE AUTHOR OR CONTRIBUTORS BE LIABLE * FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL * DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS * OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) * HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT * LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY * OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF * SUCH DAMAGE. */ #if HAVE_NBTOOL_CONFIG_H #include "nbtool_config.h" #endif #include <sys/cdefs.h> #if defined(LIBC_SCCS) && !defined(lint) __RCSID("$NetBSD: strnlen.c,v 1.2 2014/01/09 11:25:11 apb Exp $"); #endif /* LIBC_SCCS and not lint */ /* FreeBSD: src/lib/libc/string/strnlen.c,v 1.1 2009/02/28 06:00:58 das Exp */ #if !defined(_KERNEL) && !defined(_STANDALONE) #include <string.h> #else #include <lib/libkern/libkern.h> #endif #if !HAVE_STRNLEN size_t strnlen(const char *s, size_t maxlen) { size_t len; for (len = 0; len < maxlen; len++, s++) { if (!*s) break; } return (len); } #endif /* !HAVE_STRNLEN */ |
| 5 1 1 2 1 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 197 198 199 200 201 202 203 204 205 206 207 208 209 210 211 212 213 214 215 216 217 218 219 220 221 222 223 224 225 226 227 228 229 230 231 232 233 234 235 236 237 238 239 240 241 242 243 244 245 246 247 248 249 250 251 252 253 254 255 256 257 258 259 260 261 262 263 264 265 266 267 268 269 270 271 272 273 274 275 276 277 278 279 280 281 282 283 284 285 286 287 288 289 290 291 292 293 294 295 296 297 298 299 300 301 302 303 304 305 306 307 308 309 310 311 312 313 314 315 316 317 318 319 320 321 322 323 324 325 326 327 328 329 330 331 332 333 334 335 336 337 338 339 340 341 342 343 344 345 346 347 348 349 350 351 352 353 354 355 356 357 358 359 360 361 362 363 364 365 366 367 368 369 370 371 372 373 374 375 376 377 378 379 380 381 382 383 384 385 386 387 388 389 390 391 392 393 394 395 396 397 398 399 400 401 402 403 404 405 406 407 408 409 410 411 412 413 414 415 416 417 418 419 420 421 422 423 424 425 426 427 428 429 430 431 432 433 434 435 436 437 438 439 440 441 442 443 444 445 446 447 448 449 450 451 452 453 454 455 456 457 458 459 460 461 462 463 464 465 466 467 468 469 470 471 472 473 474 475 476 477 478 479 480 481 482 483 484 485 486 487 488 489 490 491 492 493 494 495 496 497 498 499 500 501 502 503 504 505 506 507 508 509 510 511 512 513 514 515 516 517 518 519 520 521 522 523 524 525 526 527 528 529 530 531 532 533 534 535 536 537 538 539 540 541 542 543 544 545 546 547 548 549 550 551 552 553 554 555 556 557 558 559 560 561 562 563 564 565 566 567 568 569 570 571 572 573 574 575 576 577 578 579 580 581 582 583 584 585 586 587 588 589 590 591 592 593 594 595 596 597 598 599 600 601 602 603 604 605 606 607 608 609 610 611 612 613 614 615 616 617 618 619 620 621 622 623 624 625 626 627 628 629 630 631 632 633 634 635 636 637 638 639 640 641 642 643 644 645 646 647 648 649 650 651 652 653 654 655 656 657 658 659 660 661 662 663 664 665 666 667 668 669 670 671 672 673 674 675 676 677 678 679 680 681 682 683 684 685 686 687 688 689 690 691 692 693 694 695 696 697 698 699 700 701 702 703 704 705 706 707 708 709 710 711 712 713 714 715 716 717 718 719 720 721 722 723 724 725 726 727 728 729 730 731 732 733 734 735 736 737 738 739 740 741 742 743 744 745 746 747 748 749 750 751 752 753 754 755 756 757 758 759 760 761 762 763 764 765 766 767 768 769 770 771 772 773 774 775 776 777 778 779 780 781 782 783 784 785 786 787 788 789 790 791 792 793 794 795 796 797 798 799 800 801 802 803 804 805 806 807 808 809 810 811 812 813 814 815 816 817 818 819 820 821 822 823 824 825 826 827 828 829 830 831 832 833 834 835 836 837 838 839 840 841 842 843 844 845 846 847 848 849 850 851 852 853 854 855 856 857 858 859 860 861 862 863 864 865 866 867 868 869 870 871 872 873 874 875 876 877 878 879 880 881 882 883 884 885 886 887 888 889 890 891 892 893 894 895 896 897 898 899 900 901 902 903 904 905 906 907 908 909 910 911 912 913 914 915 916 917 918 919 920 921 922 923 924 925 926 927 928 929 930 931 932 933 934 935 936 937 938 939 940 941 942 943 944 945 946 947 948 949 950 951 952 953 954 955 956 957 958 959 960 961 962 963 964 965 966 967 968 969 970 971 972 973 974 975 976 977 978 979 980 981 982 983 984 985 986 987 988 989 990 991 992 993 994 995 996 997 998 999 1000 1001 1002 1003 1004 1005 1006 1007 1008 1009 1010 1011 1012 1013 1014 1015 1016 1017 1018 1019 1020 1021 1022 1023 1024 1025 1026 1027 1028 1029 1030 1031 1032 1033 1034 1035 1036 1037 1038 1039 1040 1041 1042 1043 1044 1045 1046 1047 1048 1049 1050 1051 1052 1053 1054 1055 1056 1057 1058 1059 1060 1061 1062 1063 1064 1065 1066 1067 1068 1069 1070 1071 1072 1073 1074 1075 1076 1077 1078 1079 1080 1081 1082 1083 1084 1085 1086 1087 1088 1089 1090 1091 1092 1093 1094 1095 1096 1097 1098 1099 1100 1101 1102 1103 1104 1105 1106 1107 1108 1109 1110 1111 1112 1113 1114 1115 1116 1117 1118 1119 1120 1121 1122 1123 1124 1125 1126 1127 1128 1129 1130 1131 1132 1133 1134 1135 1136 1137 1138 1139 1140 1141 1142 1143 1144 1145 1146 1147 1148 1149 1150 1151 1152 1153 1154 1155 1156 1157 1158 1159 1160 1161 1162 1163 1164 1165 1166 1167 1168 1169 1170 1171 1172 1173 1174 1175 1176 1177 1178 1179 1180 1181 1182 1183 1184 1185 1186 1187 1188 1189 1190 1191 1192 1193 1194 1195 1196 1197 1198 1199 1200 1201 1202 1203 1204 1205 1206 1207 1208 1209 1210 1211 1212 1213 1214 1215 1216 1217 1218 1219 1220 1221 1222 1223 1224 1225 1226 1227 1228 1229 1230 1231 1232 1233 1234 1235 1236 1237 1238 1239 1240 1241 1242 1243 1244 1245 1246 1247 1248 1249 1250 1251 1252 1253 1254 1255 1256 1257 1258 1259 1260 1261 1262 1263 1264 1265 1266 1267 1268 1269 1270 1271 1272 1273 1274 1275 1276 1277 1278 1279 1280 1281 1282 1283 1284 1285 1286 1287 1288 1289 1290 1291 1292 1293 1294 1295 1296 1297 1298 1299 1300 1301 1302 1303 1304 1305 1306 1307 1308 1309 1310 1311 1312 1313 1314 1315 1316 1317 1318 1319 1320 1321 1322 1323 1324 1325 1326 1327 1328 1329 1330 1331 1332 1333 1334 1335 1336 1337 1338 1339 1340 1341 1342 1343 1344 1345 1346 1347 1348 1349 1350 1351 1352 1353 1354 1355 1356 1357 1358 1359 1360 1361 1362 1363 1364 1365 1366 1367 1368 1369 1370 1371 1372 1373 1374 1375 1376 1377 1378 1379 1380 1381 1382 1383 1384 1385 1386 1387 1388 1389 1390 1391 1392 1393 1394 1395 1396 1397 1398 1399 1400 1401 1402 1403 | /* $NetBSD: ext2fs_vfsops.c,v 1.221 2022/05/22 11:27:36 andvar Exp $ */ /* * Copyright (c) 1989, 1991, 1993, 1994 * The Regents of the University of California. All rights reserved. * * Redistribution and use in source and binary forms, with or without * modification, are permitted provided that the following conditions * are met: * 1. Redistributions of source code must retain the above copyright * notice, this list of conditions and the following disclaimer. * 2. Redistributions in binary form must reproduce the above copyright * notice, this list of conditions and the following disclaimer in the * documentation and/or other materials provided with the distribution. * 3. Neither the name of the University nor the names of its contributors * may be used to endorse or promote products derived from this software * without specific prior written permission. * * THIS SOFTWARE IS PROVIDED BY THE REGENTS AND CONTRIBUTORS ``AS IS'' AND * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE * ARE DISCLAIMED. IN NO EVENT SHALL THE REGENTS OR CONTRIBUTORS BE LIABLE * FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL * DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS * OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) * HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT * LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY * OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF * SUCH DAMAGE. * * @(#)ffs_vfsops.c 8.14 (Berkeley) 11/28/94 * Modified for ext2fs by Manuel Bouyer. */ /* * Copyright (c) 1997 Manuel Bouyer. * * Redistribution and use in source and binary forms, with or without * modification, are permitted provided that the following conditions * are met: * 1. Redistributions of source code must retain the above copyright * notice, this list of conditions and the following disclaimer. * 2. Redistributions in binary form must reproduce the above copyright * notice, this list of conditions and the following disclaimer in the * documentation and/or other materials provided with the distribution. * * THIS SOFTWARE IS PROVIDED BY THE AUTHOR ``AS IS'' AND ANY EXPRESS OR * IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES * OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. * IN NO EVENT SHALL THE AUTHOR BE LIABLE FOR ANY DIRECT, INDIRECT, * INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT * NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, * DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY * THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT * (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF * THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. * * @(#)ffs_vfsops.c 8.14 (Berkeley) 11/28/94 * Modified for ext2fs by Manuel Bouyer. */ #include <sys/cdefs.h> __KERNEL_RCSID(0, "$NetBSD: ext2fs_vfsops.c,v 1.221 2022/05/22 11:27:36 andvar Exp $"); #if defined(_KERNEL_OPT) #include "opt_compat_netbsd.h" #endif #include <sys/param.h> #include <sys/systm.h> #include <sys/sysctl.h> #include <sys/namei.h> #include <sys/proc.h> #include <sys/kernel.h> #include <sys/vnode.h> #include <sys/socket.h> #include <sys/mount.h> #include <sys/buf.h> #include <sys/device.h> #include <sys/file.h> #include <sys/disklabel.h> #include <sys/ioctl.h> #include <sys/errno.h> #include <sys/pool.h> #include <sys/lock.h> #include <sys/conf.h> #include <sys/kauth.h> #include <sys/module.h> #include <miscfs/genfs/genfs.h> #include <miscfs/specfs/specdev.h> #include <ufs/ufs/quota.h> #include <ufs/ufs/ufsmount.h> #include <ufs/ufs/inode.h> #include <ufs/ufs/dir.h> #include <ufs/ufs/ufs_extern.h> #include <ufs/ext2fs/ext2fs.h> #include <ufs/ext2fs/ext2fs_dir.h> #include <ufs/ext2fs/ext2fs_extern.h> MODULE(MODULE_CLASS_VFS, ext2fs, "ufs"); int ext2fs_sbupdate(struct ufsmount *, int); static int ext2fs_sbfill(struct m_ext2fs *, int); extern const struct vnodeopv_desc ext2fs_vnodeop_opv_desc; extern const struct vnodeopv_desc ext2fs_specop_opv_desc; extern const struct vnodeopv_desc ext2fs_fifoop_opv_desc; const struct vnodeopv_desc * const ext2fs_vnodeopv_descs[] = { &ext2fs_vnodeop_opv_desc, &ext2fs_specop_opv_desc, &ext2fs_fifoop_opv_desc, NULL, }; struct vfsops ext2fs_vfsops = { .vfs_name = MOUNT_EXT2FS, .vfs_min_mount_data = sizeof (struct ufs_args), .vfs_mount = ext2fs_mount, .vfs_start = ufs_start, .vfs_unmount = ext2fs_unmount, .vfs_root = ufs_root, .vfs_quotactl = ufs_quotactl, .vfs_statvfs = ext2fs_statvfs, .vfs_sync = ext2fs_sync, .vfs_vget = ufs_vget, .vfs_loadvnode = ext2fs_loadvnode, .vfs_newvnode = ext2fs_newvnode, .vfs_fhtovp = ext2fs_fhtovp, .vfs_vptofh = ext2fs_vptofh, .vfs_init = ext2fs_init, .vfs_reinit = ext2fs_reinit, .vfs_done = ext2fs_done, .vfs_mountroot = ext2fs_mountroot, .vfs_snapshot = (void *)eopnotsupp, .vfs_extattrctl = vfs_stdextattrctl, .vfs_suspendctl = genfs_suspendctl, .vfs_renamelock_enter = genfs_renamelock_enter, .vfs_renamelock_exit = genfs_renamelock_exit, .vfs_fsync = (void *)eopnotsupp, .vfs_opv_descs = ext2fs_vnodeopv_descs }; static const struct genfs_ops ext2fs_genfsops = { .gop_size = genfs_size, .gop_alloc = ext2fs_gop_alloc, .gop_write = genfs_gop_write, .gop_markupdate = ufs_gop_markupdate, .gop_putrange = genfs_gop_putrange, }; static const struct ufs_ops ext2fs_ufsops = { .uo_itimes = ext2fs_itimes, .uo_update = ext2fs_update, .uo_bufrd = ext2fs_bufrd, .uo_bufwr = ext2fs_bufwr, }; /* Fill in the inode uid/gid from ext2 halves. */ void ext2fs_set_inode_guid(struct inode *ip) { ip->i_gid = ip->i_e2fs_gid; ip->i_uid = ip->i_e2fs_uid; if (ip->i_e2fs->e2fs.e2fs_rev > E2FS_REV0) { ip->i_gid |= ip->i_e2fs_gid_high << 16; ip->i_uid |= ip->i_e2fs_uid_high << 16; } } SYSCTL_SETUP(ext2fs_sysctl_setup, "ext2fs sysctl") { sysctl_createv(clog, 0, NULL, NULL, CTLFLAG_PERMANENT, CTLTYPE_NODE, "ext2fs", SYSCTL_DESCR("Linux EXT2FS file system"), NULL, 0, NULL, 0, CTL_VFS, 17, CTL_EOL); /* * XXX the "17" above could be dynamic, thereby eliminating * one more instance of the "number to vfs" mapping problem, * but "17" is the order as taken from sys/mount.h */ } static int ext2fs_modcmd(modcmd_t cmd, void *arg) { int error; switch (cmd) { case MODULE_CMD_INIT: error = vfs_attach(&ext2fs_vfsops); if (error != 0) break; break; case MODULE_CMD_FINI: error = vfs_detach(&ext2fs_vfsops); if (error != 0) break; break; default: error = ENOTTY; break; } return error; } /* * XXX Same structure as FFS inodes? Should we share a common pool? */ struct pool ext2fs_inode_pool; extern u_long ext2gennumber; void ext2fs_init(void) { pool_init(&ext2fs_inode_pool, sizeof(struct inode), 0, 0, 0, "ext2fsinopl", &pool_allocator_nointr, IPL_NONE); ufs_init(); } void ext2fs_reinit(void) { ufs_reinit(); } void ext2fs_done(void) { ufs_done(); pool_destroy(&ext2fs_inode_pool); } static void ext2fs_sb_setmountinfo(struct m_ext2fs *fs, struct mount *mp) { (void)strlcpy(fs->e2fs_fsmnt, mp->mnt_stat.f_mntonname, sizeof(fs->e2fs_fsmnt)); if (fs->e2fs_ronly == 0 && fs->e2fs.e2fs_rev > E2FS_REV0) { (void)strlcpy(fs->e2fs.e2fs_fsmnt, mp->mnt_stat.f_mntonname, sizeof(fs->e2fs.e2fs_fsmnt)); fs->e2fs.e2fs_mtime = time_second; fs->e2fs.e2fs_mnt_count++; fs->e2fs_fmod = 1; } } /* * Called by main() when ext2fs is going to be mounted as root. * * Name is updated by mount(8) after booting. */ int ext2fs_mountroot(void) { extern struct vnode *rootvp; struct m_ext2fs *fs; struct mount *mp; struct ufsmount *ump; int error; if (device_class(root_device) != DV_DISK) return ENODEV; if ((error = vfs_rootmountalloc(MOUNT_EXT2FS, "root_device", &mp))) { vrele(rootvp); return error; } if ((error = ext2fs_mountfs(rootvp, mp)) != 0) { vfs_unbusy(mp); vfs_rele(mp); return error; } mountlist_append(mp); ump = VFSTOUFS(mp); fs = ump->um_e2fs; ext2fs_sb_setmountinfo(fs, mp); (void)ext2fs_statvfs(mp, &mp->mnt_stat); vfs_unbusy(mp); setrootfstime((time_t)fs->e2fs.e2fs_wtime); return 0; } /* * VFS Operations. * * mount system call */ int ext2fs_mount(struct mount *mp, const char *path, void *data, size_t *data_len) { struct lwp *l = curlwp; struct vnode *devvp; struct ufs_args *args = data; struct ufsmount *ump = NULL; struct m_ext2fs *fs; int error = 0, flags, update; mode_t accessmode; if (args == NULL) return EINVAL; if (*data_len < sizeof *args) return EINVAL; if (mp->mnt_flag & MNT_GETARGS) { ump = VFSTOUFS(mp); if (ump == NULL) return EIO; memset(args, 0, sizeof *args); args->fspec = NULL; *data_len = sizeof *args; return 0; } update = mp->mnt_flag & MNT_UPDATE; /* Check arguments */ if (args->fspec != NULL) { /* * Look up the name and verify that it's sane. */ error = namei_simple_user(args->fspec, NSM_FOLLOW_NOEMULROOT, &devvp); if (error != 0) return error; if (!update) { /* * Be sure this is a valid block device */ if (devvp->v_type != VBLK) error = ENOTBLK; else if (bdevsw_lookup(devvp->v_rdev) == NULL) error = ENXIO; } else { /* * Be sure we're still naming the same device * used for our initial mount */ ump = VFSTOUFS(mp); if (devvp != ump->um_devvp) { if (devvp->v_rdev != ump->um_devvp->v_rdev) error = EINVAL; else { vrele(devvp); devvp = ump->um_devvp; vref(devvp); } } } } else { if (!update) { /* New mounts must have a filename for the device */ return EINVAL; } else { ump = VFSTOUFS(mp); devvp = ump->um_devvp; vref(devvp); } } /* * If mount by non-root, then verify that user has necessary * permissions on the device. * * Permission to update a mount is checked higher, so here we presume * updating the mount is okay (for example, as far as securelevel goes) * which leaves us with the normal check. */ if (error == 0) { accessmode = VREAD; if (update ? (mp->mnt_iflag & IMNT_WANTRDWR) != 0 : (mp->mnt_flag & MNT_RDONLY) == 0) accessmode |= VWRITE; vn_lock(devvp, LK_EXCLUSIVE | LK_RETRY); error = kauth_authorize_system(l->l_cred, KAUTH_SYSTEM_MOUNT, KAUTH_REQ_SYSTEM_MOUNT_DEVICE, mp, devvp, KAUTH_ARG(accessmode)); VOP_UNLOCK(devvp); } if (error) { vrele(devvp); return error; } if (!update) { int xflags; if (mp->mnt_flag & MNT_RDONLY) xflags = FREAD; else xflags = FREAD|FWRITE; vn_lock(devvp, LK_EXCLUSIVE | LK_RETRY); error = VOP_OPEN(devvp, xflags, FSCRED); VOP_UNLOCK(devvp); if (error) goto fail; error = ext2fs_mountfs(devvp, mp); if (error) { vn_lock(devvp, LK_EXCLUSIVE | LK_RETRY); (void)VOP_CLOSE(devvp, xflags, NOCRED); VOP_UNLOCK(devvp); goto fail; } ump = VFSTOUFS(mp); fs = ump->um_e2fs; } else { /* * Update the mount. */ /* * The initial mount got a reference on this * device, so drop the one obtained via * namei(), above. */ vrele(devvp); ump = VFSTOUFS(mp); fs = ump->um_e2fs; if (fs->e2fs_ronly == 0 && (mp->mnt_flag & MNT_RDONLY)) { /* * Changing from r/w to r/o */ flags = WRITECLOSE; if (mp->mnt_flag & MNT_FORCE) flags |= FORCECLOSE; error = ext2fs_flushfiles(mp, flags); if (error == 0 && ext2fs_cgupdate(ump, MNT_WAIT) == 0 && (fs->e2fs.e2fs_state & E2FS_ERRORS) == 0) { fs->e2fs.e2fs_state = E2FS_ISCLEAN; (void) ext2fs_sbupdate(ump, MNT_WAIT); } if (error) return error; fs->e2fs_ronly = 1; } if (mp->mnt_flag & MNT_RELOAD) { error = ext2fs_reload(mp, l->l_cred, l); if (error) return error; } if (fs->e2fs_ronly && (mp->mnt_iflag & IMNT_WANTRDWR)) { /* * Changing from read-only to read/write */ fs->e2fs_ronly = 0; if (fs->e2fs.e2fs_state == E2FS_ISCLEAN) fs->e2fs.e2fs_state = 0; else fs->e2fs.e2fs_state = E2FS_ERRORS; fs->e2fs_fmod = 1; } if (args->fspec == NULL) return 0; } error = set_statvfs_info(path, UIO_USERSPACE, args->fspec, UIO_USERSPACE, mp->mnt_op->vfs_name, mp, l); if (error == 0) ext2fs_sb_setmountinfo(fs, mp); if (fs->e2fs_fmod != 0) { /* XXX */ fs->e2fs_fmod = 0; if (fs->e2fs.e2fs_state == 0) fs->e2fs.e2fs_wtime = time_second; else printf("%s: file system not clean; please fsck(8)\n", mp->mnt_stat.f_mntfromname); (void) ext2fs_cgupdate(ump, MNT_WAIT); } return error; fail: vrele(devvp); return error; } /* * Sanity check the disk vnode content, and copy it over to inode structure. */ static int ext2fs_loadvnode_content(struct m_ext2fs *fs, ino_t ino, struct buf *bp, struct inode *ip) { struct ext2fs_dinode *din; int error = 0; din = (struct ext2fs_dinode *)((char *)bp->b_data + (ino_to_fsbo(fs, ino) * EXT2_DINODE_SIZE(fs))); /* sanity checks - inode data NOT byteswapped at this point */ if (EXT2_DINODE_FITS(din, e2di_extra_isize, EXT2_DINODE_SIZE(fs)) && (EXT2_DINODE_SIZE(fs) - EXT2_REV0_DINODE_SIZE) < fs2h16(din->e2di_extra_isize)) { printf("ext2fs: inode %"PRIu64" bad extra_isize %u", ino, din->e2di_extra_isize); error = EINVAL; goto bad; } /* everything alright, proceed with copy */ if (ip->i_din.e2fs_din == NULL) ip->i_din.e2fs_din = kmem_alloc(EXT2_DINODE_SIZE(fs), KM_SLEEP); e2fs_iload(din, ip->i_din.e2fs_din, EXT2_DINODE_SIZE(fs)); ext2fs_set_inode_guid(ip); bad: return error; } /* * Reload all incore data for a filesystem (used after running fsck on * the root filesystem and finding things to fix). The filesystem must * be mounted read-only. * * Things to do to update the mount: * 1) invalidate all cached meta-data. * 2) re-read superblock from disk. * 3) re-read summary information from disk. * 4) invalidate all inactive vnodes. * 5) invalidate all cached file data. * 6) re-read inode data for all active vnodes. */ int ext2fs_reload(struct mount *mp, kauth_cred_t cred, struct lwp *l) { struct vnode *vp, *devvp; struct inode *ip; struct buf *bp; struct m_ext2fs *fs; struct ext2fs *newfs; int i, error; struct ufsmount *ump; struct vnode_iterator *marker; if ((mp->mnt_flag & MNT_RDONLY) == 0) return EINVAL; ump = VFSTOUFS(mp); /* * Step 1: invalidate all cached meta-data. */ devvp = ump->um_devvp; vn_lock(devvp, LK_EXCLUSIVE | LK_RETRY); error = vinvalbuf(devvp, 0, cred, l, 0, 0); VOP_UNLOCK(devvp); if (error) panic("ext2fs_reload: dirty1"); fs = ump->um_e2fs; /* * Step 2: re-read superblock from disk. Copy in new superblock, and compute * in-memory values. */ error = bread(devvp, SBLOCK, SBSIZE, 0, &bp); if (error) return error; newfs = (struct ext2fs *)bp->b_data; e2fs_sbload(newfs, &fs->e2fs); brelse(bp, 0); error = ext2fs_sbfill(fs, (mp->mnt_flag & MNT_RDONLY) != 0); if (error) return error; /* * Step 3: re-read summary information from disk. */ for (i = 0; i < fs->e2fs_ngdb; i++) { error = bread(devvp , EXT2_FSBTODB(fs, fs->e2fs.e2fs_first_dblock + 1 /* superblock */ + i), fs->e2fs_bsize, 0, &bp); if (error) { return error; } e2fs_cgload((struct ext2_gd *)bp->b_data, &fs->e2fs_gd[i * fs->e2fs_bsize / sizeof(struct ext2_gd)], fs->e2fs_bsize); brelse(bp, 0); } vfs_vnode_iterator_init(mp, &marker); while ((vp = vfs_vnode_iterator_next(marker, NULL, NULL))) { /* * Step 4: invalidate all inactive vnodes. */ if (vrecycle(vp)) continue; /* * Step 5: invalidate all cached file data. */ if (vn_lock(vp, LK_EXCLUSIVE)) { vrele(vp); continue; } if (vinvalbuf(vp, 0, cred, l, 0, 0)) panic("ext2fs_reload: dirty2"); /* * Step 6: re-read inode data for all active vnodes. */ ip = VTOI(vp); error = bread(devvp, EXT2_FSBTODB(fs, ino_to_fsba(fs, ip->i_number)), (int)fs->e2fs_bsize, 0, &bp); if (error) { vput(vp); break; } error = ext2fs_loadvnode_content(fs, ip->i_number, bp, ip); brelse(bp, 0); if (error) { vput(vp); break; } vput(vp); } vfs_vnode_iterator_destroy(marker); return error; } /* * Common code for mount and mountroot */ int ext2fs_mountfs(struct vnode *devvp, struct mount *mp) { struct lwp *l = curlwp; struct ufsmount *ump; struct buf *bp; struct ext2fs *fs; struct m_ext2fs *m_fs; dev_t dev; int error, i, ronly; kauth_cred_t cred; dev = devvp->v_rdev; cred = l->l_cred; /* Flush out any old buffers remaining from a previous use. */ vn_lock(devvp, LK_EXCLUSIVE | LK_RETRY); error = vinvalbuf(devvp, V_SAVE, cred, l, 0, 0); VOP_UNLOCK(devvp); if (error) return error; ronly = (mp->mnt_flag & MNT_RDONLY) != 0; bp = NULL; ump = NULL; /* Read the superblock from disk, and swap it directly. */ error = bread(devvp, SBLOCK, SBSIZE, 0, &bp); if (error) goto out; fs = (struct ext2fs *)bp->b_data; m_fs = kmem_zalloc(sizeof(*m_fs), KM_SLEEP); e2fs_sbload(fs, &m_fs->e2fs); brelse(bp, 0); bp = NULL; /* Once swapped, validate and fill in the superblock. */ error = ext2fs_sbfill(m_fs, ronly); if (error) { kmem_free(m_fs, sizeof(*m_fs)); goto out; } m_fs->e2fs_ronly = ronly; ump = kmem_zalloc(sizeof(*ump), KM_SLEEP); ump->um_fstype = UFS1; ump->um_ops = &ext2fs_ufsops; ump->um_e2fs = m_fs; if (ronly == 0) { if (m_fs->e2fs.e2fs_state == E2FS_ISCLEAN) m_fs->e2fs.e2fs_state = 0; else m_fs->e2fs.e2fs_state = E2FS_ERRORS; m_fs->e2fs_fmod = 1; } /* XXX: should be added in ext2fs_sbfill()? */ m_fs->e2fs_gd = kmem_alloc(m_fs->e2fs_ngdb * m_fs->e2fs_bsize, KM_SLEEP); for (i = 0; i < m_fs->e2fs_ngdb; i++) { error = bread(devvp, EXT2_FSBTODB(m_fs, m_fs->e2fs.e2fs_first_dblock + 1 /* superblock */ + i), m_fs->e2fs_bsize, 0, &bp); if (error) { kmem_free(m_fs->e2fs_gd, m_fs->e2fs_ngdb * m_fs->e2fs_bsize); goto out; } e2fs_cgload((struct ext2_gd *)bp->b_data, &m_fs->e2fs_gd[ i * m_fs->e2fs_bsize / sizeof(struct ext2_gd)], m_fs->e2fs_bsize); brelse(bp, 0); bp = NULL; } error = ext2fs_cg_verify_and_initialize(devvp, m_fs, ronly); if (error) { kmem_free(m_fs->e2fs_gd, m_fs->e2fs_ngdb * m_fs->e2fs_bsize); goto out; } mp->mnt_data = ump; mp->mnt_stat.f_fsidx.__fsid_val[0] = (long)dev; mp->mnt_stat.f_fsidx.__fsid_val[1] = makefstype(MOUNT_EXT2FS); mp->mnt_stat.f_fsid = mp->mnt_stat.f_fsidx.__fsid_val[0]; mp->mnt_stat.f_namemax = EXT2FS_MAXNAMLEN; mp->mnt_flag |= MNT_LOCAL; mp->mnt_dev_bshift = DEV_BSHIFT; /* XXX */ mp->mnt_fs_bshift = m_fs->e2fs_bshift; mp->mnt_iflag |= IMNT_DTYPE | IMNT_SHRLOOKUP; ump->um_flags = 0; ump->um_mountp = mp; ump->um_dev = dev; ump->um_devvp = devvp; ump->um_nindir = EXT2_NINDIR(m_fs); ump->um_lognindir = ffs(EXT2_NINDIR(m_fs)) - 1; ump->um_bptrtodb = m_fs->e2fs_fsbtodb; ump->um_seqinc = 1; /* no frags */ ump->um_maxsymlinklen = EXT2_MAXSYMLINKLEN; ump->um_dirblksiz = m_fs->e2fs_bsize; ump->um_maxfilesize = ((uint64_t)0x80000000 * m_fs->e2fs_bsize - 1); spec_node_setmountedfs(devvp, mp); return 0; out: if (bp != NULL) brelse(bp, 0); if (ump) { kmem_free(ump->um_e2fs, sizeof(*m_fs)); kmem_free(ump, sizeof(*ump)); mp->mnt_data = NULL; } return error; } /* * unmount system call */ int ext2fs_unmount(struct mount *mp, int mntflags) { struct ufsmount *ump; struct m_ext2fs *fs; int error, flags; flags = 0; if (mntflags & MNT_FORCE) flags |= FORCECLOSE; if ((error = ext2fs_flushfiles(mp, flags)) != 0) return error; ump = VFSTOUFS(mp); fs = ump->um_e2fs; if (fs->e2fs_ronly == 0 && ext2fs_cgupdate(ump, MNT_WAIT) == 0 && (fs->e2fs.e2fs_state & E2FS_ERRORS) == 0) { fs->e2fs.e2fs_state = E2FS_ISCLEAN; (void) ext2fs_sbupdate(ump, MNT_WAIT); } if (ump->um_devvp->v_type != VBAD) spec_node_setmountedfs(ump->um_devvp, NULL); vn_lock(ump->um_devvp, LK_EXCLUSIVE | LK_RETRY); error = VOP_CLOSE(ump->um_devvp, fs->e2fs_ronly ? FREAD : FREAD|FWRITE, NOCRED); vput(ump->um_devvp); kmem_free(fs->e2fs_gd, fs->e2fs_ngdb * fs->e2fs_bsize); kmem_free(fs, sizeof(*fs)); kmem_free(ump, sizeof(*ump)); mp->mnt_data = NULL; mp->mnt_flag &= ~MNT_LOCAL; return error; } /* * Flush out all the files in a filesystem. */ int ext2fs_flushfiles(struct mount *mp, int flags) { extern int doforce; int error; if (!doforce) flags &= ~FORCECLOSE; error = vflush(mp, NULLVP, flags); return error; } /* * Get file system statistics. */ int ext2fs_statvfs(struct mount *mp, struct statvfs *sbp) { struct ufsmount *ump; struct m_ext2fs *fs; uint32_t overhead, overhead_per_group, ngdb; int i, ngroups; ump = VFSTOUFS(mp); fs = ump->um_e2fs; if (fs->e2fs.e2fs_magic != E2FS_MAGIC) panic("ext2fs_statvfs"); /* * Compute the overhead (FS structures) */ overhead_per_group = 1 /* block bitmap */ + 1 /* inode bitmap */ + fs->e2fs_itpg; overhead = fs->e2fs.e2fs_first_dblock + fs->e2fs_ncg * overhead_per_group; if (EXT2F_HAS_COMPAT_FEATURE(fs, EXT2F_COMPAT_SPARSESUPER2)) { /* * Superblock and group descriptions is in group zero, * then optionally 0, 1 or 2 extra copies. */ ngroups = 1 + (fs->e2fs.e4fs_backup_bgs[0] ? 1 : 0) + (fs->e2fs.e4fs_backup_bgs[1] ? 1 : 0); } else if (EXT2F_HAS_ROCOMPAT_FEATURE(fs, EXT2F_ROCOMPAT_SPARSESUPER)) { for (i = 0, ngroups = 0; i < fs->e2fs_ncg; i++) { if (cg_has_sb(i)) ngroups++; } } else { ngroups = fs->e2fs_ncg; } ngdb = fs->e2fs_ngdb; if (EXT2F_HAS_COMPAT_FEATURE(fs, EXT2F_COMPAT_RESIZE)) ngdb += fs->e2fs.e2fs_reserved_ngdb; overhead += ngroups * (1 /* superblock */ + ngdb); sbp->f_bsize = fs->e2fs_bsize; sbp->f_frsize = MINBSIZE << fs->e2fs.e2fs_fsize; sbp->f_iosize = fs->e2fs_bsize; sbp->f_blocks = fs->e2fs.e2fs_bcount - overhead; sbp->f_bfree = fs->e2fs.e2fs_fbcount; sbp->f_bresvd = fs->e2fs.e2fs_rbcount; if (sbp->f_bfree > sbp->f_bresvd) sbp->f_bavail = sbp->f_bfree - sbp->f_bresvd; else sbp->f_bavail = 0; sbp->f_files = fs->e2fs.e2fs_icount; sbp->f_ffree = fs->e2fs.e2fs_ficount; sbp->f_favail = fs->e2fs.e2fs_ficount; sbp->f_fresvd = 0; copy_statvfs_info(sbp, mp); return 0; } static bool ext2fs_sync_selector(void *cl, struct vnode *vp) { struct inode *ip; KASSERT(mutex_owned(vp->v_interlock)); ip = VTOI(vp); /* * Skip the vnode/inode if inaccessible. */ if (ip == NULL || vp->v_type == VNON) return false; if (((ip->i_flag & (IN_CHANGE | IN_UPDATE | IN_MODIFIED)) == 0 && LIST_EMPTY(&vp->v_dirtyblkhd) && (vp->v_iflag & VI_ONWORKLST) == 0)) return false; return true; } /* * Go through the disk queues to initiate sandbagged IO; * go through the inodes to write those that have been modified; * initiate the writing of the super block if it has been modified. * * Note: we are always called with the filesystem marked `MPBUSY'. */ int ext2fs_sync(struct mount *mp, int waitfor, kauth_cred_t cred) { struct vnode *vp; struct ufsmount *ump = VFSTOUFS(mp); struct m_ext2fs *fs; struct vnode_iterator *marker; int error, allerror = 0; fs = ump->um_e2fs; if (fs->e2fs_fmod != 0 && fs->e2fs_ronly != 0) { /* XXX */ printf("fs = %s\n", fs->e2fs_fsmnt); panic("update: rofs mod"); } /* * Write back each (modified) inode. */ vfs_vnode_iterator_init(mp, &marker); while ((vp = vfs_vnode_iterator_next(marker, ext2fs_sync_selector, NULL))) { error = vn_lock(vp, LK_EXCLUSIVE); if (error) { vrele(vp); continue; } if (vp->v_type == VREG && waitfor == MNT_LAZY) error = ext2fs_update(vp, NULL, NULL, 0); else error = VOP_FSYNC(vp, cred, waitfor == MNT_WAIT ? FSYNC_WAIT : 0, 0, 0); if (error) allerror = error; vput(vp); } vfs_vnode_iterator_destroy(marker); /* * Force stale file system control information to be flushed. */ if (waitfor != MNT_LAZY) { vn_lock(ump->um_devvp, LK_EXCLUSIVE | LK_RETRY); if ((error = VOP_FSYNC(ump->um_devvp, cred, waitfor == MNT_WAIT ? FSYNC_WAIT : 0, 0, 0)) != 0) allerror = error; VOP_UNLOCK(ump->um_devvp); } /* * Write back modified superblock. */ if (fs->e2fs_fmod != 0) { fs->e2fs_fmod = 0; fs->e2fs.e2fs_wtime = time_second; if ((error = ext2fs_cgupdate(ump, waitfor))) allerror = error; } return allerror; } /* * Load inode from disk and initialize vnode. */ static int ext2fs_init_vnode(struct ufsmount *ump, struct vnode *vp, ino_t ino) { struct m_ext2fs *fs; struct inode *ip; struct buf *bp; int error; fs = ump->um_e2fs; /* Read in the disk contents for the inode, copy into the inode. */ error = bread(ump->um_devvp, EXT2_FSBTODB(fs, ino_to_fsba(fs, ino)), (int)fs->e2fs_bsize, 0, &bp); if (error) return error; /* Allocate and initialize inode. */ ip = pool_get(&ext2fs_inode_pool, PR_WAITOK); memset(ip, 0, sizeof(struct inode)); ip->i_vnode = vp; ip->i_ump = ump; ip->i_e2fs = fs; ip->i_dev = ump->um_dev; ip->i_number = ino; ip->i_e2fs_last_lblk = 0; ip->i_e2fs_last_blk = 0; error = ext2fs_loadvnode_content(fs, ino, bp, ip); brelse(bp, 0); if (error) { pool_put(&ext2fs_inode_pool, ip); return error; } /* If the inode was deleted, reset all fields */ if (ip->i_e2fs_dtime != 0) { ip->i_e2fs_mode = 0; (void)ext2fs_setsize(ip, 0); (void)ext2fs_setnblock(ip, 0); memset(ip->i_e2fs_blocks, 0, sizeof(ip->i_e2fs_blocks)); } /* Initialise vnode with this inode. */ vp->v_tag = VT_EXT2FS; vp->v_op = ext2fs_vnodeop_p; vp->v_data = ip; /* Initialize genfs node. */ genfs_node_init(vp, &ext2fs_genfsops); return 0; } /* * Read an inode from disk and initialize this vnode / inode pair. * Caller assures no other thread will try to load this inode. */ int ext2fs_loadvnode(struct mount *mp, struct vnode *vp, const void *key, size_t key_len, const void **new_key) { ino_t ino; struct inode *ip; struct ufsmount *ump; int error; KASSERT(key_len == sizeof(ino)); memcpy(&ino, key, key_len); ump = VFSTOUFS(mp); error = ext2fs_init_vnode(ump, vp, ino); if (error) return error; ip = VTOI(vp); /* Initialize the vnode from the inode. */ ext2fs_vinit(mp, ext2fs_specop_p, ext2fs_fifoop_p, &vp); /* Finish inode initialization. */ ip->i_devvp = ump->um_devvp; vref(ip->i_devvp); /* * Set up a generation number for this inode if it does not * already have one. This should only happen on old filesystems. */ if (ip->i_e2fs_gen == 0) { if (++ext2gennumber < (u_long)time_second) ext2gennumber = time_second; ip->i_e2fs_gen = ext2gennumber; if ((mp->mnt_flag & MNT_RDONLY) == 0) ip->i_flag |= IN_MODIFIED; } uvm_vnp_setsize(vp, ext2fs_size(ip)); *new_key = &ip->i_number; return 0; } /* * Create a new inode on disk and initialize this vnode / inode pair. */ int ext2fs_newvnode(struct mount *mp, struct vnode *dvp, struct vnode *vp, struct vattr *vap, kauth_cred_t cred, void *extra, size_t *key_len, const void **new_key) { ino_t ino; struct inode *ip, *pdir; struct m_ext2fs *fs; struct ufsmount *ump; int error, mode; KASSERT(dvp->v_mount == mp); KASSERT(vap->va_type != VNON); *key_len = sizeof(ino); pdir = VTOI(dvp); fs = pdir->i_e2fs; ump = VFSTOUFS(mp); mode = MAKEIMODE(vap->va_type, vap->va_mode); /* Allocate fresh inode. */ error = ext2fs_valloc(dvp, mode, cred, &ino); if (error) return error; /* Attach inode to vnode. */ error = ext2fs_init_vnode(ump, vp, ino); if (error) { ext2fs_vfree(dvp, ino, mode); return error; } ip = VTOI(vp); KASSERT(!E2FS_HAS_GD_CSUM(fs) || (fs->e2fs_gd[ino_to_cg(fs, ino)].ext2bgd_flags & h2fs16(E2FS_BG_INODE_ZEROED)) != 0); /* check for already used inode; makes sense only for ZEROED itable */ if (__predict_false(ip->i_e2fs_mode && ip->i_e2fs_nlink != 0)) { printf("mode = 0%o, nlinks %d, inum = %llu, fs = %s\n", ip->i_e2fs_mode, ip->i_e2fs_nlink, (unsigned long long)ip->i_number, fs->e2fs_fsmnt); panic("ext2fs_valloc: dup alloc"); } memset(ip->i_din.e2fs_din, 0, EXT2_DINODE_SIZE(fs)); /* * Set up a new generation number for this inode. */ if (++ext2gennumber < time_second) ext2gennumber = time_second; ip->i_e2fs_gen = ext2gennumber; ip->i_uid = kauth_cred_geteuid(cred); ip->i_e2fs_uid = ip->i_uid & 0xffff; ip->i_e2fs_gid = pdir->i_e2fs_gid; if (ip->i_e2fs->e2fs.e2fs_rev > E2FS_REV0) { ip->i_e2fs_uid_high = (ip->i_uid >> 16) & 0xffff; ip->i_e2fs_gid_high = pdir->i_e2fs_gid_high; } else { ip->i_e2fs_uid_high = 0; ip->i_e2fs_gid_high = 0; } ip->i_gid = ip->i_e2fs_gid | (ip->i_e2fs_gid_high << 16); ip->i_flag |= IN_ACCESS | IN_CHANGE | IN_UPDATE; ip->i_e2fs_mode = mode; vp->v_type = IFTOVT(mode); ip->i_e2fs_nlink = 1; /* Authorize setting SGID if needed. */ if (ip->i_e2fs_mode & ISGID) { error = kauth_authorize_vnode(cred, KAUTH_VNODE_WRITE_SECURITY, vp, NULL, genfs_can_chmod(vp, cred, ip->i_uid, ip->i_gid, mode)); if (error) ip->i_e2fs_mode &= ~ISGID; } /* Initialize extra_isize according to what is set in superblock */ if (EXT2F_HAS_ROCOMPAT_FEATURE(ip->i_e2fs, EXT2F_ROCOMPAT_EXTRA_ISIZE) && EXT2_DINODE_SIZE(ip->i_e2fs) > EXT2_REV0_DINODE_SIZE) { ip->i_din.e2fs_din->e2di_extra_isize = ip->i_e2fs->e2fs.e4fs_want_extra_isize; } /* Set create time if possible */ if (EXT2_DINODE_FITS(ip->i_din.e2fs_din, e2di_crtime, EXT2_DINODE_SIZE(ip->i_e2fs))) { struct timespec now; vfs_timestamp(&now); EXT2_DINODE_TIME_SET(&now, ip->i_din.e2fs_din, e2di_crtime, EXT2_DINODE_SIZE(ip->i_e2fs)); } /* Initialize the vnode from the inode. */ ext2fs_vinit(mp, ext2fs_specop_p, ext2fs_fifoop_p, &vp); /* Finish inode initialization. */ ip->i_devvp = ump->um_devvp; vref(ip->i_devvp); uvm_vnp_setsize(vp, ext2fs_size(ip)); *new_key = &ip->i_number; return 0; } /* * File handle to vnode * * Have to be really careful about stale file handles: * - check that the inode number is valid * - call ext2fs_vget() to get the locked inode * - check for an unallocated inode (i_mode == 0) */ int ext2fs_fhtovp(struct mount *mp, struct fid *fhp, int lktype, struct vnode **vpp) { struct inode *ip; struct vnode *nvp; int error; struct ufid ufh; struct m_ext2fs *fs; if (fhp->fid_len != sizeof(struct ufid)) return EINVAL; memcpy(&ufh, fhp, sizeof(struct ufid)); fs = VFSTOUFS(mp)->um_e2fs; if ((ufh.ufid_ino < EXT2_FIRSTINO && ufh.ufid_ino != EXT2_ROOTINO) || ufh.ufid_ino >= fs->e2fs_ncg * fs->e2fs.e2fs_ipg) return ESTALE; if ((error = VFS_VGET(mp, ufh.ufid_ino, lktype, &nvp)) != 0) { *vpp = NULLVP; return error; } ip = VTOI(nvp); if (ip->i_e2fs_mode == 0 || ip->i_e2fs_dtime != 0 || ip->i_e2fs_gen != ufh.ufid_gen) { vput(nvp); *vpp = NULLVP; return ESTALE; } *vpp = nvp; return 0; } /* * Vnode pointer to File handle */ /* ARGSUSED */ int ext2fs_vptofh(struct vnode *vp, struct fid *fhp, size_t *fh_size) { struct inode *ip; struct ufid ufh; if (*fh_size < sizeof(struct ufid)) { *fh_size = sizeof(struct ufid); return E2BIG; } *fh_size = sizeof(struct ufid); ip = VTOI(vp); memset(&ufh, 0, sizeof(ufh)); ufh.ufid_len = sizeof(struct ufid); ufh.ufid_ino = ip->i_number; ufh.ufid_gen = ip->i_e2fs_gen; memcpy(fhp, &ufh, sizeof(ufh)); return 0; } /* * Write a superblock and associated information back to disk. */ int ext2fs_sbupdate(struct ufsmount *mp, int waitfor) { struct m_ext2fs *fs = mp->um_e2fs; struct buf *bp; int error = 0; bp = getblk(mp->um_devvp, SBLOCK, SBSIZE, 0, 0); e2fs_sbsave(&fs->e2fs, (struct ext2fs*)bp->b_data); if (waitfor == MNT_WAIT) error = bwrite(bp); else bawrite(bp); return error; } int ext2fs_cgupdate(struct ufsmount *mp, int waitfor) { struct m_ext2fs *fs = mp->um_e2fs; struct buf *bp; int i, error = 0, allerror = 0; allerror = ext2fs_sbupdate(mp, waitfor); for (i = 0; i < fs->e2fs_ngdb; i++) { bp = getblk(mp->um_devvp, EXT2_FSBTODB(fs, fs->e2fs.e2fs_first_dblock + 1 /* superblock */ + i), fs->e2fs_bsize, 0, 0); e2fs_cgsave(&fs->e2fs_gd[ i * fs->e2fs_bsize / sizeof(struct ext2_gd)], (struct ext2_gd *)bp->b_data, fs->e2fs_bsize); if (waitfor == MNT_WAIT) error = bwrite(bp); else bawrite(bp); } if (!allerror && error) allerror = error; return allerror; } /* * Fill in the m_fs structure, and validate the fields of the superblock. * NOTE: here, the superblock is already swapped. */ static int ext2fs_sbfill(struct m_ext2fs *m_fs, int ronly) { uint32_t u32; struct ext2fs *fs = &m_fs->e2fs; /* * General sanity checks */ if (fs->e2fs_magic != E2FS_MAGIC) return EINVAL; if (fs->e2fs_rev > E2FS_REV1) { printf("ext2fs: unsupported revision number: %x\n", fs->e2fs_rev); return EINVAL; } if (fs->e2fs_log_bsize > 2) { /* block size = 1024|2048|4096 */ printf("ext2fs: bad block size: %d\n", fs->e2fs_log_bsize); return EINVAL; } if (fs->e2fs_bpg == 0) { printf("ext2fs: zero blocks per group\n"); return EINVAL; } if (fs->e2fs_ipg == 0) { printf("ext2fs: zero inodes per group\n"); return EINVAL; } if (fs->e2fs_first_dblock >= fs->e2fs_bcount) { printf("ext2fs: invalid first data block\n"); return EINVAL; } if (fs->e2fs_rbcount > fs->e2fs_bcount || fs->e2fs_fbcount > fs->e2fs_bcount) { printf("ext2fs: invalid block count\n"); return EINVAL; } /* * Compute the fields of the superblock */ u32 = fs->e2fs_bcount - fs->e2fs_first_dblock; /* > 0 */ m_fs->e2fs_ncg = howmany(u32, fs->e2fs_bpg); if (m_fs->e2fs_ncg == 0) { printf("ext2fs: invalid number of cylinder groups\n"); return EINVAL; } m_fs->e2fs_fsbtodb = fs->e2fs_log_bsize + LOG_MINBSIZE - DEV_BSHIFT; m_fs->e2fs_bsize = MINBSIZE << fs->e2fs_log_bsize; m_fs->e2fs_bshift = LOG_MINBSIZE + fs->e2fs_log_bsize; m_fs->e2fs_qbmask = m_fs->e2fs_bsize - 1; m_fs->e2fs_bmask = ~m_fs->e2fs_qbmask; if ((u32 = m_fs->e2fs_bsize / sizeof(struct ext2_gd)) == 0) { /* Unlikely to happen */ printf("ext2fs: invalid block size\n"); return EINVAL; } m_fs->e2fs_ngdb = howmany(m_fs->e2fs_ncg, u32); if (m_fs->e2fs_ngdb == 0) { printf("ext2fs: invalid number of group descriptor blocks\n"); return EINVAL; } if (m_fs->e2fs_bsize < EXT2_DINODE_SIZE(m_fs)) { printf("ext2fs: invalid inode size\n"); return EINVAL; } m_fs->e2fs_ipb = m_fs->e2fs_bsize / EXT2_DINODE_SIZE(m_fs); m_fs->e2fs_itpg = fs->e2fs_ipg / m_fs->e2fs_ipb; /* * Revision-specific checks */ if (fs->e2fs_rev > E2FS_REV0) { char buf[256]; if (fs->e2fs_first_ino != EXT2_FIRSTINO) { printf("ext2fs: unsupported first inode position\n"); return EINVAL; } u32 = fs->e2fs_features_incompat & ~EXT2F_INCOMPAT_SUPP; if (u32) { snprintb(buf, sizeof(buf), EXT2F_INCOMPAT_BITS, u32); printf("ext2fs: unsupported incompat features: %s\n", buf); #ifndef EXT2_IGNORE_INCOMPAT_FEATURES return EINVAL; #endif } u32 = fs->e2fs_features_rocompat & ~EXT2F_ROCOMPAT_SUPP; if (!ronly && u32) { snprintb(buf, sizeof(buf), EXT2F_ROCOMPAT_BITS, u32); printf("ext2fs: unsupported ro-incompat features: %s\n", buf); #ifndef EXT2_IGNORE_ROCOMPAT_FEATURES return EROFS; #endif } if (fs->e2fs_inode_size == 0 || !powerof2(fs->e2fs_inode_size) || fs->e2fs_inode_size > m_fs->e2fs_bsize) { printf("ext2fs: bad inode size\n"); return EINVAL; } } return 0; } |
| 479 28 725 23 712 514 14 504 8 1 1 1 3 2 23 23 1 23 23 23 12 2 11 2 2 2 2 12 12 30 29 1 1 1 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 197 198 199 200 201 202 203 204 205 206 207 208 209 210 211 212 213 214 215 216 217 218 219 220 221 222 223 224 225 226 227 228 229 230 231 232 233 234 235 236 237 238 239 240 241 242 243 244 245 246 247 248 249 250 251 252 253 254 255 256 257 258 259 260 261 262 263 264 265 266 267 268 269 270 271 272 273 274 275 276 277 278 279 280 281 282 283 284 285 286 287 288 289 290 291 292 293 294 295 296 297 298 299 300 301 302 303 304 305 306 307 308 309 310 311 312 313 314 315 316 317 318 319 320 321 322 323 324 325 326 327 328 329 330 331 332 333 334 335 336 337 338 339 340 341 342 343 344 345 346 347 348 349 350 351 352 353 354 355 356 357 358 359 360 361 362 363 364 365 366 367 368 369 370 371 372 373 374 375 376 377 378 379 380 381 382 383 384 385 386 387 388 389 390 391 392 393 394 395 396 397 398 399 400 401 402 403 404 405 406 407 408 409 410 411 412 413 414 415 416 417 418 419 420 421 422 423 424 425 426 427 428 429 430 431 432 433 434 435 436 437 438 439 440 441 442 443 444 445 446 447 448 449 450 451 452 453 454 455 456 457 458 459 460 461 462 463 464 465 466 467 468 469 470 471 472 473 474 475 476 477 478 479 480 481 482 483 484 485 486 487 488 489 490 491 492 493 494 495 496 497 498 499 500 501 502 503 504 505 506 507 508 509 510 511 512 513 514 515 516 517 518 519 520 521 522 523 524 525 526 527 528 529 530 531 532 533 534 535 536 537 538 539 540 541 542 543 544 545 546 547 548 549 550 551 552 553 554 555 556 557 558 559 560 561 562 563 564 565 566 567 568 569 570 571 572 573 574 575 576 577 578 579 580 581 582 583 584 585 586 587 588 589 590 591 592 593 594 595 596 597 598 599 600 601 602 603 604 605 606 607 608 609 610 611 612 613 614 615 616 617 618 619 620 621 622 623 624 625 626 627 628 629 630 631 632 633 634 635 636 637 638 639 640 641 642 643 644 645 646 647 648 649 650 651 652 653 654 655 656 657 658 659 660 661 662 663 664 665 666 667 668 669 670 671 672 673 674 675 676 677 678 679 680 681 682 683 684 685 686 687 688 689 690 691 692 693 694 695 696 697 698 699 700 701 702 703 704 705 706 707 708 709 710 711 712 713 714 715 716 717 718 719 720 721 722 723 724 725 726 727 728 729 730 731 732 733 734 735 736 737 738 739 740 741 742 743 744 745 746 747 748 749 750 751 752 753 754 755 756 757 758 759 760 761 762 763 764 765 766 767 768 769 770 771 772 773 774 775 776 777 778 779 780 781 782 783 784 785 786 787 788 789 790 791 792 793 794 795 796 797 798 799 800 801 802 803 804 805 806 807 808 809 810 811 812 813 814 815 816 817 818 819 820 821 822 823 824 825 826 827 828 829 830 831 832 833 834 835 836 837 838 839 840 841 842 843 844 845 846 847 848 849 850 851 852 853 854 855 856 857 858 859 860 861 862 863 864 865 866 867 868 869 870 871 872 873 874 875 876 877 878 879 880 881 882 883 884 885 886 887 888 889 890 891 892 893 894 895 896 897 898 899 900 901 902 903 904 905 906 907 908 909 910 911 912 913 914 915 916 917 918 919 920 921 922 923 924 925 926 927 928 929 930 931 932 933 934 935 936 937 938 939 940 941 942 943 944 945 946 947 948 949 950 951 952 953 954 955 956 957 958 959 960 961 962 963 964 965 966 967 968 969 970 971 972 973 974 975 976 977 978 979 980 981 | /* $NetBSD: ufs_quota.c,v 1.117 2014/06/28 22:27:51 dholland Exp $ */ /* * Copyright (c) 1982, 1986, 1990, 1993, 1995 * The Regents of the University of California. All rights reserved. * * This code is derived from software contributed to Berkeley by * Robert Elz at The University of Melbourne. * * Redistribution and use in source and binary forms, with or without * modification, are permitted provided that the following conditions * are met: * 1. Redistributions of source code must retain the above copyright * notice, this list of conditions and the following disclaimer. * 2. Redistributions in binary form must reproduce the above copyright * notice, this list of conditions and the following disclaimer in the * documentation and/or other materials provided with the distribution. * 3. Neither the name of the University nor the names of its contributors * may be used to endorse or promote products derived from this software * without specific prior written permission. * * THIS SOFTWARE IS PROVIDED BY THE REGENTS AND CONTRIBUTORS ``AS IS'' AND * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE * ARE DISCLAIMED. IN NO EVENT SHALL THE REGENTS OR CONTRIBUTORS BE LIABLE * FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL * DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS * OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) * HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT * LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY * OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF * SUCH DAMAGE. * * @(#)ufs_quota.c 8.5 (Berkeley) 5/20/95 */ #include <sys/cdefs.h> __KERNEL_RCSID(0, "$NetBSD: ufs_quota.c,v 1.117 2014/06/28 22:27:51 dholland Exp $"); #if defined(_KERNEL_OPT) #include "opt_quota.h" #endif #include <sys/param.h> #include <sys/kernel.h> #include <sys/systm.h> #include <sys/namei.h> #include <sys/file.h> #include <sys/proc.h> #include <sys/vnode.h> #include <sys/mount.h> #include <sys/kauth.h> #include <sys/quotactl.h> #include <ufs/ufs/quota.h> #include <ufs/ufs/inode.h> #include <ufs/ufs/ufsmount.h> #include <ufs/ufs/ufs_extern.h> #include <ufs/ufs/ufs_quota.h> kmutex_t dqlock; kcondvar_t dqcv; const char *quotatypes[MAXQUOTAS] = INITQFNAMES; /* * Code pertaining to management of the in-core dquot data structures. */ #define DQHASH(dqvp, id) \ (((((long)(dqvp)) >> 8) + id) & dqhash) static LIST_HEAD(dqhashhead, dquot) *dqhashtbl; static u_long dqhash; static pool_cache_t dquot_cache; static int quota_handle_cmd_stat(struct mount *, struct lwp *, struct quotactl_args *args); static int quota_handle_cmd_idtypestat(struct mount *, struct lwp *, struct quotactl_args *args); static int quota_handle_cmd_objtypestat(struct mount *, struct lwp *, struct quotactl_args *args); static int quota_handle_cmd_get(struct mount *, struct lwp *, struct quotactl_args *args); static int quota_handle_cmd_put(struct mount *, struct lwp *, struct quotactl_args *args); static int quota_handle_cmd_cursorget(struct mount *, struct lwp *, struct quotactl_args *args); static int quota_handle_cmd_del(struct mount *, struct lwp *, struct quotactl_args *args); static int quota_handle_cmd_quotaon(struct mount *, struct lwp *, struct quotactl_args *args); static int quota_handle_cmd_quotaoff(struct mount *, struct lwp *, struct quotactl_args *args); static int quota_handle_cmd_cursoropen(struct mount *, struct lwp *, struct quotactl_args *args); static int quota_handle_cmd_cursorclose(struct mount *, struct lwp *, struct quotactl_args *args); static int quota_handle_cmd_cursorskipidtype(struct mount *, struct lwp *, struct quotactl_args *args); static int quota_handle_cmd_cursoratend(struct mount *, struct lwp *, struct quotactl_args *args); static int quota_handle_cmd_cursorrewind(struct mount *, struct lwp *, struct quotactl_args *args); /* * Initialize the quota fields of an inode. */ void ufsquota_init(struct inode *ip) { int i; for (i = 0; i < MAXQUOTAS; i++) ip->i_dquot[i] = NODQUOT; } /* * Release the quota fields from an inode. */ void ufsquota_free(struct inode *ip) { int i; for (i = 0; i < MAXQUOTAS; i++) { dqrele(ITOV(ip), ip->i_dquot[i]); ip->i_dquot[i] = NODQUOT; } } /* * Update disk usage, and take corrective action. */ int chkdq(struct inode *ip, int64_t change, kauth_cred_t cred, int flags) { /* do not track snapshot usage, or we will deadlock */ if ((ip->i_flags & SF_SNAPSHOT) != 0) return 0; #ifdef QUOTA if (ip->i_ump->um_flags & UFS_QUOTA) return chkdq1(ip, change, cred, flags); #endif #ifdef QUOTA2 if (ip->i_ump->um_flags & UFS_QUOTA2) return chkdq2(ip, change, cred, flags); #endif return 0; } /* * Check the inode limit, applying corrective action. */ int chkiq(struct inode *ip, int32_t change, kauth_cred_t cred, int flags) { /* do not track snapshot usage, or we will deadlock */ if ((ip->i_flags & SF_SNAPSHOT) != 0) return 0; #ifdef QUOTA if (ip->i_ump->um_flags & UFS_QUOTA) return chkiq1(ip, change, cred, flags); #endif #ifdef QUOTA2 if (ip->i_ump->um_flags & UFS_QUOTA2) return chkiq2(ip, change, cred, flags); #endif return 0; } int quota_handle_cmd(struct mount *mp, struct lwp *l, struct quotactl_args *args) { int error = 0; switch (args->qc_op) { case QUOTACTL_STAT: error = quota_handle_cmd_stat(mp, l, args); break; case QUOTACTL_IDTYPESTAT: error = quota_handle_cmd_idtypestat(mp, l, args); break; case QUOTACTL_OBJTYPESTAT: error = quota_handle_cmd_objtypestat(mp, l, args); break; case QUOTACTL_QUOTAON: error = quota_handle_cmd_quotaon(mp, l, args); break; case QUOTACTL_QUOTAOFF: error = quota_handle_cmd_quotaoff(mp, l, args); break; case QUOTACTL_GET: error = quota_handle_cmd_get(mp, l, args); break; case QUOTACTL_PUT: error = quota_handle_cmd_put(mp, l, args); break; case QUOTACTL_CURSORGET: error = quota_handle_cmd_cursorget(mp, l, args); break; case QUOTACTL_DEL: error = quota_handle_cmd_del(mp, l, args); break; case QUOTACTL_CURSOROPEN: error = quota_handle_cmd_cursoropen(mp, l, args); break; case QUOTACTL_CURSORCLOSE: error = quota_handle_cmd_cursorclose(mp, l, args); break; case QUOTACTL_CURSORSKIPIDTYPE: error = quota_handle_cmd_cursorskipidtype(mp, l, args); break; case QUOTACTL_CURSORATEND: error = quota_handle_cmd_cursoratend(mp, l, args); break; case QUOTACTL_CURSORREWIND: error = quota_handle_cmd_cursorrewind(mp, l, args); break; default: panic("Invalid quotactl operation %d\n", args->qc_op); } return error; } static int quota_handle_cmd_stat(struct mount *mp, struct lwp *l, struct quotactl_args *args) { struct ufsmount *ump = VFSTOUFS(mp); struct quotastat *info; KASSERT(args->qc_op == QUOTACTL_STAT); info = args->u.stat.qc_info; if ((ump->um_flags & (UFS_QUOTA|UFS_QUOTA2)) == 0) return EOPNOTSUPP; #ifdef QUOTA if (ump->um_flags & UFS_QUOTA) { strcpy(info->qs_implname, "ufs/ffs quota v1"); info->qs_numidtypes = MAXQUOTAS; /* XXX no define for this */ info->qs_numobjtypes = 2; info->qs_restrictions = 0; info->qs_restrictions |= QUOTA_RESTRICT_NEEDSQUOTACHECK; info->qs_restrictions |= QUOTA_RESTRICT_UNIFORMGRACE; info->qs_restrictions |= QUOTA_RESTRICT_32BIT; } else #endif #ifdef QUOTA2 if (ump->um_flags & UFS_QUOTA2) { strcpy(info->qs_implname, "ufs/ffs quota v2"); info->qs_numidtypes = MAXQUOTAS; info->qs_numobjtypes = N_QL; info->qs_restrictions = 0; } else #endif return EOPNOTSUPP; return 0; } static int quota_handle_cmd_idtypestat(struct mount *mp, struct lwp *l, struct quotactl_args *args) { struct ufsmount *ump = VFSTOUFS(mp); int idtype; struct quotaidtypestat *info; const char *name; KASSERT(args->qc_op == QUOTACTL_IDTYPESTAT); idtype = args->u.idtypestat.qc_idtype; info = args->u.idtypestat.qc_info; if ((ump->um_flags & (UFS_QUOTA|UFS_QUOTA2)) == 0) return EOPNOTSUPP; /* * These are the same for both QUOTA and QUOTA2. */ switch (idtype) { case QUOTA_IDTYPE_USER: name = "user"; break; case QUOTA_IDTYPE_GROUP: name = "group"; break; default: return EINVAL; } strlcpy(info->qis_name, name, sizeof(info->qis_name)); return 0; } static int quota_handle_cmd_objtypestat(struct mount *mp, struct lwp *l, struct quotactl_args *args) { struct ufsmount *ump = VFSTOUFS(mp); int objtype; struct quotaobjtypestat *info; const char *name; int isbytes; KASSERT(args->qc_op == QUOTACTL_OBJTYPESTAT); objtype = args->u.objtypestat.qc_objtype; info = args->u.objtypestat.qc_info; if ((ump->um_flags & (UFS_QUOTA|UFS_QUOTA2)) == 0) return EOPNOTSUPP; /* * These are the same for both QUOTA and QUOTA2. */ switch (objtype) { case QUOTA_OBJTYPE_BLOCKS: name = "block"; isbytes = 1; break; case QUOTA_OBJTYPE_FILES: name = "file"; isbytes = 0; break; default: return EINVAL; } strlcpy(info->qos_name, name, sizeof(info->qos_name)); info->qos_isbytes = isbytes; return 0; } /* XXX shouldn't all this be in kauth ? */ static int quota_get_auth(struct mount *mp, struct lwp *l, uid_t id) { /* The user can always query about his own quota. */ if (id == kauth_cred_geteuid(l->l_cred)) return 0; return kauth_authorize_system(l->l_cred, KAUTH_SYSTEM_FS_QUOTA, KAUTH_REQ_SYSTEM_FS_QUOTA_GET, mp, KAUTH_ARG(id), NULL); } static int quota_handle_cmd_get(struct mount *mp, struct lwp *l, struct quotactl_args *args) { struct ufsmount *ump = VFSTOUFS(mp); int error; const struct quotakey *qk; struct quotaval *qv; KASSERT(args->qc_op == QUOTACTL_GET); qk = args->u.get.qc_key; qv = args->u.get.qc_val; if ((ump->um_flags & (UFS_QUOTA|UFS_QUOTA2)) == 0) return EOPNOTSUPP; error = quota_get_auth(mp, l, qk->qk_id); if (error != 0) return error; #ifdef QUOTA if (ump->um_flags & UFS_QUOTA) { error = quota1_handle_cmd_get(ump, qk, qv); } else #endif #ifdef QUOTA2 if (ump->um_flags & UFS_QUOTA2) { error = quota2_handle_cmd_get(ump, qk, qv); } else #endif panic("quota_handle_cmd_get: no support ?"); if (error != 0) return error; return error; } static int quota_handle_cmd_put(struct mount *mp, struct lwp *l, struct quotactl_args *args) { struct ufsmount *ump = VFSTOUFS(mp); const struct quotakey *qk; const struct quotaval *qv; id_t kauth_id; int error; KASSERT(args->qc_op == QUOTACTL_PUT); qk = args->u.put.qc_key; qv = args->u.put.qc_val; if ((ump->um_flags & (UFS_QUOTA|UFS_QUOTA2)) == 0) return EOPNOTSUPP; kauth_id = qk->qk_id; if (kauth_id == QUOTA_DEFAULTID) { kauth_id = 0; } error = kauth_authorize_system(l->l_cred, KAUTH_SYSTEM_FS_QUOTA, KAUTH_REQ_SYSTEM_FS_QUOTA_MANAGE, mp, KAUTH_ARG(kauth_id), NULL); if (error != 0) { return error; } #ifdef QUOTA if (ump->um_flags & UFS_QUOTA) error = quota1_handle_cmd_put(ump, qk, qv); else #endif #ifdef QUOTA2 if (ump->um_flags & UFS_QUOTA2) { error = quota2_handle_cmd_put(ump, qk, qv); } else #endif panic("quota_handle_cmd_get: no support ?"); if (error == ENOENT) { error = 0; } return error; } static int quota_handle_cmd_del(struct mount *mp, struct lwp *l, struct quotactl_args *args) { struct ufsmount *ump = VFSTOUFS(mp); const struct quotakey *qk; id_t kauth_id; int error; KASSERT(args->qc_op == QUOTACTL_DEL); qk = args->u.del.qc_key; kauth_id = qk->qk_id; if (kauth_id == QUOTA_DEFAULTID) { kauth_id = 0; } if ((ump->um_flags & UFS_QUOTA2) == 0) return EOPNOTSUPP; /* avoid whitespace changes */ { error = kauth_authorize_system(l->l_cred, KAUTH_SYSTEM_FS_QUOTA, KAUTH_REQ_SYSTEM_FS_QUOTA_MANAGE, mp, KAUTH_ARG(kauth_id), NULL); if (error != 0) goto err; #ifdef QUOTA2 if (ump->um_flags & UFS_QUOTA2) { error = quota2_handle_cmd_del(ump, qk); } else #endif panic("quota_handle_cmd_get: no support ?"); if (error && error != ENOENT) goto err; } return 0; err: return error; } static int quota_handle_cmd_cursorget(struct mount *mp, struct lwp *l, struct quotactl_args *args) { struct ufsmount *ump = VFSTOUFS(mp); int error; KASSERT(args->qc_op == QUOTACTL_CURSORGET); if ((ump->um_flags & UFS_QUOTA2) == 0) return EOPNOTSUPP; error = kauth_authorize_system(l->l_cred, KAUTH_SYSTEM_FS_QUOTA, KAUTH_REQ_SYSTEM_FS_QUOTA_GET, mp, NULL, NULL); if (error) return error; #ifdef QUOTA2 if (ump->um_flags & UFS_QUOTA2) { struct quotakcursor *cursor = args->u.cursorget.qc_cursor; struct quotakey *keys = args->u.cursorget.qc_keys; struct quotaval *vals = args->u.cursorget.qc_vals; unsigned maxnum = args->u.cursorget.qc_maxnum; unsigned *ret = args->u.cursorget.qc_ret; error = quota2_handle_cmd_cursorget(ump, cursor, keys, vals, maxnum, ret); } else #endif panic("quota_handle_cmd_cursorget: no support ?"); return error; } static int quota_handle_cmd_cursoropen(struct mount *mp, struct lwp *l, struct quotactl_args *args) { #ifdef QUOTA2 struct ufsmount *ump = VFSTOUFS(mp); struct quotakcursor *cursor = args->u.cursoropen.qc_cursor; #endif int error; KASSERT(args->qc_op == QUOTACTL_CURSOROPEN); error = kauth_authorize_system(l->l_cred, KAUTH_SYSTEM_FS_QUOTA, KAUTH_REQ_SYSTEM_FS_QUOTA_GET, mp, NULL, NULL); if (error) return error; #ifdef QUOTA2 if (ump->um_flags & UFS_QUOTA2) { error = quota2_handle_cmd_cursoropen(ump, cursor); } else #endif error = EOPNOTSUPP; return error; } static int quota_handle_cmd_cursorclose(struct mount *mp, struct lwp *l, struct quotactl_args *args) { #ifdef QUOTA2 struct ufsmount *ump = VFSTOUFS(mp); struct quotakcursor *cursor = args->u.cursorclose.qc_cursor; #endif int error; KASSERT(args->qc_op == QUOTACTL_CURSORCLOSE); error = kauth_authorize_system(l->l_cred, KAUTH_SYSTEM_FS_QUOTA, KAUTH_REQ_SYSTEM_FS_QUOTA_GET, mp, NULL, NULL); if (error) return error; #ifdef QUOTA2 if (ump->um_flags & UFS_QUOTA2) { error = quota2_handle_cmd_cursorclose(ump, cursor); } else #endif error = EOPNOTSUPP; return error; } static int quota_handle_cmd_cursorskipidtype(struct mount *mp, struct lwp *l, struct quotactl_args *args) { #ifdef QUOTA2 struct ufsmount *ump = VFSTOUFS(mp); struct quotakcursor *cursor = args->u.cursorskipidtype.qc_cursor; int idtype = args->u.cursorskipidtype.qc_idtype; #endif int error; KASSERT(args->qc_op == QUOTACTL_CURSORSKIPIDTYPE); #ifdef QUOTA2 if (ump->um_flags & UFS_QUOTA2) { error = quota2_handle_cmd_cursorskipidtype(ump, cursor, idtype); } else #endif error = EOPNOTSUPP; return error; } static int quota_handle_cmd_cursoratend(struct mount *mp, struct lwp *l, struct quotactl_args *args) { #ifdef QUOTA2 struct ufsmount *ump = VFSTOUFS(mp); struct quotakcursor *cursor = args->u.cursoratend.qc_cursor; unsigned *ret = args->u.cursoratend.qc_ret; #endif int error; KASSERT(args->qc_op == QUOTACTL_CURSORATEND); #ifdef QUOTA2 if (ump->um_flags & UFS_QUOTA2) { error = quota2_handle_cmd_cursoratend(ump, cursor, ret); } else #endif error = EOPNOTSUPP; return error; } static int quota_handle_cmd_cursorrewind(struct mount *mp, struct lwp *l, struct quotactl_args *args) { #ifdef QUOTA2 struct ufsmount *ump = VFSTOUFS(mp); struct quotakcursor *cursor = args->u.cursorrewind.qc_cursor; #endif int error; KASSERT(args->qc_op == QUOTACTL_CURSORREWIND); #ifdef QUOTA2 if (ump->um_flags & UFS_QUOTA2) { error = quota2_handle_cmd_cursorrewind(ump, cursor); } else #endif error = EOPNOTSUPP; return error; } static int quota_handle_cmd_quotaon(struct mount *mp, struct lwp *l, struct quotactl_args *args) { struct ufsmount *ump = VFSTOUFS(mp); int error; KASSERT(args->qc_op == QUOTACTL_QUOTAON); if ((ump->um_flags & UFS_QUOTA2) != 0) return EBUSY; error = kauth_authorize_system(l->l_cred, KAUTH_SYSTEM_FS_QUOTA, KAUTH_REQ_SYSTEM_FS_QUOTA_ONOFF, mp, NULL, NULL); if (error != 0) { return error; } #ifdef QUOTA int idtype = args->u.quotaon.qc_idtype; const char *qfile = args->u.quotaon.qc_quotafile; error = quota1_handle_cmd_quotaon(l, ump, idtype, qfile); #else error = EOPNOTSUPP; #endif return error; } static int quota_handle_cmd_quotaoff(struct mount *mp, struct lwp *l, struct quotactl_args *args) { struct ufsmount *ump = VFSTOUFS(mp); int error; KASSERT(args->qc_op == QUOTACTL_QUOTAOFF); if ((ump->um_flags & UFS_QUOTA2) != 0) return EOPNOTSUPP; error = kauth_authorize_system(l->l_cred, KAUTH_SYSTEM_FS_QUOTA, KAUTH_REQ_SYSTEM_FS_QUOTA_ONOFF, mp, NULL, NULL); if (error != 0) { return error; } #ifdef QUOTA int idtype = args->u.quotaoff.qc_idtype; error = quota1_handle_cmd_quotaoff(l, ump, idtype); #else error = EOPNOTSUPP; #endif return error; } /* * Initialize the quota system. */ void dqinit(void) { mutex_init(&dqlock, MUTEX_DEFAULT, IPL_NONE); cv_init(&dqcv, "quota"); dqhashtbl = hashinit(desiredvnodes, HASH_LIST, true, &dqhash); dquot_cache = pool_cache_init(sizeof(struct dquot), 0, 0, 0, "ufsdq", NULL, IPL_NONE, NULL, NULL, NULL); } void dqreinit(void) { struct dquot *dq; struct dqhashhead *oldhash, *hash; struct vnode *dqvp; u_long oldmask, mask, hashval; int i; hash = hashinit(desiredvnodes, HASH_LIST, true, &mask); mutex_enter(&dqlock); oldhash = dqhashtbl; oldmask = dqhash; dqhashtbl = hash; dqhash = mask; for (i = 0; i <= oldmask; i++) { while ((dq = LIST_FIRST(&oldhash[i])) != NULL) { dqvp = dq->dq_ump->um_quotas[dq->dq_type]; LIST_REMOVE(dq, dq_hash); hashval = DQHASH(dqvp, dq->dq_id); LIST_INSERT_HEAD(&dqhashtbl[hashval], dq, dq_hash); } } mutex_exit(&dqlock); hashdone(oldhash, HASH_LIST, oldmask); } /* * Free resources held by quota system. */ void dqdone(void) { pool_cache_destroy(dquot_cache); hashdone(dqhashtbl, HASH_LIST, dqhash); cv_destroy(&dqcv); mutex_destroy(&dqlock); } /* * Set up the quotas for an inode. * * This routine completely defines the semantics of quotas. * If other criterion want to be used to establish quotas, the * MAXQUOTAS value in quotas.h should be increased, and the * additional dquots set up here. */ int getinoquota(struct inode *ip) { struct ufsmount *ump = ip->i_ump; struct vnode *vp = ITOV(ip); int i, error; u_int32_t ino_ids[MAXQUOTAS]; /* * To avoid deadlocks never update quotas for quota files * on the same file system */ for (i = 0; i < MAXQUOTAS; i++) if (vp == ump->um_quotas[i]) return 0; ino_ids[USRQUOTA] = ip->i_uid; ino_ids[GRPQUOTA] = ip->i_gid; for (i = 0; i < MAXQUOTAS; i++) { /* * If the file id changed the quota needs update. */ if (ip->i_dquot[i] != NODQUOT && ip->i_dquot[i]->dq_id != ino_ids[i]) { dqrele(ITOV(ip), ip->i_dquot[i]); ip->i_dquot[i] = NODQUOT; } /* * Set up the quota based on file id. * ENODEV means that quotas are not enabled. */ if (ip->i_dquot[i] == NODQUOT && (error = dqget(vp, ino_ids[i], ump, i, &ip->i_dquot[i])) && error != ENODEV) return (error); } return 0; } /* * Obtain a dquot structure for the specified identifier and quota file * reading the information from the file if necessary. */ int dqget(struct vnode *vp, u_long id, struct ufsmount *ump, int type, struct dquot **dqp) { struct dquot *dq, *ndq; struct dqhashhead *dqh; struct vnode *dqvp; int error = 0; /* XXX gcc */ /* Lock to see an up to date value for QTF_CLOSING. */ mutex_enter(&dqlock); if ((ump->um_flags & (UFS_QUOTA|UFS_QUOTA2)) == 0) { mutex_exit(&dqlock); *dqp = NODQUOT; return (ENODEV); } dqvp = ump->um_quotas[type]; #ifdef QUOTA if (ump->um_flags & UFS_QUOTA) { if (dqvp == NULLVP || (ump->umq1_qflags[type] & QTF_CLOSING)) { mutex_exit(&dqlock); *dqp = NODQUOT; return (ENODEV); } } #endif #ifdef QUOTA2 if (ump->um_flags & UFS_QUOTA2) { if (dqvp == NULLVP) { mutex_exit(&dqlock); *dqp = NODQUOT; return (ENODEV); } } #endif KASSERT(dqvp != vp); /* * Check the cache first. */ dqh = &dqhashtbl[DQHASH(dqvp, id)]; LIST_FOREACH(dq, dqh, dq_hash) { if (dq->dq_id != id || dq->dq_ump->um_quotas[dq->dq_type] != dqvp) continue; KASSERT(dq->dq_cnt > 0); dqref(dq); mutex_exit(&dqlock); *dqp = dq; return (0); } /* * Not in cache, allocate a new one. */ mutex_exit(&dqlock); ndq = pool_cache_get(dquot_cache, PR_WAITOK); /* * Initialize the contents of the dquot structure. */ memset((char *)ndq, 0, sizeof *ndq); ndq->dq_flags = 0; ndq->dq_id = id; ndq->dq_ump = ump; ndq->dq_type = type; mutex_init(&ndq->dq_interlock, MUTEX_DEFAULT, IPL_NONE); mutex_enter(&dqlock); dqh = &dqhashtbl[DQHASH(dqvp, id)]; LIST_FOREACH(dq, dqh, dq_hash) { if (dq->dq_id != id || dq->dq_ump->um_quotas[dq->dq_type] != dqvp) continue; /* * Another thread beat us allocating this dquot. */ KASSERT(dq->dq_cnt > 0); dqref(dq); mutex_exit(&dqlock); mutex_destroy(&ndq->dq_interlock); pool_cache_put(dquot_cache, ndq); *dqp = dq; return 0; } dq = ndq; LIST_INSERT_HEAD(dqh, dq, dq_hash); dqref(dq); mutex_enter(&dq->dq_interlock); mutex_exit(&dqlock); #ifdef QUOTA if (ump->um_flags & UFS_QUOTA) error = dq1get(dqvp, id, ump, type, dq); #endif #ifdef QUOTA2 if (ump->um_flags & UFS_QUOTA2) error = dq2get(dqvp, id, ump, type, dq); #endif /* * I/O error in reading quota file, release * quota structure and reflect problem to caller. */ if (error) { mutex_enter(&dqlock); LIST_REMOVE(dq, dq_hash); mutex_exit(&dqlock); mutex_exit(&dq->dq_interlock); dqrele(vp, dq); *dqp = NODQUOT; return (error); } mutex_exit(&dq->dq_interlock); *dqp = dq; return (0); } /* * Obtain a reference to a dquot. */ void dqref(struct dquot *dq) { KASSERT(mutex_owned(&dqlock)); dq->dq_cnt++; KASSERT(dq->dq_cnt > 0); } /* * Release a reference to a dquot. */ void dqrele(struct vnode *vp, struct dquot *dq) { if (dq == NODQUOT) return; mutex_enter(&dq->dq_interlock); for (;;) { mutex_enter(&dqlock); if (dq->dq_cnt > 1) { dq->dq_cnt--; mutex_exit(&dqlock); mutex_exit(&dq->dq_interlock); return; } if ((dq->dq_flags & DQ_MOD) == 0) break; mutex_exit(&dqlock); #ifdef QUOTA if (dq->dq_ump->um_flags & UFS_QUOTA) (void) dq1sync(vp, dq); #endif #ifdef QUOTA2 if (dq->dq_ump->um_flags & UFS_QUOTA2) (void) dq2sync(vp, dq); #endif } KASSERT(dq->dq_cnt == 1 && (dq->dq_flags & DQ_MOD) == 0); LIST_REMOVE(dq, dq_hash); mutex_exit(&dqlock); mutex_exit(&dq->dq_interlock); mutex_destroy(&dq->dq_interlock); pool_cache_put(dquot_cache, dq); } int qsync(struct mount *mp) { struct ufsmount *ump = VFSTOUFS(mp); #ifdef QUOTA if (ump->um_flags & UFS_QUOTA) return q1sync(mp); #endif #ifdef QUOTA2 if (ump->um_flags & UFS_QUOTA2) return q2sync(mp); #endif return 0; } #ifdef DIAGNOSTIC /* * Check the hash chains for stray dquot's. */ void dqflush(struct vnode *vp) { struct dquot *dq; int i; mutex_enter(&dqlock); for (i = 0; i <= dqhash; i++) LIST_FOREACH(dq, &dqhashtbl[i], dq_hash) KASSERT(dq->dq_ump->um_quotas[dq->dq_type] != vp); mutex_exit(&dqlock); } #endif |
| 3 3 217 1 8 1 1 2 1 1 2 213 213 213 213 206 206 213 213 206 213 213 206 213 1 213 210 1 208 2 208 208 207 207 208 201 3 206 206 206 206 206 206 206 1 1 2 1 1 1 3 1 2 2 2 1 1 2 2 1 2 2 206 206 206 206 206 205 206 206 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 197 198 199 200 201 202 203 204 205 206 207 208 209 210 211 212 213 214 215 216 217 218 219 220 221 222 223 224 225 226 227 228 229 230 231 232 233 234 235 236 237 238 239 240 241 242 243 244 245 246 247 248 249 250 251 252 253 254 255 256 257 258 259 260 261 262 263 264 265 266 267 268 269 270 271 272 273 274 275 276 277 278 279 280 281 282 283 284 285 286 287 288 289 290 291 292 293 294 295 296 297 298 299 300 301 302 303 304 305 306 307 308 309 310 311 312 313 314 315 316 317 318 319 320 321 322 323 324 325 326 327 328 329 330 331 332 333 334 335 336 337 338 339 340 341 342 343 344 345 346 347 348 349 350 351 352 353 354 355 356 357 358 359 360 361 362 363 364 365 366 367 368 369 370 371 372 373 374 375 376 377 378 379 380 381 382 383 384 385 386 387 388 389 390 391 392 393 394 395 396 397 398 399 400 401 402 403 404 405 406 407 408 409 410 411 412 413 414 415 416 417 418 419 420 421 422 423 424 425 426 427 428 429 430 431 432 433 434 435 436 437 438 439 440 441 442 443 444 445 446 447 448 449 450 451 452 453 454 455 456 457 458 459 460 461 462 463 464 465 466 467 468 469 470 471 472 473 474 475 476 477 478 479 480 481 482 483 484 485 486 487 488 489 490 491 492 493 494 495 496 497 498 499 500 501 502 503 504 505 506 507 508 509 510 511 512 513 514 515 516 517 518 519 520 521 522 523 524 525 526 527 528 529 530 531 532 533 534 535 536 537 538 539 540 541 542 543 544 545 546 547 548 549 550 551 552 553 554 555 556 557 558 559 560 561 562 563 564 565 566 567 568 569 570 571 572 573 574 575 576 577 578 579 580 581 582 583 584 585 586 587 588 589 590 591 592 593 594 595 596 597 598 599 600 601 602 603 604 605 606 607 608 609 610 611 612 613 614 615 616 617 618 619 620 621 622 623 624 625 626 627 628 629 630 631 632 633 634 635 636 637 638 639 640 641 642 643 644 645 646 647 648 649 650 651 652 653 654 655 656 657 658 659 660 661 662 663 664 665 666 667 668 669 670 671 672 673 674 675 676 677 678 679 680 681 682 683 684 685 686 687 688 689 690 691 692 693 694 695 696 697 698 699 700 701 702 703 704 705 706 707 708 709 710 711 712 713 714 715 716 717 718 719 720 721 722 723 724 725 726 727 728 729 730 731 732 733 734 735 736 737 738 739 740 741 742 743 744 745 746 747 748 749 750 751 752 753 754 755 756 757 758 759 760 761 762 763 764 765 766 767 768 769 770 771 772 773 774 775 776 777 778 779 780 781 782 783 784 785 786 787 788 789 790 791 792 793 794 795 796 797 798 799 800 801 802 803 804 805 806 807 808 809 810 811 812 813 814 815 816 817 818 819 820 821 822 823 824 825 826 827 828 829 830 831 832 833 834 835 836 837 838 839 840 841 842 843 844 845 846 847 848 849 850 851 852 853 854 855 856 857 858 859 860 861 862 863 864 865 866 867 868 869 870 871 872 873 874 875 876 877 878 879 880 881 882 883 884 885 886 887 888 889 890 891 892 893 894 895 896 897 898 899 900 901 902 903 904 905 906 907 908 909 910 911 912 913 914 915 916 917 918 919 920 921 922 923 924 925 926 927 928 929 930 931 932 933 934 935 936 937 938 939 940 941 942 943 944 945 946 947 948 949 950 951 952 953 954 955 956 957 958 959 960 961 962 963 964 965 966 967 968 969 970 971 972 973 974 975 976 977 978 979 980 981 982 983 984 985 986 987 988 989 990 991 992 993 994 995 996 997 998 999 1000 1001 1002 1003 1004 1005 1006 1007 1008 1009 1010 1011 1012 1013 1014 1015 1016 1017 1018 1019 1020 1021 1022 1023 1024 1025 1026 1027 1028 1029 1030 1031 1032 1033 1034 1035 1036 1037 1038 1039 1040 1041 1042 1043 1044 1045 1046 1047 1048 1049 1050 1051 1052 1053 1054 1055 1056 1057 1058 1059 1060 1061 1062 1063 1064 1065 1066 1067 1068 1069 1070 1071 1072 1073 1074 1075 1076 1077 1078 1079 1080 1081 1082 1083 1084 1085 1086 1087 1088 1089 1090 1091 1092 1093 1094 1095 1096 1097 1098 1099 1100 1101 1102 1103 1104 1105 1106 1107 1108 1109 1110 1111 1112 1113 1114 1115 1116 1117 1118 1119 1120 1121 1122 1123 1124 1125 1126 1127 1128 1129 1130 1131 1132 1133 1134 1135 1136 1137 1138 1139 1140 1141 1142 1143 1144 1145 1146 1147 1148 1149 1150 1151 1152 1153 1154 1155 1156 1157 1158 1159 1160 1161 1162 1163 1164 1165 1166 1167 1168 1169 1170 1171 1172 1173 1174 1175 1176 1177 1178 1179 1180 1181 1182 1183 1184 1185 1186 1187 1188 1189 1190 1191 1192 1193 1194 1195 1196 1197 1198 1199 1200 1201 1202 1203 1204 1205 1206 1207 1208 1209 1210 1211 1212 1213 1214 1215 1216 1217 1218 1219 1220 1221 1222 1223 1224 1225 1226 1227 1228 1229 1230 1231 1232 1233 1234 1235 1236 1237 1238 1239 1240 1241 1242 1243 1244 1245 1246 1247 1248 1249 1250 1251 1252 1253 1254 1255 1256 1257 1258 1259 1260 1261 1262 1263 1264 1265 1266 1267 1268 1269 1270 1271 1272 1273 1274 1275 1276 1277 1278 1279 1280 1281 1282 1283 1284 1285 1286 1287 1288 1289 1290 1291 1292 1293 1294 1295 1296 1297 1298 1299 1300 1301 1302 1303 1304 1305 1306 1307 1308 1309 1310 1311 1312 1313 1314 1315 1316 1317 1318 1319 1320 1321 1322 1323 1324 1325 1326 1327 1328 1329 1330 1331 1332 1333 1334 1335 1336 1337 1338 1339 1340 1341 1342 1343 1344 1345 1346 1347 1348 1349 1350 1351 1352 1353 1354 1355 1356 1357 1358 1359 1360 1361 1362 1363 1364 1365 1366 1367 1368 1369 1370 1371 1372 1373 1374 1375 1376 1377 1378 1379 1380 1381 1382 1383 1384 1385 1386 1387 1388 1389 1390 1391 1392 1393 1394 1395 1396 1397 1398 1399 1400 1401 1402 1403 1404 1405 1406 1407 1408 1409 1410 1411 1412 1413 1414 1415 1416 1417 1418 1419 1420 1421 1422 1423 1424 1425 1426 1427 1428 1429 1430 1431 1432 1433 1434 1435 1436 1437 1438 1439 1440 1441 1442 1443 1444 1445 1446 1447 1448 1449 1450 1451 1452 1453 1454 1455 1456 1457 1458 1459 1460 1461 1462 1463 1464 1465 1466 1467 1468 1469 1470 1471 1472 1473 1474 1475 1476 1477 1478 1479 1480 1481 1482 1483 1484 1485 1486 1487 1488 1489 1490 1491 1492 1493 1494 1495 1496 1497 1498 1499 1500 1501 1502 1503 1504 1505 1506 1507 1508 1509 1510 1511 1512 1513 1514 1515 1516 1517 1518 1519 1520 1521 1522 1523 1524 1525 1526 1527 1528 1529 1530 1531 1532 1533 1534 1535 1536 1537 1538 1539 1540 1541 1542 1543 1544 1545 1546 1547 1548 1549 1550 1551 1552 1553 1554 1555 1556 1557 1558 1559 1560 1561 1562 1563 1564 1565 1566 1567 1568 1569 1570 1571 1572 1573 1574 1575 1576 1577 1578 1579 1580 1581 1582 1583 1584 1585 1586 1587 1588 1589 1590 1591 1592 1593 1594 1595 1596 1597 1598 1599 1600 1601 1602 1603 1604 1605 1606 1607 1608 1609 1610 1611 1612 1613 1614 1615 1616 1617 1618 1619 1620 1621 1622 1623 1624 1625 1626 1627 1628 1629 1630 1631 1632 1633 1634 1635 1636 1637 1638 1639 1640 1641 1642 1643 1644 1645 1646 1647 1648 1649 1650 1651 1652 1653 1654 1655 1656 1657 1658 1659 1660 1661 1662 1663 1664 1665 1666 1667 1668 1669 1670 1671 1672 1673 1674 1675 1676 1677 1678 1679 1680 1681 1682 1683 1684 1685 1686 1687 1688 1689 1690 1691 1692 1693 1694 1695 1696 1697 1698 1699 1700 1701 1702 1703 1704 1705 1706 1707 1708 1709 1710 1711 1712 1713 1714 1715 1716 1717 1718 1719 1720 1721 1722 1723 1724 1725 1726 1727 1728 1729 1730 1731 1732 1733 1734 1735 1736 1737 1738 1739 1740 1741 1742 1743 1744 1745 1746 1747 1748 1749 1750 1751 1752 1753 1754 | /* $NetBSD: procfs_vnops.c,v 1.229 2022/06/17 14:30:37 shm Exp $ */ /*- * Copyright (c) 2006, 2007, 2008, 2020 The NetBSD Foundation, Inc. * All rights reserved. * * This code is derived from software contributed to The NetBSD Foundation * by Andrew Doran. * * Redistribution and use in source and binary forms, with or without * modification, are permitted provided that the following conditions * are met: * 1. Redistributions of source code must retain the above copyright * notice, this list of conditions and the following disclaimer. * 2. Redistributions in binary form must reproduce the above copyright * notice, this list of conditions and the following disclaimer in the * documentation and/or other materials provided with the distribution. * * THIS SOFTWARE IS PROVIDED BY THE NETBSD FOUNDATION, INC. AND CONTRIBUTORS * ``AS IS'' AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED * TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR * PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE FOUNDATION OR CONTRIBUTORS * BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR * CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF * SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS * INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN * CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) * ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE * POSSIBILITY OF SUCH DAMAGE. */ /* * Copyright (c) 1993, 1995 * The Regents of the University of California. All rights reserved. * * This code is derived from software contributed to Berkeley by * Jan-Simon Pendry. * * Redistribution and use in source and binary forms, with or without * modification, are permitted provided that the following conditions * are met: * 1. Redistributions of source code must retain the above copyright * notice, this list of conditions and the following disclaimer. * 2. Redistributions in binary form must reproduce the above copyright * notice, this list of conditions and the following disclaimer in the * documentation and/or other materials provided with the distribution. * 3. Neither the name of the University nor the names of its contributors * may be used to endorse or promote products derived from this software * without specific prior written permission. * * THIS SOFTWARE IS PROVIDED BY THE REGENTS AND CONTRIBUTORS ``AS IS'' AND * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE * ARE DISCLAIMED. IN NO EVENT SHALL THE REGENTS OR CONTRIBUTORS BE LIABLE * FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL * DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS * OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) * HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT * LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY * OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF * SUCH DAMAGE. * * @(#)procfs_vnops.c 8.18 (Berkeley) 5/21/95 */ /* * Copyright (c) 1993 Jan-Simon Pendry * * This code is derived from software contributed to Berkeley by * Jan-Simon Pendry. * * Redistribution and use in source and binary forms, with or without * modification, are permitted provided that the following conditions * are met: * 1. Redistributions of source code must retain the above copyright * notice, this list of conditions and the following disclaimer. * 2. Redistributions in binary form must reproduce the above copyright * notice, this list of conditions and the following disclaimer in the * documentation and/or other materials provided with the distribution. * 3. All advertising materials mentioning features or use of this software * must display the following acknowledgement: * This product includes software developed by the University of * California, Berkeley and its contributors. * 4. Neither the name of the University nor the names of its contributors * may be used to endorse or promote products derived from this software * without specific prior written permission. * * THIS SOFTWARE IS PROVIDED BY THE REGENTS AND CONTRIBUTORS ``AS IS'' AND * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE * ARE DISCLAIMED. IN NO EVENT SHALL THE REGENTS OR CONTRIBUTORS BE LIABLE * FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL * DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS * OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) * HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT * LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY * OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF * SUCH DAMAGE. * * @(#)procfs_vnops.c 8.18 (Berkeley) 5/21/95 */ /* * procfs vnode interface */ #include <sys/cdefs.h> __KERNEL_RCSID(0, "$NetBSD: procfs_vnops.c,v 1.229 2022/06/17 14:30:37 shm Exp $"); #include <sys/param.h> #include <sys/atomic.h> #include <sys/systm.h> #include <sys/time.h> #include <sys/kernel.h> #include <sys/file.h> #include <sys/filedesc.h> #include <sys/proc.h> #include <sys/vnode.h> #include <sys/namei.h> #include <sys/malloc.h> #include <sys/mount.h> #include <sys/dirent.h> #include <sys/resourcevar.h> #include <sys/stat.h> #include <sys/ptrace.h> #include <sys/kauth.h> #include <sys/exec.h> #include <uvm/uvm_extern.h> /* for PAGE_SIZE */ #include <machine/reg.h> #include <miscfs/genfs/genfs.h> #include <miscfs/procfs/procfs.h> /* * Vnode Operations. * */ static int procfs_validfile_linux(struct lwp *, struct mount *); static int procfs_root_readdir_callback(struct proc *, void *); static void procfs_dir(pfstype, struct lwp *, struct proc *, char **, char *, size_t); /* * This is a list of the valid names in the * process-specific sub-directories. It is * used in procfs_lookup and procfs_readdir */ static const struct proc_target { u_char pt_type; u_char pt_namlen; const char *pt_name; pfstype pt_pfstype; int (*pt_valid)(struct lwp *, struct mount *); } proc_targets[] = { #define N(s) sizeof(s)-1, s /* name type validp */ { DT_DIR, N("."), PFSproc, NULL }, { DT_DIR, N(".."), PFSroot, NULL }, { DT_DIR, N("fd"), PFSfd, NULL }, { DT_DIR, N("task"), PFStask, procfs_validfile_linux }, { DT_LNK, N("cwd"), PFScwd, NULL }, { DT_REG, N("emul"), PFSemul, NULL }, { DT_LNK, N("root"), PFSchroot, NULL }, { DT_REG, N("auxv"), PFSauxv, procfs_validauxv }, { DT_REG, N("cmdline"), PFScmdline, NULL }, { DT_REG, N("environ"), PFSenviron, NULL }, { DT_LNK, N("exe"), PFSexe, procfs_validfile }, { DT_REG, N("file"), PFSfile, procfs_validfile }, { DT_REG, N("fpregs"), PFSfpregs, procfs_validfpregs }, { DT_REG, N("limit"), PFSlimit, NULL }, { DT_REG, N("map"), PFSmap, procfs_validmap }, { DT_REG, N("maps"), PFSmaps, procfs_validmap }, { DT_REG, N("mem"), PFSmem, NULL }, { DT_REG, N("note"), PFSnote, NULL }, { DT_REG, N("notepg"), PFSnotepg, NULL }, { DT_REG, N("regs"), PFSregs, procfs_validregs }, { DT_REG, N("stat"), PFSstat, procfs_validfile_linux }, { DT_REG, N("statm"), PFSstatm, procfs_validfile_linux }, { DT_REG, N("status"), PFSstatus, NULL }, #ifdef __HAVE_PROCFS_MACHDEP PROCFS_MACHDEP_NODETYPE_DEFNS #endif #undef N }; static const int nproc_targets = sizeof(proc_targets) / sizeof(proc_targets[0]); /* * List of files in the root directory. Note: the validate function will * be called with p == NULL for these ones. */ static const struct proc_target proc_root_targets[] = { #define N(s) sizeof(s)-1, s /* name type validp */ { DT_REG, N("meminfo"), PFSmeminfo, procfs_validfile_linux }, { DT_REG, N("cpuinfo"), PFScpuinfo, procfs_validfile_linux }, { DT_REG, N("uptime"), PFSuptime, procfs_validfile_linux }, { DT_REG, N("mounts"), PFSmounts, procfs_validfile_linux }, { DT_REG, N("devices"), PFSdevices, procfs_validfile_linux }, { DT_REG, N("stat"), PFScpustat, procfs_validfile_linux }, { DT_REG, N("loadavg"), PFSloadavg, procfs_validfile_linux }, { DT_REG, N("version"), PFSversion, procfs_validfile_linux }, #undef N }; static const int nproc_root_targets = sizeof(proc_root_targets) / sizeof(proc_root_targets[0]); int procfs_lookup(void *); int procfs_open(void *); int procfs_close(void *); int procfs_access(void *); int procfs_getattr(void *); int procfs_setattr(void *); int procfs_readdir(void *); int procfs_readlink(void *); int procfs_inactive(void *); int procfs_reclaim(void *); int procfs_print(void *); int procfs_pathconf(void *); int procfs_getpages(void *); static uint8_t fttodt(file_t *); static int atoi(const char *, size_t); /* * procfs vnode operations. */ int (**procfs_vnodeop_p)(void *); const struct vnodeopv_entry_desc procfs_vnodeop_entries[] = { { &vop_default_desc, vn_default_error }, { &vop_parsepath_desc, genfs_parsepath }, /* parsepath */ { &vop_lookup_desc, procfs_lookup }, /* lookup */ { &vop_create_desc, genfs_eopnotsupp }, /* create */ { &vop_mknod_desc, genfs_eopnotsupp }, /* mknod */ { &vop_open_desc, procfs_open }, /* open */ { &vop_close_desc, procfs_close }, /* close */ { &vop_access_desc, procfs_access }, /* access */ { &vop_accessx_desc, genfs_accessx }, /* accessx */ { &vop_getattr_desc, procfs_getattr }, /* getattr */ { &vop_setattr_desc, procfs_setattr }, /* setattr */ { &vop_read_desc, procfs_rw }, /* read */ { &vop_write_desc, procfs_rw }, /* write */ { &vop_fallocate_desc, genfs_eopnotsupp }, /* fallocate */ { &vop_fdiscard_desc, genfs_eopnotsupp }, /* fdiscard */ { &vop_fcntl_desc, genfs_fcntl }, /* fcntl */ { &vop_ioctl_desc, genfs_enoioctl }, /* ioctl */ { &vop_poll_desc, genfs_poll }, /* poll */ { &vop_kqfilter_desc, genfs_kqfilter }, /* kqfilter */ { &vop_revoke_desc, genfs_revoke }, /* revoke */ { &vop_fsync_desc, genfs_nullop }, /* fsync */ { &vop_seek_desc, genfs_nullop }, /* seek */ { &vop_remove_desc, genfs_eopnotsupp }, /* remove */ { &vop_link_desc, genfs_erofs_link }, /* link */ { &vop_rename_desc, genfs_eopnotsupp }, /* rename */ { &vop_mkdir_desc, genfs_eopnotsupp }, /* mkdir */ { &vop_rmdir_desc, genfs_eopnotsupp }, /* rmdir */ { &vop_symlink_desc, genfs_erofs_symlink }, /* symlink */ { &vop_readdir_desc, procfs_readdir }, /* readdir */ { &vop_readlink_desc, procfs_readlink }, /* readlink */ { &vop_abortop_desc, genfs_abortop }, /* abortop */ { &vop_inactive_desc, procfs_inactive }, /* inactive */ { &vop_reclaim_desc, procfs_reclaim }, /* reclaim */ { &vop_lock_desc, genfs_lock }, /* lock */ { &vop_unlock_desc, genfs_unlock }, /* unlock */ { &vop_bmap_desc, genfs_eopnotsupp }, /* bmap */ { &vop_strategy_desc, genfs_badop }, /* strategy */ { &vop_print_desc, procfs_print }, /* print */ { &vop_islocked_desc, genfs_islocked }, /* islocked */ { &vop_pathconf_desc, procfs_pathconf }, /* pathconf */ { &vop_advlock_desc, genfs_einval }, /* advlock */ { &vop_getpages_desc, procfs_getpages }, /* getpages */ { &vop_putpages_desc, genfs_null_putpages }, /* putpages */ { NULL, NULL } }; const struct vnodeopv_desc procfs_vnodeop_opv_desc = { &procfs_vnodeop_p, procfs_vnodeop_entries }; /* * set things up for doing i/o on * the pfsnode (vp). (vp) is locked * on entry, and should be left locked * on exit. * * for procfs we don't need to do anything * in particular for i/o. all that is done * is to support exclusive open on process * memory images. */ int procfs_open(void *v) { struct vop_open_args /* { struct vnode *a_vp; int a_mode; kauth_cred_t a_cred; } */ *ap = v; struct vnode *vp = ap->a_vp; struct pfsnode *pfs = VTOPFS(vp); struct lwp *l1; struct proc *p2; int error; if ((error = procfs_proc_lock(vp->v_mount, pfs->pfs_pid, &p2, ENOENT)) != 0) return error; l1 = curlwp; /* tracer */ #define M2K(m) (((m) & FREAD) && ((m) & FWRITE) ? \ KAUTH_REQ_PROCESS_PROCFS_RW : \ (m) & FWRITE ? KAUTH_REQ_PROCESS_PROCFS_WRITE : \ KAUTH_REQ_PROCESS_PROCFS_READ) mutex_enter(p2->p_lock); error = kauth_authorize_process(l1->l_cred, KAUTH_PROCESS_PROCFS, p2, pfs, KAUTH_ARG(M2K(ap->a_mode)), NULL); mutex_exit(p2->p_lock); if (error) { procfs_proc_unlock(p2); return (error); } #undef M2K switch (pfs->pfs_type) { case PFSmem: if (((pfs->pfs_flags & FWRITE) && (ap->a_mode & O_EXCL)) || ((pfs->pfs_flags & O_EXCL) && (ap->a_mode & FWRITE))) { error = EBUSY; break; } if (!proc_isunder(p2, l1)) { error = EPERM; break; } if (ap->a_mode & FWRITE) pfs->pfs_flags = ap->a_mode & (FWRITE|O_EXCL); break; case PFSregs: case PFSfpregs: if (!proc_isunder(p2, l1)) { error = EPERM; break; } break; default: break; } procfs_proc_unlock(p2); return (error); } /* * close the pfsnode (vp) after doing i/o. * (vp) is not locked on entry or exit. * * nothing to do for procfs other than undo * any exclusive open flag (see _open above). */ int procfs_close(void *v) { struct vop_close_args /* { struct vnode *a_vp; int a_fflag; kauth_cred_t a_cred; } */ *ap = v; struct pfsnode *pfs = VTOPFS(ap->a_vp); switch (pfs->pfs_type) { case PFSmem: if ((ap->a_fflag & FWRITE) && (pfs->pfs_flags & O_EXCL)) pfs->pfs_flags &= ~(FWRITE|O_EXCL); break; default: break; } return (0); } /* * _inactive is called when the pfsnode * is vrele'd and the reference count goes * to zero. (vp) will be on the vnode free * list, so to get it back vget() must be * used. * * (vp) is locked on entry, but must be unlocked on exit. */ int procfs_inactive(void *v) { struct vop_inactive_v2_args /* { struct vnode *a_vp; bool *a_recycle; } */ *ap = v; struct vnode *vp = ap->a_vp; struct pfsnode *pfs = VTOPFS(vp); mutex_enter(&proc_lock); *ap->a_recycle = (procfs_proc_find(vp->v_mount, pfs->pfs_pid) == NULL); mutex_exit(&proc_lock); return (0); } /* * _reclaim is called when getnewvnode() * wants to make use of an entry on the vnode * free list. at this time the filesystem needs * to free any private data and remove the node * from any private lists. */ int procfs_reclaim(void *v) { struct vop_reclaim_v2_args /* { struct vnode *a_vp; } */ *ap = v; struct vnode *vp = ap->a_vp; struct pfsnode *pfs = VTOPFS(vp); VOP_UNLOCK(vp); /* * To interlock with procfs_revoke_vnodes(). */ mutex_enter(vp->v_interlock); vp->v_data = NULL; mutex_exit(vp->v_interlock); kmem_free(pfs, sizeof(*pfs)); return 0; } /* * Return POSIX pathconf information applicable to special devices. */ int procfs_pathconf(void *v) { struct vop_pathconf_args /* { struct vnode *a_vp; int a_name; register_t *a_retval; } */ *ap = v; switch (ap->a_name) { case _PC_LINK_MAX: *ap->a_retval = LINK_MAX; return (0); case _PC_MAX_CANON: *ap->a_retval = MAX_CANON; return (0); case _PC_MAX_INPUT: *ap->a_retval = MAX_INPUT; return (0); case _PC_PIPE_BUF: *ap->a_retval = PIPE_BUF; return (0); case _PC_CHOWN_RESTRICTED: *ap->a_retval = 1; return (0); case _PC_VDISABLE: *ap->a_retval = _POSIX_VDISABLE; return (0); case _PC_SYNC_IO: *ap->a_retval = 1; return (0); default: return genfs_pathconf(ap); } /* NOTREACHED */ } /* * _print is used for debugging. * just print a readable description * of (vp). */ int procfs_print(void *v) { struct vop_print_args /* { struct vnode *a_vp; } */ *ap = v; struct pfsnode *pfs = VTOPFS(ap->a_vp); printf("tag VT_PROCFS, type %d, pid %d, mode %x, flags %lx\n", pfs->pfs_type, pfs->pfs_pid, pfs->pfs_mode, pfs->pfs_flags); return 0; } /* * Works out the path to the target process's current * working directory or chroot. If the caller is in a chroot and * can't "reach" the target's cwd or root (or some other error * occurs), a "/" is returned for the path. */ static void procfs_dir(pfstype t, struct lwp *caller, struct proc *target, char **bpp, char *path, size_t len) { struct cwdinfo *cwdi; struct vnode *vp, *rvp; char *bp; /* * Lock target cwdi and take a reference to the vnode * we are interested in to prevent it from disappearing * before getcwd_common() below. */ rw_enter(&target->p_cwdi->cwdi_lock, RW_READER); switch (t) { case PFScwd: vp = target->p_cwdi->cwdi_cdir; break; case PFSchroot: vp = target->p_cwdi->cwdi_rdir; break; default: rw_exit(&target->p_cwdi->cwdi_lock); return; } if (vp != NULL) vref(vp); rw_exit(&target->p_cwdi->cwdi_lock); cwdi = caller->l_proc->p_cwdi; rw_enter(&cwdi->cwdi_lock, RW_READER); rvp = cwdi->cwdi_rdir; bp = bpp ? *bpp : NULL; /* * XXX: this horrible kludge avoids locking panics when * attempting to lookup links that point to within procfs */ if (vp != NULL && vp->v_tag == VT_PROCFS) { if (bpp) { *--bp = '/'; *bpp = bp; } vrele(vp); rw_exit(&cwdi->cwdi_lock); return; } if (rvp == NULL) rvp = rootvnode; if (vp == NULL || getcwd_common(vp, rvp, bp ? &bp : NULL, path, len / 2, 0, caller) != 0) { if (bpp) { bp = *bpp; *--bp = '/'; } } if (bpp) *bpp = bp; if (vp != NULL) vrele(vp); rw_exit(&cwdi->cwdi_lock); } /* * Invent attributes for pfsnode (vp) and store * them in (vap). * Directories lengths are returned as zero since * any real length would require the genuine size * to be computed, and nothing cares anyway. * * this is relatively minimal for procfs. */ int procfs_getattr(void *v) { struct vop_getattr_args /* { struct vnode *a_vp; struct vattr *a_vap; kauth_cred_t a_cred; } */ *ap = v; struct vnode *vp = ap->a_vp; struct pfsnode *pfs = VTOPFS(vp); struct vattr *vap = ap->a_vap; struct proc *procp; char *path, *bp, bf[16]; int error; /* first check the process still exists */ switch (pfs->pfs_type) { case PFSroot: case PFScurproc: case PFSself: procp = NULL; break; default: error = procfs_proc_lock(vp->v_mount, pfs->pfs_pid, &procp, ENOENT); if (error != 0) return (error); break; } switch (pfs->pfs_type) { case PFStask: if (pfs->pfs_fd == -1) { path = NULL; break; } /*FALLTHROUGH*/ case PFScwd: case PFSchroot: path = malloc(MAXPATHLEN + 4, M_TEMP, M_WAITOK); if (path == NULL && procp != NULL) { procfs_proc_unlock(procp); return (ENOMEM); } break; default: path = NULL; break; } if (procp != NULL) { mutex_enter(procp->p_lock); error = kauth_authorize_process(kauth_cred_get(), KAUTH_PROCESS_CANSEE, procp, KAUTH_ARG(KAUTH_REQ_PROCESS_CANSEE_ENTRY), NULL, NULL); mutex_exit(procp->p_lock); if (error != 0) { procfs_proc_unlock(procp); if (path != NULL) free(path, M_TEMP); return (ENOENT); } } error = 0; /* start by zeroing out the attributes */ vattr_null(vap); /* next do all the common fields */ vap->va_type = ap->a_vp->v_type; vap->va_mode = pfs->pfs_mode; vap->va_fileid = pfs->pfs_fileno; vap->va_flags = 0; vap->va_blocksize = PAGE_SIZE; /* * Make all times be current TOD. * * It would be possible to get the process start * time from the p_stats structure, but there's * no "file creation" time stamp anyway, and the * p_stats structure is not addressable if u. gets * swapped out for that process. */ getnanotime(&vap->va_ctime); vap->va_atime = vap->va_mtime = vap->va_ctime; if (procp) TIMEVAL_TO_TIMESPEC(&procp->p_stats->p_start, &vap->va_birthtime); else getnanotime(&vap->va_birthtime); switch (pfs->pfs_type) { case PFSmem: case PFSregs: case PFSfpregs: #if defined(__HAVE_PROCFS_MACHDEP) && defined(PROCFS_MACHDEP_PROTECT_CASES) PROCFS_MACHDEP_PROTECT_CASES #endif /* * If the process has exercised some setuid or setgid * privilege, then rip away read/write permission so * that only root can gain access. */ if (procp->p_flag & PK_SUGID) vap->va_mode &= ~(S_IRUSR|S_IWUSR); /* FALLTHROUGH */ case PFSstatus: case PFSstat: case PFSnote: case PFSnotepg: case PFScmdline: case PFSenviron: case PFSemul: case PFSstatm: case PFSmap: case PFSmaps: case PFSlimit: case PFSauxv: vap->va_nlink = 1; vap->va_uid = kauth_cred_geteuid(procp->p_cred); vap->va_gid = kauth_cred_getegid(procp->p_cred); break; case PFScwd: case PFSchroot: case PFSmeminfo: case PFSdevices: case PFScpuinfo: case PFSuptime: case PFSmounts: case PFScpustat: case PFSloadavg: case PFSversion: case PFSexe: case PFSself: case PFScurproc: case PFSroot: vap->va_nlink = 1; vap->va_uid = vap->va_gid = 0; break; case PFSproc: case PFStask: case PFSfile: case PFSfd: break; default: panic("%s: %d/1", __func__, pfs->pfs_type); } /* * now do the object specific fields * * The size could be set from struct reg, but it's hardly * worth the trouble, and it puts some (potentially) machine * dependent data into this machine-independent code. If it * becomes important then this function should break out into * a per-file stat function in the corresponding .c file. */ switch (pfs->pfs_type) { case PFSroot: vap->va_bytes = vap->va_size = DEV_BSIZE; break; case PFSself: case PFScurproc: vap->va_bytes = vap->va_size = snprintf(bf, sizeof(bf), "%ld", (long)curproc->p_pid); break; case PFStask: if (pfs->pfs_fd != -1) { vap->va_nlink = 1; vap->va_uid = 0; vap->va_gid = 0; vap->va_bytes = vap->va_size = snprintf(bf, sizeof(bf), ".."); break; } /*FALLTHROUGH*/ case PFSfd: if (pfs->pfs_fd != -1) { file_t *fp; fp = fd_getfile2(procp, pfs->pfs_fd); if (fp == NULL) { error = EBADF; break; } vap->va_nlink = 1; vap->va_uid = kauth_cred_geteuid(fp->f_cred); vap->va_gid = kauth_cred_getegid(fp->f_cred); switch (fp->f_type) { case DTYPE_VNODE: vap->va_bytes = vap->va_size = fp->f_vnode->v_size; break; default: vap->va_bytes = vap->va_size = 0; break; } closef(fp); break; } /*FALLTHROUGH*/ case PFSproc: vap->va_nlink = 2; vap->va_uid = kauth_cred_geteuid(procp->p_cred); vap->va_gid = kauth_cred_getegid(procp->p_cred); vap->va_bytes = vap->va_size = DEV_BSIZE; break; case PFSfile: error = EOPNOTSUPP; break; case PFSmem: vap->va_bytes = vap->va_size = ctob(procp->p_vmspace->vm_tsize + procp->p_vmspace->vm_dsize + procp->p_vmspace->vm_ssize); break; case PFSauxv: vap->va_bytes = vap->va_size = procp->p_execsw->es_arglen; break; #if defined(PT_GETREGS) || defined(PT_SETREGS) case PFSregs: vap->va_bytes = vap->va_size = sizeof(struct reg); break; #endif #if defined(PT_GETFPREGS) || defined(PT_SETFPREGS) case PFSfpregs: vap->va_bytes = vap->va_size = sizeof(struct fpreg); break; #endif case PFSstatus: case PFSstat: case PFSnote: case PFSnotepg: case PFScmdline: case PFSenviron: case PFSmeminfo: case PFSdevices: case PFScpuinfo: case PFSuptime: case PFSmounts: case PFScpustat: case PFSloadavg: case PFSstatm: case PFSversion: vap->va_bytes = vap->va_size = 0; break; case PFSlimit: case PFSmap: case PFSmaps: /* * Advise a larger blocksize for the map files, so that * they may be read in one pass. */ vap->va_blocksize = 4 * PAGE_SIZE; vap->va_bytes = vap->va_size = 0; break; case PFScwd: case PFSchroot: bp = path + MAXPATHLEN; *--bp = '\0'; procfs_dir(pfs->pfs_type, curlwp, procp, &bp, path, MAXPATHLEN); vap->va_bytes = vap->va_size = strlen(bp); break; case PFSexe: vap->va_bytes = vap->va_size = strlen(procp->p_path); break; case PFSemul: vap->va_bytes = vap->va_size = strlen(procp->p_emul->e_name); break; #ifdef __HAVE_PROCFS_MACHDEP PROCFS_MACHDEP_NODETYPE_CASES error = procfs_machdep_getattr(ap->a_vp, vap, procp); break; #endif default: panic("%s: %d/2", __func__, pfs->pfs_type); } if (procp != NULL) procfs_proc_unlock(procp); if (path != NULL) free(path, M_TEMP); return (error); } /*ARGSUSED*/ int procfs_setattr(void *v) { /* * just fake out attribute setting * it's not good to generate an error * return, otherwise things like creat() * will fail when they try to set the * file length to 0. worse, this means * that echo $note > /proc/$pid/note will fail. */ return (0); } /* * implement access checking. * * actually, the check for super-user is slightly * broken since it will allow read access to write-only * objects. this doesn't cause any particular trouble * but does mean that the i/o entry points need to check * that the operation really does make sense. */ int procfs_access(void *v) { struct vop_access_args /* { struct vnode *a_vp; accmode_t a_accmode; kauth_cred_t a_cred; } */ *ap = v; struct vattr va; int error; if ((error = VOP_GETATTR(ap->a_vp, &va, ap->a_cred)) != 0) return (error); return kauth_authorize_vnode(ap->a_cred, KAUTH_ACCESS_ACTION(ap->a_accmode, ap->a_vp->v_type, va.va_mode), ap->a_vp, NULL, genfs_can_access(ap->a_vp, ap->a_cred, va.va_uid, va.va_gid, va.va_mode, NULL, ap->a_accmode)); } /* * lookup. this is incredibly complicated in the * general case, however for most pseudo-filesystems * very little needs to be done. * * Locking isn't hard here, just poorly documented. * * If we're looking up ".", just vref the parent & return it. * * If we're looking up "..", unlock the parent, and lock "..". If everything * went ok, and we're on the last component and the caller requested the * parent locked, try to re-lock the parent. We do this to prevent lock * races. * * For anything else, get the needed node. Then unlock the parent if not * the last component or not LOCKPARENT (i.e. if we wouldn't re-lock the * parent in the .. case). * * We try to exit with the parent locked in error cases. */ int procfs_lookup(void *v) { struct vop_lookup_v2_args /* { struct vnode * a_dvp; struct vnode ** a_vpp; struct componentname * a_cnp; } */ *ap = v; struct componentname *cnp = ap->a_cnp; struct vnode **vpp = ap->a_vpp; struct vnode *dvp = ap->a_dvp; const char *pname = cnp->cn_nameptr; const struct proc_target *pt = NULL; struct vnode *fvp; pid_t pid, vnpid; struct pfsnode *pfs; struct proc *p = NULL; struct lwp *plwp; int i, error; pfstype type; *vpp = NULL; if ((error = VOP_ACCESS(dvp, VEXEC, cnp->cn_cred)) != 0) return (error); if (cnp->cn_nameiop == DELETE || cnp->cn_nameiop == RENAME) return (EROFS); if (cnp->cn_namelen == 1 && *pname == '.') { *vpp = dvp; vref(dvp); return (0); } pfs = VTOPFS(dvp); switch (pfs->pfs_type) { case PFSroot: /* * Shouldn't get here with .. in the root node. */ if (cnp->cn_flags & ISDOTDOT) return (EIO); for (i = 0; i < nproc_root_targets; i++) { pt = &proc_root_targets[i]; /* * check for node match. proc is always NULL here, * so call pt_valid with constant NULL lwp. */ if (cnp->cn_namelen == pt->pt_namlen && memcmp(pt->pt_name, pname, cnp->cn_namelen) == 0 && (pt->pt_valid == NULL || (*pt->pt_valid)(NULL, dvp->v_mount))) break; } if (i != nproc_root_targets) { error = procfs_allocvp(dvp->v_mount, vpp, 0, pt->pt_pfstype, -1); return (error); } if (CNEQ(cnp, "curproc", 7)) { pid = curproc->p_pid; vnpid = 0; type = PFScurproc; } else if (CNEQ(cnp, "self", 4)) { pid = curproc->p_pid; vnpid = 0; type = PFSself; } else { pid = (pid_t)atoi(pname, cnp->cn_namelen); vnpid = pid; type = PFSproc; } if (procfs_proc_lock(dvp->v_mount, pid, &p, ESRCH) != 0) break; error = procfs_allocvp(dvp->v_mount, vpp, vnpid, type, -1); procfs_proc_unlock(p); return (error); case PFSproc: if (cnp->cn_flags & ISDOTDOT) { error = procfs_allocvp(dvp->v_mount, vpp, 0, PFSroot, -1); return (error); } if (procfs_proc_lock(dvp->v_mount, pfs->pfs_pid, &p, ESRCH) != 0) break; mutex_enter(p->p_lock); LIST_FOREACH(plwp, &p->p_lwps, l_sibling) { if (plwp->l_stat != LSZOMB) break; } /* Process is exiting if no-LWPS or all LWPs are LSZOMB */ if (plwp == NULL) { mutex_exit(p->p_lock); procfs_proc_unlock(p); return ESRCH; } lwp_addref(plwp); mutex_exit(p->p_lock); for (pt = proc_targets, i = 0; i < nproc_targets; pt++, i++) { int found; found = cnp->cn_namelen == pt->pt_namlen && memcmp(pt->pt_name, pname, cnp->cn_namelen) == 0 && (pt->pt_valid == NULL || (*pt->pt_valid)(plwp, dvp->v_mount)); if (found) break; } lwp_delref(plwp); if (i == nproc_targets) { procfs_proc_unlock(p); break; } if (pt->pt_pfstype == PFSfile) { fvp = p->p_textvp; /* We already checked that it exists. */ vref(fvp); procfs_proc_unlock(p); *vpp = fvp; return (0); } error = procfs_allocvp(dvp->v_mount, vpp, pfs->pfs_pid, pt->pt_pfstype, -1); procfs_proc_unlock(p); return (error); case PFSfd: { int fd; file_t *fp; if ((error = procfs_proc_lock(dvp->v_mount, pfs->pfs_pid, &p, ENOENT)) != 0) return error; if (cnp->cn_flags & ISDOTDOT) { error = procfs_allocvp(dvp->v_mount, vpp, pfs->pfs_pid, PFSproc, -1); procfs_proc_unlock(p); return (error); } fd = atoi(pname, cnp->cn_namelen); fp = fd_getfile2(p, fd); if (fp == NULL) { procfs_proc_unlock(p); return ENOENT; } fvp = fp->f_vnode; /* Don't show directories */ if (fp->f_type == DTYPE_VNODE && fvp->v_type != VDIR && !procfs_proc_is_linux_compat()) { vref(fvp); closef(fp); procfs_proc_unlock(p); *vpp = fvp; return 0; } closef(fp); error = procfs_allocvp(dvp->v_mount, vpp, pfs->pfs_pid, PFSfd, fd); procfs_proc_unlock(p); return error; } case PFStask: { int xpid; if ((error = procfs_proc_lock(dvp->v_mount, pfs->pfs_pid, &p, ENOENT)) != 0) return error; if (cnp->cn_flags & ISDOTDOT) { error = procfs_allocvp(dvp->v_mount, vpp, pfs->pfs_pid, PFSproc, -1); procfs_proc_unlock(p); return (error); } xpid = atoi(pname, cnp->cn_namelen); if (xpid != pfs->pfs_pid) { procfs_proc_unlock(p); return ENOENT; } error = procfs_allocvp(dvp->v_mount, vpp, pfs->pfs_pid, PFStask, 0); procfs_proc_unlock(p); return error; } default: return (ENOTDIR); } return (cnp->cn_nameiop == LOOKUP ? ENOENT : EROFS); } int procfs_validfile(struct lwp *l, struct mount *mp) { return l != NULL && l->l_proc != NULL && l->l_proc->p_textvp != NULL; } static int procfs_validfile_linux(struct lwp *l, struct mount *mp) { return procfs_use_linux_compat(mp) && (l == NULL || l->l_proc == NULL || procfs_validfile(l, mp)); } struct procfs_root_readdir_ctx { struct uio *uiop; off_t *cookies; int ncookies; off_t off; off_t startoff; int error; }; static int procfs_root_readdir_callback(struct proc *p, void *arg) { struct procfs_root_readdir_ctx *ctxp = arg; struct dirent d; struct uio *uiop; int error; uiop = ctxp->uiop; if (uiop->uio_resid < UIO_MX) return -1; /* no space */ if (kauth_authorize_process(kauth_cred_get(), KAUTH_PROCESS_CANSEE, p, KAUTH_ARG(KAUTH_REQ_PROCESS_CANSEE_ENTRY), NULL, NULL) != 0) return 0; if (ctxp->off < ctxp->startoff) { ctxp->off++; return 0; } memset(&d, 0, UIO_MX); d.d_reclen = UIO_MX; d.d_fileno = PROCFS_FILENO(p->p_pid, PFSproc, -1); d.d_namlen = snprintf(d.d_name, UIO_MX - offsetof(struct dirent, d_name), "%ld", (long)p->p_pid); d.d_type = DT_DIR; mutex_exit(&proc_lock); error = uiomove(&d, UIO_MX, uiop); mutex_enter(&proc_lock); if (error) { ctxp->error = error; return -1; } ctxp->ncookies++; if (ctxp->cookies) *(ctxp->cookies)++ = ctxp->off + 1; ctxp->off++; return 0; } /* * readdir returns directory entries from pfsnode (vp). * * the strategy here with procfs is to generate a single * directory entry at a time (struct dirent) and then * copy that out to userland using uiomove. a more efficient * though more complex implementation, would try to minimize * the number of calls to uiomove(). for procfs, this is * hardly worth the added code complexity. * * this should just be done through read() */ int procfs_readdir(void *v) { struct vop_readdir_args /* { struct vnode *a_vp; struct uio *a_uio; kauth_cred_t a_cred; int *a_eofflag; off_t **a_cookies; int *a_ncookies; } */ *ap = v; struct uio *uio = ap->a_uio; struct dirent d; struct pfsnode *pfs; off_t i; int error; off_t *cookies = NULL; int ncookies; struct vnode *vp; const struct proc_target *pt; struct procfs_root_readdir_ctx ctx; struct proc *p = NULL; struct lwp *l; int nfd; int nc = 0; vp = ap->a_vp; pfs = VTOPFS(vp); if (uio->uio_resid < UIO_MX) return (EINVAL); if (uio->uio_offset < 0) return (EINVAL); error = 0; i = uio->uio_offset; memset(&d, 0, UIO_MX); d.d_reclen = UIO_MX; ncookies = uio->uio_resid / UIO_MX; switch (pfs->pfs_type) { /* * this is for the process-specific sub-directories. * all that is needed to is copy out all the entries * from the procent[] table (top of this file). */ case PFSproc: { if (i >= nproc_targets) return 0; if (procfs_proc_lock(vp->v_mount, pfs->pfs_pid, &p, ESRCH) != 0) break; if (ap->a_ncookies) { ncookies = uimin(ncookies, (nproc_targets - i)); cookies = malloc(ncookies * sizeof (off_t), M_TEMP, M_WAITOK); *ap->a_cookies = cookies; } for (pt = &proc_targets[i]; uio->uio_resid >= UIO_MX && i < nproc_targets; pt++, i++) { if (pt->pt_valid) { /* XXXSMP LWP can disappear */ mutex_enter(p->p_lock); l = LIST_FIRST(&p->p_lwps); KASSERT(l != NULL); mutex_exit(p->p_lock); if ((*pt->pt_valid)(l, vp->v_mount) == 0) continue; } d.d_fileno = PROCFS_FILENO(pfs->pfs_pid, pt->pt_pfstype, -1); d.d_namlen = pt->pt_namlen; memcpy(d.d_name, pt->pt_name, pt->pt_namlen + 1); d.d_type = pt->pt_type; if ((error = uiomove(&d, UIO_MX, uio)) != 0) break; if (cookies) *cookies++ = i + 1; } procfs_proc_unlock(p); break; } case PFSfd: { file_t *fp; int lim; if ((error = procfs_proc_lock(vp->v_mount, pfs->pfs_pid, &p, ESRCH)) != 0) return error; /* XXX Should this be by file as well? */ if (kauth_authorize_process(kauth_cred_get(), KAUTH_PROCESS_CANSEE, p, KAUTH_ARG(KAUTH_REQ_PROCESS_CANSEE_OPENFILES), NULL, NULL) != 0) { procfs_proc_unlock(p); return ESRCH; } nfd = atomic_load_consume(&p->p_fd->fd_dt)->dt_nfiles; lim = uimin((int)p->p_rlimit[RLIMIT_NOFILE].rlim_cur, maxfiles); if (i >= lim) { procfs_proc_unlock(p); return 0; } if (ap->a_ncookies) { ncookies = uimin(ncookies, (nfd + 2 - i)); cookies = malloc(ncookies * sizeof (off_t), M_TEMP, M_WAITOK); *ap->a_cookies = cookies; } for (; i < 2 && uio->uio_resid >= UIO_MX; i++) { pt = &proc_targets[i]; d.d_namlen = pt->pt_namlen; d.d_fileno = PROCFS_FILENO(pfs->pfs_pid, pt->pt_pfstype, -1); (void)memcpy(d.d_name, pt->pt_name, pt->pt_namlen + 1); d.d_type = pt->pt_type; if ((error = uiomove(&d, UIO_MX, uio)) != 0) break; if (cookies) *cookies++ = i + 1; nc++; } if (error) goto out; for (; uio->uio_resid >= UIO_MX && i < nfd; i++) { /* check the descriptor exists */ if ((fp = fd_getfile2(p, i - 2)) == NULL) continue; closef(fp); d.d_fileno = PROCFS_FILENO(pfs->pfs_pid, PFSfd, i - 2); d.d_namlen = snprintf(d.d_name, sizeof(d.d_name), "%lld", (long long)(i - 2)); d.d_type = fttodt(fp); if ((error = uiomove(&d, UIO_MX, uio)) != 0) break; if (cookies) *cookies++ = i + 1; nc++; } goto out; } case PFStask: { if ((error = procfs_proc_lock(vp->v_mount, pfs->pfs_pid, &p, ESRCH)) != 0) return error; nfd = 3; /* ., .., pid */ if (ap->a_ncookies) { ncookies = uimin(ncookies, (nfd + 2 - i)); cookies = malloc(ncookies * sizeof (off_t), M_TEMP, M_WAITOK); *ap->a_cookies = cookies; } for (; i < 2 && uio->uio_resid >= UIO_MX; i++) { pt = &proc_targets[i]; d.d_namlen = pt->pt_namlen; d.d_fileno = PROCFS_FILENO(pfs->pfs_pid, pt->pt_pfstype, -1); (void)memcpy(d.d_name, pt->pt_name, pt->pt_namlen + 1); d.d_type = pt->pt_type; if ((error = uiomove(&d, UIO_MX, uio)) != 0) break; if (cookies) *cookies++ = i + 1; nc++; } if (error) goto out; for (; uio->uio_resid >= UIO_MX && i < nfd; i++) { /* check the descriptor exists */ d.d_fileno = PROCFS_FILENO(pfs->pfs_pid, PFStask, i - 2); d.d_namlen = snprintf(d.d_name, sizeof(d.d_name), "%ld", (long)pfs->pfs_pid); d.d_type = DT_LNK; if ((error = uiomove(&d, UIO_MX, uio)) != 0) break; if (cookies) *cookies++ = i + 1; nc++; } goto out; } /* * this is for the root of the procfs filesystem * what is needed are special entries for "curproc" * and "self" followed by an entry for each process * on allproc. */ case PFSroot: { if (ap->a_ncookies) { /* * XXX Potentially allocating too much space here, * but I'm lazy. This loop needs some work. */ cookies = malloc(ncookies * sizeof (off_t), M_TEMP, M_WAITOK); *ap->a_cookies = cookies; } /* 0 ... 3 are static entries. */ for (; i <= 3 && uio->uio_resid >= UIO_MX; i++) { switch (i) { case 0: /* `.' */ case 1: /* `..' */ d.d_fileno = PROCFS_FILENO(0, PFSroot, -1); d.d_namlen = i + 1; memcpy(d.d_name, "..", d.d_namlen); d.d_name[i + 1] = '\0'; d.d_type = DT_DIR; break; case 2: d.d_fileno = PROCFS_FILENO(0, PFScurproc, -1); d.d_namlen = sizeof("curproc") - 1; memcpy(d.d_name, "curproc", sizeof("curproc")); d.d_type = DT_LNK; break; case 3: d.d_fileno = PROCFS_FILENO(0, PFSself, -1); d.d_namlen = sizeof("self") - 1; memcpy(d.d_name, "self", sizeof("self")); d.d_type = DT_LNK; break; } if ((error = uiomove(&d, UIO_MX, uio)) != 0) break; nc++; if (cookies) *cookies++ = i + 1; } if (error) break; /* 4 ... are process entries. */ ctx.uiop = uio; ctx.error = 0; ctx.off = 4; ctx.startoff = i; ctx.cookies = cookies; ctx.ncookies = nc; proclist_foreach_call(&allproc, procfs_root_readdir_callback, &ctx); cookies = ctx.cookies; nc = ctx.ncookies; error = ctx.error; if (error) break; /* misc entries. */ if (i < ctx.off) i = ctx.off; if (i >= ctx.off + nproc_root_targets) break; error = procfs_proc_lock(vp->v_mount, pfs->pfs_pid, &p, ESRCH); if (error) break; for (pt = &proc_root_targets[i - ctx.off]; uio->uio_resid >= UIO_MX && pt < &proc_root_targets[nproc_root_targets]; pt++, i++) { if (pt->pt_valid && (*pt->pt_valid)(NULL, vp->v_mount) == 0) continue; if (kauth_authorize_process(kauth_cred_get(), KAUTH_PROCESS_CANSEE, p, KAUTH_ARG(KAUTH_REQ_PROCESS_CANSEE_ENTRY), NULL, NULL) != 0) continue; d.d_fileno = PROCFS_FILENO(0, pt->pt_pfstype, -1); d.d_namlen = pt->pt_namlen; memcpy(d.d_name, pt->pt_name, pt->pt_namlen + 1); d.d_type = pt->pt_type; if ((error = uiomove(&d, UIO_MX, uio)) != 0) break; nc++; if (cookies) *cookies++ = i + 1; } out: KASSERT(p != NULL); ncookies = nc; procfs_proc_unlock(p); break; } default: error = ENOTDIR; break; } if (ap->a_ncookies) { if (error) { if (cookies) free(*ap->a_cookies, M_TEMP); *ap->a_ncookies = 0; *ap->a_cookies = NULL; } else *ap->a_ncookies = ncookies; } uio->uio_offset = i; return (error); } /* * readlink reads the link of `curproc' and others */ int procfs_readlink(void *v) { struct vop_readlink_args *ap = v; char bf[16]; /* should be enough */ char *bp = bf; char *path = NULL; int len = 0; int error = 0; struct vnode *vp = ap->a_vp; struct pfsnode *pfs = VTOPFS(vp); struct proc *pown = NULL; if (pfs->pfs_fileno == PROCFS_FILENO(0, PFScurproc, -1)) len = snprintf(bf, sizeof(bf), "%ld", (long)curproc->p_pid); else if (pfs->pfs_fileno == PROCFS_FILENO(0, PFSself, -1)) len = snprintf(bf, sizeof(bf), "%s", "curproc"); else if (pfs->pfs_fileno == PROCFS_FILENO(pfs->pfs_pid, PFStask, 0)) len = snprintf(bf, sizeof(bf), ".."); else if (pfs->pfs_fileno == PROCFS_FILENO(pfs->pfs_pid, PFSexe, -1)) { if ((error = procfs_proc_lock(vp->v_mount, pfs->pfs_pid, &pown, ESRCH)) != 0) return error; bp = pown->p_path; len = strlen(bp); } else if (pfs->pfs_fileno == PROCFS_FILENO(pfs->pfs_pid, PFScwd, -1) || pfs->pfs_fileno == PROCFS_FILENO(pfs->pfs_pid, PFSchroot, -1)) { if ((error = procfs_proc_lock(vp->v_mount, pfs->pfs_pid, &pown, ESRCH)) != 0) return error; path = malloc(MAXPATHLEN + 4, M_TEMP, M_WAITOK); if (path == NULL) { procfs_proc_unlock(pown); return (ENOMEM); } bp = path + MAXPATHLEN; *--bp = '\0'; procfs_dir(PROCFS_TYPE(pfs->pfs_fileno), curlwp, pown, &bp, path, MAXPATHLEN); len = strlen(bp); } else { file_t *fp; struct vnode *vxp; if ((error = procfs_proc_lock(vp->v_mount, pfs->pfs_pid, &pown, ESRCH)) != 0) return error; fp = fd_getfile2(pown, pfs->pfs_fd); if (fp == NULL) { procfs_proc_unlock(pown); return EBADF; } switch (fp->f_type) { case DTYPE_VNODE: vxp = fp->f_vnode; if (vxp->v_type != VDIR && !procfs_proc_is_linux_compat()) { error = EINVAL; break; } if ((path = malloc(MAXPATHLEN, M_TEMP, M_WAITOK)) == NULL) { error = ENOMEM; break; } bp = path + MAXPATHLEN; *--bp = '\0'; /* * XXX: kludge to avoid locking against ourselves * in getcwd() */ if (vxp->v_tag == VT_PROCFS) { *--bp = '/'; } else { rw_enter(&curproc->p_cwdi->cwdi_lock, RW_READER); vp = curproc->p_cwdi->cwdi_rdir; if (vp == NULL) vp = rootvnode; error = getcwd_common(vxp, vp, &bp, path, MAXPATHLEN / 2, 0, curlwp); rw_exit(&curproc->p_cwdi->cwdi_lock); } if (error) break; len = strlen(bp); break; case DTYPE_MISC: len = snprintf(bf, sizeof(bf), "%s", "[misc]"); break; case DTYPE_KQUEUE: len = snprintf(bf, sizeof(bf), "%s", "[kqueue]"); break; case DTYPE_SEM: len = snprintf(bf, sizeof(bf), "%s", "[ksem]"); break; default: error = EINVAL; break; } closef(fp); } if (error == 0) error = uiomove(bp, len, ap->a_uio); if (pown) procfs_proc_unlock(pown); if (path) free(path, M_TEMP); return error; } int procfs_getpages(void *v) { struct vop_getpages_args /* { struct vnode *a_vp; voff_t a_offset; struct vm_page **a_m; int *a_count; int a_centeridx; vm_prot_t a_access_type; int a_advice; int a_flags; } */ *ap = v; if ((ap->a_flags & PGO_LOCKED) == 0) rw_exit(ap->a_vp->v_uobj.vmobjlock); return (EFAULT); } /* * convert decimal ascii to int */ static int atoi(const char *b, size_t len) { int p = 0; while (len--) { char c = *b++; if (c < '0' || c > '9') return -1; p = 10 * p + (c - '0'); } return p; } /** * convert DTYPE_XXX to corresponding DT_XXX * matching what procfs_loadvnode() does. */ static uint8_t fttodt(file_t *fp) { switch (fp->f_type) { case DTYPE_VNODE: switch (fp->f_vnode->v_type) { case VREG: return DT_REG; case VDIR: return DT_LNK; /* symlink */ case VBLK: return DT_BLK; case VCHR: return DT_CHR; case VLNK: return DT_LNK; case VSOCK: return DT_SOCK; case VFIFO: return DT_FIFO; default: return DT_UNKNOWN; } case DTYPE_PIPE: return DT_FIFO; case DTYPE_SOCKET: return DT_SOCK; case DTYPE_KQUEUE: /*FALLTHROUGH*/ case DTYPE_MISC: /*FALLTHROUGH*/ case DTYPE_SEM: return DT_LNK; /* symlinks */ default: return DT_UNKNOWN; } } |
| 8 8 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 197 198 199 200 201 202 203 204 205 206 207 208 209 210 211 212 213 214 215 216 217 218 219 220 221 222 223 224 225 226 227 228 229 230 231 232 233 234 235 236 237 238 239 240 241 242 243 244 245 246 247 248 249 250 251 252 253 254 255 256 257 258 259 260 261 262 263 264 265 266 267 268 269 270 271 272 273 274 275 276 277 278 279 280 281 282 283 284 285 286 287 288 289 290 291 292 293 294 295 296 297 298 299 300 301 302 303 304 305 306 307 308 309 310 311 312 313 314 315 316 317 318 319 320 321 322 323 324 325 326 327 328 329 330 331 332 333 334 335 336 337 338 339 340 341 342 343 344 345 346 347 348 349 350 351 352 353 354 355 356 357 358 359 360 361 362 363 364 365 366 367 368 369 370 371 372 373 374 375 376 377 378 379 380 381 382 383 384 385 386 387 388 389 390 391 392 393 394 395 396 397 398 399 400 401 402 403 404 405 406 407 408 409 410 411 412 413 414 415 416 417 418 419 420 421 422 423 424 425 426 427 428 429 430 431 432 433 434 435 436 437 438 439 440 441 442 443 444 445 446 447 448 449 450 451 452 453 454 455 456 457 458 459 460 461 | /* $NetBSD: uep.c,v 1.25 2021/08/07 16:19:17 thorpej Exp $ */ /* * Copyright (c) 2004 The NetBSD Foundation, Inc. * All rights reserved. * * This code is derived from software contributed to The NetBSD Foundation * by Tyler C. Sarna (tsarna@netbsd.org). * * Redistribution and use in source and binary forms, with or without * modification, are permitted provided that the following conditions * are met: * 1. Redistributions of source code must retain the above copyright * notice, this list of conditions and the following disclaimer. * 2. Redistributions in binary form must reproduce the above copyright * notice, this list of conditions and the following disclaimer in the * documentation and/or other materials provided with the distribution. * * THIS SOFTWARE IS PROVIDED BY THE NETBSD FOUNDATION, INC. AND CONTRIBUTORS * ``AS IS'' AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED * TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR * PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE FOUNDATION OR CONTRIBUTORS * BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR * CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF * SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS * INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN * CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) * ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE * POSSIBILITY OF SUCH DAMAGE. */ /* * eGalax USB touchpanel controller driver. */ #include <sys/cdefs.h> __KERNEL_RCSID(0, "$NetBSD: uep.c,v 1.25 2021/08/07 16:19:17 thorpej Exp $"); #include <sys/param.h> #include <sys/systm.h> #include <sys/kernel.h> #include <sys/kmem.h> #include <sys/device.h> #include <sys/ioctl.h> #include <sys/vnode.h> #include <dev/usb/usb.h> #include <dev/usb/usbdi.h> #include <dev/usb/usbdi_util.h> #include <dev/usb/usbdevs.h> #include <dev/usb/usb_quirks.h> #include <dev/wscons/wsconsio.h> #include <dev/wscons/wsmousevar.h> #include <dev/wscons/tpcalibvar.h> #define UIDSTR "eGalax USB SN000000" /* calibration - integer values, perhaps sysctls? */ #define X_RATIO 293 #define X_OFFSET -28 #define Y_RATIO -348 #define Y_OFFSET 537 /* an X_RATIO of ``312'' means : reduce by a factor 3.12 x axis amplitude */ /* an Y_RATIO of ``-157'' means : reduce by a factor 1.57 y axis amplitude, * and reverse y motion */ struct uep_softc { device_t sc_dev; struct usbd_device *sc_udev; /* device */ struct usbd_interface *sc_iface; /* interface */ int sc_iface_number; int sc_intr_number; /* interrupt number */ struct usbd_pipe * sc_intr_pipe; /* interrupt pipe */ u_char *sc_ibuf; int sc_isize; device_t sc_wsmousedev; /* wsmouse device */ struct tpcalib_softc sc_tpcalib; /* calibration */ u_char sc_enabled; u_char sc_dying; }; static struct wsmouse_calibcoords default_calib = { .minx = 0, .miny = 0, .maxx = 2047, .maxy = 2047, .samplelen = WSMOUSE_CALIBCOORDS_RESET, }; Static void uep_intr(struct usbd_xfer *, void *, usbd_status); Static int uep_enable(void *); Static void uep_disable(void *); Static int uep_ioctl(void *, u_long, void *, int, struct lwp *); static const struct wsmouse_accessops uep_accessops = { uep_enable, uep_ioctl, uep_disable, }; static int uep_match(device_t, cfdata_t, void *); static void uep_attach(device_t, device_t, void *); static void uep_childdet(device_t, device_t); static int uep_detach(device_t, int); static int uep_activate(device_t, enum devact); CFATTACH_DECL2_NEW(uep, sizeof(struct uep_softc), uep_match, uep_attach, uep_detach, uep_activate, NULL, uep_childdet); static int uep_match(device_t parent, cfdata_t match, void *aux) { struct usb_attach_arg *uaa = aux; if ((uaa->uaa_vendor == USB_VENDOR_EGALAX) && ( (uaa->uaa_product == USB_PRODUCT_EGALAX_TPANEL) || (uaa->uaa_product == USB_PRODUCT_EGALAX_TPANEL2))) return UMATCH_VENDOR_PRODUCT; if ((uaa->uaa_vendor == USB_VENDOR_EGALAX2) && (uaa->uaa_product == USB_PRODUCT_EGALAX2_TPANEL)) return UMATCH_VENDOR_PRODUCT; return UMATCH_NONE; } static void uep_attach(device_t parent, device_t self, void *aux) { struct uep_softc *sc = device_private(self); struct usb_attach_arg *uaa = aux; struct usbd_device *dev = uaa->uaa_device; usb_config_descriptor_t *cdesc; usb_interface_descriptor_t *id; usb_endpoint_descriptor_t *ed; usb_device_request_t req; uByte act; struct wsmousedev_attach_args a; char *devinfop; usbd_status err; int i; sc->sc_dev = self; aprint_naive("\n"); aprint_normal("\n"); devinfop = usbd_devinfo_alloc(dev, 0); aprint_normal_dev(self, "%s\n", devinfop); usbd_devinfo_free(devinfop); sc->sc_udev = dev; sc->sc_intr_number = -1; sc->sc_intr_pipe = NULL; sc->sc_enabled = sc->sc_isize = 0; /* Move the device into the configured state. */ err = usbd_set_config_index(dev, 0, 1); if (err) { aprint_error("\n%s: failed to set configuration, err=%s\n", device_xname(sc->sc_dev), usbd_errstr(err)); sc->sc_dying = 1; return; } /* get the config descriptor */ cdesc = usbd_get_config_descriptor(sc->sc_udev); if (cdesc == NULL) { aprint_error_dev(self, "failed to get configuration descriptor\n"); sc->sc_dying = 1; return; } /* get the interface */ err = usbd_device2interface_handle(dev, 0, &sc->sc_iface); if (err) { aprint_error("\n%s: failed to get interface, err=%s\n", device_xname(sc->sc_dev), usbd_errstr(err)); sc->sc_dying = 1; return; } /* Find the interrupt endpoint */ id = usbd_get_interface_descriptor(sc->sc_iface); sc->sc_iface_number = id->bInterfaceNumber; for (i = 0; i < id->bNumEndpoints; i++) { ed = usbd_interface2endpoint_descriptor(sc->sc_iface, i); if (ed == NULL) { aprint_error_dev(self, "no endpoint descriptor for %d\n", i); sc->sc_dying = 1; return; } if (UE_GET_DIR(ed->bEndpointAddress) == UE_DIR_IN && UE_GET_XFERTYPE(ed->bmAttributes) == UE_INTERRUPT) { sc->sc_intr_number = ed->bEndpointAddress; sc->sc_isize = UGETW(ed->wMaxPacketSize); } } if (sc->sc_intr_number== -1) { aprint_error_dev(self, "Could not find interrupt in\n"); sc->sc_dying = 1; return; } /* Newer controllers need an activation command */ req.bmRequestType = UT_WRITE_VENDOR_DEVICE; req.bRequest = 0x0a; USETW(req.wValue, 'A'); USETW(req.wIndex, 0); USETW(req.wLength, 1); usbd_do_request(dev, &req, &act); usbd_add_drv_event(USB_EVENT_DRIVER_ATTACH, sc->sc_udev, sc->sc_dev); a.accessops = &uep_accessops; a.accesscookie = sc; sc->sc_wsmousedev = config_found(self, &a, wsmousedevprint, CFARGS_NONE); tpcalib_init(&sc->sc_tpcalib); tpcalib_ioctl(&sc->sc_tpcalib, WSMOUSEIO_SCALIBCOORDS, (void *)&default_calib, 0, 0); return; } static int uep_detach(device_t self, int flags) { struct uep_softc *sc = device_private(self); int rv = 0; if (sc->sc_intr_pipe != NULL) { usbd_abort_pipe(sc->sc_intr_pipe); usbd_close_pipe(sc->sc_intr_pipe); sc->sc_intr_pipe = NULL; } sc->sc_dying = 1; /* save current calib as defaults */ default_calib = sc->sc_tpcalib.sc_saved; if (sc->sc_wsmousedev != NULL) rv = config_detach(sc->sc_wsmousedev, flags); usbd_add_drv_event(USB_EVENT_DRIVER_DETACH, sc->sc_udev, sc->sc_dev); return rv; } static void uep_childdet(device_t self, device_t child) { struct uep_softc *sc = device_private(self); KASSERT(sc->sc_wsmousedev == child); sc->sc_wsmousedev = NULL; } static int uep_activate(device_t self, enum devact act) { struct uep_softc *sc = device_private(self); switch (act) { case DVACT_DEACTIVATE: sc->sc_dying = 1; return 0; default: return EOPNOTSUPP; } } Static int uep_enable(void *v) { struct uep_softc *sc = v; int err; if (sc->sc_dying) return EIO; if (sc->sc_enabled) return EBUSY; if (sc->sc_isize == 0) return 0; sc->sc_ibuf = kmem_alloc(sc->sc_isize, KM_SLEEP); err = usbd_open_pipe_intr(sc->sc_iface, sc->sc_intr_number, USBD_SHORT_XFER_OK, &sc->sc_intr_pipe, sc, sc->sc_ibuf, sc->sc_isize, uep_intr, USBD_DEFAULT_INTERVAL); if (err) { kmem_free(sc->sc_ibuf, sc->sc_isize); sc->sc_intr_pipe = NULL; return EIO; } sc->sc_enabled = 1; return 0; } Static void uep_disable(void *v) { struct uep_softc *sc = v; if (!sc->sc_enabled) { printf("uep_disable: already disabled!\n"); return; } /* Disable interrupts. */ if (sc->sc_intr_pipe != NULL) { usbd_abort_pipe(sc->sc_intr_pipe); usbd_close_pipe(sc->sc_intr_pipe); sc->sc_intr_pipe = NULL; } if (sc->sc_ibuf != NULL) { kmem_free(sc->sc_ibuf, sc->sc_isize); sc->sc_ibuf = NULL; } sc->sc_enabled = 0; } Static int uep_ioctl(void *v, u_long cmd, void *data, int flag, struct lwp *l) { struct uep_softc *sc = v; struct wsmouse_id *id; switch (cmd) { case WSMOUSEIO_GTYPE: *(u_int *)data = WSMOUSE_TYPE_TPANEL; return 0; case WSMOUSEIO_GETID: /* * return unique ID string * "<vendor> <model> <serial number>" * unfortunately we have no serial number... */ id = (struct wsmouse_id *)data; if (id->type != WSMOUSE_ID_TYPE_UIDSTR) return EINVAL; strcpy(id->data, UIDSTR); id->length = strlen(UIDSTR); return 0; case WSMOUSEIO_SCALIBCOORDS: case WSMOUSEIO_GCALIBCOORDS: return tpcalib_ioctl(&sc->sc_tpcalib, cmd, data, flag, l); } return EPASSTHROUGH; } static int uep_adjust(int v, int off, int rat) { int num = 100 * v; int quot = num / rat; int rem = num % rat; if (num >= 0 && rem < 0) quot++; return quot + off; } void uep_intr(struct usbd_xfer *xfer, void *addr, usbd_status status) { struct uep_softc *sc = addr; u_char *p = sc->sc_ibuf; u_char msk; uint32_t len; int x = 0, y = 0, s; usbd_get_xfer_status(xfer, NULL, NULL, &len, NULL); if (status == USBD_CANCELLED) return; if (status != USBD_NORMAL_COMPLETION) { aprint_error_dev(sc->sc_dev, "status %d\n", status); usbd_clear_endpoint_stall_async(sc->sc_intr_pipe); return; } /* First bit is always set to 1 */ if ((p[0] & 0x80) != 0x80) { aprint_error_dev(sc->sc_dev, "bad input packet format\n"); return; } if (sc->sc_wsmousedev != NULL) { /* * Each report package may contain 5 or 6 bytes as below: * * Byte 0 1ZM00HLT * Byte 1 0AAAAAAA * Byte 2 0AAAAAAA * Byte 3 0BBBBBBB * Byte 4 0BBBBBBB * Byte 5 0PPPPPPP * * Z: 1=byte 5 is pressure information, 0=no pressure * M: 1=byte 5 is play id, 0=no player id * T: 1=touched, 0=not touched * H,L: Resolution * 0,0: 11 bits * 0,1: 12 bits * 1,0: 13 bits * 1,1: 14 bits * A: bits of axis A position, MSB to LSB * B: bits of axis B position, MSB to LSB * * The packet has six bytes only if Z or M is set. * Byte 5, if sent, is ignored. * * For the unit I have, A = Y and B = X. * I don't know if units exist with A=X and B=Y, * if so we'll cross that bridge when we come to it. * * The controller sends a stream of T=1 events while the * panel is touched, followed by a single T=0 event. */ switch (p[0] & 0x06) { case 0x02: msk = 0x1f; break; case 0x04: msk = 0x3f; break; case 0x06: msk = 0x7f; break; default: msk = 0x0f; /* H=0, L=0 */ } x = uep_adjust(((p[3] & msk) << 7) | p[4], X_OFFSET, X_RATIO); y = uep_adjust(((p[1] & msk) << 7) | p[2], Y_OFFSET, Y_RATIO); tpcalib_trans(&sc->sc_tpcalib, x, y, &x, &y); s = spltty(); wsmouse_input(sc->sc_wsmousedev, p[0] & 0x01, x, y, 0, 0, WSMOUSE_INPUT_ABSOLUTE_X | WSMOUSE_INPUT_ABSOLUTE_Y); splx(s); } } |
| 2 2 1 2 2 2 2 2 2 2 2 2 2 2 1 1 1 1 1 1 1 2 2 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 2 2 1 2 1 2 1 1 1 2 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 197 198 199 200 201 202 203 204 205 206 207 208 209 210 211 212 213 214 215 216 217 218 219 220 221 222 223 224 225 226 227 228 229 230 231 232 233 234 235 236 237 238 239 240 241 242 243 244 245 246 247 248 249 250 251 252 253 254 255 256 257 258 259 260 261 262 263 264 265 266 267 268 269 270 271 272 273 274 275 276 277 278 279 280 281 282 283 284 285 286 287 288 289 290 291 292 293 294 295 296 297 298 299 300 301 302 303 304 305 306 307 308 309 310 311 312 313 314 315 316 317 318 319 320 321 322 323 324 325 326 327 328 329 330 331 332 333 334 335 336 337 338 339 340 341 342 343 344 345 346 347 348 349 350 351 352 353 354 355 356 357 358 359 360 361 362 363 364 365 366 367 368 369 370 371 372 373 374 375 376 377 378 379 380 381 382 383 384 385 386 387 388 389 390 391 392 393 394 395 396 397 398 399 400 401 402 403 404 405 406 407 408 409 410 411 412 413 414 415 416 417 418 419 420 421 422 423 424 425 426 427 428 429 430 431 432 433 434 435 436 437 438 439 440 441 442 443 444 445 446 447 448 449 450 451 452 453 454 455 456 457 458 459 460 461 462 463 464 465 466 467 468 469 470 471 472 473 474 475 476 477 478 479 480 481 482 483 484 485 486 487 488 489 490 491 492 493 494 495 496 497 498 499 500 501 502 503 504 505 506 507 508 509 510 511 512 513 514 515 516 517 518 519 520 521 522 523 524 525 526 527 528 529 530 531 532 533 534 535 536 537 538 539 540 541 542 543 544 545 546 547 548 549 550 551 552 553 554 555 556 557 558 559 560 561 562 563 564 565 566 567 568 569 570 571 572 573 574 575 576 577 578 579 580 581 582 583 584 585 586 587 588 589 590 591 592 593 594 595 596 597 598 599 600 601 602 603 604 605 606 607 608 609 610 611 612 613 614 615 616 617 618 619 620 621 622 623 624 625 626 627 628 629 630 631 632 633 634 635 636 637 638 639 640 641 642 643 644 645 646 647 648 649 650 651 652 653 654 655 656 657 658 659 660 661 662 663 664 665 666 667 668 669 670 671 672 673 674 675 676 677 678 679 680 681 682 683 684 685 686 687 688 689 690 691 692 693 694 695 696 697 698 699 700 701 702 703 704 705 706 707 708 709 710 711 712 713 714 715 716 717 718 719 720 721 722 723 724 725 726 727 728 729 730 731 732 733 734 735 736 737 738 739 740 741 742 743 744 745 746 747 748 749 750 751 752 753 754 755 756 757 758 759 760 761 762 763 764 765 766 767 768 769 770 771 772 773 774 775 776 777 778 779 780 781 782 783 784 785 786 787 788 789 790 791 792 793 794 795 796 797 798 799 800 801 802 803 804 805 806 807 808 809 810 811 812 813 814 815 816 817 818 819 820 821 822 823 824 825 826 827 828 829 830 831 832 833 834 835 836 837 838 839 840 841 842 843 844 845 846 847 848 849 850 851 852 853 854 855 856 857 858 859 860 861 862 863 864 865 866 867 868 869 870 871 872 873 874 875 876 877 878 879 880 881 882 883 884 885 886 887 888 889 890 891 892 893 894 895 896 897 898 899 900 901 902 903 904 905 906 907 908 909 910 911 912 913 914 915 916 917 918 919 920 921 922 923 924 925 926 927 928 929 930 931 932 933 934 935 936 937 938 939 940 941 942 943 944 945 946 947 948 949 950 951 952 953 954 955 956 957 958 959 960 961 962 963 964 965 966 967 968 969 970 971 972 973 974 975 976 977 978 979 980 981 982 983 984 985 986 987 988 989 990 991 992 993 994 995 996 997 998 999 1000 1001 1002 1003 1004 1005 1006 1007 1008 1009 1010 1011 1012 1013 1014 1015 1016 1017 1018 1019 1020 1021 1022 1023 1024 1025 1026 1027 1028 1029 1030 1031 1032 1033 1034 1035 1036 1037 1038 1039 1040 1041 1042 1043 1044 1045 1046 1047 1048 1049 1050 1051 1052 1053 1054 1055 1056 1057 1058 1059 1060 1061 1062 1063 1064 1065 1066 1067 1068 1069 1070 1071 1072 1073 1074 1075 1076 1077 1078 1079 1080 1081 1082 1083 1084 1085 1086 1087 1088 1089 1090 1091 1092 1093 1094 1095 1096 1097 1098 1099 1100 1101 1102 1103 1104 1105 1106 1107 1108 1109 1110 1111 1112 1113 1114 1115 1116 1117 1118 1119 1120 1121 1122 1123 1124 1125 1126 1127 1128 1129 1130 1131 1132 1133 1134 1135 1136 1137 1138 1139 1140 1141 1142 1143 1144 1145 1146 1147 1148 1149 1150 1151 1152 1153 1154 1155 1156 1157 1158 1159 1160 1161 1162 1163 1164 1165 1166 1167 1168 1169 1170 1171 1172 1173 1174 1175 1176 1177 1178 1179 1180 1181 1182 1183 1184 1185 1186 1187 1188 1189 1190 1191 1192 1193 1194 1195 1196 1197 1198 1199 1200 1201 1202 1203 1204 1205 1206 1207 1208 1209 1210 1211 1212 1213 1214 1215 1216 1217 1218 1219 1220 1221 1222 1223 1224 1225 1226 1227 1228 1229 1230 1231 1232 1233 1234 1235 1236 1237 1238 1239 1240 1241 1242 1243 1244 1245 1246 1247 1248 1249 1250 1251 1252 1253 1254 1255 1256 1257 1258 1259 1260 1261 1262 1263 1264 1265 1266 1267 1268 1269 1270 1271 1272 1273 1274 1275 1276 1277 1278 1279 1280 1281 1282 1283 1284 1285 1286 1287 1288 1289 1290 1291 1292 1293 1294 1295 1296 1297 1298 1299 1300 1301 1302 1303 1304 1305 1306 1307 1308 1309 1310 1311 1312 1313 1314 1315 1316 1317 1318 1319 1320 1321 1322 1323 1324 1325 1326 1327 1328 1329 1330 1331 1332 1333 1334 1335 1336 1337 1338 1339 1340 1341 1342 1343 1344 1345 1346 1347 1348 1349 1350 1351 1352 1353 1354 1355 1356 1357 1358 1359 1360 1361 1362 1363 1364 1365 1366 1367 1368 1369 1370 1371 1372 1373 1374 1375 1376 1377 1378 1379 1380 1381 1382 1383 1384 1385 1386 1387 1388 1389 1390 1391 1392 1393 1394 1395 1396 1397 1398 1399 1400 1401 1402 1403 1404 1405 1406 1407 1408 1409 1410 1411 1412 1413 1414 1415 1416 1417 1418 1419 1420 1421 1422 1423 1424 1425 1426 1427 1428 1429 1430 1431 1432 1433 1434 1435 1436 1437 1438 1439 1440 1441 1442 1443 1444 1445 1446 1447 1448 1449 1450 1451 1452 1453 1454 1455 1456 1457 1458 1459 1460 1461 1462 1463 1464 1465 1466 1467 1468 1469 1470 1471 1472 1473 1474 1475 1476 1477 1478 1479 1480 1481 1482 1483 1484 1485 1486 1487 1488 1489 1490 1491 1492 1493 1494 1495 1496 1497 1498 1499 1500 1501 1502 1503 1504 1505 1506 1507 1508 1509 1510 1511 1512 1513 1514 1515 1516 1517 1518 1519 1520 1521 1522 1523 1524 1525 1526 1527 1528 1529 1530 1531 1532 1533 1534 1535 1536 1537 1538 1539 1540 1541 1542 1543 1544 1545 1546 1547 1548 1549 1550 1551 1552 1553 1554 1555 1556 1557 1558 1559 1560 1561 1562 1563 1564 1565 1566 1567 1568 1569 1570 1571 1572 1573 1574 1575 1576 1577 1578 1579 1580 1581 1582 1583 1584 1585 1586 1587 1588 1589 1590 1591 1592 1593 1594 1595 1596 1597 1598 1599 1600 1601 1602 1603 1604 1605 1606 1607 1608 1609 1610 1611 1612 1613 1614 1615 1616 1617 1618 1619 1620 1621 1622 1623 1624 1625 1626 1627 1628 1629 1630 1631 1632 1633 1634 1635 1636 1637 1638 1639 1640 1641 1642 1643 1644 1645 1646 1647 1648 1649 1650 1651 1652 1653 1654 1655 1656 1657 1658 1659 1660 1661 1662 1663 1664 1665 1666 1667 1668 1669 1670 1671 1672 1673 1674 1675 1676 1677 1678 1679 1680 1681 1682 1683 1684 1685 1686 1687 1688 1689 1690 1691 1692 1693 1694 1695 1696 1697 1698 1699 1700 1701 1702 1703 1704 1705 1706 1707 1708 1709 1710 1711 1712 1713 1714 1715 1716 1717 1718 1719 1720 1721 1722 1723 1724 1725 1726 1727 1728 1729 1730 1731 1732 1733 1734 1735 1736 1737 1738 1739 1740 1741 1742 1743 1744 1745 1746 1747 1748 1749 1750 1751 1752 1753 1754 1755 1756 1757 1758 1759 1760 1761 1762 1763 1764 1765 1766 1767 1768 1769 1770 1771 1772 1773 1774 1775 1776 1777 1778 1779 1780 1781 1782 1783 1784 1785 1786 1787 1788 1789 1790 1791 1792 1793 1794 1795 1796 1797 1798 1799 1800 1801 1802 1803 1804 1805 1806 1807 1808 1809 1810 1811 1812 1813 1814 1815 1816 1817 1818 1819 1820 1821 1822 1823 1824 1825 1826 1827 1828 1829 1830 1831 1832 1833 1834 1835 1836 1837 1838 1839 1840 1841 1842 1843 1844 1845 1846 1847 1848 1849 1850 1851 1852 1853 1854 1855 1856 1857 1858 1859 1860 1861 1862 1863 1864 1865 1866 1867 1868 1869 1870 1871 1872 1873 1874 1875 1876 1877 1878 1879 1880 1881 1882 1883 1884 1885 1886 1887 1888 1889 1890 1891 1892 1893 1894 1895 1896 1897 1898 1899 1900 1901 1902 1903 1904 1905 1906 1907 1908 1909 1910 1911 1912 1913 1914 1915 1916 1917 1918 1919 1920 1921 1922 1923 1924 1925 1926 1927 1928 1929 1930 1931 1932 1933 1934 1935 1936 1937 1938 1939 1940 1941 1942 1943 1944 1945 1946 1947 1948 1949 1950 1951 1952 1953 1954 1955 1956 1957 1958 1959 1960 1961 1962 1963 1964 1965 1966 1967 1968 1969 1970 1971 1972 1973 1974 1975 1976 1977 1978 1979 1980 1981 1982 1983 1984 1985 1986 1987 1988 1989 1990 1991 1992 1993 1994 1995 1996 1997 1998 1999 2000 2001 2002 2003 2004 2005 2006 2007 2008 2009 2010 2011 2012 2013 2014 2015 2016 2017 2018 2019 2020 2021 2022 2023 2024 2025 2026 2027 2028 2029 2030 2031 2032 2033 2034 2035 2036 2037 2038 2039 2040 2041 2042 2043 2044 2045 2046 2047 2048 2049 2050 2051 2052 2053 2054 2055 2056 2057 2058 2059 2060 2061 2062 2063 2064 2065 2066 2067 2068 2069 2070 2071 2072 2073 2074 2075 2076 2077 2078 2079 2080 2081 2082 2083 2084 2085 2086 2087 2088 2089 2090 2091 2092 2093 2094 2095 2096 2097 2098 2099 2100 2101 2102 2103 2104 2105 2106 2107 2108 2109 2110 2111 2112 2113 2114 2115 2116 2117 2118 2119 2120 2121 2122 2123 2124 2125 2126 2127 2128 2129 2130 2131 2132 2133 2134 2135 2136 2137 2138 2139 2140 2141 2142 2143 2144 2145 2146 2147 2148 2149 2150 2151 2152 2153 2154 2155 2156 2157 2158 2159 2160 2161 2162 2163 2164 2165 2166 2167 2168 2169 2170 2171 2172 2173 | /* $NetBSD: nvpair.c,v 1.11 2019/07/24 14:25:56 martin Exp $ */ /*- * SPDX-License-Identifier: BSD-2-Clause-FreeBSD * * Copyright (c) 2009-2013 The FreeBSD Foundation * Copyright (c) 2013-2015 Mariusz Zaborski <oshogbo@FreeBSD.org> * All rights reserved. * * This software was developed by Pawel Jakub Dawidek under sponsorship from * the FreeBSD Foundation. * * Redistribution and use in source and binary forms, with or without * modification, are permitted provided that the following conditions * are met: * 1. Redistributions of source code must retain the above copyright * notice, this list of conditions and the following disclaimer. * 2. Redistributions in binary form must reproduce the above copyright * notice, this list of conditions and the following disclaimer in the * documentation and/or other materials provided with the distribution. * * THIS SOFTWARE IS PROVIDED BY THE AUTHORS AND CONTRIBUTORS ``AS IS'' AND * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE * ARE DISCLAIMED. IN NO EVENT SHALL THE AUTHORS OR CONTRIBUTORS BE LIABLE * FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL * DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS * OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) * HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT * LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY * OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF * SUCH DAMAGE. */ #include <sys/cdefs.h> #ifdef __FreeBSD__ __FBSDID("$FreeBSD: head/sys/contrib/libnv/nvpair.c 335382 2018-06-19 18:43:02Z lwhsu $"); #else __RCSID("$NetBSD: nvpair.c,v 1.11 2019/07/24 14:25:56 martin Exp $"); #endif #include <sys/param.h> #include <sys/endian.h> #include <sys/queue.h> #if defined(_KERNEL) || defined(_STANDALONE) #include <sys/errno.h> #include <sys/lock.h> #include <sys/malloc.h> #include <sys/systm.h> #include <sys/kmem.h> #ifdef __FreeBSD__ #include <machine/stdarg.h> #endif #else #include <errno.h> #include <fcntl.h> #include <stdarg.h> #include <stdbool.h> #include <stdint.h> #include <stdlib.h> #include <string.h> #include <unistd.h> #include "common_impl.h" #endif #ifdef HAVE_PJDLOG #include <pjdlog.h> #endif #ifdef __FreeBSD__ #include <sys/nv.h> #else #include "nv.h" #endif #include "nv_impl.h" #include "nvlist_impl.h" #include "nvpair_impl.h" #ifndef HAVE_PJDLOG #if defined(_KERNEL) || defined(_STANDALONE) #ifdef __FreeBSD__ #define PJDLOG_ASSERT(...) MPASS(__VA_ARGS__) #else #define PJDLOG_ASSERT(...) KASSERT(__VA_ARGS__) #endif #define PJDLOG_RASSERT(expr, ...) KASSERT(expr, (__VA_ARGS__)) #define PJDLOG_ABORT(...) panic(__VA_ARGS__) #else #ifndef __lint__ #include <assert.h> #define PJDLOG_ASSERT(...) assert(__VA_ARGS__) #define PJDLOG_RASSERT(expr, ...) assert(expr) #define PJDLOG_ABORT(...) abort() #else #define PJDLOG_ASSERT(...) #define PJDLOG_RASSERT(expr, ...) #define PJDLOG_ABORT(...) #endif #endif #endif #define NVPAIR_MAGIC 0x6e7670 /* "nvp" */ struct nvpair { int nvp_magic; char *nvp_name; int nvp_type; uint64_t nvp_data; size_t nvp_datasize; size_t nvp_nitems; /* Used only for array types. */ nvlist_t *nvp_list; TAILQ_ENTRY(nvpair) nvp_next; }; #define NVPAIR_ASSERT(nvp) do { \ PJDLOG_ASSERT((nvp) != NULL); \ PJDLOG_ASSERT((nvp)->nvp_magic == NVPAIR_MAGIC); \ } while (/*CONSTCOND*/0) struct nvpair_header { uint8_t nvph_type; uint16_t nvph_namesize; uint64_t nvph_datasize; uint64_t nvph_nitems; } __packed; void nvpair_assert(const nvpair_t *nvp) { NVPAIR_ASSERT(nvp); } static nvpair_t * nvpair_allocv(const char *name, int type, uint64_t data, size_t datasize, size_t nitems) { nvpair_t *nvp; size_t namelen; PJDLOG_ASSERT(type >= NV_TYPE_FIRST && type <= NV_TYPE_LAST); namelen = strlen(name); if (namelen >= NV_NAME_MAX) { ERRNO_SET(ENAMETOOLONG); return (NULL); } nvp = nv_calloc(1, sizeof(*nvp) + namelen + 1); if (nvp != NULL) { nvp->nvp_name = (char *)(nvp + 1); memcpy(nvp->nvp_name, name, namelen); nvp->nvp_name[namelen] = '\0'; nvp->nvp_type = type; nvp->nvp_data = data; nvp->nvp_datasize = datasize; nvp->nvp_nitems = nitems; nvp->nvp_magic = NVPAIR_MAGIC; } return (nvp); } static int nvpair_append(nvpair_t *nvp, const void *value, size_t valsize, size_t datasize) { void *olddata, *data, *valp; size_t oldlen; oldlen = nvp->nvp_nitems * valsize; olddata = (void *)(uintptr_t)nvp->nvp_data; data = nv_realloc(olddata, oldlen + valsize); if (data == NULL) { ERRNO_SET(ENOMEM); return (-1); } valp = (unsigned char *)data + oldlen; memcpy(valp, value, valsize); nvp->nvp_data = (uint64_t)(uintptr_t)data; nvp->nvp_datasize += datasize; nvp->nvp_nitems++; return (0); } nvlist_t * nvpair_nvlist(const nvpair_t *nvp) { NVPAIR_ASSERT(nvp); return (nvp->nvp_list); } nvpair_t * nvpair_next(const nvpair_t *nvp) { NVPAIR_ASSERT(nvp); PJDLOG_ASSERT(nvp->nvp_list != NULL); return (TAILQ_NEXT(nvp, nvp_next)); } nvpair_t * nvpair_prev(const nvpair_t *nvp) { NVPAIR_ASSERT(nvp); PJDLOG_ASSERT(nvp->nvp_list != NULL); return (TAILQ_PREV(nvp, nvl_head, nvp_next)); } void nvpair_insert(struct nvl_head *head, nvpair_t *nvp, nvlist_t *nvl) { NVPAIR_ASSERT(nvp); PJDLOG_ASSERT(nvp->nvp_list == NULL); PJDLOG_ASSERT((nvlist_flags(nvl) & NV_FLAG_NO_UNIQUE) != 0 || !nvlist_exists(nvl, nvpair_name(nvp))); TAILQ_INSERT_TAIL(head, nvp, nvp_next); nvp->nvp_list = nvl; } static void nvpair_remove_nvlist(nvpair_t *nvp) { nvlist_t *nvl; /* XXX: DECONST is bad, mkay? */ nvl = __DECONST(nvlist_t *, nvpair_get_nvlist(nvp)); PJDLOG_ASSERT(nvl != NULL); nvlist_set_parent(nvl, NULL); } static void nvpair_remove_nvlist_array(nvpair_t *nvp) { nvlist_t **nvlarray; size_t count, i; /* XXX: DECONST is bad, mkay? */ nvlarray = __DECONST(nvlist_t **, nvpair_get_nvlist_array(nvp, &count)); for (i = 0; i < count; i++) { nvlist_set_array_next(nvlarray[i], NULL); nvlist_set_parent(nvlarray[i], NULL); } } void nvpair_remove(struct nvl_head *head, nvpair_t *nvp, const nvlist_t *nvl) { NVPAIR_ASSERT(nvp); PJDLOG_ASSERT(nvp->nvp_list == nvl); if (nvpair_type(nvp) == NV_TYPE_NVLIST) nvpair_remove_nvlist(nvp); else if (nvpair_type(nvp) == NV_TYPE_NVLIST_ARRAY) nvpair_remove_nvlist_array(nvp); TAILQ_REMOVE(head, nvp, nvp_next); nvp->nvp_list = NULL; } nvpair_t * nvpair_clone(const nvpair_t *nvp) { nvpair_t *newnvp; const char *name; const void *data; size_t datasize; NVPAIR_ASSERT(nvp); name = nvpair_name(nvp); switch (nvpair_type(nvp)) { case NV_TYPE_NULL: newnvp = nvpair_create_null(name); break; case NV_TYPE_BOOL: newnvp = nvpair_create_bool(name, nvpair_get_bool(nvp)); break; case NV_TYPE_NUMBER: newnvp = nvpair_create_number(name, nvpair_get_number(nvp)); break; case NV_TYPE_STRING: newnvp = nvpair_create_string(name, nvpair_get_string(nvp)); break; case NV_TYPE_NVLIST: newnvp = nvpair_create_nvlist(name, nvpair_get_nvlist(nvp)); break; case NV_TYPE_BINARY: data = nvpair_get_binary(nvp, &datasize); newnvp = nvpair_create_binary(name, data, datasize); break; case NV_TYPE_BOOL_ARRAY: data = nvpair_get_bool_array(nvp, &datasize); newnvp = nvpair_create_bool_array(name, data, datasize); break; case NV_TYPE_NUMBER_ARRAY: data = nvpair_get_number_array(nvp, &datasize); newnvp = nvpair_create_number_array(name, data, datasize); break; case NV_TYPE_STRING_ARRAY: data = nvpair_get_string_array(nvp, &datasize); newnvp = nvpair_create_string_array(name, data, datasize); break; case NV_TYPE_NVLIST_ARRAY: data = nvpair_get_nvlist_array(nvp, &datasize); newnvp = nvpair_create_nvlist_array(name, data, datasize); break; #if !defined(_KERNEL) && !defined(_STANDALONE) case NV_TYPE_DESCRIPTOR: newnvp = nvpair_create_descriptor(name, nvpair_get_descriptor(nvp)); break; case NV_TYPE_DESCRIPTOR_ARRAY: data = nvpair_get_descriptor_array(nvp, &datasize); newnvp = nvpair_create_descriptor_array(name, data, datasize); break; #endif default: PJDLOG_ABORT("Unknown type: %d.", nvpair_type(nvp)); } return (newnvp); } size_t nvpair_header_size(void) { return (sizeof(struct nvpair_header)); } size_t nvpair_size(const nvpair_t *nvp) { NVPAIR_ASSERT(nvp); return (nvp->nvp_datasize); } unsigned char * nvpair_pack_header(const nvpair_t *nvp, unsigned char *ptr, size_t *leftp) { struct nvpair_header nvphdr; size_t namesize; NVPAIR_ASSERT(nvp); nvphdr.nvph_type = nvp->nvp_type; namesize = strlen(nvp->nvp_name) + 1; PJDLOG_ASSERT(namesize > 0 && namesize <= UINT16_MAX); nvphdr.nvph_namesize = namesize; nvphdr.nvph_datasize = nvp->nvp_datasize; nvphdr.nvph_nitems = nvp->nvp_nitems; PJDLOG_ASSERT(*leftp >= sizeof(nvphdr)); memcpy(ptr, &nvphdr, sizeof(nvphdr)); ptr += sizeof(nvphdr); *leftp -= sizeof(nvphdr); PJDLOG_ASSERT(*leftp >= namesize); memcpy(ptr, nvp->nvp_name, namesize); ptr += namesize; *leftp -= namesize; return (ptr); } unsigned char * nvpair_pack_null(const nvpair_t *nvp, unsigned char *ptr, size_t *leftp __unused) { NVPAIR_ASSERT(nvp); PJDLOG_ASSERT(nvp->nvp_type == NV_TYPE_NULL); return (ptr); } unsigned char * nvpair_pack_bool(const nvpair_t *nvp, unsigned char *ptr, size_t *leftp) { uint8_t value; NVPAIR_ASSERT(nvp); PJDLOG_ASSERT(nvp->nvp_type == NV_TYPE_BOOL); value = (uint8_t)nvp->nvp_data; PJDLOG_ASSERT(*leftp >= sizeof(value)); memcpy(ptr, &value, sizeof(value)); ptr += sizeof(value); *leftp -= sizeof(value); return (ptr); } unsigned char * nvpair_pack_number(const nvpair_t *nvp, unsigned char *ptr, size_t *leftp) { uint64_t value; NVPAIR_ASSERT(nvp); PJDLOG_ASSERT(nvp->nvp_type == NV_TYPE_NUMBER); value = (uint64_t)nvp->nvp_data; PJDLOG_ASSERT(*leftp >= sizeof(value)); memcpy(ptr, &value, sizeof(value)); ptr += sizeof(value); *leftp -= sizeof(value); return (ptr); } unsigned char * nvpair_pack_string(const nvpair_t *nvp, unsigned char *ptr, size_t *leftp) { NVPAIR_ASSERT(nvp); PJDLOG_ASSERT(nvp->nvp_type == NV_TYPE_STRING); PJDLOG_ASSERT(*leftp >= nvp->nvp_datasize); memcpy(ptr, (const void *)(intptr_t)nvp->nvp_data, nvp->nvp_datasize); ptr += nvp->nvp_datasize; *leftp -= nvp->nvp_datasize; return (ptr); } unsigned char * nvpair_pack_nvlist_up(unsigned char *ptr, size_t *leftp) { struct nvpair_header nvphdr; size_t namesize; const char *name = ""; namesize = 1; nvphdr.nvph_type = NV_TYPE_NVLIST_UP; nvphdr.nvph_namesize = namesize; nvphdr.nvph_datasize = 0; nvphdr.nvph_nitems = 0; PJDLOG_ASSERT(*leftp >= sizeof(nvphdr)); memcpy(ptr, &nvphdr, sizeof(nvphdr)); ptr += sizeof(nvphdr); *leftp -= sizeof(nvphdr); PJDLOG_ASSERT(*leftp >= namesize); memcpy(ptr, name, namesize); ptr += namesize; *leftp -= namesize; return (ptr); } unsigned char * nvpair_pack_nvlist_array_next(unsigned char *ptr, size_t *leftp) { struct nvpair_header nvphdr; size_t namesize; const char *name = ""; namesize = 1; nvphdr.nvph_type = NV_TYPE_NVLIST_ARRAY_NEXT; nvphdr.nvph_namesize = namesize; nvphdr.nvph_datasize = 0; nvphdr.nvph_nitems = 0; PJDLOG_ASSERT(*leftp >= sizeof(nvphdr)); memcpy(ptr, &nvphdr, sizeof(nvphdr)); ptr += sizeof(nvphdr); *leftp -= sizeof(nvphdr); PJDLOG_ASSERT(*leftp >= namesize); memcpy(ptr, name, namesize); ptr += namesize; *leftp -= namesize; return (ptr); } #if !defined(_KERNEL) && !defined(_STANDALONE) unsigned char * nvpair_pack_descriptor(const nvpair_t *nvp, unsigned char *ptr, int64_t *fdidxp, size_t *leftp) { int64_t value; NVPAIR_ASSERT(nvp); PJDLOG_ASSERT(nvp->nvp_type == NV_TYPE_DESCRIPTOR); value = (int64_t)nvp->nvp_data; if (value != -1) { /* * If there is a real descriptor here, we change its number * to position in the array of descriptors send via control * message. */ PJDLOG_ASSERT(fdidxp != NULL); value = *fdidxp; (*fdidxp)++; } PJDLOG_ASSERT(*leftp >= sizeof(value)); memcpy(ptr, &value, sizeof(value)); ptr += sizeof(value); *leftp -= sizeof(value); return (ptr); } #endif unsigned char * nvpair_pack_binary(const nvpair_t *nvp, unsigned char *ptr, size_t *leftp) { NVPAIR_ASSERT(nvp); PJDLOG_ASSERT(nvp->nvp_type == NV_TYPE_BINARY); PJDLOG_ASSERT(*leftp >= nvp->nvp_datasize); memcpy(ptr, (const void *)(intptr_t)nvp->nvp_data, nvp->nvp_datasize); ptr += nvp->nvp_datasize; *leftp -= nvp->nvp_datasize; return (ptr); } unsigned char * nvpair_pack_bool_array(const nvpair_t *nvp, unsigned char *ptr, size_t *leftp) { NVPAIR_ASSERT(nvp); PJDLOG_ASSERT(nvp->nvp_type == NV_TYPE_BOOL_ARRAY); PJDLOG_ASSERT(*leftp >= nvp->nvp_datasize); memcpy(ptr, (const void *)(intptr_t)nvp->nvp_data, nvp->nvp_datasize); ptr += nvp->nvp_datasize; *leftp -= nvp->nvp_datasize; return (ptr); } unsigned char * nvpair_pack_number_array(const nvpair_t *nvp, unsigned char *ptr, size_t *leftp) { NVPAIR_ASSERT(nvp); PJDLOG_ASSERT(nvp->nvp_type == NV_TYPE_NUMBER_ARRAY); PJDLOG_ASSERT(*leftp >= nvp->nvp_datasize); memcpy(ptr, (const void *)(intptr_t)nvp->nvp_data, nvp->nvp_datasize); ptr += nvp->nvp_datasize; *leftp -= nvp->nvp_datasize; return (ptr); } unsigned char * nvpair_pack_string_array(const nvpair_t *nvp, unsigned char *ptr, size_t *leftp) { unsigned int ii; size_t size, len; const char * const *array; NVPAIR_ASSERT(nvp); PJDLOG_ASSERT(nvp->nvp_type == NV_TYPE_STRING_ARRAY); PJDLOG_ASSERT(*leftp >= nvp->nvp_datasize); size = 0; array = nvpair_get_string_array(nvp, NULL); PJDLOG_ASSERT(array != NULL); for (ii = 0; ii < nvp->nvp_nitems; ii++) { len = strlen(array[ii]) + 1; PJDLOG_ASSERT(*leftp >= len); memcpy(ptr, (const void *)array[ii], len); size += len; ptr += len; *leftp -= len; } PJDLOG_ASSERT(size == nvp->nvp_datasize); return (ptr); } #if !defined(_KERNEL) && !defined(_STANDALONE) unsigned char * nvpair_pack_descriptor_array(const nvpair_t *nvp, unsigned char *ptr, int64_t *fdidxp, size_t *leftp) { int64_t value; const int *array; unsigned int ii; NVPAIR_ASSERT(nvp); PJDLOG_ASSERT(nvp->nvp_type == NV_TYPE_DESCRIPTOR_ARRAY); PJDLOG_ASSERT(*leftp >= nvp->nvp_datasize); array = nvpair_get_descriptor_array(nvp, NULL); PJDLOG_ASSERT(array != NULL); for (ii = 0; ii < nvp->nvp_nitems; ii++) { PJDLOG_ASSERT(*leftp >= sizeof(value)); value = array[ii]; if (value != -1) { /* * If there is a real descriptor here, we change its * number to position in the array of descriptors send * via control message. */ PJDLOG_ASSERT(fdidxp != NULL); value = *fdidxp; (*fdidxp)++; } memcpy(ptr, &value, sizeof(value)); ptr += sizeof(value); *leftp -= sizeof(value); } return (ptr); } #endif void nvpair_init_datasize(nvpair_t *nvp) { NVPAIR_ASSERT(nvp); if (nvp->nvp_type == NV_TYPE_NVLIST) { if (nvp->nvp_data == 0) { nvp->nvp_datasize = 0; } else { nvp->nvp_datasize = nvlist_size((const nvlist_t *)(intptr_t)nvp->nvp_data); } } } const unsigned char * nvpair_unpack_header(bool isbe, nvpair_t *nvp, const unsigned char *ptr, size_t *leftp) { struct nvpair_header nvphdr; if (*leftp < sizeof(nvphdr)) goto fail; memcpy(&nvphdr, ptr, sizeof(nvphdr)); ptr += sizeof(nvphdr); *leftp -= sizeof(nvphdr); #if NV_TYPE_FIRST > 0 if (nvphdr.nvph_type < NV_TYPE_FIRST) goto fail; #endif if (nvphdr.nvph_type > NV_TYPE_LAST && nvphdr.nvph_type != NV_TYPE_NVLIST_UP && nvphdr.nvph_type != NV_TYPE_NVLIST_ARRAY_NEXT) { goto fail; } #if BYTE_ORDER == BIG_ENDIAN if (!isbe) { nvphdr.nvph_namesize = le16toh(nvphdr.nvph_namesize); nvphdr.nvph_datasize = le64toh(nvphdr.nvph_datasize); } #else if (isbe) { nvphdr.nvph_namesize = be16toh(nvphdr.nvph_namesize); nvphdr.nvph_datasize = be64toh(nvphdr.nvph_datasize); } #endif if (nvphdr.nvph_namesize > NV_NAME_MAX) goto fail; if (*leftp < nvphdr.nvph_namesize) goto fail; if (nvphdr.nvph_namesize < 1) goto fail; if (strnlen((const char *)ptr, nvphdr.nvph_namesize) != (size_t)(nvphdr.nvph_namesize - 1)) { goto fail; } memcpy(nvp->nvp_name, ptr, nvphdr.nvph_namesize); ptr += nvphdr.nvph_namesize; *leftp -= nvphdr.nvph_namesize; if (*leftp < nvphdr.nvph_datasize) goto fail; nvp->nvp_type = nvphdr.nvph_type; nvp->nvp_data = 0; nvp->nvp_datasize = nvphdr.nvph_datasize; nvp->nvp_nitems = nvphdr.nvph_nitems; return (ptr); fail: ERRNO_SET(EINVAL); return (NULL); } const unsigned char * nvpair_unpack_null(bool isbe __unused, nvpair_t *nvp, const unsigned char *ptr, size_t *leftp __unused) { PJDLOG_ASSERT(nvp->nvp_type == NV_TYPE_NULL); if (nvp->nvp_datasize != 0) { ERRNO_SET(EINVAL); return (NULL); } return (ptr); } const unsigned char * nvpair_unpack_bool(bool isbe __unused, nvpair_t *nvp, const unsigned char *ptr, size_t *leftp) { uint8_t value; PJDLOG_ASSERT(nvp->nvp_type == NV_TYPE_BOOL); if (nvp->nvp_datasize != sizeof(value)) { ERRNO_SET(EINVAL); return (NULL); } if (*leftp < sizeof(value)) { ERRNO_SET(EINVAL); return (NULL); } memcpy(&value, ptr, sizeof(value)); ptr += sizeof(value); *leftp -= sizeof(value); if (value != 0 && value != 1) { ERRNO_SET(EINVAL); return (NULL); } nvp->nvp_data = (uint64_t)value; return (ptr); } const unsigned char * nvpair_unpack_number(bool isbe, nvpair_t *nvp, const unsigned char *ptr, size_t *leftp) { PJDLOG_ASSERT(nvp->nvp_type == NV_TYPE_NUMBER); if (nvp->nvp_datasize != sizeof(uint64_t)) { ERRNO_SET(EINVAL); return (NULL); } if (*leftp < sizeof(uint64_t)) { ERRNO_SET(EINVAL); return (NULL); } if (isbe) nvp->nvp_data = be64dec(ptr); else nvp->nvp_data = le64dec(ptr); ptr += sizeof(uint64_t); *leftp -= sizeof(uint64_t); return (ptr); } const unsigned char * nvpair_unpack_string(bool isbe __unused, nvpair_t *nvp, const unsigned char *ptr, size_t *leftp) { PJDLOG_ASSERT(nvp->nvp_type == NV_TYPE_STRING); if (*leftp < nvp->nvp_datasize || nvp->nvp_datasize == 0) { ERRNO_SET(EINVAL); return (NULL); } if (strnlen((const char *)ptr, nvp->nvp_datasize) != nvp->nvp_datasize - 1) { ERRNO_SET(EINVAL); return (NULL); } nvp->nvp_data = (uint64_t)(uintptr_t)nv_strdup((const char *)ptr); if (nvp->nvp_data == 0) return (NULL); ptr += nvp->nvp_datasize; *leftp -= nvp->nvp_datasize; return (ptr); } const unsigned char * nvpair_unpack_nvlist(bool isbe __unused, nvpair_t *nvp, const unsigned char *ptr, size_t *leftp, size_t nfds, nvlist_t **child) { nvlist_t *value; PJDLOG_ASSERT(nvp->nvp_type == NV_TYPE_NVLIST); if (*leftp < nvp->nvp_datasize || nvp->nvp_datasize == 0) { ERRNO_SET(EINVAL); return (NULL); } value = nvlist_create(0); if (value == NULL) return (NULL); ptr = nvlist_unpack_header(value, ptr, nfds, NULL, leftp); if (ptr == NULL) return (NULL); nvp->nvp_data = (uint64_t)(uintptr_t)value; *child = value; return (ptr); } #if !defined(_KERNEL) && !defined(_STANDALONE) const unsigned char * nvpair_unpack_descriptor(bool isbe, nvpair_t *nvp, const unsigned char *ptr, size_t *leftp, const int *fds, size_t nfds) { int64_t idx; PJDLOG_ASSERT(nvp->nvp_type == NV_TYPE_DESCRIPTOR); if (nvp->nvp_datasize != sizeof(idx)) { ERRNO_SET(EINVAL); return (NULL); } if (*leftp < sizeof(idx)) { ERRNO_SET(EINVAL); return (NULL); } if (isbe) idx = be64dec(ptr); else idx = le64dec(ptr); if (idx < 0) { ERRNO_SET(EINVAL); return (NULL); } if ((size_t)idx >= nfds) { ERRNO_SET(EINVAL); return (NULL); } nvp->nvp_data = (uint64_t)fds[idx]; ptr += sizeof(idx); *leftp -= sizeof(idx); return (ptr); } #endif const unsigned char * nvpair_unpack_binary(bool isbe __unused, nvpair_t *nvp, const unsigned char *ptr, size_t *leftp) { void *value; PJDLOG_ASSERT(nvp->nvp_type == NV_TYPE_BINARY); if (*leftp < nvp->nvp_datasize || nvp->nvp_datasize == 0) { ERRNO_SET(EINVAL); return (NULL); } value = nv_malloc(nvp->nvp_datasize); if (value == NULL) return (NULL); memcpy(value, ptr, nvp->nvp_datasize); ptr += nvp->nvp_datasize; *leftp -= nvp->nvp_datasize; nvp->nvp_data = (uint64_t)(uintptr_t)value; return (ptr); } const unsigned char * nvpair_unpack_bool_array(bool isbe __unused, nvpair_t *nvp, const unsigned char *ptr, size_t *leftp) { uint8_t *value; size_t size; unsigned int i; PJDLOG_ASSERT(nvp->nvp_type == NV_TYPE_BOOL_ARRAY); size = sizeof(*value) * nvp->nvp_nitems; if (nvp->nvp_datasize != size || *leftp < size || nvp->nvp_nitems == 0 || size < nvp->nvp_nitems) { ERRNO_SET(EINVAL); return (NULL); } value = nv_malloc(size); if (value == NULL) return (NULL); for (i = 0; i < nvp->nvp_nitems; i++) { value[i] = *(const uint8_t *)ptr; ptr += sizeof(*value); *leftp -= sizeof(*value); } nvp->nvp_data = (uint64_t)(uintptr_t)value; return (ptr); } const unsigned char * nvpair_unpack_number_array(bool isbe, nvpair_t *nvp, const unsigned char *ptr, size_t *leftp) { uint64_t *value; size_t size; unsigned int i; PJDLOG_ASSERT(nvp->nvp_type == NV_TYPE_NUMBER_ARRAY); size = sizeof(*value) * nvp->nvp_nitems; if (nvp->nvp_datasize != size || *leftp < size || nvp->nvp_nitems == 0 || size < nvp->nvp_nitems) { ERRNO_SET(EINVAL); return (NULL); } value = nv_malloc(size); if (value == NULL) return (NULL); for (i = 0; i < nvp->nvp_nitems; i++) { if (isbe) value[i] = be64dec(ptr); else value[i] = le64dec(ptr); ptr += sizeof(*value); *leftp -= sizeof(*value); } nvp->nvp_data = (uint64_t)(uintptr_t)value; return (ptr); } const unsigned char * nvpair_unpack_string_array(bool isbe __unused, nvpair_t *nvp, const unsigned char *ptr, size_t *leftp) { ssize_t size; size_t len; const char *tmp; char **value; unsigned int ii, j; PJDLOG_ASSERT(nvp->nvp_type == NV_TYPE_STRING_ARRAY); if (*leftp < nvp->nvp_datasize || nvp->nvp_datasize == 0 || nvp->nvp_nitems == 0) { ERRNO_SET(EINVAL); return (NULL); } size = nvp->nvp_datasize; tmp = (const char *)ptr; for (ii = 0; ii < nvp->nvp_nitems; ii++) { len = strnlen(tmp, size - 1) + 1; size -= len; if (size < 0) { ERRNO_SET(EINVAL); return (NULL); } tmp += len; } if (size != 0) { ERRNO_SET(EINVAL); return (NULL); } value = nv_malloc(sizeof(*value) * nvp->nvp_nitems); if (value == NULL) return (NULL); for (ii = 0; ii < nvp->nvp_nitems; ii++) { value[ii] = nv_strdup((const char *)ptr); if (value[ii] == NULL) goto out; len = strlen(value[ii]) + 1; ptr += len; *leftp -= len; } nvp->nvp_data = (uint64_t)(uintptr_t)value; return (ptr); out: for (j = 0; j < ii; j++) nv_free(value[j]); nv_free(value); return (NULL); } #if !defined(_KERNEL) && !defined(_STANDALONE) && !defined(__NetBSD__) const unsigned char * nvpair_unpack_descriptor_array(bool isbe, nvpair_t *nvp, const unsigned char *ptr, size_t *leftp, const int *fds, size_t nfds) { int64_t idx; size_t size; unsigned int ii; int *array; PJDLOG_ASSERT(nvp->nvp_type == NV_TYPE_DESCRIPTOR_ARRAY); size = sizeof(idx) * nvp->nvp_nitems; if (nvp->nvp_datasize != size || *leftp < size || nvp->nvp_nitems == 0 || size < nvp->nvp_nitems) { ERRNO_SET(EINVAL); return (NULL); } array = (int *)nv_malloc(size); if (array == NULL) return (NULL); for (ii = 0; ii < nvp->nvp_nitems; ii++) { if (isbe) idx = be64dec(ptr); else idx = le64dec(ptr); if (idx < 0) { ERRNO_SET(EINVAL); nv_free(array); return (NULL); } if ((size_t)idx >= nfds) { ERRNO_SET(EINVAL); nv_free(array); return (NULL); } array[ii] = (uint64_t)fds[idx]; ptr += sizeof(idx); *leftp -= sizeof(idx); } nvp->nvp_data = (uint64_t)(uintptr_t)array; return (ptr); } #endif const unsigned char * nvpair_unpack_nvlist_array(bool isbe __unused, nvpair_t *nvp, const unsigned char *ptr, size_t *leftp, nvlist_t **firstel) { nvlist_t **value; nvpair_t *tmpnvp; unsigned int ii, j; size_t sizeup; PJDLOG_ASSERT(nvp->nvp_type == NV_TYPE_NVLIST_ARRAY); sizeup = sizeof(struct nvpair_header) * nvp->nvp_nitems; if (nvp->nvp_nitems == 0 || sizeup < nvp->nvp_nitems || sizeup > *leftp) { ERRNO_SET(EINVAL); return (NULL); } value = nv_malloc(nvp->nvp_nitems * sizeof(*value)); if (value == NULL) return (NULL); for (ii = 0; ii < nvp->nvp_nitems; ii++) { value[ii] = nvlist_create(0); if (value[ii] == NULL) goto fail; if (ii > 0) { tmpnvp = nvpair_allocv(" ", NV_TYPE_NVLIST, (uint64_t)(uintptr_t)value[ii], 0, 0); if (tmpnvp == NULL) goto fail; nvlist_set_array_next(value[ii - 1], tmpnvp); } } nvlist_set_flags(value[nvp->nvp_nitems - 1], NV_FLAG_IN_ARRAY); nvp->nvp_data = (uint64_t)(uintptr_t)value; *firstel = value[0]; return (ptr); fail: ERRNO_SAVE(); for (j = 0; j <= ii; j++) nvlist_destroy(value[j]); nv_free(value); ERRNO_RESTORE(); return (NULL); } const unsigned char * nvpair_unpack(bool isbe, const unsigned char *ptr, size_t *leftp, nvpair_t **nvpp) { nvpair_t *nvp, *tmp; nvp = nv_calloc(1, sizeof(*nvp) + NV_NAME_MAX); if (nvp == NULL) return (NULL); nvp->nvp_name = (char *)(nvp + 1); ptr = nvpair_unpack_header(isbe, nvp, ptr, leftp); if (ptr == NULL) goto fail; tmp = nv_realloc(nvp, sizeof(*nvp) + strlen(nvp->nvp_name) + 1); if (tmp == NULL) goto fail; nvp = tmp; /* Update nvp_name after realloc(). */ nvp->nvp_name = (char *)(nvp + 1); nvp->nvp_data = 0x00; nvp->nvp_magic = NVPAIR_MAGIC; *nvpp = nvp; return (ptr); fail: nv_free(nvp); return (NULL); } int nvpair_type(const nvpair_t *nvp) { NVPAIR_ASSERT(nvp); return (nvp->nvp_type); } const char * nvpair_name(const nvpair_t *nvp) { NVPAIR_ASSERT(nvp); return (nvp->nvp_name); } #if !defined(_STANDALONE) nvpair_t * nvpair_create_stringf(const char *name, const char *valuefmt, ...) { va_list valueap; nvpair_t *nvp; va_start(valueap, valuefmt); nvp = nvpair_create_stringv(name, valuefmt, valueap); va_end(valueap); return (nvp); } nvpair_t * nvpair_create_stringv(const char *name, const char *valuefmt, va_list valueap) { nvpair_t *nvp; char *str; int len; len = vasprintf(&str, valuefmt, valueap); if (len < 0) return (NULL); nvp = nvpair_create_string(name, str); nv_kmem_free(str, len+1); return (nvp); } #endif nvpair_t * nvpair_create_null(const char *name) { return (nvpair_allocv(name, NV_TYPE_NULL, 0, 0, 0)); } nvpair_t * nvpair_create_bool(const char *name, bool value) { return (nvpair_allocv(name, NV_TYPE_BOOL, value ? 1 : 0, sizeof(uint8_t), 0)); } nvpair_t * nvpair_create_number(const char *name, uint64_t value) { return (nvpair_allocv(name, NV_TYPE_NUMBER, value, sizeof(value), 0)); } nvpair_t * nvpair_create_string(const char *name, const char *value) { nvpair_t *nvp; size_t size; char *data; if (value == NULL) { ERRNO_SET(EINVAL); return (NULL); } data = nv_strdup(value); if (data == NULL) return (NULL); size = strlen(value) + 1; nvp = nvpair_allocv(name, NV_TYPE_STRING, (uint64_t)(uintptr_t)data, size, 0); if (nvp == NULL) nv_free(data); return (nvp); } nvpair_t * nvpair_create_nvlist(const char *name, const nvlist_t *value) { nvlist_t *nvl; nvpair_t *nvp; if (value == NULL) { ERRNO_SET(EINVAL); return (NULL); } nvl = nvlist_clone(value); if (nvl == NULL) return (NULL); nvp = nvpair_allocv(name, NV_TYPE_NVLIST, (uint64_t)(uintptr_t)nvl, 0, 0); if (nvp == NULL) nvlist_destroy(nvl); else nvlist_set_parent(nvl, nvp); return (nvp); } #if !defined(_KERNEL) && !defined(_STANDALONE) nvpair_t * nvpair_create_descriptor(const char *name, int value) { nvpair_t *nvp; if (value < 0 || !fd_is_valid(value)) { ERRNO_SET(EBADF); return (NULL); } value = fcntl(value, F_DUPFD_CLOEXEC, 0); if (value < 0) return (NULL); nvp = nvpair_allocv(name, NV_TYPE_DESCRIPTOR, (uint64_t)value, sizeof(int64_t), 0); if (nvp == NULL) { ERRNO_SAVE(); close(value); ERRNO_RESTORE(); } return (nvp); } #endif nvpair_t * nvpair_create_binary(const char *name, const void *value, size_t size) { nvpair_t *nvp; void *data; if (value == NULL || size == 0) { ERRNO_SET(EINVAL); return (NULL); } data = nv_malloc(size); if (data == NULL) return (NULL); memcpy(data, value, size); nvp = nvpair_allocv(name, NV_TYPE_BINARY, (uint64_t)(uintptr_t)data, size, 0); if (nvp == NULL) nv_free(data); return (nvp); } nvpair_t * nvpair_create_bool_array(const char *name, const bool *value, size_t nitems) { nvpair_t *nvp; size_t size; void *data; if (value == NULL || nitems == 0) { ERRNO_SET(EINVAL); return (NULL); } size = sizeof(value[0]) * nitems; data = nv_malloc(size); if (data == NULL) return (NULL); memcpy(data, value, size); nvp = nvpair_allocv(name, NV_TYPE_BOOL_ARRAY, (uint64_t)(uintptr_t)data, size, nitems); if (nvp == NULL) { ERRNO_SAVE(); nv_free(data); ERRNO_RESTORE(); } return (nvp); } nvpair_t * nvpair_create_number_array(const char *name, const uint64_t *value, size_t nitems) { nvpair_t *nvp; size_t size; void *data; if (value == NULL || nitems == 0) { ERRNO_SET(EINVAL); return (NULL); } size = sizeof(value[0]) * nitems; data = nv_malloc(size); if (data == NULL) return (NULL); memcpy(data, value, size); nvp = nvpair_allocv(name, NV_TYPE_NUMBER_ARRAY, (uint64_t)(uintptr_t)data, size, nitems); if (nvp == NULL) { ERRNO_SAVE(); nv_free(data); ERRNO_RESTORE(); } return (nvp); } nvpair_t * nvpair_create_string_array(const char *name, const char * const *value, size_t nitems) { nvpair_t *nvp; unsigned int ii; size_t datasize, size; char **data; if (value == NULL || nitems == 0) { ERRNO_SET(EINVAL); return (NULL); } nvp = NULL; datasize = 0; data = nv_malloc(sizeof(value[0]) * nitems); if (data == NULL) return (NULL); for (ii = 0; ii < nitems; ii++) { if (value[ii] == NULL) { ERRNO_SET(EINVAL); goto fail; } size = strlen(value[ii]) + 1; datasize += size; data[ii] = nv_strdup(value[ii]); if (data[ii] == NULL) goto fail; } nvp = nvpair_allocv(name, NV_TYPE_STRING_ARRAY, (uint64_t)(uintptr_t)data, datasize, nitems); fail: if (nvp == NULL) { ERRNO_SAVE(); for (; ii > 0; ii--) nv_free(data[ii - 1]); nv_free(data); ERRNO_RESTORE(); } return (nvp); } nvpair_t * nvpair_create_nvlist_array(const char *name, const nvlist_t * const *value, size_t nitems) { unsigned int ii; nvlist_t **nvls; nvpair_t *parent; int flags; nvls = NULL; if (value == NULL || nitems == 0) { ERRNO_SET(EINVAL); return (NULL); } nvls = nv_malloc(sizeof(value[0]) * nitems); if (nvls == NULL) return (NULL); for (ii = 0; ii < nitems; ii++) { if (value[ii] == NULL) { ERRNO_SET(EINVAL); goto fail; } nvls[ii] = nvlist_clone(value[ii]); if (nvls[ii] == NULL) goto fail; if (ii > 0) { nvpair_t *nvp; nvp = nvpair_allocv(" ", NV_TYPE_NVLIST, (uint64_t)(uintptr_t)nvls[ii], 0, 0); if (nvp == NULL) { ERRNO_SAVE(); nvlist_destroy(nvls[ii]); ERRNO_RESTORE(); goto fail; } nvlist_set_array_next(nvls[ii - 1], nvp); } } flags = nvlist_flags(nvls[nitems - 1]) | NV_FLAG_IN_ARRAY; nvlist_set_flags(nvls[nitems - 1], flags); parent = nvpair_allocv(name, NV_TYPE_NVLIST_ARRAY, (uint64_t)(uintptr_t)nvls, 0, nitems); if (parent == NULL) goto fail; for (ii = 0; ii < nitems; ii++) nvlist_set_parent(nvls[ii], parent); return (parent); fail: ERRNO_SAVE(); for (; ii > 0; ii--) nvlist_destroy(nvls[ii - 1]); nv_free(nvls); ERRNO_RESTORE(); return (NULL); } #if !defined(_KERNEL) && !defined(_STANDALONE) nvpair_t * nvpair_create_descriptor_array(const char *name, const int *value, size_t nitems) { unsigned int ii; nvpair_t *nvp; int *fds; if (value == NULL) { ERRNO_SET(EINVAL); return (NULL); } nvp = NULL; fds = nv_malloc(sizeof(value[0]) * nitems); if (fds == NULL) return (NULL); for (ii = 0; ii < nitems; ii++) { if (value[ii] == -1) { fds[ii] = -1; } else { if (!fd_is_valid(value[ii])) { ERRNO_SET(EBADF); goto fail; } fds[ii] = fcntl(value[ii], F_DUPFD_CLOEXEC, 0); if (fds[ii] == -1) goto fail; } } nvp = nvpair_allocv(name, NV_TYPE_DESCRIPTOR_ARRAY, (uint64_t)(uintptr_t)fds, sizeof(int64_t) * nitems, nitems); fail: if (nvp == NULL) { ERRNO_SAVE(); for (; ii > 0; ii--) { if (fds[ii - 1] != -1) close(fds[ii - 1]); } nv_free(fds); ERRNO_RESTORE(); } return (nvp); } #endif nvpair_t * nvpair_move_string(const char *name, char *value) { nvpair_t *nvp; if (value == NULL) { ERRNO_SET(EINVAL); return (NULL); } nvp = nvpair_allocv(name, NV_TYPE_STRING, (uint64_t)(uintptr_t)value, strlen(value) + 1, 0); if (nvp == NULL) { ERRNO_SAVE(); nv_free(value); ERRNO_RESTORE(); } return (nvp); } nvpair_t * nvpair_move_nvlist(const char *name, nvlist_t *value) { nvpair_t *nvp; if (value == NULL || nvlist_get_nvpair_parent(value) != NULL) { ERRNO_SET(EINVAL); return (NULL); } if (nvlist_error(value) != 0) { ERRNO_SET(nvlist_error(value)); nvlist_destroy(value); return (NULL); } nvp = nvpair_allocv(name, NV_TYPE_NVLIST, (uint64_t)(uintptr_t)value, 0, 0); if (nvp == NULL) nvlist_destroy(value); else nvlist_set_parent(value, nvp); return (nvp); } #if !defined(_KERNEL) && !defined(_STANDALONE) nvpair_t * nvpair_move_descriptor(const char *name, int value) { nvpair_t *nvp; if (value < 0 || !fd_is_valid(value)) { ERRNO_SET(EBADF); return (NULL); } nvp = nvpair_allocv(name, NV_TYPE_DESCRIPTOR, (uint64_t)value, sizeof(int64_t), 0); if (nvp == NULL) { ERRNO_SAVE(); close(value); ERRNO_RESTORE(); } return (nvp); } #endif nvpair_t * nvpair_move_binary(const char *name, void *value, size_t size) { nvpair_t *nvp; if (value == NULL || size == 0) { ERRNO_SET(EINVAL); return (NULL); } nvp = nvpair_allocv(name, NV_TYPE_BINARY, (uint64_t)(uintptr_t)value, size, 0); if (nvp == NULL) { ERRNO_SAVE(); nv_free(value); ERRNO_RESTORE(); } return (nvp); } nvpair_t * nvpair_move_bool_array(const char *name, bool *value, size_t nitems) { nvpair_t *nvp; if (value == NULL || nitems == 0) { ERRNO_SET(EINVAL); return (NULL); } nvp = nvpair_allocv(name, NV_TYPE_BOOL_ARRAY, (uint64_t)(uintptr_t)value, sizeof(value[0]) * nitems, nitems); if (nvp == NULL) { ERRNO_SAVE(); nv_free(value); ERRNO_RESTORE(); } return (nvp); } nvpair_t * nvpair_move_string_array(const char *name, char **value, size_t nitems) { nvpair_t *nvp; size_t i, size; if (value == NULL || nitems == 0) { ERRNO_SET(EINVAL); return (NULL); } size = 0; for (i = 0; i < nitems; i++) { if (value[i] == NULL) { ERRNO_SET(EINVAL); return (NULL); } size += strlen(value[i]) + 1; } nvp = nvpair_allocv(name, NV_TYPE_STRING_ARRAY, (uint64_t)(uintptr_t)value, size, nitems); if (nvp == NULL) { ERRNO_SAVE(); for (i = 0; i < nitems; i++) nv_free(value[i]); nv_free(value); ERRNO_RESTORE(); } return (nvp); } nvpair_t * nvpair_move_number_array(const char *name, uint64_t *value, size_t nitems) { nvpair_t *nvp; if (value == NULL || nitems == 0) { ERRNO_SET(EINVAL); return (NULL); } nvp = nvpair_allocv(name, NV_TYPE_NUMBER_ARRAY, (uint64_t)(uintptr_t)value, sizeof(value[0]) * nitems, nitems); if (nvp == NULL) { ERRNO_SAVE(); nv_free(value); ERRNO_RESTORE(); } return (nvp); } nvpair_t * nvpair_move_nvlist_array(const char *name, nvlist_t **value, size_t nitems) { nvpair_t *parent; unsigned int ii; int flags; if (value == NULL || nitems == 0) { ERRNO_SET(EINVAL); return (NULL); } for (ii = 0; ii < nitems; ii++) { if (value == NULL || nvlist_error(value[ii]) != 0 || nvlist_get_pararr(value[ii], NULL) != NULL) { ERRNO_SET(EINVAL); goto fail; } if (ii > 0) { nvpair_t *nvp; nvp = nvpair_allocv(" ", NV_TYPE_NVLIST, (uint64_t)(uintptr_t)value[ii], 0, 0); if (nvp == NULL) goto fail; nvlist_set_array_next(value[ii - 1], nvp); } } flags = nvlist_flags(value[nitems - 1]) | NV_FLAG_IN_ARRAY; nvlist_set_flags(value[nitems - 1], flags); parent = nvpair_allocv(name, NV_TYPE_NVLIST_ARRAY, (uint64_t)(uintptr_t)value, 0, nitems); if (parent == NULL) goto fail; for (ii = 0; ii < nitems; ii++) nvlist_set_parent(value[ii], parent); return (parent); fail: ERRNO_SAVE(); for (ii = 0; ii < nitems; ii++) { if (value[ii] != NULL && nvlist_get_pararr(value[ii], NULL) != NULL) { nvlist_destroy(value[ii]); } } nv_free(value); ERRNO_RESTORE(); return (NULL); } #if !defined(_KERNEL) && !defined(_STANDALONE) nvpair_t * nvpair_move_descriptor_array(const char *name, int *value, size_t nitems) { nvpair_t *nvp; size_t i; if (value == NULL || nitems == 0) { ERRNO_SET(EINVAL); return (NULL); } for (i = 0; i < nitems; i++) { if (value[i] != -1 && !fd_is_valid(value[i])) { ERRNO_SET(EBADF); goto fail; } } nvp = nvpair_allocv(name, NV_TYPE_DESCRIPTOR_ARRAY, (uint64_t)(uintptr_t)value, sizeof(value[0]) * nitems, nitems); if (nvp == NULL) goto fail; return (nvp); fail: ERRNO_SAVE(); for (i = 0; i < nitems; i++) { if (fd_is_valid(value[i])) close(value[i]); } nv_free(value); ERRNO_RESTORE(); return (NULL); } #endif bool nvpair_get_bool(const nvpair_t *nvp) { NVPAIR_ASSERT(nvp); return (nvp->nvp_data == 1); } uint64_t nvpair_get_number(const nvpair_t *nvp) { NVPAIR_ASSERT(nvp); return (nvp->nvp_data); } const char * nvpair_get_string(const nvpair_t *nvp) { NVPAIR_ASSERT(nvp); PJDLOG_ASSERT(nvp->nvp_type == NV_TYPE_STRING); return ((const char *)(intptr_t)nvp->nvp_data); } const nvlist_t * nvpair_get_nvlist(const nvpair_t *nvp) { NVPAIR_ASSERT(nvp); PJDLOG_ASSERT(nvp->nvp_type == NV_TYPE_NVLIST); return ((const nvlist_t *)(intptr_t)nvp->nvp_data); } #if !defined(_KERNEL) && !defined(_STANDALONE) int nvpair_get_descriptor(const nvpair_t *nvp) { NVPAIR_ASSERT(nvp); PJDLOG_ASSERT(nvp->nvp_type == NV_TYPE_DESCRIPTOR); return ((int)nvp->nvp_data); } #endif const void * nvpair_get_binary(const nvpair_t *nvp, size_t *sizep) { NVPAIR_ASSERT(nvp); PJDLOG_ASSERT(nvp->nvp_type == NV_TYPE_BINARY); if (sizep != NULL) *sizep = nvp->nvp_datasize; return ((const void *)(intptr_t)nvp->nvp_data); } const bool * nvpair_get_bool_array(const nvpair_t *nvp, size_t *nitems) { NVPAIR_ASSERT(nvp); PJDLOG_ASSERT(nvp->nvp_type == NV_TYPE_BOOL_ARRAY); if (nitems != NULL) *nitems = nvp->nvp_nitems; return ((const bool *)(intptr_t)nvp->nvp_data); } const uint64_t * nvpair_get_number_array(const nvpair_t *nvp, size_t *nitems) { NVPAIR_ASSERT(nvp); PJDLOG_ASSERT(nvp->nvp_type == NV_TYPE_NUMBER_ARRAY); if (nitems != NULL) *nitems = nvp->nvp_nitems; return ((const uint64_t *)(intptr_t)nvp->nvp_data); } const char * const * nvpair_get_string_array(const nvpair_t *nvp, size_t *nitems) { NVPAIR_ASSERT(nvp); PJDLOG_ASSERT(nvp->nvp_type == NV_TYPE_STRING_ARRAY); if (nitems != NULL) *nitems = nvp->nvp_nitems; return ((const char * const *)(intptr_t)nvp->nvp_data); } const nvlist_t * const * nvpair_get_nvlist_array(const nvpair_t *nvp, size_t *nitems) { NVPAIR_ASSERT(nvp); PJDLOG_ASSERT(nvp->nvp_type == NV_TYPE_NVLIST_ARRAY); if (nitems != NULL) *nitems = nvp->nvp_nitems; return ((const nvlist_t * const *)((intptr_t)nvp->nvp_data)); } #if !defined(_KERNEL) && !defined(_STANDALONE) const int * nvpair_get_descriptor_array(const nvpair_t *nvp, size_t *nitems) { NVPAIR_ASSERT(nvp); PJDLOG_ASSERT(nvp->nvp_type == NV_TYPE_DESCRIPTOR_ARRAY); if (nitems != NULL) *nitems = nvp->nvp_nitems; return ((const int *)(intptr_t)nvp->nvp_data); } #endif int nvpair_append_bool_array(nvpair_t *nvp, const bool value) { NVPAIR_ASSERT(nvp); PJDLOG_ASSERT(nvp->nvp_type == NV_TYPE_BOOL_ARRAY); return (nvpair_append(nvp, &value, sizeof(value), sizeof(value))); } int nvpair_append_number_array(nvpair_t *nvp, const uint64_t value) { NVPAIR_ASSERT(nvp); PJDLOG_ASSERT(nvp->nvp_type == NV_TYPE_NUMBER_ARRAY); return (nvpair_append(nvp, &value, sizeof(value), sizeof(value))); } int nvpair_append_string_array(nvpair_t *nvp, const char *value) { char *str; NVPAIR_ASSERT(nvp); PJDLOG_ASSERT(nvp->nvp_type == NV_TYPE_STRING_ARRAY); if (value == NULL) { ERRNO_SET(EINVAL); return (-1); } str = nv_strdup(value); if (str == NULL) { return (-1); } if (nvpair_append(nvp, &str, sizeof(str), strlen(str) + 1) == -1) { nv_free(str); return (-1); } return (0); } int nvpair_append_nvlist_array(nvpair_t *nvp, const nvlist_t *value) { nvpair_t *tmpnvp; nvlist_t *nvl, *prev; int flags; NVPAIR_ASSERT(nvp); PJDLOG_ASSERT(nvp->nvp_type == NV_TYPE_NVLIST_ARRAY); if (value == NULL || nvlist_error(value) != 0 || nvlist_get_pararr(value, NULL) != NULL) { ERRNO_SET(EINVAL); return (-1); } nvl = nvlist_clone(value); if (nvl == NULL) { return (-1); } flags = nvlist_flags(nvl) | NV_FLAG_IN_ARRAY; nvlist_set_flags(nvl, flags); tmpnvp = NULL; prev = NULL; if (nvp->nvp_nitems > 0) { nvlist_t **nvls = (void *)(uintptr_t)nvp->nvp_data; prev = nvls[nvp->nvp_nitems - 1]; PJDLOG_ASSERT(prev != NULL); tmpnvp = nvpair_allocv(" ", NV_TYPE_NVLIST, (uint64_t)(uintptr_t)nvl, 0, 0); if (tmpnvp == NULL) { goto fail; } } if (nvpair_append(nvp, &nvl, sizeof(nvl), 0) == -1) { goto fail; } if (tmpnvp) { NVPAIR_ASSERT(tmpnvp); nvlist_set_array_next(prev, tmpnvp); } nvlist_set_parent(nvl, nvp); return (0); fail: if (tmpnvp) { nvpair_free(tmpnvp); } nvlist_destroy(nvl); return (-1); } #if !defined(_KERNEL) && !defined(_STANDALONE) int nvpair_append_descriptor_array(nvpair_t *nvp, const int value) { int fd; NVPAIR_ASSERT(nvp); PJDLOG_ASSERT(nvp->nvp_type == NV_TYPE_DESCRIPTOR_ARRAY); if (value < 0 || !fd_is_valid(value)) { ERRNO_SET(EBADF); return -1; } fd = fcntl(value, F_DUPFD_CLOEXEC, 0); if (fd == -1) { return (-1); } if (nvpair_append(nvp, &fd, sizeof(fd), sizeof(fd)) == -1) { close(fd); return (-1); } return (0); } #endif void nvpair_free(nvpair_t *nvp) { size_t i; NVPAIR_ASSERT(nvp); PJDLOG_ASSERT(nvp->nvp_list == NULL); nvp->nvp_magic = 0; switch (nvp->nvp_type) { #if !defined(_KERNEL) && !defined(_STANDALONE) case NV_TYPE_DESCRIPTOR: close((int)nvp->nvp_data); break; case NV_TYPE_DESCRIPTOR_ARRAY: for (i = 0; i < nvp->nvp_nitems; i++) close(((int *)(intptr_t)nvp->nvp_data)[i]); nv_free((int *)(intptr_t)nvp->nvp_data); break; #endif case NV_TYPE_NVLIST: nvlist_destroy((nvlist_t *)(intptr_t)nvp->nvp_data); break; case NV_TYPE_STRING: nv_free((char *)(intptr_t)nvp->nvp_data); break; case NV_TYPE_BINARY: nv_free((void *)(intptr_t)nvp->nvp_data); break; case NV_TYPE_NVLIST_ARRAY: for (i = 0; i < nvp->nvp_nitems; i++) { nvlist_destroy( ((nvlist_t **)(intptr_t)nvp->nvp_data)[i]); } nv_free(((nvlist_t **)(intptr_t)nvp->nvp_data)); break; case NV_TYPE_NUMBER_ARRAY: nv_free((uint64_t *)(intptr_t)nvp->nvp_data); break; case NV_TYPE_BOOL_ARRAY: nv_free((bool *)(intptr_t)nvp->nvp_data); break; case NV_TYPE_STRING_ARRAY: for (i = 0; i < nvp->nvp_nitems; i++) nv_free(((char **)(intptr_t)nvp->nvp_data)[i]); nv_free((char **)(intptr_t)nvp->nvp_data); break; } nv_free(nvp); } void nvpair_free_structure(nvpair_t *nvp) { NVPAIR_ASSERT(nvp); PJDLOG_ASSERT(nvp->nvp_list == NULL); nvp->nvp_magic = 0; nv_free(nvp); } const char * nvpair_type_string(int type) { switch (type) { case NV_TYPE_NULL: return ("NULL"); case NV_TYPE_BOOL: return ("BOOL"); case NV_TYPE_NUMBER: return ("NUMBER"); case NV_TYPE_STRING: return ("STRING"); case NV_TYPE_NVLIST: return ("NVLIST"); case NV_TYPE_DESCRIPTOR: return ("DESCRIPTOR"); case NV_TYPE_BINARY: return ("BINARY"); case NV_TYPE_BOOL_ARRAY: return ("BOOL ARRAY"); case NV_TYPE_NUMBER_ARRAY: return ("NUMBER ARRAY"); case NV_TYPE_STRING_ARRAY: return ("STRING ARRAY"); case NV_TYPE_NVLIST_ARRAY: return ("NVLIST ARRAY"); case NV_TYPE_DESCRIPTOR_ARRAY: return ("DESCRIPTOR ARRAY"); default: return ("<UNKNOWN>"); } } |
| 14 14 14 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 197 198 199 200 201 202 203 204 205 206 207 208 209 210 211 212 213 214 215 216 217 218 219 220 221 222 223 224 225 226 227 228 229 230 231 232 233 234 235 236 237 238 239 240 241 242 243 244 245 246 247 248 249 250 251 252 253 254 255 256 257 258 259 260 261 262 263 264 265 266 267 268 269 270 271 272 273 274 275 276 277 278 279 280 281 282 283 284 285 286 287 288 289 290 291 292 293 294 295 296 297 298 299 300 301 302 303 304 305 306 307 308 309 310 311 312 313 314 315 316 317 318 319 320 321 322 323 324 325 326 327 328 329 330 331 332 333 334 335 336 337 338 339 340 341 342 343 344 345 346 347 348 349 350 351 352 353 354 355 356 357 358 359 360 361 362 363 364 365 366 367 368 369 370 371 372 373 374 375 376 377 378 | /* $NetBSD: pcppi.c,v 1.47 2021/08/07 16:19:12 thorpej Exp $ */ /* * Copyright (c) 1996 Carnegie-Mellon University. * All rights reserved. * * Author: Chris G. Demetriou * * Permission to use, copy, modify and distribute this software and * its documentation is hereby granted, provided that both the copyright * notice and this permission notice appear in all copies of the * software, derivative works or modified versions, and any portions * thereof, and that both notices appear in supporting documentation. * * CARNEGIE MELLON ALLOWS FREE USE OF THIS SOFTWARE IN ITS "AS IS" * CONDITION. CARNEGIE MELLON DISCLAIMS ANY LIABILITY OF ANY KIND * FOR ANY DAMAGES WHATSOEVER RESULTING FROM THE USE OF THIS SOFTWARE. * * Carnegie Mellon requests users of this software to return to * * Software Distribution Coordinator or Software.Distribution@CS.CMU.EDU * School of Computer Science * Carnegie Mellon University * Pittsburgh PA 15213-3890 * * any improvements or extensions that they make and grant Carnegie the * rights to redistribute these changes. */ #include <sys/cdefs.h> __KERNEL_RCSID(0, "$NetBSD: pcppi.c,v 1.47 2021/08/07 16:19:12 thorpej Exp $"); #include "attimer.h" #include <sys/param.h> #include <sys/systm.h> #include <sys/callout.h> #include <sys/kernel.h> #include <sys/proc.h> #include <sys/device.h> #include <sys/errno.h> #include <sys/bus.h> #include <sys/mutex.h> #include <sys/condvar.h> #include <sys/tty.h> #include <dev/ic/attimervar.h> #include <dev/isa/isareg.h> #include <dev/isa/isavar.h> #include <dev/isa/pcppireg.h> #include <dev/isa/pcppivar.h> #include "pckbd.h" #if NPCKBD > 0 #include <dev/pckbport/pckbdvar.h> void pcppi_pckbd_bell(void *, u_int, u_int, u_int, int); #endif int pcppi_match(device_t, cfdata_t, void *); void pcppi_isa_attach(device_t, device_t, void *); void pcppi_childdet(device_t, device_t); int pcppi_rescan(device_t, const char *, const int *); CFATTACH_DECL3_NEW(pcppi, sizeof(struct pcppi_softc), pcppi_match, pcppi_isa_attach, pcppi_detach, NULL, pcppi_rescan, pcppi_childdet, DVF_DETACH_SHUTDOWN); static int pcppisearch(device_t, cfdata_t, const int *, void *); static void pcppi_bell_stop(struct pcppi_softc *); static void pcppi_bell_callout(void *); #if NATTIMER > 0 static void pcppi_attach_speaker(device_t); static void pcppi_detach_speaker(struct pcppi_softc *); #endif int pcppi_match(device_t parent, cfdata_t match, void *aux) { struct isa_attach_args *ia = aux; bus_space_handle_t ppi_ioh; int have_ppi, rv; u_int8_t v, nv; if (ISA_DIRECT_CONFIG(ia)) return (0); /* If values are hardwired to something that they can't be, punt. */ if (ia->ia_nio < 1 || (ia->ia_io[0].ir_addr != ISA_UNKNOWN_PORT && ia->ia_io[0].ir_addr != IO_PPI)) return (0); if (ia->ia_niomem > 0 && (ia->ia_iomem[0].ir_addr != ISA_UNKNOWN_IOMEM)) return (0); if (ia->ia_nirq > 0 && (ia->ia_irq[0].ir_irq != ISA_UNKNOWN_IRQ)) return (0); if (ia->ia_ndrq > 0 && (ia->ia_drq[0].ir_drq != ISA_UNKNOWN_DRQ)) return (0); rv = 0; have_ppi = 0; if (bus_space_map(ia->ia_iot, IO_PPI, 1, 0, &ppi_ioh)) goto lose; have_ppi = 1; /* * Check for existence of PPI. Realistically, this is either going to * be here or nothing is going to be here. * * We don't want to have any chance of changing speaker output (which * this test might, if it crashes in the middle, or something; * normally it's be to quick to produce anthing audible), but * many "combo chip" mock-PPI's don't seem to support the top bit * of Port B as a settable bit. The bottom bit has to be settable, * since the speaker driver hardware still uses it. */ v = bus_space_read_1(ia->ia_iot, ppi_ioh, 0); /* XXX */ bus_space_write_1(ia->ia_iot, ppi_ioh, 0, v ^ 0x01); /* XXX */ nv = bus_space_read_1(ia->ia_iot, ppi_ioh, 0); /* XXX */ if (((nv ^ v) & 0x01) == 0x01) rv = 1; bus_space_write_1(ia->ia_iot, ppi_ioh, 0, v); /* XXX */ nv = bus_space_read_1(ia->ia_iot, ppi_ioh, 0); /* XXX */ if (((nv ^ v) & 0x01) != 0x00) { rv = 0; goto lose; } /* * We assume that the programmable interval timer is there. */ lose: if (have_ppi) bus_space_unmap(ia->ia_iot, ppi_ioh, 1); if (rv) { ia->ia_io[0].ir_addr = IO_PPI; ia->ia_io[0].ir_size = 1; ia->ia_nio = 1; ia->ia_niomem = 0; ia->ia_nirq = 0; ia->ia_ndrq = 0; } return (rv); } void pcppi_isa_attach(device_t parent, device_t self, void *aux) { struct pcppi_softc *sc = device_private(self); struct isa_attach_args *ia = aux; bus_space_tag_t iot; sc->sc_dv = self; sc->sc_iot = iot = ia->ia_iot; sc->sc_size = 1; if (bus_space_map(iot, IO_PPI, sc->sc_size, 0, &sc->sc_ppi_ioh)) panic("pcppi_attach: couldn't map"); aprint_naive("\n"); aprint_normal("\n"); pcppi_attach(sc); } void pcppi_childdet(device_t self, device_t child) { /* we hold no child references, so do nothing */ } int pcppi_detach(device_t self, int flags) { int rc; struct pcppi_softc *sc = device_private(self); #if NATTIMER > 0 pcppi_detach_speaker(sc); #endif if ((rc = config_detach_children(sc->sc_dv, flags)) != 0) return rc; pmf_device_deregister(self); #if NPCKBD > 0 pckbd_unhook_bell(pcppi_pckbd_bell, sc); #endif mutex_spin_enter(&tty_lock); pcppi_bell_stop(sc); mutex_spin_exit(&tty_lock); callout_halt(&sc->sc_bell_ch, NULL); callout_destroy(&sc->sc_bell_ch); cv_destroy(&sc->sc_slp); bus_space_unmap(sc->sc_iot, sc->sc_ppi_ioh, sc->sc_size); return 0; } void pcppi_attach(struct pcppi_softc *sc) { device_t self = sc->sc_dv; callout_init(&sc->sc_bell_ch, CALLOUT_MPSAFE); callout_setfunc(&sc->sc_bell_ch, pcppi_bell_callout, sc); cv_init(&sc->sc_slp, "bell"); sc->sc_bellactive = sc->sc_bellpitch = 0; #if NPCKBD > 0 /* Provide a beeper for the PC Keyboard, if there isn't one already. */ pckbd_hookup_bell(pcppi_pckbd_bell, sc); #endif #if NATTIMER > 0 config_defer(sc->sc_dv, pcppi_attach_speaker); #endif if (!pmf_device_register(self, NULL, NULL)) aprint_error_dev(self, "couldn't establish power handler\n"); pcppi_rescan(self, NULL, NULL); } int pcppi_rescan(device_t self, const char *ifattr, const int *locators) { struct pcppi_softc *sc = device_private(self); struct pcppi_attach_args pa; pa.pa_cookie = sc; pa.pa_bell_func = pcppi_bell; config_search(sc->sc_dv, &pa, CFARGS(.search = pcppisearch)); return 0; } static int pcppisearch(device_t parent, cfdata_t cf, const int *locs, void *aux) { if (config_probe(parent, cf, aux)) config_attach(parent, cf, aux, NULL, CFARGS(.locators = locs)); return 0; } #if NATTIMER > 0 static void pcppi_detach_speaker(struct pcppi_softc *sc) { if (sc->sc_timer != NULL) { attimer_detach_speaker(sc->sc_timer); sc->sc_timer = NULL; } } static void pcppi_attach_speaker(device_t self) { struct pcppi_softc *sc = device_private(self); if ((sc->sc_timer = attimer_attach_speaker()) == NULL) aprint_error_dev(self, "could not find any available timer\n"); else { aprint_normal_dev(sc->sc_timer, "attached to %s\n", device_xname(self)); } } #endif void pcppi_bell(pcppi_tag_t self, int pitch, int period, int slp) { mutex_spin_enter(&tty_lock); pcppi_bell_locked(self, pitch, period, slp); mutex_spin_exit(&tty_lock); } void pcppi_bell_locked(pcppi_tag_t self, int pitch, int period, int slp) { struct pcppi_softc *sc = self; if (sc->sc_bellactive) { if (sc->sc_timeout) { sc->sc_timeout = 0; callout_stop(&sc->sc_bell_ch); } cv_broadcast(&sc->sc_slp); } if (pitch == 0 || period == 0) { pcppi_bell_stop(sc); sc->sc_bellpitch = 0; return; } if (!sc->sc_bellactive || sc->sc_bellpitch != pitch) { #if NATTIMER > 0 if (sc->sc_timer != NULL) attimer_set_pitch(sc->sc_timer, pitch); #endif /* enable speaker */ bus_space_write_1(sc->sc_iot, sc->sc_ppi_ioh, 0, bus_space_read_1(sc->sc_iot, sc->sc_ppi_ioh, 0) | PIT_SPKR); } sc->sc_bellpitch = pitch; sc->sc_bellactive = 1; if (slp & PCPPI_BELL_POLL) { delay((period * 1000000) / hz); pcppi_bell_stop(sc); } else { sc->sc_timeout = 1; callout_schedule(&sc->sc_bell_ch, period); if (slp & PCPPI_BELL_SLEEP) { cv_wait_sig(&sc->sc_slp, &tty_lock); } } } static void pcppi_bell_callout(void *arg) { struct pcppi_softc *sc = arg; mutex_spin_enter(&tty_lock); if (sc->sc_timeout != 0) { pcppi_bell_stop(sc); } mutex_spin_exit(&tty_lock); } static void pcppi_bell_stop(struct pcppi_softc *sc) { sc->sc_timeout = 0; /* disable bell */ bus_space_write_1(sc->sc_iot, sc->sc_ppi_ioh, 0, bus_space_read_1(sc->sc_iot, sc->sc_ppi_ioh, 0) & ~PIT_SPKR); sc->sc_bellactive = 0; cv_broadcast(&sc->sc_slp); } #if NPCKBD > 0 void pcppi_pckbd_bell(void *arg, u_int pitch, u_int period, u_int volume, int poll) { /* * Comes in as ms, goes out at ticks; volume ignored. */ pcppi_bell_locked(arg, pitch, (period * hz) / 1000, poll ? PCPPI_BELL_POLL : 0); } #endif /* NPCKBD > 0 */ |
| 5766 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 | /* $NetBSD: userret.h,v 1.13 2018/07/26 09:29:08 maxv Exp $ */ /* * XXXfvdl same as i386 counterpart, but should probably be independent. */ /*- * Copyright (c) 1998, 2000 The NetBSD Foundation, Inc. * All rights reserved. * * This code is derived from software contributed to The NetBSD Foundation * by Charles M. Hannum. * * Redistribution and use in source and binary forms, with or without * modification, are permitted provided that the following conditions * are met: * 1. Redistributions of source code must retain the above copyright * notice, this list of conditions and the following disclaimer. * 2. Redistributions in binary form must reproduce the above copyright * notice, this list of conditions and the following disclaimer in the * documentation and/or other materials provided with the distribution. * * THIS SOFTWARE IS PROVIDED BY THE NETBSD FOUNDATION, INC. AND CONTRIBUTORS * ``AS IS'' AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED * TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR * PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE FOUNDATION OR CONTRIBUTORS * BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR * CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF * SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS * INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN * CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) * ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE * POSSIBILITY OF SUCH DAMAGE. */ /*- * Copyright (c) 1990 The Regents of the University of California. * All rights reserved. * * This code is derived from software contributed to Berkeley by * William Jolitz. * * Redistribution and use in source and binary forms, with or without * modification, are permitted provided that the following conditions * are met: * 1. Redistributions of source code must retain the above copyright * notice, this list of conditions and the following disclaimer. * 2. Redistributions in binary form must reproduce the above copyright * notice, this list of conditions and the following disclaimer in the * documentation and/or other materials provided with the distribution. * 3. Neither the name of the University nor the names of its contributors * may be used to endorse or promote products derived from this software * without specific prior written permission. * * THIS SOFTWARE IS PROVIDED BY THE REGENTS AND CONTRIBUTORS ``AS IS'' AND * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE * ARE DISCLAIMED. IN NO EVENT SHALL THE REGENTS OR CONTRIBUTORS BE LIABLE * FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL * DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS * OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) * HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT * LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY * OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF * SUCH DAMAGE. * */ #include <sys/userret.h> static __inline void userret(struct lwp *); /* * Define the code needed before returning to user mode, for * trap and syscall. */ static __inline void userret(struct lwp *l) { /* Invoke MI userret code */ mi_userret(l); } |
| 2 2 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 | /* $NetBSD: rf_mcpair.c,v 1.25 2021/07/23 00:54:45 oster Exp $ */ /* * Copyright (c) 1995 Carnegie-Mellon University. * All rights reserved. * * Author: Jim Zelenka * * Permission to use, copy, modify and distribute this software and * its documentation is hereby granted, provided that both the copyright * notice and this permission notice appear in all copies of the * software, derivative works or modified versions, and any portions * thereof, and that both notices appear in supporting documentation. * * CARNEGIE MELLON ALLOWS FREE USE OF THIS SOFTWARE IN ITS "AS IS" * CONDITION. CARNEGIE MELLON DISCLAIMS ANY LIABILITY OF ANY KIND * FOR ANY DAMAGES WHATSOEVER RESULTING FROM THE USE OF THIS SOFTWARE. * * Carnegie Mellon requests users of this software to return to * * Software Distribution Coordinator or Software.Distribution@CS.CMU.EDU * School of Computer Science * Carnegie Mellon University * Pittsburgh PA 15213-3890 * * any improvements or extensions that they make and grant Carnegie the * rights to redistribute these changes. */ /* rf_mcpair.c * an mcpair is a structure containing a mutex and a condition variable. * it's used to block the current thread until some event occurs. */ #include <sys/cdefs.h> __KERNEL_RCSID(0, "$NetBSD: rf_mcpair.c,v 1.25 2021/07/23 00:54:45 oster Exp $"); #include <dev/raidframe/raidframevar.h> #include "rf_archs.h" #include "rf_threadstuff.h" #include "rf_mcpair.h" #include "rf_debugMem.h" #include "rf_general.h" #include "rf_shutdown.h" #include "rf_netbsd.h" #include "rf_raid.h" #include <sys/pool.h> #include <sys/proc.h> #define RF_MAX_FREE_MCPAIR 128 #define RF_MIN_FREE_MCPAIR 24 static void rf_ShutdownMCPair(void *); static void rf_ShutdownMCPair(void *arg) { RF_Raid_t *raidPtr; raidPtr = (RF_Raid_t *) arg; pool_destroy(&raidPtr->pools.mcpair); } int rf_ConfigureMCPair(RF_ShutdownList_t **listp, RF_Raid_t *raidPtr, RF_Config_t *cfgPtr) { rf_pool_init(raidPtr, raidPtr->poolNames.mcpair, &raidPtr->pools.mcpair, sizeof(RF_MCPair_t), "mcpair", RF_MIN_FREE_MCPAIR, RF_MAX_FREE_MCPAIR); rf_ShutdownCreate(listp, rf_ShutdownMCPair, raidPtr); return (0); } RF_MCPair_t * rf_AllocMCPair(RF_Raid_t *raidPtr) { RF_MCPair_t *t; t = pool_get(&raidPtr->pools.mcpair, PR_WAITOK); rf_init_mutex2(t->mutex, IPL_VM); rf_init_cond2(t->cond, "mcpair"); t->flag = 0; return (t); } void rf_FreeMCPair(RF_Raid_t *raidPtr, RF_MCPair_t *t) { rf_destroy_cond2(t->cond); rf_destroy_mutex2(t->mutex); pool_put(&raidPtr->pools.mcpair, t); } /* the callback function used to wake you up when you use an mcpair to wait for something */ void rf_MCPairWakeupFunc(RF_MCPair_t *mcpair) { RF_LOCK_MCPAIR(mcpair); mcpair->flag = 1; rf_broadcast_cond2(mcpair->cond); RF_UNLOCK_MCPAIR(mcpair); } |
| 367 888 885 822 59 361 397 39 6 281 357 150 9 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 197 198 199 200 201 202 203 204 205 206 207 208 209 210 211 212 213 214 215 216 217 218 219 220 221 222 223 224 225 226 227 228 229 230 231 232 233 234 235 236 237 238 239 240 241 242 243 244 245 246 247 248 249 250 251 252 253 254 255 256 257 258 259 260 261 262 263 264 265 266 267 268 269 270 271 272 273 274 275 276 277 278 279 280 281 282 283 284 285 286 287 288 289 290 291 292 293 294 295 296 297 298 299 300 301 302 303 304 305 306 307 308 309 310 311 312 313 314 315 316 317 318 319 320 321 322 323 324 325 326 327 328 329 330 331 332 333 334 335 336 337 338 339 340 341 342 343 344 345 346 347 348 349 350 351 352 353 354 355 356 357 358 359 360 361 362 363 364 365 366 367 368 369 370 371 372 373 374 375 376 377 378 379 380 381 382 383 384 385 386 387 388 389 390 391 392 393 394 395 396 397 398 399 400 401 402 403 404 405 406 407 408 409 410 411 412 413 414 415 416 417 418 419 420 421 422 423 424 425 426 427 428 429 430 431 432 433 434 435 436 437 438 439 440 441 442 443 444 445 446 447 448 449 450 451 452 453 454 455 456 457 458 459 460 461 462 463 464 465 466 467 468 469 470 471 472 473 474 475 476 477 478 479 480 481 482 483 484 485 486 487 488 489 490 491 492 493 494 495 496 497 498 499 500 501 502 503 504 505 506 507 508 509 510 511 512 513 514 515 516 517 518 519 520 521 522 523 524 525 526 527 528 529 530 531 532 533 534 535 536 537 538 539 540 541 542 543 544 545 546 547 548 549 550 551 552 553 554 555 556 557 558 559 560 561 562 563 564 565 566 567 568 569 570 571 572 573 574 575 576 577 578 579 580 581 582 583 584 585 586 587 588 589 590 591 592 593 594 595 596 597 598 599 600 601 602 603 604 605 606 607 608 609 610 611 612 613 614 615 616 617 618 619 620 621 622 623 624 625 626 627 628 629 630 631 632 633 634 635 636 637 638 639 640 641 642 643 644 645 646 647 648 649 650 651 652 653 654 655 656 657 658 659 660 661 662 663 664 665 666 667 668 669 670 671 672 673 674 675 676 677 678 679 680 681 682 683 684 685 686 687 688 689 690 691 692 693 694 695 696 697 698 699 700 701 702 703 704 705 706 707 708 709 | /* $NetBSD: subr_copy.c,v 1.16 2022/04/09 23:51:09 riastradh Exp $ */ /*- * Copyright (c) 1997, 1998, 1999, 2002, 2007, 2008, 2019 * The NetBSD Foundation, Inc. * All rights reserved. * * This code is derived from software contributed to The NetBSD Foundation * by Jason R. Thorpe of the Numerical Aerospace Simulation Facility, * NASA Ames Research Center. * * Redistribution and use in source and binary forms, with or without * modification, are permitted provided that the following conditions * are met: * 1. Redistributions of source code must retain the above copyright * notice, this list of conditions and the following disclaimer. * 2. Redistributions in binary form must reproduce the above copyright * notice, this list of conditions and the following disclaimer in the * documentation and/or other materials provided with the distribution. * * THIS SOFTWARE IS PROVIDED BY THE NETBSD FOUNDATION, INC. AND CONTRIBUTORS * ``AS IS'' AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED * TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR * PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE FOUNDATION OR CONTRIBUTORS * BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR * CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF * SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS * INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN * CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) * ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE * POSSIBILITY OF SUCH DAMAGE. */ /* * Copyright (c) 1982, 1986, 1991, 1993 * The Regents of the University of California. All rights reserved. * (c) UNIX System Laboratories, Inc. * All or some portions of this file are derived from material licensed * to the University of California by American Telephone and Telegraph * Co. or Unix System Laboratories, Inc. and are reproduced herein with * the permission of UNIX System Laboratories, Inc. * * Copyright (c) 1992, 1993 * The Regents of the University of California. All rights reserved. * * This software was developed by the Computer Systems Engineering group * at Lawrence Berkeley Laboratory under DARPA contract BG 91-66 and * contributed to Berkeley. * * All advertising materials mentioning features or use of this software * must display the following acknowledgement: * This product includes software developed by the University of * California, Lawrence Berkeley Laboratory. * * Redistribution and use in source and binary forms, with or without * modification, are permitted provided that the following conditions * are met: * 1. Redistributions of source code must retain the above copyright * notice, this list of conditions and the following disclaimer. * 2. Redistributions in binary form must reproduce the above copyright * notice, this list of conditions and the following disclaimer in the * documentation and/or other materials provided with the distribution. * 3. Neither the name of the University nor the names of its contributors * may be used to endorse or promote products derived from this software * without specific prior written permission. * * THIS SOFTWARE IS PROVIDED BY THE REGENTS AND CONTRIBUTORS ``AS IS'' AND * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE * ARE DISCLAIMED. IN NO EVENT SHALL THE REGENTS OR CONTRIBUTORS BE LIABLE * FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL * DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS * OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) * HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT * LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY * OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF * SUCH DAMAGE. * * @(#)kern_subr.c 8.4 (Berkeley) 2/14/95 */ #include <sys/cdefs.h> __KERNEL_RCSID(0, "$NetBSD: subr_copy.c,v 1.16 2022/04/09 23:51:09 riastradh Exp $"); #define __UFETCHSTORE_PRIVATE #define __UCAS_PRIVATE #include <sys/param.h> #include <sys/fcntl.h> #include <sys/proc.h> #include <sys/systm.h> #include <uvm/uvm_extern.h> void uio_setup_sysspace(struct uio *uio) { uio->uio_vmspace = vmspace_kernel(); } int uiomove(void *buf, size_t n, struct uio *uio) { struct vmspace *vm = uio->uio_vmspace; struct iovec *iov; size_t cnt; int error = 0; char *cp = buf; ASSERT_SLEEPABLE(); KASSERT(uio->uio_rw == UIO_READ || uio->uio_rw == UIO_WRITE); while (n > 0 && uio->uio_resid) { iov = uio->uio_iov; cnt = iov->iov_len; if (cnt == 0) { KASSERT(uio->uio_iovcnt > 0); uio->uio_iov++; uio->uio_iovcnt--; continue; } if (cnt > n) cnt = n; if (!VMSPACE_IS_KERNEL_P(vm)) { preempt_point(); } if (uio->uio_rw == UIO_READ) { error = copyout_vmspace(vm, cp, iov->iov_base, cnt); } else { error = copyin_vmspace(vm, iov->iov_base, cp, cnt); } if (error) { break; } iov->iov_base = (char *)iov->iov_base + cnt; iov->iov_len -= cnt; uio->uio_resid -= cnt; uio->uio_offset += cnt; cp += cnt; KDASSERT(cnt <= n); n -= cnt; } return (error); } /* * Wrapper for uiomove() that validates the arguments against a known-good * kernel buffer. */ int uiomove_frombuf(void *buf, size_t buflen, struct uio *uio) { size_t offset; if (uio->uio_offset < 0 || /* uio->uio_resid < 0 || */ (offset = uio->uio_offset) != uio->uio_offset) return (EINVAL); if (offset >= buflen) return (0); return (uiomove((char *)buf + offset, buflen - offset, uio)); } /* * Give next character to user as result of read. */ int ureadc(int c, struct uio *uio) { struct iovec *iov; if (uio->uio_resid <= 0) panic("ureadc: non-positive resid"); again: if (uio->uio_iovcnt <= 0) panic("ureadc: non-positive iovcnt"); iov = uio->uio_iov; if (iov->iov_len <= 0) { uio->uio_iovcnt--; uio->uio_iov++; goto again; } if (!VMSPACE_IS_KERNEL_P(uio->uio_vmspace)) { int error; if ((error = ustore_char(iov->iov_base, c)) != 0) return (error); } else { *(char *)iov->iov_base = c; } iov->iov_base = (char *)iov->iov_base + 1; iov->iov_len--; uio->uio_resid--; uio->uio_offset++; return (0); } /* * Like copyin(), but operates on an arbitrary vmspace. */ int copyin_vmspace(struct vmspace *vm, const void *uaddr, void *kaddr, size_t len) { struct iovec iov; struct uio uio; int error; if (len == 0) return (0); if (VMSPACE_IS_KERNEL_P(vm)) { return kcopy(uaddr, kaddr, len); } if (__predict_true(vm == curproc->p_vmspace)) { return copyin(uaddr, kaddr, len); } iov.iov_base = kaddr; iov.iov_len = len; uio.uio_iov = &iov; uio.uio_iovcnt = 1; uio.uio_offset = (off_t)(uintptr_t)uaddr; uio.uio_resid = len; uio.uio_rw = UIO_READ; UIO_SETUP_SYSSPACE(&uio); error = uvm_io(&vm->vm_map, &uio, 0); return (error); } /* * Like copyout(), but operates on an arbitrary vmspace. */ int copyout_vmspace(struct vmspace *vm, const void *kaddr, void *uaddr, size_t len) { struct iovec iov; struct uio uio; int error; if (len == 0) return (0); if (VMSPACE_IS_KERNEL_P(vm)) { return kcopy(kaddr, uaddr, len); } if (__predict_true(vm == curproc->p_vmspace)) { return copyout(kaddr, uaddr, len); } iov.iov_base = __UNCONST(kaddr); /* XXXUNCONST cast away const */ iov.iov_len = len; uio.uio_iov = &iov; uio.uio_iovcnt = 1; uio.uio_offset = (off_t)(uintptr_t)uaddr; uio.uio_resid = len; uio.uio_rw = UIO_WRITE; UIO_SETUP_SYSSPACE(&uio); error = uvm_io(&vm->vm_map, &uio, 0); return (error); } /* * Like copyin(), but operates on an arbitrary process. */ int copyin_proc(struct proc *p, const void *uaddr, void *kaddr, size_t len) { struct vmspace *vm; int error; error = proc_vmspace_getref(p, &vm); if (error) { return error; } error = copyin_vmspace(vm, uaddr, kaddr, len); uvmspace_free(vm); return error; } /* * Like copyout(), but operates on an arbitrary process. */ int copyout_proc(struct proc *p, const void *kaddr, void *uaddr, size_t len) { struct vmspace *vm; int error; error = proc_vmspace_getref(p, &vm); if (error) { return error; } error = copyout_vmspace(vm, kaddr, uaddr, len); uvmspace_free(vm); return error; } /* * Like copyin(), but operates on an arbitrary pid. */ int copyin_pid(pid_t pid, const void *uaddr, void *kaddr, size_t len) { struct proc *p; struct vmspace *vm; int error; mutex_enter(&proc_lock); p = proc_find(pid); if (p == NULL) { mutex_exit(&proc_lock); return ESRCH; } mutex_enter(p->p_lock); error = proc_vmspace_getref(p, &vm); mutex_exit(p->p_lock); mutex_exit(&proc_lock); if (error == 0) { error = copyin_vmspace(vm, uaddr, kaddr, len); uvmspace_free(vm); } return error; } /* * Like copyin(), except it operates on kernel addresses when the FKIOCTL * flag is passed in `ioctlflags' from the ioctl call. */ int ioctl_copyin(int ioctlflags, const void *src, void *dst, size_t len) { if (ioctlflags & FKIOCTL) return kcopy(src, dst, len); return copyin(src, dst, len); } /* * Like copyout(), except it operates on kernel addresses when the FKIOCTL * flag is passed in `ioctlflags' from the ioctl call. */ int ioctl_copyout(int ioctlflags, const void *src, void *dst, size_t len) { if (ioctlflags & FKIOCTL) return kcopy(src, dst, len); return copyout(src, dst, len); } /* * User-space CAS / fetch / store */ #ifdef __NO_STRICT_ALIGNMENT #define CHECK_ALIGNMENT(x) __nothing #else /* ! __NO_STRICT_ALIGNMENT */ static bool ufetchstore_aligned(uintptr_t uaddr, size_t size) { return (uaddr & (size - 1)) == 0; } #define CHECK_ALIGNMENT() \ do { \ if (!ufetchstore_aligned((uintptr_t)uaddr, sizeof(*uaddr))) \ return EFAULT; \ } while (/*CONSTCOND*/0) #endif /* __NO_STRICT_ALIGNMENT */ /* * __HAVE_UCAS_FULL platforms provide _ucas_32() and _ucas_64() themselves. * _RUMPKERNEL also provides it's own _ucas_32() and _ucas_64(). * * In all other cases, we provide generic implementations that work on * all platforms. */ #if !defined(__HAVE_UCAS_FULL) && !defined(_RUMPKERNEL) #if !defined(__HAVE_UCAS_MP) && defined(MULTIPROCESSOR) #include <sys/atomic.h> #include <sys/cpu.h> #include <sys/once.h> #include <sys/mutex.h> #include <sys/ipi.h> static int ucas_critical_splcookie; static volatile u_int ucas_critical_pausing_cpus; static u_int ucas_critical_ipi; static ONCE_DECL(ucas_critical_init_once) static void ucas_critical_cpu_gate(void *arg __unused) { int count = SPINLOCK_BACKOFF_MIN; KASSERT(atomic_load_relaxed(&ucas_critical_pausing_cpus) > 0); /* * Notify ucas_critical_wait that we have stopped. Using * store-release ensures all our memory operations up to the * IPI happen before the ucas -- no buffered stores on our end * can clobber it later on, for instance. * * Matches atomic_load_acquire in ucas_critical_wait -- turns * the following atomic_dec_uint into a store-release. */ #ifndef __HAVE_ATOMIC_AS_MEMBAR membar_release(); #endif atomic_dec_uint(&ucas_critical_pausing_cpus); /* * Wait for ucas_critical_exit to reopen the gate and let us * proceed. Using a load-acquire ensures the ucas happens * before any of our memory operations when we return from the * IPI and proceed -- we won't observe any stale cached value * that the ucas overwrote, for instance. * * Matches atomic_store_release in ucas_critical_exit. */ while (atomic_load_acquire(&ucas_critical_pausing_cpus) != (u_int)-1) { SPINLOCK_BACKOFF(count); } } static int ucas_critical_init(void) { ucas_critical_ipi = ipi_register(ucas_critical_cpu_gate, NULL); return 0; } static void ucas_critical_wait(void) { int count = SPINLOCK_BACKOFF_MIN; /* * Wait for all CPUs to stop at the gate. Using a load-acquire * ensures all memory operations before they stop at the gate * happen before the ucas -- no buffered stores in other CPUs * can clobber it later on, for instance. * * Matches membar_release/atomic_dec_uint (store-release) in * ucas_critical_cpu_gate. */ while (atomic_load_acquire(&ucas_critical_pausing_cpus) > 0) { SPINLOCK_BACKOFF(count); } } #endif /* ! __HAVE_UCAS_MP && MULTIPROCESSOR */ static inline void ucas_critical_enter(lwp_t * const l) { #if !defined(__HAVE_UCAS_MP) && defined(MULTIPROCESSOR) if (ncpu > 1) { RUN_ONCE(&ucas_critical_init_once, ucas_critical_init); /* * Acquire the mutex first, then go to splhigh() and * broadcast the IPI to lock all of the other CPUs * behind the gate. * * N.B. Going to splhigh() implicitly disables preemption, * so there's no need to do it explicitly. */ mutex_enter(&cpu_lock); ucas_critical_splcookie = splhigh(); ucas_critical_pausing_cpus = ncpu - 1; ipi_trigger_broadcast(ucas_critical_ipi, true); ucas_critical_wait(); return; } #endif /* ! __HAVE_UCAS_MP && MULTIPROCESSOR */ KPREEMPT_DISABLE(l); } static inline void ucas_critical_exit(lwp_t * const l) { #if !defined(__HAVE_UCAS_MP) && defined(MULTIPROCESSOR) if (ncpu > 1) { /* * Open the gate and notify all CPUs in * ucas_critical_cpu_gate that they can now proceed. * Using a store-release ensures the ucas happens * before any memory operations they issue after the * IPI -- they won't observe any stale cache of the * target word, for instance. * * Matches atomic_load_acquire in ucas_critical_cpu_gate. */ atomic_store_release(&ucas_critical_pausing_cpus, (u_int)-1); splx(ucas_critical_splcookie); mutex_exit(&cpu_lock); return; } #endif /* ! __HAVE_UCAS_MP && MULTIPROCESSOR */ KPREEMPT_ENABLE(l); } int _ucas_32(volatile uint32_t *uaddr, uint32_t old, uint32_t new, uint32_t *ret) { lwp_t * const l = curlwp; uint32_t *uva = ((void *)(uintptr_t)uaddr); int error; /* * Wire the user address down to avoid taking a page fault during * the critical section. */ error = uvm_vslock(l->l_proc->p_vmspace, uva, sizeof(*uaddr), VM_PROT_READ | VM_PROT_WRITE); if (error) return error; ucas_critical_enter(l); error = _ufetch_32(uva, ret); if (error == 0 && *ret == old) { error = _ustore_32(uva, new); } ucas_critical_exit(l); uvm_vsunlock(l->l_proc->p_vmspace, uva, sizeof(*uaddr)); return error; } #ifdef _LP64 int _ucas_64(volatile uint64_t *uaddr, uint64_t old, uint64_t new, uint64_t *ret) { lwp_t * const l = curlwp; uint64_t *uva = ((void *)(uintptr_t)uaddr); int error; /* * Wire the user address down to avoid taking a page fault during * the critical section. */ error = uvm_vslock(l->l_proc->p_vmspace, uva, sizeof(*uaddr), VM_PROT_READ | VM_PROT_WRITE); if (error) return error; ucas_critical_enter(l); error = _ufetch_64(uva, ret); if (error == 0 && *ret == old) { error = _ustore_64(uva, new); } ucas_critical_exit(l); uvm_vsunlock(l->l_proc->p_vmspace, uva, sizeof(*uaddr)); return error; } #endif /* _LP64 */ #endif /* ! __HAVE_UCAS_FULL && ! _RUMPKERNEL */ int ucas_32(volatile uint32_t *uaddr, uint32_t old, uint32_t new, uint32_t *ret) { ASSERT_SLEEPABLE(); CHECK_ALIGNMENT(); #if (defined(__HAVE_UCAS_MP) && defined(MULTIPROCESSOR)) && \ !defined(_RUMPKERNEL) if (ncpu > 1) { return _ucas_32_mp(uaddr, old, new, ret); } #endif /* __HAVE_UCAS_MP && MULTIPROCESSOR */ return _ucas_32(uaddr, old, new, ret); } #ifdef _LP64 int ucas_64(volatile uint64_t *uaddr, uint64_t old, uint64_t new, uint64_t *ret) { ASSERT_SLEEPABLE(); CHECK_ALIGNMENT(); #if (defined(__HAVE_UCAS_MP) && defined(MULTIPROCESSOR)) && \ !defined(_RUMPKERNEL) if (ncpu > 1) { return _ucas_64_mp(uaddr, old, new, ret); } #endif /* __HAVE_UCAS_MP && MULTIPROCESSOR */ return _ucas_64(uaddr, old, new, ret); } #endif /* _LP64 */ __strong_alias(ucas_int,ucas_32); #ifdef _LP64 __strong_alias(ucas_ptr,ucas_64); #else __strong_alias(ucas_ptr,ucas_32); #endif /* _LP64 */ int ufetch_8(const uint8_t *uaddr, uint8_t *valp) { ASSERT_SLEEPABLE(); CHECK_ALIGNMENT(); return _ufetch_8(uaddr, valp); } int ufetch_16(const uint16_t *uaddr, uint16_t *valp) { ASSERT_SLEEPABLE(); CHECK_ALIGNMENT(); return _ufetch_16(uaddr, valp); } int ufetch_32(const uint32_t *uaddr, uint32_t *valp) { ASSERT_SLEEPABLE(); CHECK_ALIGNMENT(); return _ufetch_32(uaddr, valp); } #ifdef _LP64 int ufetch_64(const uint64_t *uaddr, uint64_t *valp) { ASSERT_SLEEPABLE(); CHECK_ALIGNMENT(); return _ufetch_64(uaddr, valp); } #endif /* _LP64 */ __strong_alias(ufetch_char,ufetch_8); __strong_alias(ufetch_short,ufetch_16); __strong_alias(ufetch_int,ufetch_32); #ifdef _LP64 __strong_alias(ufetch_long,ufetch_64); __strong_alias(ufetch_ptr,ufetch_64); #else __strong_alias(ufetch_long,ufetch_32); __strong_alias(ufetch_ptr,ufetch_32); #endif /* _LP64 */ int ustore_8(uint8_t *uaddr, uint8_t val) { ASSERT_SLEEPABLE(); CHECK_ALIGNMENT(); return _ustore_8(uaddr, val); } int ustore_16(uint16_t *uaddr, uint16_t val) { ASSERT_SLEEPABLE(); CHECK_ALIGNMENT(); return _ustore_16(uaddr, val); } int ustore_32(uint32_t *uaddr, uint32_t val) { ASSERT_SLEEPABLE(); CHECK_ALIGNMENT(); return _ustore_32(uaddr, val); } #ifdef _LP64 int ustore_64(uint64_t *uaddr, uint64_t val) { ASSERT_SLEEPABLE(); CHECK_ALIGNMENT(); return _ustore_64(uaddr, val); } #endif /* _LP64 */ __strong_alias(ustore_char,ustore_8); __strong_alias(ustore_short,ustore_16); __strong_alias(ustore_int,ustore_32); #ifdef _LP64 __strong_alias(ustore_long,ustore_64); __strong_alias(ustore_ptr,ustore_64); #else __strong_alias(ustore_long,ustore_32); __strong_alias(ustore_ptr,ustore_32); #endif /* _LP64 */ |
| 228 223 15 13 15 15 233 23 222 12 9 227 41 211 8 23 233 233 233 233 8 233 233 233 226 233 1 1 1 1 25 25 27 25 25 25 8 7 8 8 8 23 23 2 2 2 191 6 6 421 150 58 528 528 232 528 528 338 432 232 1 231 20 232 232 417 1 417 1 422 421 22 19 430 19 432 233 448 70 70 233 448 447 14 11 448 61 448 447 448 227 528 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 197 198 199 200 201 202 203 204 205 206 207 208 209 210 211 212 213 214 215 216 217 218 219 220 221 222 223 224 225 226 227 228 229 230 231 232 233 234 235 236 237 238 239 240 241 242 243 244 245 246 247 248 249 250 251 252 253 254 255 256 257 258 259 260 261 262 263 264 265 266 267 268 269 270 271 272 273 274 275 276 277 278 279 280 281 282 283 284 285 286 287 288 289 290 291 292 293 294 295 296 297 298 299 300 301 302 303 304 305 306 307 308 309 310 311 312 313 314 315 316 317 318 319 320 321 322 323 324 325 326 327 328 329 330 331 332 333 334 335 336 337 338 339 340 341 342 343 344 345 346 347 348 349 350 351 352 353 354 355 356 357 358 359 360 361 362 363 364 365 366 367 368 369 370 371 372 373 374 375 376 377 378 379 380 381 382 383 384 385 386 387 388 389 390 391 392 393 394 395 396 397 398 399 400 401 402 403 404 405 406 407 408 409 410 411 412 413 414 415 416 417 418 419 420 421 422 423 424 425 426 427 428 429 430 431 432 433 434 435 436 437 438 439 440 441 442 443 444 445 446 447 448 449 450 451 452 453 454 455 456 457 458 459 460 461 462 463 464 465 466 467 468 469 470 471 472 473 474 475 476 477 478 479 480 481 482 483 484 485 486 487 488 489 490 491 492 493 494 495 496 497 498 499 500 501 502 503 504 505 506 507 508 509 510 511 512 513 514 515 516 517 518 519 520 521 522 523 524 525 526 527 528 529 530 531 532 533 534 535 536 537 538 539 540 541 542 543 544 545 546 547 548 549 550 551 552 553 554 555 556 557 558 559 560 561 562 563 564 565 566 567 568 569 570 571 572 573 574 575 576 577 578 579 580 581 582 583 584 585 586 587 588 589 590 591 592 593 594 595 596 597 598 599 600 601 602 603 604 605 606 607 608 609 610 611 612 613 614 615 616 617 618 619 620 621 622 623 624 625 626 627 628 629 630 631 632 633 634 635 636 637 638 639 640 641 642 643 644 645 646 647 648 649 650 651 652 653 654 655 656 657 658 659 660 661 662 663 664 665 666 667 668 669 670 671 672 673 674 675 676 677 678 679 680 681 682 683 684 685 686 687 688 689 690 691 692 693 694 695 696 697 698 699 700 701 702 703 704 705 706 707 708 709 710 711 712 713 714 715 716 717 718 719 720 721 722 723 724 725 726 727 728 729 730 731 732 733 734 735 736 737 738 739 740 741 742 743 744 745 746 747 748 749 750 751 752 753 754 755 756 757 758 759 760 761 762 763 764 765 766 767 768 769 770 771 772 773 774 775 776 777 778 779 780 781 782 783 784 785 786 787 788 789 790 791 792 793 794 795 796 797 798 799 800 801 802 803 804 805 806 807 808 809 810 811 812 813 814 815 816 817 818 819 820 821 822 823 824 825 826 827 828 829 830 831 832 833 834 835 836 837 838 839 840 841 842 843 844 845 846 847 848 849 850 851 852 853 854 855 856 857 858 859 860 861 862 863 864 865 866 867 868 869 870 871 872 873 874 875 876 877 878 879 880 881 882 883 884 885 886 887 888 889 890 891 892 893 894 895 896 897 898 899 900 901 902 903 904 905 906 907 908 909 910 911 912 913 914 915 916 917 918 919 920 921 922 923 924 925 926 927 928 929 930 931 932 933 934 935 936 937 938 939 940 941 942 943 944 945 946 947 948 949 950 951 952 953 954 955 956 957 958 959 960 961 962 963 964 965 966 967 968 969 970 971 972 973 974 975 976 977 978 979 980 981 982 983 984 985 986 987 988 989 990 991 992 993 994 995 996 997 998 999 1000 1001 1002 1003 1004 1005 1006 1007 1008 1009 1010 1011 1012 1013 1014 1015 1016 1017 1018 1019 1020 1021 1022 1023 1024 1025 1026 1027 1028 1029 1030 1031 1032 1033 1034 1035 1036 1037 1038 1039 1040 1041 1042 1043 1044 1045 1046 1047 1048 1049 1050 1051 1052 1053 1054 1055 1056 1057 1058 1059 1060 1061 1062 1063 1064 1065 1066 1067 1068 1069 1070 1071 1072 1073 1074 1075 1076 1077 1078 1079 1080 1081 1082 1083 1084 1085 1086 1087 1088 1089 1090 1091 1092 1093 1094 1095 1096 1097 1098 1099 1100 1101 1102 1103 1104 1105 1106 1107 1108 1109 1110 1111 1112 1113 1114 1115 1116 1117 1118 1119 1120 1121 1122 1123 1124 1125 1126 1127 1128 1129 1130 1131 1132 1133 1134 1135 1136 1137 1138 1139 1140 1141 1142 1143 1144 1145 1146 1147 1148 1149 1150 1151 1152 1153 1154 1155 1156 1157 1158 1159 1160 1161 1162 1163 1164 1165 1166 1167 1168 1169 1170 1171 1172 1173 1174 1175 1176 1177 1178 1179 1180 1181 1182 1183 1184 1185 1186 1187 1188 1189 1190 1191 1192 1193 1194 1195 1196 1197 1198 1199 1200 1201 1202 1203 1204 1205 1206 1207 1208 1209 1210 1211 1212 1213 1214 1215 1216 1217 1218 1219 1220 1221 1222 1223 1224 1225 1226 1227 1228 1229 1230 1231 1232 1233 1234 1235 1236 1237 1238 1239 1240 1241 1242 1243 1244 1245 1246 1247 1248 1249 1250 1251 1252 1253 1254 1255 1256 1257 1258 1259 1260 1261 1262 1263 1264 1265 1266 1267 1268 1269 1270 1271 1272 1273 1274 1275 1276 1277 1278 1279 1280 1281 1282 1283 1284 1285 1286 1287 1288 1289 1290 1291 1292 1293 1294 1295 1296 1297 1298 1299 1300 1301 1302 1303 1304 1305 1306 1307 1308 1309 1310 1311 1312 1313 1314 1315 1316 1317 1318 1319 1320 1321 1322 1323 1324 1325 1326 1327 1328 1329 1330 1331 1332 1333 1334 1335 1336 1337 1338 1339 1340 1341 1342 1343 1344 1345 1346 1347 1348 1349 1350 1351 1352 1353 1354 1355 1356 1357 1358 1359 1360 1361 1362 1363 1364 1365 1366 1367 1368 1369 1370 1371 1372 1373 1374 1375 1376 1377 1378 1379 1380 1381 1382 1383 1384 1385 1386 1387 1388 1389 1390 1391 1392 1393 1394 1395 1396 1397 1398 1399 1400 1401 1402 1403 1404 1405 1406 1407 1408 1409 1410 1411 1412 1413 1414 1415 1416 1417 1418 1419 1420 1421 1422 1423 1424 1425 1426 1427 1428 1429 1430 1431 1432 1433 1434 1435 1436 1437 1438 1439 1440 1441 1442 1443 1444 1445 1446 1447 1448 1449 1450 1451 1452 1453 1454 1455 1456 1457 1458 1459 1460 1461 1462 1463 1464 1465 1466 1467 1468 1469 1470 1471 1472 1473 1474 1475 1476 1477 1478 1479 1480 1481 1482 1483 1484 1485 1486 1487 1488 1489 1490 1491 1492 1493 1494 1495 1496 1497 1498 1499 1500 1501 1502 1503 1504 1505 1506 1507 1508 1509 1510 1511 1512 1513 1514 1515 1516 1517 1518 1519 1520 1521 1522 1523 1524 1525 1526 1527 1528 1529 1530 1531 1532 1533 1534 1535 1536 1537 1538 1539 1540 1541 1542 1543 1544 1545 1546 1547 1548 1549 1550 1551 1552 1553 1554 1555 1556 1557 1558 1559 1560 1561 1562 1563 1564 1565 1566 1567 1568 1569 1570 1571 1572 1573 1574 1575 1576 1577 1578 1579 1580 1581 1582 1583 1584 1585 1586 1587 1588 1589 1590 1591 1592 1593 1594 1595 1596 1597 1598 1599 1600 1601 1602 1603 1604 1605 1606 1607 1608 1609 1610 1611 1612 | /* $NetBSD: subr_prf.c,v 1.189 2022/08/11 23:53:03 gutteridge Exp $ */ /*- * Copyright (c) 1986, 1988, 1991, 1993 * The Regents of the University of California. All rights reserved. * (c) UNIX System Laboratories, Inc. * All or some portions of this file are derived from material licensed * to the University of California by American Telephone and Telegraph * Co. or Unix System Laboratories, Inc. and are reproduced herein with * the permission of UNIX System Laboratories, Inc. * * Redistribution and use in source and binary forms, with or without * modification, are permitted provided that the following conditions * are met: * 1. Redistributions of source code must retain the above copyright * notice, this list of conditions and the following disclaimer. * 2. Redistributions in binary form must reproduce the above copyright * notice, this list of conditions and the following disclaimer in the * documentation and/or other materials provided with the distribution. * 3. Neither the name of the University nor the names of its contributors * may be used to endorse or promote products derived from this software * without specific prior written permission. * * THIS SOFTWARE IS PROVIDED BY THE REGENTS AND CONTRIBUTORS ``AS IS'' AND * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE * ARE DISCLAIMED. IN NO EVENT SHALL THE REGENTS OR CONTRIBUTORS BE LIABLE * FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL * DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS * OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) * HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT * LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY * OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF * SUCH DAMAGE. * * @(#)subr_prf.c 8.4 (Berkeley) 5/4/95 */ #include <sys/cdefs.h> __KERNEL_RCSID(0, "$NetBSD: subr_prf.c,v 1.189 2022/08/11 23:53:03 gutteridge Exp $"); #ifdef _KERNEL_OPT #include "opt_ddb.h" #include "opt_kgdb.h" #include "opt_dump.h" #include "opt_rnd_printf.h" #endif #include <sys/param.h> #include <sys/stdint.h> #include <sys/systm.h> #include <sys/buf.h> #include <sys/device.h> #include <sys/reboot.h> #include <sys/msgbuf.h> #include <sys/proc.h> #include <sys/ioctl.h> #include <sys/vnode.h> #include <sys/file.h> #include <sys/tty.h> #include <sys/tprintf.h> #include <sys/spldebug.h> #include <sys/syslog.h> #include <sys/kprintf.h> #include <sys/atomic.h> #include <sys/kernel.h> #include <sys/cpu.h> #include <sys/rndsource.h> #include <sys/kmem.h> #include <dev/cons.h> #include <net/if.h> static kmutex_t kprintf_mtx; static bool kprintf_inited = false; #ifdef KGDB #include <sys/kgdb.h> #endif #ifdef DDB #include <ddb/ddbvar.h> /* db_panic */ #include <ddb/db_output.h> /* db_printf, db_putchar prototypes */ #endif /* * defines */ #define KLOG_PRI 0x80000000 /* * local prototypes */ static void putchar(int, int, struct tty *); static void kprintf_internal(const char *, int, void *, char *, ...); /* * globals */ extern struct tty *constty; /* pointer to console "window" tty */ extern int log_open; /* subr_log: is /dev/klog open? */ const char *panicstr; /* arg to first call to panic (used as a flag to indicate that panic has already been called). */ struct cpu_info *paniccpu; /* cpu that first panicked */ long panicstart, panicend; /* position in the msgbuf of the start and end of the formatted panicstr. */ int doing_shutdown; /* set to indicate shutdown in progress */ #ifdef RND_PRINTF static krndsource_t rnd_printf_source; #endif #ifndef DUMP_ON_PANIC #define DUMP_ON_PANIC 1 #endif int dumponpanic = DUMP_ON_PANIC; /* * v_putc: routine to putc on virtual console * * the v_putc pointer can be used to redirect the console cnputc elsewhere * [e.g. to a "virtual console"]. */ void (*v_putc)(int) = cnputc; /* start with cnputc (normal cons) */ void (*v_flush)(void) = cnflush; /* start with cnflush (normal cons) */ const char hexdigits[] = "0123456789abcdef"; const char HEXDIGITS[] = "0123456789ABCDEF"; /* * functions */ /* * Locking is inited fairly early in MI bootstrap. Before that * prints are done unlocked. But that doesn't really matter, * since nothing can preempt us before interrupts are enabled. */ void kprintf_init(void) { KASSERT(!kprintf_inited && cold); /* not foolproof, but ... */ mutex_init(&kprintf_mtx, MUTEX_DEFAULT, IPL_HIGH); #ifdef RND_PRINTF rnd_attach_source(&rnd_printf_source, "printf", RND_TYPE_UNKNOWN, RND_FLAG_COLLECT_TIME|RND_FLAG_COLLECT_VALUE); #endif kprintf_inited = true; } void kprintf_lock(void) { if (__predict_true(kprintf_inited)) mutex_enter(&kprintf_mtx); } void kprintf_unlock(void) { if (__predict_true(kprintf_inited)) { /* assert kprintf wasn't somehow inited while we were in */ KASSERT(mutex_owned(&kprintf_mtx)); mutex_exit(&kprintf_mtx); } } /* * twiddle: spin a little propellor on the console. */ void twiddle(void) { static const char twiddle_chars[] = "|/-\\"; static int pos; kprintf_lock(); putchar(twiddle_chars[pos++ & 3], TOCONS|NOTSTAMP, NULL); putchar('\b', TOCONS|NOTSTAMP, NULL); kprintf_unlock(); } /* * panic: handle an unresolvable fatal error * * prints "panic: <message>" and reboots. if called twice (i.e. recursive * call) we avoid trying to dump and just reboot (to avoid recursive panics). */ void panic(const char *fmt, ...) { va_list ap; va_start(ap, fmt); vpanic(fmt, ap); va_end(ap); } void vpanic(const char *fmt, va_list ap) { CPU_INFO_ITERATOR cii; struct cpu_info *ci, *oci; int bootopt; static char scratchstr[384]; /* stores panic message */ spldebug_stop(); if (lwp0.l_cpu && curlwp) { /* * Disable preemption. If already panicking on another CPU, sit * here and spin until the system is rebooted. Allow the CPU that * first panicked to panic again. */ kpreempt_disable(); ci = curcpu(); oci = atomic_cas_ptr((void *)&paniccpu, NULL, ci); if (oci != NULL && oci != ci) { /* Give interrupts a chance to try and prevent deadlock. */ for (;;) { #ifndef _RUMPKERNEL /* XXXpooka: temporary build fix, see kern/40505 */ DELAY(10); #endif /* _RUMPKERNEL */ } } /* * Convert the current thread to a bound thread and prevent all * CPUs from scheduling unbound jobs. Do so without taking any * locks. */ curlwp->l_pflag |= LP_BOUND; for (CPU_INFO_FOREACH(cii, ci)) { ci->ci_schedstate.spc_flags |= SPCF_OFFLINE; } } bootopt = RB_AUTOBOOT | RB_NOSYNC; if (!doing_shutdown) { if (dumponpanic) bootopt |= RB_DUMP; } else printf("Skipping crash dump on recursive panic\n"); doing_shutdown = 1; if (logenabled(msgbufp)) panicstart = msgbufp->msg_bufx; kprintf_lock(); kprintf_internal("panic: ", TOLOG|TOCONS, NULL, NULL); if (panicstr == NULL) { /* first time in panic - store fmt first for precaution */ panicstr = fmt; vsnprintf(scratchstr, sizeof(scratchstr), fmt, ap); kprintf_internal("%s", TOLOG|TOCONS, NULL, NULL, scratchstr); panicstr = scratchstr; } else { kprintf(fmt, TOLOG|TOCONS, NULL, NULL, ap); } kprintf_internal("\n", TOLOG|TOCONS, NULL, NULL); kprintf_unlock(); if (logenabled(msgbufp)) panicend = msgbufp->msg_bufx; #ifdef KGDB kgdb_panic(); #endif #ifdef KADB if (boothowto & RB_KDB) kdbpanic(); #endif #ifdef DDB db_panic(); #endif kern_reboot(bootopt, NULL); } /* * kernel logging functions: log, logpri, addlog */ /* * log: write to the log buffer * * => will not sleep [so safe to call from interrupt] * => will log to console if /dev/klog isn't open */ void log(int level, const char *fmt, ...) { va_list ap; kprintf_lock(); klogpri(level); /* log the level first */ va_start(ap, fmt); kprintf(fmt, TOLOG, NULL, NULL, ap); va_end(ap); if (!log_open) { va_start(ap, fmt); kprintf(fmt, TOCONS, NULL, NULL, ap); va_end(ap); } kprintf_unlock(); logwakeup(); /* wake up anyone waiting for log msgs */ } /* * vlog: write to the log buffer [already have va_list] */ void vlog(int level, const char *fmt, va_list ap) { va_list cap; va_copy(cap, ap); kprintf_lock(); klogpri(level); /* log the level first */ kprintf(fmt, TOLOG, NULL, NULL, ap); if (!log_open) kprintf(fmt, TOCONS, NULL, NULL, cap); kprintf_unlock(); va_end(cap); logwakeup(); /* wake up anyone waiting for log msgs */ } /* * logpri: log the priority level to the klog */ void logpri(int level) { kprintf_lock(); klogpri(level); kprintf_unlock(); } /* * Note: we must be in the mutex here! */ void klogpri(int level) { KASSERT((level & KLOG_PRI) == 0); putchar(level | KLOG_PRI, TOLOG, NULL); } /* * addlog: add info to previous log message */ void addlog(const char *fmt, ...) { va_list ap; kprintf_lock(); va_start(ap, fmt); kprintf(fmt, TOLOG, NULL, NULL, ap); va_end(ap); if (!log_open) { va_start(ap, fmt); kprintf(fmt, TOCONS, NULL, NULL, ap); va_end(ap); } kprintf_unlock(); logwakeup(); } static void putone(int c, int flags, struct tty *tp) { if (panicstr) constty = NULL; if ((flags & TOCONS) && tp == NULL && constty) { tp = constty; flags |= TOTTY; } if ((flags & TOTTY) && tp && tputchar(c, flags, tp) < 0 && (flags & TOCONS) && tp == constty) constty = NULL; if ((flags & TOLOG) && c != '\0' && c != '\r' && c != 0177) logputchar(c); if ((flags & TOCONS) && constty == NULL && c != '\0') (*v_putc)(c); } static void putlogpri(int level) { char *p; char snbuf[KPRINTF_BUFSIZE]; putone('<', TOLOG, NULL); snprintf(snbuf, sizeof(snbuf), "%d", level); for (p = snbuf ; *p ; p++) putone(*p, TOLOG, NULL); putone('>', TOLOG, NULL); } #ifndef KLOG_NOTIMESTAMP static int needtstamp = 1; int log_ts_prec = 7; static void addtstamp(int flags, struct tty *tp) { char buf[64]; struct timespec ts; int n, prec; long fsec; prec = log_ts_prec; if (prec < 0) { prec = 0; log_ts_prec = prec; } else if (prec > 9) { prec = 9; log_ts_prec = prec; } getnanouptime(&ts); for (n = prec, fsec = ts.tv_nsec; n < 8; n++) fsec /= 10; if (n < 9) fsec = (fsec / 10) + ((fsec % 10) >= 5); n = snprintf(buf, sizeof(buf), "[% 4jd.%.*ld] ", (intmax_t)ts.tv_sec, prec, fsec); for (int i = 0; i < n; i++) putone(buf[i], flags, tp); } #endif /* * putchar: print a single character on console or user terminal. * * => if console, then the last MSGBUFS chars are saved in msgbuf * for inspection later (e.g. dmesg/syslog) * => we must already be in the mutex! */ static void putchar(int c, int flags, struct tty *tp) { if (c & KLOG_PRI) { putlogpri(c & ~KLOG_PRI); return; } #ifndef KLOG_NOTIMESTAMP if (c != '\0' && c != '\n' && needtstamp && (flags & NOTSTAMP) == 0) { addtstamp(flags, tp); needtstamp = 0; } if (c == '\n') needtstamp = 1; #endif putone(c, flags, tp); #ifdef DDB if (flags & TODDB) { db_putchar(c); return; } #endif #ifdef RND_PRINTF if (__predict_true(kprintf_inited)) { unsigned char ch = c; rnd_add_data(&rnd_printf_source, &ch, 1, 0); } #endif } /* * tablefull: warn that a system table is full */ void tablefull(const char *tab, const char *hint) { if (hint) log(LOG_ERR, "%s: table is full - %s\n", tab, hint); else log(LOG_ERR, "%s: table is full\n", tab); } /* * uprintf: print to the controlling tty of the current process * * => we may block if the tty queue is full * => no message is printed if the queue doesn't clear in a reasonable * time */ void uprintf(const char *fmt, ...) { struct proc *p = curproc; va_list ap; /* mutex_enter(&proc_lock); XXXSMP */ if (p->p_lflag & PL_CONTROLT && p->p_session->s_ttyvp) { /* No mutex needed; going to process TTY. */ va_start(ap, fmt); kprintf(fmt, TOTTY, p->p_session->s_ttyp, NULL, ap); va_end(ap); } /* mutex_exit(&proc_lock); XXXSMP */ } void uprintf_locked(const char *fmt, ...) { struct proc *p = curproc; va_list ap; if (p->p_lflag & PL_CONTROLT && p->p_session->s_ttyvp) { /* No mutex needed; going to process TTY. */ va_start(ap, fmt); kprintf(fmt, TOTTY, p->p_session->s_ttyp, NULL, ap); va_end(ap); } } /* * tprintf functions: used to send messages to a specific process * * usage: * get a tpr_t handle on a process "p" by using "tprintf_open(p)" * use the handle when calling "tprintf" * when done, do a "tprintf_close" to drop the handle */ /* * tprintf_open: get a tprintf handle on a process "p" * * => returns NULL if process can't be printed to */ tpr_t tprintf_open(struct proc *p) { tpr_t cookie; cookie = NULL; mutex_enter(&proc_lock); if (p->p_lflag & PL_CONTROLT && p->p_session->s_ttyvp) { proc_sesshold(p->p_session); cookie = (tpr_t)p->p_session; } mutex_exit(&proc_lock); return cookie; } /* * tprintf_close: dispose of a tprintf handle obtained with tprintf_open */ void tprintf_close(tpr_t sess) { if (sess) { mutex_enter(&proc_lock); /* Releases proc_lock. */ proc_sessrele((struct session *)sess); } } /* * tprintf: given tprintf handle to a process [obtained with tprintf_open], * send a message to the controlling tty for that process. * * => also sends message to /dev/klog */ void tprintf(tpr_t tpr, const char *fmt, ...) { struct session *sess = (struct session *)tpr; struct tty *tp = NULL; int flags = TOLOG; va_list ap; /* mutex_enter(&proc_lock); XXXSMP */ if (sess && sess->s_ttyvp && ttycheckoutq(sess->s_ttyp, 0)) { flags |= TOTTY; tp = sess->s_ttyp; } kprintf_lock(); klogpri(LOG_INFO); va_start(ap, fmt); kprintf(fmt, flags, tp, NULL, ap); va_end(ap); kprintf_unlock(); /* mutex_exit(&proc_lock); XXXSMP */ logwakeup(); } /* * ttyprintf: send a message to a specific tty * * => should be used only by tty driver or anything that knows the * underlying tty will not be revoked(2)'d away. [otherwise, * use tprintf] */ void ttyprintf(struct tty *tp, const char *fmt, ...) { va_list ap; /* No mutex needed; going to process TTY. */ va_start(ap, fmt); kprintf(fmt, TOTTY, tp, NULL, ap); va_end(ap); } #ifdef DDB /* * db_printf: printf for DDB (via db_putchar) */ void db_printf(const char *fmt, ...) { va_list ap; /* No mutex needed; DDB pauses all processors. */ va_start(ap, fmt); kprintf(fmt, TODDB, NULL, NULL, ap); va_end(ap); if (db_tee_msgbuf) { va_start(ap, fmt); kprintf(fmt, TOLOG, NULL, NULL, ap); va_end(ap); } } void db_vprintf(const char *fmt, va_list ap) { va_list cap; va_copy(cap, ap); /* No mutex needed; DDB pauses all processors. */ kprintf(fmt, TODDB, NULL, NULL, ap); if (db_tee_msgbuf) kprintf(fmt, TOLOG, NULL, NULL, cap); va_end(cap); } #endif /* DDB */ static void kprintf_internal(const char *fmt, int oflags, void *vp, char *sbuf, ...) { va_list ap; va_start(ap, sbuf); (void)kprintf(fmt, oflags, vp, sbuf, ap); va_end(ap); } /* * Device autoconfiguration printf routines. These change their * behavior based on the AB_* flags in boothowto. If AB_SILENT * is set, messages never go to the console (but they still always * go to the log). AB_VERBOSE overrides AB_SILENT. */ /* * aprint_normal: Send to console unless AB_QUIET. Always goes * to the log. */ static void aprint_normal_internal(const char *prefix, const char *fmt, va_list ap) { int flags = TOLOG; if ((boothowto & (AB_SILENT|AB_QUIET)) == 0 || (boothowto & AB_VERBOSE) != 0) flags |= TOCONS; kprintf_lock(); if (prefix) kprintf_internal("%s: ", flags, NULL, NULL, prefix); kprintf(fmt, flags, NULL, NULL, ap); kprintf_unlock(); if (!panicstr) logwakeup(); } void aprint_normal(const char *fmt, ...) { va_list ap; va_start(ap, fmt); aprint_normal_internal(NULL, fmt, ap); va_end(ap); } void aprint_normal_dev(device_t dv, const char *fmt, ...) { va_list ap; KASSERT(dv != NULL); va_start(ap, fmt); aprint_normal_internal(device_xname(dv), fmt, ap); va_end(ap); } void aprint_normal_ifnet(struct ifnet *ifp, const char *fmt, ...) { va_list ap; KASSERT(ifp != NULL); va_start(ap, fmt); aprint_normal_internal(ifp->if_xname, fmt, ap); va_end(ap); } /* * aprint_error: Send to console unless AB_QUIET. Always goes * to the log. Also counts the number of times called so other * parts of the kernel can report the number of errors during a * given phase of system startup. */ static int aprint_error_count; int aprint_get_error_count(void) { int count; kprintf_lock(); count = aprint_error_count; aprint_error_count = 0; kprintf_unlock(); return (count); } static void aprint_error_internal(const char *prefix, const char *fmt, va_list ap) { int flags = TOLOG; if ((boothowto & (AB_SILENT|AB_QUIET)) == 0 || (boothowto & AB_VERBOSE) != 0) flags |= TOCONS; kprintf_lock(); aprint_error_count++; if (prefix) kprintf_internal("%s: ", flags, NULL, NULL, prefix); kprintf_internal("autoconfiguration error: ", TOLOG, NULL, NULL); kprintf(fmt, flags, NULL, NULL, ap); kprintf_unlock(); if (!panicstr) logwakeup(); } void aprint_error(const char *fmt, ...) { va_list ap; va_start(ap, fmt); aprint_error_internal(NULL, fmt, ap); va_end(ap); } void aprint_error_dev(device_t dv, const char *fmt, ...) { va_list ap; KASSERT(dv != NULL); va_start(ap, fmt); aprint_error_internal(device_xname(dv), fmt, ap); va_end(ap); } void aprint_error_ifnet(struct ifnet *ifp, const char *fmt, ...) { va_list ap; KASSERT(ifp != NULL); va_start(ap, fmt); aprint_error_internal(ifp->if_xname, fmt, ap); va_end(ap); } /* * aprint_naive: Send to console only if AB_QUIET. Never goes * to the log. */ static void aprint_naive_internal(const char *prefix, const char *fmt, va_list ap) { if ((boothowto & (AB_QUIET|AB_SILENT|AB_VERBOSE)) != AB_QUIET) return; kprintf_lock(); if (prefix) kprintf_internal("%s: ", TOCONS, NULL, NULL, prefix); kprintf(fmt, TOCONS, NULL, NULL, ap); kprintf_unlock(); } void aprint_naive(const char *fmt, ...) { va_list ap; va_start(ap, fmt); aprint_naive_internal(NULL, fmt, ap); va_end(ap); } void aprint_naive_dev(device_t dv, const char *fmt, ...) { va_list ap; KASSERT(dv != NULL); va_start(ap, fmt); aprint_naive_internal(device_xname(dv), fmt, ap); va_end(ap); } void aprint_naive_ifnet(struct ifnet *ifp, const char *fmt, ...) { va_list ap; KASSERT(ifp != NULL); va_start(ap, fmt); aprint_naive_internal(ifp->if_xname, fmt, ap); va_end(ap); } /* * aprint_verbose: Send to console only if AB_VERBOSE. Always * goes to the log. */ static void aprint_verbose_internal(const char *prefix, const char *fmt, va_list ap) { int flags = TOLOG; if (boothowto & AB_VERBOSE) flags |= TOCONS; kprintf_lock(); if (prefix) kprintf_internal("%s: ", flags, NULL, NULL, prefix); kprintf(fmt, flags, NULL, NULL, ap); kprintf_unlock(); if (!panicstr) logwakeup(); } void aprint_verbose(const char *fmt, ...) { va_list ap; va_start(ap, fmt); aprint_verbose_internal(NULL, fmt, ap); va_end(ap); } void aprint_verbose_dev(device_t dv, const char *fmt, ...) { va_list ap; KASSERT(dv != NULL); va_start(ap, fmt); aprint_verbose_internal(device_xname(dv), fmt, ap); va_end(ap); } void aprint_verbose_ifnet(struct ifnet *ifp, const char *fmt, ...) { va_list ap; KASSERT(ifp != NULL); va_start(ap, fmt); aprint_verbose_internal(ifp->if_xname, fmt, ap); va_end(ap); } /* * aprint_debug: Send to console and log only if AB_DEBUG. */ static void aprint_debug_internal(const char *prefix, const char *fmt, va_list ap) { if ((boothowto & AB_DEBUG) == 0) return; kprintf_lock(); if (prefix) kprintf_internal("%s: ", TOCONS | TOLOG, NULL, NULL, prefix); kprintf(fmt, TOCONS | TOLOG, NULL, NULL, ap); kprintf_unlock(); } void aprint_debug(const char *fmt, ...) { va_list ap; va_start(ap, fmt); aprint_debug_internal(NULL, fmt, ap); va_end(ap); } void aprint_debug_dev(device_t dv, const char *fmt, ...) { va_list ap; KASSERT(dv != NULL); va_start(ap, fmt); aprint_debug_internal(device_xname(dv), fmt, ap); va_end(ap); } void aprint_debug_ifnet(struct ifnet *ifp, const char *fmt, ...) { va_list ap; KASSERT(ifp != NULL); va_start(ap, fmt); aprint_debug_internal(ifp->if_xname, fmt, ap); va_end(ap); } void vprintf_flags(int flags, const char *fmt, va_list ap) { kprintf_lock(); kprintf(fmt, flags, NULL, NULL, ap); kprintf_unlock(); } void printf_flags(int flags, const char *fmt, ...) { va_list ap; va_start(ap, fmt); vprintf_flags(flags, fmt, ap); va_end(ap); } void printf_tolog(const char *fmt, ...) { va_list ap; va_start(ap, fmt); vprintf_flags(TOLOG, fmt, ap); va_end(ap); } /* * printf_nolog: Like printf(), but does not send message to the log. */ void printf_nolog(const char *fmt, ...) { va_list ap; va_start(ap, fmt); vprintf_flags(TOCONS, fmt, ap); va_end(ap); } /* * printf_nostamp: Like printf(), but does not prepend a timestamp. */ void printf_nostamp(const char *fmt, ...) { va_list ap; va_start(ap, fmt); vprintf_flags(TOCONS|NOTSTAMP, fmt, ap); va_end(ap); } /* * normal kernel printf functions: printf, vprintf, snprintf, vsnprintf */ /* * printf: print a message to the console and the log */ void printf(const char *fmt, ...) { va_list ap; va_start(ap, fmt); vprintf_flags(TOCONS | TOLOG, fmt, ap); va_end(ap); } /* * vprintf: print a message to the console and the log [already have * va_list] */ void vprintf(const char *fmt, va_list ap) { vprintf_flags(TOCONS | TOLOG, fmt, ap); if (!panicstr) logwakeup(); } /* * snprintf: print a message to a buffer */ int snprintf(char *bf, size_t size, const char *fmt, ...) { int retval; va_list ap; va_start(ap, fmt); retval = vsnprintf(bf, size, fmt, ap); va_end(ap); return retval; } /* * vsnprintf: print a message to a buffer [already have va_list] */ int vsnprintf(char *bf, size_t size, const char *fmt, va_list ap) { int retval; char *p; p = bf + size; retval = kprintf(fmt, TOBUFONLY, &p, bf, ap); if (bf && size > 0) { /* nul terminate */ if (size <= (size_t)retval) bf[size - 1] = '\0'; else bf[retval] = '\0'; } return retval; } int vasprintf(char **bf, const char *fmt, va_list ap) { int retval; va_list cap; va_copy(cap, ap); retval = kprintf(fmt, TOBUFONLY, NULL, NULL, cap) + 1; va_end(cap); *bf = kmem_alloc(retval, KM_SLEEP); return vsnprintf(*bf, retval, fmt, ap); } /* * kprintf: scaled down version of printf(3). * * this version based on vfprintf() from libc which was derived from * software contributed to Berkeley by Chris Torek. * * NOTE: The kprintf mutex must be held if we're going TOBUF or TOCONS! */ /* * macros for converting digits to letters and vice versa */ #define to_digit(c) ((c) - '0') #define is_digit(c) ((unsigned)to_digit(c) <= 9) #define to_char(n) ((n) + '0') /* * flags used during conversion. */ #define ALT 0x001 /* alternate form */ #define HEXPREFIX 0x002 /* add 0x or 0X prefix */ #define LADJUST 0x004 /* left adjustment */ #define LONGDBL 0x008 /* long double; unimplemented */ #define LONGINT 0x010 /* long integer */ #define QUADINT 0x020 /* quad integer */ #define SHORTINT 0x040 /* short integer */ #define MAXINT 0x080 /* intmax_t */ #define PTRINT 0x100 /* intptr_t */ #define SIZEINT 0x200 /* size_t */ #define ZEROPAD 0x400 /* zero (as opposed to blank) pad */ #define FPT 0x800 /* Floating point number */ /* * To extend shorts properly, we need both signed and unsigned * argument extraction methods. */ #define SARG() \ (flags&MAXINT ? va_arg(ap, intmax_t) : \ flags&PTRINT ? va_arg(ap, intptr_t) : \ flags&SIZEINT ? va_arg(ap, ssize_t) : /* XXX */ \ flags&QUADINT ? va_arg(ap, quad_t) : \ flags&LONGINT ? va_arg(ap, long) : \ flags&SHORTINT ? (long)(short)va_arg(ap, int) : \ (long)va_arg(ap, int)) #define UARG() \ (flags&MAXINT ? va_arg(ap, uintmax_t) : \ flags&PTRINT ? va_arg(ap, uintptr_t) : \ flags&SIZEINT ? va_arg(ap, size_t) : \ flags&QUADINT ? va_arg(ap, u_quad_t) : \ flags&LONGINT ? va_arg(ap, u_long) : \ flags&SHORTINT ? (u_long)(u_short)va_arg(ap, int) : \ (u_long)va_arg(ap, u_int)) #define KPRINTF_PUTCHAR(C) { \ if (oflags == TOBUFONLY) { \ if (sbuf && ((vp == NULL) || (sbuf < tailp))) \ *sbuf++ = (C); \ } else { \ putchar((C), oflags, vp); \ } \ } void device_printf(device_t dev, const char *fmt, ...) { va_list ap; va_start(ap, fmt); printf("%s: ", device_xname(dev)); vprintf(fmt, ap); va_end(ap); return; } /* * Guts of kernel printf. Note, we already expect to be in a mutex! */ int kprintf(const char *fmt0, int oflags, void *vp, char *sbuf, va_list ap) { const char *fmt; /* format string */ int ch; /* character from fmt */ int n; /* handy integer (short term usage) */ char *cp; /* handy char pointer (short term usage) */ int flags; /* flags as above */ int ret; /* return value accumulator */ int width; /* width from format (%8d), or 0 */ int prec; /* precision from format (%.3d), or -1 */ char sign; /* sign prefix (' ', '+', '-', or \0) */ u_quad_t _uquad; /* integer arguments %[diouxX] */ enum { OCT, DEC, HEX } base;/* base for [diouxX] conversion */ int dprec; /* a copy of prec if [diouxX], 0 otherwise */ int realsz; /* field size expanded by dprec */ int size; /* size of converted field or string */ const char *xdigs; /* digits for [xX] conversion */ char bf[KPRINTF_BUFSIZE]; /* space for %c, %[diouxX] */ char *tailp; /* tail pointer for snprintf */ if (oflags == TOBUFONLY && (vp != NULL)) tailp = *(char **)vp; else tailp = NULL; cp = NULL; /* XXX: shutup gcc */ size = 0; /* XXX: shutup gcc */ fmt = fmt0; ret = 0; xdigs = NULL; /* XXX: shut up gcc warning */ /* * Scan the format for conversions (`%' character). */ for (;;) { for (; *fmt != '%' && *fmt; fmt++) { ret++; KPRINTF_PUTCHAR(*fmt); } if (*fmt == 0) goto done; fmt++; /* skip over '%' */ flags = 0; dprec = 0; width = 0; prec = -1; sign = '\0'; rflag: ch = *fmt++; reswitch: switch (ch) { case ' ': /* * ``If the space and + flags both appear, the space * flag will be ignored.'' * -- ANSI X3J11 */ if (!sign) sign = ' '; goto rflag; case '#': flags |= ALT; goto rflag; case '*': /* * ``A negative field width argument is taken as a * - flag followed by a positive field width.'' * -- ANSI X3J11 * They don't exclude field widths read from args. */ if ((width = va_arg(ap, int)) >= 0) goto rflag; width = -width; /* FALLTHROUGH */ case '-': flags |= LADJUST; goto rflag; case '+': sign = '+'; goto rflag; case '.': if ((ch = *fmt++) == '*') { n = va_arg(ap, int); prec = n < 0 ? -1 : n; goto rflag; } n = 0; while (is_digit(ch)) { n = 10 * n + to_digit(ch); ch = *fmt++; } prec = n < 0 ? -1 : n; goto reswitch; case '0': /* * ``Note that 0 is taken as a flag, not as the * beginning of a field width.'' * -- ANSI X3J11 */ flags |= ZEROPAD; goto rflag; case '1': case '2': case '3': case '4': case '5': case '6': case '7': case '8': case '9': n = 0; do { n = 10 * n + to_digit(ch); ch = *fmt++; } while (is_digit(ch)); width = n; goto reswitch; case 'h': flags |= SHORTINT; goto rflag; case 'j': flags |= MAXINT; goto rflag; case 'l': if (*fmt == 'l') { fmt++; flags |= QUADINT; } else { flags |= LONGINT; } goto rflag; case 'q': flags |= QUADINT; goto rflag; case 't': flags |= PTRINT; goto rflag; case 'z': flags |= SIZEINT; goto rflag; case 'c': *(cp = bf) = va_arg(ap, int); size = 1; sign = '\0'; break; case 'D': flags |= LONGINT; /*FALLTHROUGH*/ case 'd': case 'i': _uquad = SARG(); if ((quad_t)_uquad < 0) { _uquad = -_uquad; sign = '-'; } base = DEC; goto number; case 'n': /* no %n support in the kernel, consume and skip */ if (flags & MAXINT) (void)va_arg(ap, intmax_t *); else if (flags & PTRINT) (void)va_arg(ap, intptr_t *); else if (flags & SIZEINT) (void)va_arg(ap, ssize_t *); else if (flags & QUADINT) (void)va_arg(ap, quad_t *); else if (flags & LONGINT) (void)va_arg(ap, long *); else if (flags & SHORTINT) (void)va_arg(ap, short *); else (void)va_arg(ap, int *); continue; /* no output */ case 'O': flags |= LONGINT; /*FALLTHROUGH*/ case 'o': _uquad = UARG(); base = OCT; goto nosign; case 'p': /* * ``The argument shall be a pointer to void. The * value of the pointer is converted to a sequence * of printable characters, in an implementation- * defined manner.'' * -- ANSI X3J11 */ /* NOSTRICT */ _uquad = (u_long)va_arg(ap, void *); base = HEX; xdigs = hexdigits; flags |= HEXPREFIX; ch = 'x'; goto nosign; case 's': if ((cp = va_arg(ap, char *)) == NULL) /*XXXUNCONST*/ cp = __UNCONST("(null)"); if (prec >= 0) { /* * can't use strlen; can only look for the * NUL in the first `prec' characters, and * strlen() will go further. */ char *p = memchr(cp, 0, prec); if (p != NULL) { size = p - cp; if (size > prec) size = prec; } else size = prec; } else size = strlen(cp); sign = '\0'; break; case 'U': flags |= LONGINT; /*FALLTHROUGH*/ case 'u': _uquad = UARG(); base = DEC; goto nosign; case 'X': xdigs = HEXDIGITS; goto hex; case 'x': xdigs = hexdigits; hex: _uquad = UARG(); base = HEX; /* leading 0x/X only if non-zero */ if (flags & ALT && _uquad != 0) flags |= HEXPREFIX; /* unsigned conversions */ nosign: sign = '\0'; /* * ``... diouXx conversions ... if a precision is * specified, the 0 flag will be ignored.'' * -- ANSI X3J11 */ number: if ((dprec = prec) >= 0) flags &= ~ZEROPAD; /* * ``The result of converting a zero value with an * explicit precision of zero is no characters.'' * -- ANSI X3J11 */ cp = bf + KPRINTF_BUFSIZE; if (_uquad != 0 || prec != 0) { /* * Unsigned mod is hard, and unsigned mod * by a constant is easier than that by * a variable; hence this switch. */ switch (base) { case OCT: do { *--cp = to_char(_uquad & 7); _uquad >>= 3; } while (_uquad); /* handle octal leading 0 */ if (flags & ALT && *cp != '0') *--cp = '0'; break; case DEC: /* many numbers are 1 digit */ while (_uquad >= 10) { *--cp = to_char(_uquad % 10); _uquad /= 10; } *--cp = to_char(_uquad); break; case HEX: do { *--cp = xdigs[_uquad & 15]; _uquad >>= 4; } while (_uquad); break; default: /*XXXUNCONST*/ cp = __UNCONST("bug in kprintf: bad base"); size = strlen(cp); goto skipsize; } } size = bf + KPRINTF_BUFSIZE - cp; skipsize: break; default: /* "%?" prints ?, unless ? is NUL */ if (ch == '\0') goto done; /* pretend it was %c with argument ch */ cp = bf; *cp = ch; size = 1; sign = '\0'; break; } /* * All reasonable formats wind up here. At this point, `cp' * points to a string which (if not flags&LADJUST) should be * padded out to `width' places. If flags&ZEROPAD, it should * first be prefixed by any sign or other prefix; otherwise, * it should be blank padded before the prefix is emitted. * After any left-hand padding and prefixing, emit zeroes * required by a decimal [diouxX] precision, then print the * string proper, then emit zeroes required by any leftover * floating precision; finally, if LADJUST, pad with blanks. * * Compute actual size, so we know how much to pad. * size excludes decimal prec; realsz includes it. */ realsz = dprec > size ? dprec : size; if (sign) realsz++; else if (flags & HEXPREFIX) realsz+= 2; /* adjust ret */ ret += width > realsz ? width : realsz; /* right-adjusting blank padding */ if ((flags & (LADJUST|ZEROPAD)) == 0) { n = width - realsz; while (n-- > 0) KPRINTF_PUTCHAR(' '); } /* prefix */ if (sign) { KPRINTF_PUTCHAR(sign); } else if (flags & HEXPREFIX) { KPRINTF_PUTCHAR('0'); KPRINTF_PUTCHAR(ch); } /* right-adjusting zero padding */ if ((flags & (LADJUST|ZEROPAD)) == ZEROPAD) { n = width - realsz; while (n-- > 0) KPRINTF_PUTCHAR('0'); } /* leading zeroes from decimal precision */ n = dprec - size; while (n-- > 0) KPRINTF_PUTCHAR('0'); /* the string or number proper */ for (; size--; cp++) KPRINTF_PUTCHAR(*cp); /* left-adjusting padding (always blank) */ if (flags & LADJUST) { n = width - realsz; while (n-- > 0) KPRINTF_PUTCHAR(' '); } } done: if ((oflags == TOBUFONLY) && (vp != NULL)) *(char **)vp = sbuf; (*v_flush)(); #ifdef RND_PRINTF if (__predict_true(kprintf_inited)) rnd_add_data(&rnd_printf_source, NULL, 0, 0); #endif return ret; } |
| 29 29 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 197 198 199 200 201 202 203 204 205 206 207 208 209 210 211 212 213 214 215 216 217 218 219 220 221 222 223 224 225 226 227 228 229 230 231 232 233 234 235 236 237 238 239 240 241 242 243 244 245 246 247 248 249 250 251 252 253 254 255 256 257 258 259 260 261 262 263 264 265 266 267 268 269 270 271 272 273 274 275 276 277 278 279 280 281 282 283 284 285 286 287 288 289 290 291 292 293 294 295 296 297 298 299 300 301 302 303 304 305 306 307 308 309 310 311 312 313 314 315 316 317 318 319 320 321 322 323 324 325 326 327 328 329 330 331 332 333 334 335 336 337 338 339 340 341 342 343 344 345 346 347 348 349 350 351 352 353 354 355 356 357 358 359 360 361 362 363 364 365 366 367 368 369 370 371 372 373 374 375 376 377 378 379 380 381 382 383 384 385 386 387 388 389 390 391 392 393 394 395 396 397 398 399 400 401 402 403 404 405 406 407 408 409 410 411 412 413 414 415 416 417 418 419 420 421 422 423 424 425 426 427 428 429 430 431 432 433 434 435 436 437 438 439 440 441 442 443 444 445 446 447 448 449 450 451 452 453 454 455 456 457 458 459 460 461 462 463 464 465 466 467 468 469 470 471 472 473 474 475 476 477 478 479 480 481 482 483 484 485 486 487 488 489 490 491 492 493 494 495 496 497 498 499 500 501 502 503 504 505 506 507 508 509 510 511 512 513 514 515 516 517 518 519 520 521 522 523 524 525 526 527 528 529 530 531 532 533 534 535 536 537 538 539 540 541 542 543 544 545 546 547 548 549 550 551 552 553 554 555 556 557 558 559 560 561 562 563 564 565 566 567 568 569 570 571 572 573 574 575 576 577 578 579 580 581 582 583 584 585 586 587 588 589 590 591 592 593 594 595 596 597 598 599 600 601 602 603 604 605 606 607 608 609 610 611 612 613 614 615 616 617 618 619 620 621 622 623 624 625 626 627 628 629 630 631 632 633 634 635 636 637 638 639 640 641 642 643 644 645 646 647 648 649 650 651 652 653 654 655 656 657 658 659 660 661 662 663 664 665 666 667 668 669 670 671 672 673 674 675 676 677 678 679 680 681 682 683 684 685 686 687 688 689 690 691 692 693 694 695 696 697 698 699 700 701 702 703 704 705 706 707 708 709 710 711 712 713 714 715 716 717 718 719 720 721 722 723 724 725 726 727 728 729 730 731 732 733 734 735 736 737 738 739 740 741 742 743 744 745 746 747 748 749 750 751 752 753 754 755 756 757 758 759 760 761 762 763 764 765 766 767 768 769 770 771 772 773 774 775 776 777 778 779 780 781 782 783 784 785 786 787 788 789 790 791 792 793 794 795 796 797 798 799 800 801 802 803 804 805 806 807 808 809 810 811 812 813 814 815 816 817 818 819 820 821 822 823 824 825 826 827 828 829 830 831 832 833 834 835 836 837 838 839 840 841 842 843 844 845 846 847 848 849 850 851 852 853 854 855 856 857 858 859 860 861 862 863 864 865 866 867 868 869 870 871 872 873 874 875 876 877 878 879 880 881 882 883 884 885 886 887 888 889 890 891 892 893 894 895 896 897 898 899 900 901 902 903 904 905 906 907 908 909 910 911 912 913 914 915 916 917 918 919 920 921 922 923 924 925 926 927 928 929 930 931 932 933 934 935 936 937 938 939 940 941 942 943 944 945 946 947 948 949 950 951 952 953 954 955 956 957 958 959 960 961 962 963 964 965 966 967 968 969 970 971 972 973 974 975 976 977 978 979 980 981 982 983 984 985 986 987 988 989 990 991 992 993 994 995 996 997 998 999 1000 1001 1002 1003 1004 1005 1006 1007 1008 1009 1010 1011 1012 1013 1014 1015 1016 1017 1018 1019 1020 1021 1022 1023 1024 1025 1026 1027 1028 1029 1030 1031 1032 1033 1034 1035 1036 1037 1038 1039 1040 1041 1042 1043 1044 1045 1046 1047 1048 1049 1050 1051 1052 1053 1054 1055 1056 1057 1058 1059 1060 1061 1062 1063 1064 1065 1066 1067 1068 1069 1070 1071 1072 1073 1074 1075 1076 1077 1078 1079 1080 1081 1082 1083 1084 1085 1086 1087 1088 1089 1090 1091 1092 1093 1094 1095 1096 1097 1098 1099 1100 1101 1102 1103 1104 1105 1106 1107 1108 1109 1110 1111 1112 1113 1114 1115 1116 1117 1118 1119 1120 1121 1122 1123 1124 1125 1126 1127 1128 1129 1130 1131 1132 1133 1134 1135 1136 1137 1138 1139 1140 1141 1142 1143 1144 1145 1146 1147 1148 1149 1150 1151 1152 1153 1154 1155 1156 1157 1158 1159 1160 1161 1162 1163 1164 1165 1166 1167 1168 1169 1170 1171 1172 1173 1174 1175 1176 1177 1178 1179 1180 1181 1182 1183 1184 1185 1186 1187 1188 1189 1190 1191 1192 1193 1194 1195 1196 1197 1198 1199 1200 1201 1202 1203 1204 1205 1206 1207 1208 1209 1210 1211 1212 1213 1214 1215 1216 1217 1218 1219 1220 1221 1222 1223 1224 1225 1226 1227 1228 1229 1230 1231 1232 1233 1234 1235 1236 1237 1238 1239 1240 1241 1242 1243 1244 1245 1246 1247 1248 1249 1250 1251 1252 1253 1254 1255 1256 1257 1258 1259 1260 1261 1262 1263 1264 1265 1266 1267 1268 1269 1270 1271 1272 1273 1274 1275 1276 1277 1278 1279 1280 1281 1282 1283 1284 1285 1286 1287 1288 1289 1290 1291 1292 1293 1294 1295 1296 1297 1298 1299 1300 1301 1302 1303 1304 1305 1306 1307 1308 1309 1310 1311 1312 1313 1314 1315 1316 1317 1318 1319 1320 1321 1322 1323 1324 1325 1326 1327 1328 1329 1330 1331 1332 1333 1334 1335 1336 1337 1338 1339 1340 1341 1342 1343 1344 1345 1346 1347 1348 1349 1350 1351 1352 1353 1354 1355 1356 1357 1358 1359 1360 1361 1362 1363 1364 1365 1366 1367 1368 1369 1370 1371 1372 1373 1374 1375 1376 1377 1378 1379 1380 1381 1382 1383 1384 1385 1386 1387 1388 1389 1390 1391 1392 1393 1394 1395 1396 1397 1398 1399 1400 1401 1402 1403 1404 1405 1406 1407 1408 1409 1410 1411 1412 1413 1414 1415 1416 1417 1418 1419 1420 1421 1422 1423 1424 1425 1426 1427 1428 1429 1430 1431 1432 1433 1434 1435 1436 1437 1438 1439 1440 1441 1442 1443 1444 1445 1446 1447 1448 1449 1450 1451 1452 1453 1454 1455 1456 1457 1458 1459 1460 1461 1462 1463 1464 1465 1466 1467 1468 1469 1470 1471 1472 1473 1474 1475 1476 1477 1478 1479 1480 1481 1482 1483 1484 1485 1486 1487 1488 1489 1490 1491 1492 1493 1494 1495 1496 1497 1498 1499 1500 1501 1502 1503 1504 1505 1506 1507 1508 1509 1510 1511 1512 1513 1514 1515 1516 1517 1518 1519 1520 1521 1522 1523 1524 1525 1526 1527 1528 1529 1530 1531 1532 1533 1534 1535 1536 1537 1538 1539 1540 1541 1542 1543 1544 1545 1546 1547 1548 1549 1550 1551 1552 1553 1554 1555 1556 1557 1558 1559 1560 1561 1562 1563 1564 1565 1566 1567 1568 1569 1570 1571 1572 1573 1574 1575 1576 1577 1578 1579 1580 1581 1582 1583 1584 1585 1586 1587 1588 1589 1590 1591 1592 1593 1594 1595 1596 1597 1598 1599 1600 1601 1602 1603 1604 1605 1606 1607 1608 1609 1610 1611 1612 1613 1614 1615 1616 1617 1618 1619 1620 1621 1622 1623 1624 1625 1626 1627 1628 1629 1630 1631 1632 1633 1634 1635 1636 1637 1638 1639 1640 1641 1642 1643 1644 1645 1646 1647 1648 1649 1650 1651 1652 1653 1654 1655 1656 1657 1658 1659 1660 1661 1662 1663 1664 1665 1666 1667 1668 1669 1670 1671 1672 1673 1674 1675 1676 1677 1678 1679 1680 1681 1682 1683 1684 1685 1686 1687 1688 1689 1690 1691 1692 1693 1694 1695 1696 1697 1698 1699 1700 1701 1702 1703 1704 1705 1706 1707 1708 1709 1710 1711 1712 1713 1714 1715 1716 1717 1718 1719 1720 1721 1722 1723 1724 1725 1726 1727 1728 1729 1730 1731 1732 1733 1734 1735 1736 1737 1738 1739 1740 1741 1742 1743 1744 1745 1746 1747 1748 1749 1750 1751 1752 1753 1754 1755 1756 1757 1758 1759 1760 1761 1762 1763 1764 1765 1766 1767 1768 1769 1770 1771 1772 1773 1774 1775 1776 1777 1778 1779 1780 1781 1782 1783 1784 1785 1786 1787 1788 1789 1790 1791 1792 1793 1794 1795 1796 1797 1798 1799 1800 1801 1802 1803 1804 1805 1806 1807 1808 1809 1810 1811 1812 1813 1814 1815 1816 1817 1818 1819 1820 1821 1822 1823 1824 1825 1826 1827 1828 1829 1830 1831 1832 1833 1834 1835 1836 1837 1838 1839 1840 1841 1842 1843 1844 1845 1846 1847 1848 1849 1850 1851 1852 1853 1854 1855 1856 1857 1858 1859 1860 1861 1862 1863 1864 1865 1866 1867 1868 1869 1870 1871 1872 1873 1874 1875 1876 1877 1878 1879 1880 1881 1882 1883 1884 1885 1886 1887 1888 1889 1890 1891 1892 1893 1894 1895 1896 1897 1898 1899 1900 1901 1902 1903 1904 1905 1906 1907 1908 1909 1910 1911 1912 1913 1914 1915 1916 1917 1918 1919 1920 1921 1922 1923 1924 1925 1926 1927 1928 1929 1930 1931 1932 1933 1934 1935 1936 1937 1938 1939 1940 1941 1942 1943 1944 1945 1946 1947 1948 1949 1950 1951 1952 1953 1954 1955 1956 1957 1958 1959 1960 1961 1962 1963 1964 1965 1966 1967 1968 1969 1970 1971 1972 1973 1974 1975 1976 1977 1978 1979 1980 1981 1982 1983 1984 1985 1986 1987 1988 1989 1990 1991 1992 1993 1994 1995 1996 1997 1998 1999 2000 2001 2002 2003 2004 2005 2006 2007 2008 2009 2010 2011 2012 2013 2014 2015 2016 2017 2018 2019 2020 2021 2022 2023 2024 2025 2026 2027 2028 2029 2030 2031 2032 2033 2034 2035 2036 2037 2038 2039 2040 2041 2042 2043 2044 2045 2046 2047 2048 2049 2050 2051 2052 2053 2054 2055 2056 2057 2058 2059 2060 2061 2062 2063 2064 2065 2066 2067 2068 2069 2070 2071 2072 2073 2074 2075 2076 2077 2078 2079 2080 2081 2082 2083 2084 2085 2086 2087 2088 2089 2090 2091 2092 2093 2094 2095 2096 2097 2098 2099 2100 2101 2102 2103 2104 2105 2106 2107 2108 2109 2110 2111 2112 2113 2114 2115 2116 2117 2118 2119 2120 2121 2122 2123 2124 2125 2126 2127 2128 2129 2130 2131 2132 2133 2134 2135 2136 2137 2138 2139 2140 2141 2142 2143 2144 2145 2146 2147 2148 2149 2150 2151 2152 2153 2154 2155 2156 2157 2158 2159 2160 2161 2162 2163 2164 2165 2166 2167 2168 2169 2170 2171 2172 2173 2174 2175 2176 2177 2178 2179 2180 2181 2182 2183 2184 2185 2186 2187 2188 2189 2190 2191 2192 2193 2194 2195 2196 2197 2198 2199 2200 2201 2202 2203 2204 2205 2206 2207 2208 2209 2210 2211 2212 2213 2214 2215 2216 2217 2218 2219 2220 2221 2222 2223 2224 2225 2226 2227 2228 2229 2230 2231 2232 2233 2234 2235 2236 2237 2238 2239 2240 2241 2242 2243 2244 2245 2246 2247 2248 2249 2250 2251 2252 2253 2254 2255 2256 2257 2258 2259 2260 2261 2262 2263 2264 2265 2266 2267 2268 2269 2270 2271 2272 2273 2274 2275 2276 2277 2278 2279 2280 2281 2282 2283 2284 2285 2286 2287 2288 2289 2290 2291 2292 2293 2294 2295 2296 2297 2298 2299 2300 2301 2302 2303 2304 2305 2306 2307 2308 2309 2310 2311 2312 2313 2314 2315 2316 2317 2318 2319 2320 2321 2322 2323 2324 2325 2326 2327 2328 2329 2330 2331 2332 2333 2334 2335 2336 2337 2338 2339 2340 2341 2342 2343 2344 2345 2346 2347 2348 2349 2350 2351 2352 2353 2354 2355 2356 2357 2358 2359 2360 2361 2362 2363 2364 2365 2366 2367 2368 2369 2370 2371 2372 2373 2374 2375 2376 2377 2378 2379 2380 2381 2382 2383 2384 2385 2386 2387 2388 2389 2390 2391 2392 2393 2394 2395 2396 2397 2398 2399 2400 2401 2402 2403 2404 2405 2406 2407 2408 2409 2410 2411 2412 2413 2414 2415 2416 2417 2418 2419 2420 2421 2422 2423 2424 2425 2426 2427 2428 2429 2430 2431 2432 2433 2434 2435 2436 2437 2438 2439 2440 2441 2442 2443 2444 2445 2446 2447 2448 2449 2450 2451 2452 2453 2454 2455 2456 2457 2458 2459 2460 2461 2462 2463 2464 2465 2466 2467 2468 2469 2470 2471 2472 2473 2474 2475 2476 2477 2478 2479 2480 2481 2482 2483 2484 2485 2486 2487 2488 2489 2490 2491 2492 2493 2494 2495 2496 2497 2498 2499 2500 2501 2502 2503 2504 2505 2506 2507 2508 2509 2510 2511 2512 2513 2514 2515 2516 2517 2518 2519 2520 2521 2522 2523 2524 2525 2526 2527 2528 2529 2530 2531 2532 2533 2534 2535 2536 2537 2538 2539 2540 2541 2542 2543 2544 2545 2546 2547 2548 2549 2550 2551 2552 2553 2554 2555 2556 2557 2558 2559 2560 2561 2562 2563 2564 2565 2566 2567 2568 2569 2570 2571 2572 2573 2574 2575 2576 2577 2578 2579 2580 2581 2582 2583 2584 2585 2586 2587 2588 2589 2590 2591 2592 2593 2594 2595 2596 2597 2598 2599 2600 2601 2602 2603 2604 2605 2606 2607 2608 2609 2610 2611 2612 2613 2614 2615 2616 2617 2618 2619 2620 2621 2622 2623 2624 2625 2626 2627 2628 2629 2630 2631 2632 2633 2634 2635 2636 2637 2638 2639 2640 2641 2642 2643 2644 2645 2646 2647 2648 2649 2650 2651 2652 2653 2654 2655 2656 2657 2658 2659 2660 2661 2662 2663 2664 2665 2666 2667 2668 2669 2670 2671 2672 2673 2674 2675 2676 2677 2678 2679 2680 2681 2682 2683 2684 2685 2686 2687 2688 2689 2690 2691 2692 2693 2694 2695 2696 2697 2698 2699 2700 2701 2702 2703 2704 2705 2706 2707 2708 2709 2710 2711 2712 2713 2714 2715 2716 2717 2718 2719 2720 2721 2722 2723 2724 2725 2726 2727 2728 2729 2730 2731 2732 2733 2734 2735 2736 2737 2738 2739 2740 2741 2742 2743 2744 2745 2746 2747 2748 2749 2750 2751 2752 2753 2754 2755 2756 2757 2758 2759 2760 2761 2762 2763 2764 2765 2766 2767 2768 2769 2770 2771 2772 2773 2774 2775 2776 2777 2778 2779 2780 2781 2782 2783 2784 2785 2786 2787 2788 2789 2790 2791 2792 2793 2794 2795 2796 2797 2798 2799 2800 2801 2802 2803 2804 2805 2806 2807 2808 2809 2810 2811 2812 2813 2814 2815 2816 2817 2818 2819 2820 2821 2822 2823 2824 2825 2826 2827 2828 2829 2830 2831 2832 2833 2834 2835 2836 2837 2838 2839 2840 2841 2842 2843 2844 2845 2846 2847 2848 2849 2850 2851 2852 2853 2854 2855 2856 2857 2858 2859 2860 2861 2862 2863 2864 2865 2866 2867 2868 2869 2870 2871 2872 2873 2874 2875 2876 2877 2878 2879 2880 2881 2882 2883 2884 2885 2886 2887 2888 2889 2890 2891 2892 2893 2894 2895 2896 2897 2898 2899 2900 2901 2902 2903 2904 2905 2906 2907 2908 2909 2910 2911 2912 2913 2914 2915 2916 2917 2918 2919 2920 2921 2922 2923 2924 2925 2926 2927 2928 2929 2930 2931 2932 2933 2934 2935 2936 2937 2938 2939 2940 2941 2942 2943 2944 2945 2946 2947 2948 2949 2950 2951 2952 2953 2954 2955 2956 2957 2958 2959 2960 2961 2962 2963 2964 2965 2966 2967 2968 2969 2970 2971 2972 2973 2974 2975 2976 2977 2978 2979 2980 2981 2982 2983 2984 2985 2986 2987 2988 2989 2990 2991 2992 2993 2994 2995 2996 2997 2998 2999 3000 3001 3002 3003 3004 3005 3006 3007 3008 3009 3010 3011 3012 3013 3014 3015 3016 3017 3018 3019 3020 3021 3022 3023 3024 3025 3026 3027 3028 3029 3030 3031 3032 3033 3034 3035 3036 3037 3038 3039 3040 3041 3042 3043 3044 3045 3046 3047 3048 3049 3050 3051 3052 3053 3054 3055 3056 3057 3058 3059 3060 3061 3062 3063 3064 3065 3066 3067 3068 3069 3070 3071 3072 3073 3074 3075 3076 3077 3078 3079 3080 3081 3082 3083 3084 3085 3086 3087 3088 3089 3090 3091 3092 3093 3094 3095 3096 3097 3098 3099 3100 3101 3102 3103 3104 3105 3106 3107 3108 3109 3110 3111 3112 3113 3114 3115 3116 3117 3118 3119 3120 3121 3122 3123 3124 3125 3126 3127 3128 3129 3130 3131 3132 3133 3134 3135 3136 3137 3138 3139 3140 3141 3142 3143 3144 3145 3146 3147 3148 3149 3150 3151 3152 3153 3154 3155 3156 3157 3158 3159 3160 3161 3162 3163 3164 3165 3166 3167 3168 3169 3170 3171 3172 3173 3174 3175 3176 3177 3178 3179 3180 3181 3182 3183 3184 3185 3186 3187 3188 3189 3190 3191 3192 3193 3194 3195 3196 3197 3198 3199 3200 3201 3202 3203 3204 3205 3206 3207 3208 3209 3210 3211 3212 3213 3214 3215 3216 3217 3218 3219 3220 3221 3222 3223 3224 3225 3226 3227 3228 3229 3230 3231 3232 3233 3234 3235 3236 3237 3238 3239 3240 3241 3242 3243 3244 3245 3246 3247 3248 3249 3250 3251 3252 3253 3254 3255 3256 3257 3258 3259 3260 3261 3262 3263 3264 3265 3266 3267 3268 3269 3270 3271 3272 3273 3274 3275 3276 3277 3278 3279 3280 3281 3282 3283 3284 3285 3286 3287 3288 3289 3290 3291 3292 3293 3294 3295 3296 3297 3298 3299 3300 3301 3302 3303 3304 3305 3306 3307 3308 3309 3310 3311 3312 3313 3314 3315 3316 3317 3318 3319 3320 3321 3322 3323 3324 3325 3326 3327 3328 3329 3330 3331 3332 3333 3334 3335 3336 3337 3338 3339 3340 3341 3342 3343 3344 3345 3346 3347 3348 3349 3350 3351 3352 3353 3354 3355 3356 3357 3358 3359 3360 3361 3362 3363 3364 3365 3366 3367 3368 3369 3370 3371 3372 3373 3374 3375 3376 3377 3378 3379 3380 3381 3382 3383 3384 3385 3386 3387 3388 3389 3390 3391 3392 3393 3394 3395 3396 3397 3398 3399 3400 3401 3402 3403 3404 3405 3406 3407 3408 3409 3410 3411 3412 3413 3414 3415 3416 3417 3418 3419 3420 3421 3422 3423 3424 3425 3426 3427 3428 3429 3430 3431 3432 3433 3434 3435 3436 3437 3438 3439 3440 3441 3442 3443 3444 3445 3446 3447 3448 3449 3450 3451 3452 3453 3454 3455 3456 3457 3458 3459 3460 3461 3462 3463 3464 3465 3466 3467 3468 3469 3470 3471 3472 3473 3474 3475 3476 3477 3478 3479 3480 3481 3482 3483 3484 3485 3486 3487 3488 3489 3490 3491 3492 3493 3494 3495 3496 3497 3498 3499 3500 3501 3502 3503 3504 3505 3506 3507 3508 3509 3510 3511 3512 3513 3514 3515 3516 3517 3518 3519 3520 3521 3522 3523 3524 3525 3526 3527 3528 3529 3530 3531 3532 3533 3534 3535 3536 3537 3538 3539 3540 3541 3542 3543 3544 3545 3546 3547 3548 3549 3550 3551 3552 3553 3554 3555 3556 3557 3558 3559 3560 3561 3562 3563 3564 3565 3566 3567 3568 3569 3570 3571 3572 3573 3574 3575 3576 3577 3578 3579 3580 3581 3582 3583 3584 3585 3586 3587 3588 3589 3590 3591 3592 3593 3594 3595 3596 3597 3598 3599 3600 3601 3602 3603 3604 3605 3606 3607 3608 3609 3610 3611 3612 3613 3614 3615 3616 3617 3618 3619 3620 3621 3622 3623 3624 3625 3626 3627 3628 3629 3630 3631 3632 3633 3634 3635 3636 3637 3638 3639 3640 3641 3642 3643 3644 3645 3646 3647 3648 3649 3650 3651 3652 3653 3654 3655 3656 3657 3658 3659 3660 3661 3662 3663 3664 3665 3666 3667 3668 3669 3670 3671 3672 3673 3674 3675 3676 3677 3678 3679 3680 3681 3682 3683 3684 3685 3686 3687 3688 3689 3690 3691 3692 3693 3694 3695 3696 3697 3698 3699 3700 3701 3702 3703 3704 3705 3706 3707 3708 3709 3710 3711 3712 3713 3714 3715 3716 3717 3718 3719 3720 3721 3722 3723 3724 3725 3726 3727 3728 3729 3730 3731 3732 3733 3734 3735 3736 3737 3738 3739 3740 3741 3742 3743 3744 3745 3746 3747 3748 3749 3750 3751 3752 3753 3754 3755 3756 3757 3758 3759 3760 3761 3762 3763 3764 3765 3766 3767 3768 3769 3770 3771 3772 3773 3774 3775 3776 3777 3778 3779 3780 3781 3782 3783 3784 3785 3786 3787 3788 3789 3790 3791 3792 3793 3794 3795 3796 3797 3798 3799 3800 3801 3802 3803 3804 3805 3806 3807 3808 3809 3810 3811 3812 3813 3814 3815 3816 3817 3818 3819 3820 3821 3822 3823 3824 3825 3826 3827 3828 3829 3830 3831 3832 3833 3834 3835 3836 3837 3838 3839 3840 3841 3842 3843 3844 3845 3846 3847 3848 3849 3850 3851 3852 3853 3854 3855 3856 3857 3858 3859 3860 3861 3862 3863 3864 3865 3866 3867 3868 3869 3870 3871 3872 3873 3874 3875 3876 3877 3878 3879 3880 3881 3882 3883 3884 3885 3886 3887 3888 3889 3890 3891 3892 3893 3894 3895 3896 3897 3898 3899 3900 3901 3902 3903 3904 3905 3906 3907 3908 3909 3910 3911 3912 3913 3914 3915 3916 3917 3918 3919 3920 3921 3922 3923 3924 3925 3926 3927 3928 3929 3930 3931 3932 3933 3934 3935 3936 3937 3938 3939 3940 3941 3942 3943 3944 3945 3946 3947 3948 3949 3950 3951 3952 3953 3954 3955 3956 3957 3958 3959 3960 3961 3962 3963 3964 3965 3966 3967 3968 3969 3970 3971 3972 3973 3974 3975 3976 3977 3978 3979 3980 3981 3982 3983 3984 3985 3986 3987 3988 3989 3990 3991 3992 3993 3994 3995 3996 3997 3998 3999 4000 4001 4002 4003 4004 4005 4006 4007 4008 4009 4010 4011 4012 4013 4014 4015 4016 4017 4018 4019 4020 4021 4022 4023 4024 4025 4026 4027 4028 4029 4030 4031 4032 4033 4034 4035 4036 4037 4038 4039 4040 4041 4042 4043 4044 4045 4046 4047 4048 4049 4050 4051 4052 4053 4054 4055 4056 4057 4058 4059 4060 4061 4062 4063 4064 4065 4066 4067 4068 4069 4070 4071 4072 4073 4074 4075 4076 4077 4078 4079 4080 4081 4082 4083 4084 4085 4086 4087 4088 4089 4090 4091 4092 4093 4094 4095 4096 4097 4098 4099 4100 4101 4102 4103 4104 4105 4106 4107 4108 4109 4110 4111 4112 4113 4114 4115 4116 4117 4118 4119 4120 4121 4122 4123 4124 4125 4126 4127 4128 4129 4130 4131 4132 4133 4134 4135 4136 4137 4138 4139 4140 4141 4142 4143 4144 4145 4146 4147 4148 4149 4150 4151 4152 4153 4154 4155 4156 4157 4158 4159 4160 4161 4162 4163 4164 4165 4166 4167 4168 4169 4170 4171 4172 4173 4174 4175 4176 4177 4178 4179 4180 4181 4182 4183 4184 4185 4186 4187 4188 4189 4190 4191 4192 4193 4194 4195 4196 4197 4198 4199 4200 4201 4202 4203 4204 4205 4206 4207 4208 4209 4210 4211 4212 4213 4214 4215 4216 4217 4218 4219 4220 4221 4222 4223 4224 4225 4226 4227 4228 4229 4230 4231 4232 4233 4234 4235 4236 4237 4238 4239 4240 4241 4242 4243 4244 4245 4246 4247 4248 4249 4250 4251 4252 4253 4254 4255 4256 4257 4258 4259 4260 4261 4262 4263 4264 4265 4266 4267 4268 4269 4270 4271 4272 4273 4274 4275 4276 4277 4278 4279 4280 4281 4282 4283 4284 4285 4286 4287 4288 4289 4290 4291 4292 4293 4294 4295 4296 4297 4298 4299 4300 4301 4302 4303 4304 4305 4306 4307 4308 4309 4310 4311 4312 4313 4314 4315 4316 4317 4318 4319 4320 4321 4322 4323 4324 4325 4326 4327 4328 4329 4330 4331 4332 4333 4334 4335 4336 4337 4338 4339 4340 4341 4342 4343 4344 4345 4346 4347 4348 4349 4350 4351 4352 4353 4354 4355 4356 4357 4358 4359 4360 4361 4362 4363 4364 4365 4366 4367 4368 4369 4370 4371 4372 4373 4374 4375 4376 4377 4378 4379 4380 4381 4382 4383 4384 4385 4386 4387 4388 4389 4390 4391 4392 4393 4394 4395 4396 4397 4398 4399 4400 4401 4402 4403 4404 4405 4406 4407 4408 4409 4410 4411 4412 4413 4414 4415 4416 4417 4418 4419 4420 4421 4422 4423 4424 4425 4426 4427 4428 4429 4430 4431 4432 4433 4434 4435 4436 4437 4438 4439 4440 4441 4442 4443 4444 4445 4446 4447 4448 4449 4450 4451 4452 4453 4454 4455 4456 4457 4458 4459 4460 4461 4462 4463 4464 4465 4466 4467 4468 4469 4470 4471 4472 4473 4474 4475 4476 4477 4478 4479 4480 4481 4482 4483 4484 4485 4486 4487 4488 4489 4490 4491 4492 4493 4494 4495 4496 4497 4498 4499 4500 4501 4502 4503 4504 4505 4506 4507 4508 4509 4510 4511 4512 4513 4514 4515 4516 4517 4518 4519 4520 4521 4522 4523 4524 4525 4526 4527 4528 4529 4530 4531 4532 4533 4534 4535 4536 4537 4538 4539 4540 4541 4542 4543 4544 4545 4546 4547 4548 4549 4550 4551 4552 4553 4554 4555 4556 4557 4558 4559 4560 4561 4562 4563 4564 4565 4566 4567 4568 4569 4570 4571 4572 4573 4574 4575 4576 4577 4578 4579 4580 4581 4582 4583 4584 4585 4586 4587 4588 4589 4590 4591 4592 4593 4594 4595 4596 4597 4598 4599 4600 4601 4602 4603 4604 4605 4606 4607 4608 4609 4610 4611 4612 4613 4614 4615 4616 4617 4618 4619 4620 4621 4622 4623 4624 4625 4626 4627 4628 4629 4630 4631 4632 4633 4634 4635 4636 4637 4638 4639 4640 4641 | /* $NetBSD: tcp_input.c,v 1.433 2022/05/24 20:50:20 andvar Exp $ */ /* * Copyright (C) 1995, 1996, 1997, and 1998 WIDE Project. * All rights reserved. * * Redistribution and use in source and binary forms, with or without * modification, are permitted provided that the following conditions * are met: * 1. Redistributions of source code must retain the above copyright * notice, this list of conditions and the following disclaimer. * 2. Redistributions in binary form must reproduce the above copyright * notice, this list of conditions and the following disclaimer in the * documentation and/or other materials provided with the distribution. * 3. Neither the name of the project nor the names of its contributors * may be used to endorse or promote products derived from this software * without specific prior written permission. * * THIS SOFTWARE IS PROVIDED BY THE PROJECT AND CONTRIBUTORS ``AS IS'' AND * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE * ARE DISCLAIMED. IN NO EVENT SHALL THE PROJECT OR CONTRIBUTORS BE LIABLE * FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL * DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS * OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) * HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT * LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY * OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF * SUCH DAMAGE. */ /* * @(#)COPYRIGHT 1.1 (NRL) 17 January 1995 * * NRL grants permission for redistribution and use in source and binary * forms, with or without modification, of the software and documentation * created at NRL provided that the following conditions are met: * * 1. Redistributions of source code must retain the above copyright * notice, this list of conditions and the following disclaimer. * 2. Redistributions in binary form must reproduce the above copyright * notice, this list of conditions and the following disclaimer in the * documentation and/or other materials provided with the distribution. * 3. All advertising materials mentioning features or use of this software * must display the following acknowledgements: * This product includes software developed by the University of * California, Berkeley and its contributors. * This product includes software developed at the Information * Technology Division, US Naval Research Laboratory. * 4. Neither the name of the NRL nor the names of its contributors * may be used to endorse or promote products derived from this software * without specific prior written permission. * * THE SOFTWARE PROVIDED BY NRL IS PROVIDED BY NRL AND CONTRIBUTORS ``AS * IS'' AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED * TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A * PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL NRL OR * CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, * EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, * PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR * PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF * LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING * NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS * SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. * * The views and conclusions contained in the software and documentation * are those of the authors and should not be interpreted as representing * official policies, either expressed or implied, of the US Naval * Research Laboratory (NRL). */ /*- * Copyright (c) 1997, 1998, 1999, 2001, 2005, 2006, * 2011 The NetBSD Foundation, Inc. * All rights reserved. * * This code is derived from software contributed to The NetBSD Foundation * by Coyote Point Systems, Inc. * This code is derived from software contributed to The NetBSD Foundation * by Jason R. Thorpe and Kevin M. Lahey of the Numerical Aerospace Simulation * Facility, NASA Ames Research Center. * This code is derived from software contributed to The NetBSD Foundation * by Charles M. Hannum. * This code is derived from software contributed to The NetBSD Foundation * by Rui Paulo. * * Redistribution and use in source and binary forms, with or without * modification, are permitted provided that the following conditions * are met: * 1. Redistributions of source code must retain the above copyright * notice, this list of conditions and the following disclaimer. * 2. Redistributions in binary form must reproduce the above copyright * notice, this list of conditions and the following disclaimer in the * documentation and/or other materials provided with the distribution. * * THIS SOFTWARE IS PROVIDED BY THE NETBSD FOUNDATION, INC. AND CONTRIBUTORS * ``AS IS'' AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED * TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR * PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE FOUNDATION OR CONTRIBUTORS * BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR * CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF * SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS * INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN * CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) * ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE * POSSIBILITY OF SUCH DAMAGE. */ /* * Copyright (c) 1982, 1986, 1988, 1990, 1993, 1994, 1995 * The Regents of the University of California. All rights reserved. * * Redistribution and use in source and binary forms, with or without * modification, are permitted provided that the following conditions * are met: * 1. Redistributions of source code must retain the above copyright * notice, this list of conditions and the following disclaimer. * 2. Redistributions in binary form must reproduce the above copyright * notice, this list of conditions and the following disclaimer in the * documentation and/or other materials provided with the distribution. * 3. Neither the name of the University nor the names of its contributors * may be used to endorse or promote products derived from this software * without specific prior written permission. * * THIS SOFTWARE IS PROVIDED BY THE REGENTS AND CONTRIBUTORS ``AS IS'' AND * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE * ARE DISCLAIMED. IN NO EVENT SHALL THE REGENTS OR CONTRIBUTORS BE LIABLE * FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL * DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS * OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) * HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT * LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY * OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF * SUCH DAMAGE. * * @(#)tcp_input.c 8.12 (Berkeley) 5/24/95 */ /* * TODO list for SYN cache stuff: * * Find room for a "state" field, which is needed to keep a * compressed state for TIME_WAIT TCBs. It's been noted already * that this is fairly important for very high-volume web and * mail servers, which use a large number of short-lived * connections. */ #include <sys/cdefs.h> __KERNEL_RCSID(0, "$NetBSD: tcp_input.c,v 1.433 2022/05/24 20:50:20 andvar Exp $"); #ifdef _KERNEL_OPT #include "opt_inet.h" #include "opt_ipsec.h" #include "opt_inet_csum.h" #include "opt_tcp_debug.h" #endif #include <sys/param.h> #include <sys/systm.h> #include <sys/malloc.h> #include <sys/mbuf.h> #include <sys/protosw.h> #include <sys/socket.h> #include <sys/socketvar.h> #include <sys/errno.h> #include <sys/syslog.h> #include <sys/pool.h> #include <sys/domain.h> #include <sys/kernel.h> #ifdef TCP_SIGNATURE #include <sys/md5.h> #endif #include <sys/lwp.h> /* for lwp0 */ #include <sys/cprng.h> #include <net/if.h> #include <net/if_types.h> #include <netinet/in.h> #include <netinet/in_systm.h> #include <netinet/ip.h> #include <netinet/in_pcb.h> #include <netinet/in_var.h> #include <netinet/ip_var.h> #include <netinet/in_offload.h> #if NARP > 0 #include <netinet/if_inarp.h> #endif #ifdef INET6 #include <netinet/ip6.h> #include <netinet6/ip6_var.h> #include <netinet6/in6_pcb.h> #include <netinet6/ip6_var.h> #include <netinet6/in6_var.h> #include <netinet/icmp6.h> #include <netinet6/nd6.h> #ifdef TCP_SIGNATURE #include <netinet6/scope6_var.h> #endif #endif #ifndef INET6 #include <netinet/ip6.h> #endif #include <netinet/tcp.h> #include <netinet/tcp_fsm.h> #include <netinet/tcp_seq.h> #include <netinet/tcp_timer.h> #include <netinet/tcp_var.h> #include <netinet/tcp_private.h> #include <netinet/tcp_congctl.h> #include <netinet/tcp_debug.h> #ifdef INET6 #include "faith.h" #if defined(NFAITH) && NFAITH > 0 #include <net/if_faith.h> #endif #endif #ifdef IPSEC #include <netipsec/ipsec.h> #include <netipsec/key.h> #ifdef INET6 #include <netipsec/ipsec6.h> #endif #endif /* IPSEC*/ #include <netinet/tcp_vtw.h> int tcprexmtthresh = 3; int tcp_log_refused; int tcp_do_autorcvbuf = 1; int tcp_autorcvbuf_inc = 16 * 1024; int tcp_autorcvbuf_max = 256 * 1024; int tcp_msl = (TCPTV_MSL / PR_SLOWHZ); static int tcp_rst_ppslim_count = 0; static struct timeval tcp_rst_ppslim_last; static int tcp_ackdrop_ppslim_count = 0; static struct timeval tcp_ackdrop_ppslim_last; static void syn_cache_timer(void *); #define TCP_PAWS_IDLE (24U * 24 * 60 * 60 * PR_SLOWHZ) /* for modulo comparisons of timestamps */ #define TSTMP_LT(a,b) ((int)((a)-(b)) < 0) #define TSTMP_GEQ(a,b) ((int)((a)-(b)) >= 0) /* * Neighbor Discovery, Neighbor Unreachability Detection Upper layer hint. */ static void nd_hint(struct tcpcb *tp) { struct route *ro = NULL; struct rtentry *rt; if (tp == NULL) return; switch (tp->t_family) { #if NARP > 0 case AF_INET: if (tp->t_inpcb != NULL) ro = &tp->t_inpcb->inp_route; break; #endif #ifdef INET6 case AF_INET6: if (tp->t_in6pcb != NULL) ro = &tp->t_in6pcb->in6p_route; break; #endif } if (ro == NULL) return; rt = rtcache_validate(ro); if (rt == NULL) return; switch (tp->t_family) { #if NARP > 0 case AF_INET: arp_nud_hint(rt); break; #endif #ifdef INET6 case AF_INET6: nd6_nud_hint(rt); break; #endif } rtcache_unref(rt, ro); } /* * Compute ACK transmission behavior. Delay the ACK unless * we have already delayed an ACK (must send an ACK every two segments). * We also ACK immediately if we received a PUSH and the ACK-on-PUSH * option is enabled. */ static void tcp_setup_ack(struct tcpcb *tp, const struct tcphdr *th) { if (tp->t_flags & TF_DELACK || (tcp_ack_on_push && th->th_flags & TH_PUSH)) tp->t_flags |= TF_ACKNOW; else TCP_SET_DELACK(tp); } static void icmp_check(struct tcpcb *tp, const struct tcphdr *th, int acked) { /* * If we had a pending ICMP message that refers to data that have * just been acknowledged, disregard the recorded ICMP message. */ if ((tp->t_flags & TF_PMTUD_PEND) && SEQ_GT(th->th_ack, tp->t_pmtud_th_seq)) tp->t_flags &= ~TF_PMTUD_PEND; /* * Keep track of the largest chunk of data * acknowledged since last PMTU update */ if (tp->t_pmtud_mss_acked < acked) tp->t_pmtud_mss_acked = acked; } /* * Convert TCP protocol fields to host order for easier processing. */ static void tcp_fields_to_host(struct tcphdr *th) { NTOHL(th->th_seq); NTOHL(th->th_ack); NTOHS(th->th_win); NTOHS(th->th_urp); } /* * ... and reverse the above. */ static void tcp_fields_to_net(struct tcphdr *th) { HTONL(th->th_seq); HTONL(th->th_ack); HTONS(th->th_win); HTONS(th->th_urp); } static void tcp_urp_drop(struct tcphdr *th, int todrop, int *tiflags) { if (th->th_urp > todrop) { th->th_urp -= todrop; } else { *tiflags &= ~TH_URG; th->th_urp = 0; } } #ifdef TCP_CSUM_COUNTERS #include <sys/device.h> extern struct evcnt tcp_hwcsum_ok; extern struct evcnt tcp_hwcsum_bad; extern struct evcnt tcp_hwcsum_data; extern struct evcnt tcp_swcsum; #if defined(INET6) extern struct evcnt tcp6_hwcsum_ok; extern struct evcnt tcp6_hwcsum_bad; extern struct evcnt tcp6_hwcsum_data; extern struct evcnt tcp6_swcsum; #endif /* defined(INET6) */ #define TCP_CSUM_COUNTER_INCR(ev) (ev)->ev_count++ #else #define TCP_CSUM_COUNTER_INCR(ev) /* nothing */ #endif /* TCP_CSUM_COUNTERS */ #ifdef TCP_REASS_COUNTERS #include <sys/device.h> extern struct evcnt tcp_reass_; extern struct evcnt tcp_reass_empty; extern struct evcnt tcp_reass_iteration[8]; extern struct evcnt tcp_reass_prependfirst; extern struct evcnt tcp_reass_prepend; extern struct evcnt tcp_reass_insert; extern struct evcnt tcp_reass_inserttail; extern struct evcnt tcp_reass_append; extern struct evcnt tcp_reass_appendtail; extern struct evcnt tcp_reass_overlaptail; extern struct evcnt tcp_reass_overlapfront; extern struct evcnt tcp_reass_segdup; extern struct evcnt tcp_reass_fragdup; #define TCP_REASS_COUNTER_INCR(ev) (ev)->ev_count++ #else #define TCP_REASS_COUNTER_INCR(ev) /* nothing */ #endif /* TCP_REASS_COUNTERS */ static int tcp_reass(struct tcpcb *, const struct tcphdr *, struct mbuf *, int); static int tcp_dooptions(struct tcpcb *, const u_char *, int, struct tcphdr *, struct mbuf *, int, struct tcp_opt_info *); static void tcp4_log_refused(const struct ip *, const struct tcphdr *); #ifdef INET6 static void tcp6_log_refused(const struct ip6_hdr *, const struct tcphdr *); #endif #if defined(MBUFTRACE) struct mowner tcp_reass_mowner = MOWNER_INIT("tcp", "reass"); #endif /* defined(MBUFTRACE) */ static struct pool tcpipqent_pool; void tcpipqent_init(void) { pool_init(&tcpipqent_pool, sizeof(struct ipqent), 0, 0, 0, "tcpipqepl", NULL, IPL_VM); } struct ipqent * tcpipqent_alloc(void) { struct ipqent *ipqe; int s; s = splvm(); ipqe = pool_get(&tcpipqent_pool, PR_NOWAIT); splx(s); return ipqe; } void tcpipqent_free(struct ipqent *ipqe) { int s; s = splvm(); pool_put(&tcpipqent_pool, ipqe); splx(s); } /* * Insert segment ti into reassembly queue of tcp with * control block tp. Return TH_FIN if reassembly now includes * a segment with FIN. */ static int tcp_reass(struct tcpcb *tp, const struct tcphdr *th, struct mbuf *m, int tlen) { struct ipqent *p, *q, *nq, *tiqe = NULL; struct socket *so = NULL; int pkt_flags; tcp_seq pkt_seq; unsigned pkt_len; u_long rcvpartdupbyte = 0; u_long rcvoobyte; #ifdef TCP_REASS_COUNTERS u_int count = 0; #endif uint64_t *tcps; if (tp->t_inpcb) so = tp->t_inpcb->inp_socket; #ifdef INET6 else if (tp->t_in6pcb) so = tp->t_in6pcb->in6p_socket; #endif TCP_REASS_LOCK_CHECK(tp); /* * Call with th==NULL after become established to * force pre-ESTABLISHED data up to user socket. */ if (th == NULL) goto present; m_claimm(m, &tcp_reass_mowner); rcvoobyte = tlen; /* * Copy these to local variables because the TCP header gets munged * while we are collapsing mbufs. */ pkt_seq = th->th_seq; pkt_len = tlen; pkt_flags = th->th_flags; TCP_REASS_COUNTER_INCR(&tcp_reass_); if ((p = TAILQ_LAST(&tp->segq, ipqehead)) != NULL) { /* * When we miss a packet, the vast majority of time we get * packets that follow it in order. So optimize for that. */ if (pkt_seq == p->ipqe_seq + p->ipqe_len) { p->ipqe_len += pkt_len; p->ipqe_flags |= pkt_flags; m_cat(p->ipqe_m, m); m = NULL; tiqe = p; TAILQ_REMOVE(&tp->timeq, p, ipqe_timeq); TCP_REASS_COUNTER_INCR(&tcp_reass_appendtail); goto skip_replacement; } /* * While we're here, if the pkt is completely beyond * anything we have, just insert it at the tail. */ if (SEQ_GT(pkt_seq, p->ipqe_seq + p->ipqe_len)) { TCP_REASS_COUNTER_INCR(&tcp_reass_inserttail); goto insert_it; } } q = TAILQ_FIRST(&tp->segq); if (q != NULL) { /* * If this segment immediately precedes the first out-of-order * block, simply slap the segment in front of it and (mostly) * skip the complicated logic. */ if (pkt_seq + pkt_len == q->ipqe_seq) { q->ipqe_seq = pkt_seq; q->ipqe_len += pkt_len; q->ipqe_flags |= pkt_flags; m_cat(m, q->ipqe_m); q->ipqe_m = m; tiqe = q; TAILQ_REMOVE(&tp->timeq, q, ipqe_timeq); TCP_REASS_COUNTER_INCR(&tcp_reass_prependfirst); goto skip_replacement; } } else { TCP_REASS_COUNTER_INCR(&tcp_reass_empty); } /* * Find a segment which begins after this one does. */ for (p = NULL; q != NULL; q = nq) { nq = TAILQ_NEXT(q, ipqe_q); #ifdef TCP_REASS_COUNTERS count++; #endif /* * If the received segment is just right after this * fragment, merge the two together and then check * for further overlaps. */ if (q->ipqe_seq + q->ipqe_len == pkt_seq) { pkt_len += q->ipqe_len; pkt_flags |= q->ipqe_flags; pkt_seq = q->ipqe_seq; m_cat(q->ipqe_m, m); m = q->ipqe_m; TCP_REASS_COUNTER_INCR(&tcp_reass_append); goto free_ipqe; } /* * If the received segment is completely past this * fragment, we need to go to the next fragment. */ if (SEQ_LT(q->ipqe_seq + q->ipqe_len, pkt_seq)) { p = q; continue; } /* * If the fragment is past the received segment, * it (or any following) can't be concatenated. */ if (SEQ_GT(q->ipqe_seq, pkt_seq + pkt_len)) { TCP_REASS_COUNTER_INCR(&tcp_reass_insert); break; } /* * We've received all the data in this segment before. * Mark it as a duplicate and return. */ if (SEQ_LEQ(q->ipqe_seq, pkt_seq) && SEQ_GEQ(q->ipqe_seq + q->ipqe_len, pkt_seq + pkt_len)) { tcps = TCP_STAT_GETREF(); tcps[TCP_STAT_RCVDUPPACK]++; tcps[TCP_STAT_RCVDUPBYTE] += pkt_len; TCP_STAT_PUTREF(); tcp_new_dsack(tp, pkt_seq, pkt_len); m_freem(m); if (tiqe != NULL) { tcpipqent_free(tiqe); } TCP_REASS_COUNTER_INCR(&tcp_reass_segdup); goto out; } /* * Received segment completely overlaps this fragment * so we drop the fragment (this keeps the temporal * ordering of segments correct). */ if (SEQ_GEQ(q->ipqe_seq, pkt_seq) && SEQ_LEQ(q->ipqe_seq + q->ipqe_len, pkt_seq + pkt_len)) { rcvpartdupbyte += q->ipqe_len; m_freem(q->ipqe_m); TCP_REASS_COUNTER_INCR(&tcp_reass_fragdup); goto free_ipqe; } /* * Received segment extends past the end of the fragment. * Drop the overlapping bytes, merge the fragment and * segment, and treat as a longer received packet. */ if (SEQ_LT(q->ipqe_seq, pkt_seq) && SEQ_GT(q->ipqe_seq + q->ipqe_len, pkt_seq)) { int overlap = q->ipqe_seq + q->ipqe_len - pkt_seq; m_adj(m, overlap); rcvpartdupbyte += overlap; m_cat(q->ipqe_m, m); m = q->ipqe_m; pkt_seq = q->ipqe_seq; pkt_len += q->ipqe_len - overlap; rcvoobyte -= overlap; TCP_REASS_COUNTER_INCR(&tcp_reass_overlaptail); goto free_ipqe; } /* * Received segment extends past the front of the fragment. * Drop the overlapping bytes on the received packet. The * packet will then be concatenated with this fragment a * bit later. */ if (SEQ_GT(q->ipqe_seq, pkt_seq) && SEQ_LT(q->ipqe_seq, pkt_seq + pkt_len)) { int overlap = pkt_seq + pkt_len - q->ipqe_seq; m_adj(m, -overlap); pkt_len -= overlap; rcvpartdupbyte += overlap; TCP_REASS_COUNTER_INCR(&tcp_reass_overlapfront); rcvoobyte -= overlap; } /* * If the received segment immediately precedes this * fragment then tack the fragment onto this segment * and reinsert the data. */ if (q->ipqe_seq == pkt_seq + pkt_len) { pkt_len += q->ipqe_len; pkt_flags |= q->ipqe_flags; m_cat(m, q->ipqe_m); TAILQ_REMOVE(&tp->segq, q, ipqe_q); TAILQ_REMOVE(&tp->timeq, q, ipqe_timeq); tp->t_segqlen--; KASSERT(tp->t_segqlen >= 0); KASSERT(tp->t_segqlen != 0 || (TAILQ_EMPTY(&tp->segq) && TAILQ_EMPTY(&tp->timeq))); if (tiqe == NULL) { tiqe = q; } else { tcpipqent_free(q); } TCP_REASS_COUNTER_INCR(&tcp_reass_prepend); break; } /* * If the fragment is before the segment, remember it. * When this loop is terminated, p will contain the * pointer to the fragment that is right before the * received segment. */ if (SEQ_LEQ(q->ipqe_seq, pkt_seq)) p = q; continue; /* * This is a common operation. It also will allow * to save doing a malloc/free in most instances. */ free_ipqe: TAILQ_REMOVE(&tp->segq, q, ipqe_q); TAILQ_REMOVE(&tp->timeq, q, ipqe_timeq); tp->t_segqlen--; KASSERT(tp->t_segqlen >= 0); KASSERT(tp->t_segqlen != 0 || (TAILQ_EMPTY(&tp->segq) && TAILQ_EMPTY(&tp->timeq))); if (tiqe == NULL) { tiqe = q; } else { tcpipqent_free(q); } } #ifdef TCP_REASS_COUNTERS if (count > 7) TCP_REASS_COUNTER_INCR(&tcp_reass_iteration[0]); else if (count > 0) TCP_REASS_COUNTER_INCR(&tcp_reass_iteration[count]); #endif insert_it: /* * Allocate a new queue entry (block) since the received segment * did not collapse onto any other out-of-order block. If it had * collapsed, tiqe would not be NULL and we would be reusing it. * * If the allocation fails, drop the packet. */ if (tiqe == NULL) { tiqe = tcpipqent_alloc(); if (tiqe == NULL) { TCP_STATINC(TCP_STAT_RCVMEMDROP); m_freem(m); goto out; } } /* * Update the counters. */ tp->t_rcvoopack++; tcps = TCP_STAT_GETREF(); tcps[TCP_STAT_RCVOOPACK]++; tcps[TCP_STAT_RCVOOBYTE] += rcvoobyte; if (rcvpartdupbyte) { tcps[TCP_STAT_RCVPARTDUPPACK]++; tcps[TCP_STAT_RCVPARTDUPBYTE] += rcvpartdupbyte; } TCP_STAT_PUTREF(); /* * Insert the new fragment queue entry into both queues. */ tiqe->ipqe_m = m; tiqe->ipqe_seq = pkt_seq; tiqe->ipqe_len = pkt_len; tiqe->ipqe_flags = pkt_flags; if (p == NULL) { TAILQ_INSERT_HEAD(&tp->segq, tiqe, ipqe_q); } else { TAILQ_INSERT_AFTER(&tp->segq, p, tiqe, ipqe_q); } tp->t_segqlen++; skip_replacement: TAILQ_INSERT_HEAD(&tp->timeq, tiqe, ipqe_timeq); present: /* * Present data to user, advancing rcv_nxt through * completed sequence space. */ if (TCPS_HAVEESTABLISHED(tp->t_state) == 0) goto out; q = TAILQ_FIRST(&tp->segq); if (q == NULL || q->ipqe_seq != tp->rcv_nxt) goto out; if (tp->t_state == TCPS_SYN_RECEIVED && q->ipqe_len) goto out; tp->rcv_nxt += q->ipqe_len; pkt_flags = q->ipqe_flags & TH_FIN; nd_hint(tp); TAILQ_REMOVE(&tp->segq, q, ipqe_q); TAILQ_REMOVE(&tp->timeq, q, ipqe_timeq); tp->t_segqlen--; KASSERT(tp->t_segqlen >= 0); KASSERT(tp->t_segqlen != 0 || (TAILQ_EMPTY(&tp->segq) && TAILQ_EMPTY(&tp->timeq))); if (so->so_state & SS_CANTRCVMORE) m_freem(q->ipqe_m); else sbappendstream(&so->so_rcv, q->ipqe_m); tcpipqent_free(q); TCP_REASS_UNLOCK(tp); sorwakeup(so); return pkt_flags; out: TCP_REASS_UNLOCK(tp); return 0; } #ifdef INET6 int tcp6_input(struct mbuf **mp, int *offp, int proto) { struct mbuf *m = *mp; /* * draft-itojun-ipv6-tcp-to-anycast * better place to put this in? */ if (m->m_flags & M_ANYCAST6) { struct ip6_hdr *ip6; if (m->m_len < sizeof(struct ip6_hdr)) { if ((m = m_pullup(m, sizeof(struct ip6_hdr))) == NULL) { TCP_STATINC(TCP_STAT_RCVSHORT); return IPPROTO_DONE; } } ip6 = mtod(m, struct ip6_hdr *); icmp6_error(m, ICMP6_DST_UNREACH, ICMP6_DST_UNREACH_ADDR, (char *)&ip6->ip6_dst - (char *)ip6); return IPPROTO_DONE; } tcp_input(m, *offp, proto); return IPPROTO_DONE; } #endif static void tcp4_log_refused(const struct ip *ip, const struct tcphdr *th) { char src[INET_ADDRSTRLEN]; char dst[INET_ADDRSTRLEN]; if (ip) { in_print(src, sizeof(src), &ip->ip_src); in_print(dst, sizeof(dst), &ip->ip_dst); } else { strlcpy(src, "(unknown)", sizeof(src)); strlcpy(dst, "(unknown)", sizeof(dst)); } log(LOG_INFO, "Connection attempt to TCP %s:%d from %s:%d\n", dst, ntohs(th->th_dport), src, ntohs(th->th_sport)); } #ifdef INET6 static void tcp6_log_refused(const struct ip6_hdr *ip6, const struct tcphdr *th) { char src[INET6_ADDRSTRLEN]; char dst[INET6_ADDRSTRLEN]; if (ip6) { in6_print(src, sizeof(src), &ip6->ip6_src); in6_print(dst, sizeof(dst), &ip6->ip6_dst); } else { strlcpy(src, "(unknown v6)", sizeof(src)); strlcpy(dst, "(unknown v6)", sizeof(dst)); } log(LOG_INFO, "Connection attempt to TCP [%s]:%d from [%s]:%d\n", dst, ntohs(th->th_dport), src, ntohs(th->th_sport)); } #endif /* * Checksum extended TCP header and data. */ int tcp_input_checksum(int af, struct mbuf *m, const struct tcphdr *th, int toff, int off, int tlen) { struct ifnet *rcvif; int s; /* * XXX it's better to record and check if this mbuf is * already checked. */ rcvif = m_get_rcvif(m, &s); if (__predict_false(rcvif == NULL)) goto badcsum; /* XXX */ switch (af) { case AF_INET: switch (m->m_pkthdr.csum_flags & ((rcvif->if_csum_flags_rx & M_CSUM_TCPv4) | M_CSUM_TCP_UDP_BAD | M_CSUM_DATA)) { case M_CSUM_TCPv4|M_CSUM_TCP_UDP_BAD: TCP_CSUM_COUNTER_INCR(&tcp_hwcsum_bad); goto badcsum; case M_CSUM_TCPv4|M_CSUM_DATA: { u_int32_t hw_csum = m->m_pkthdr.csum_data; TCP_CSUM_COUNTER_INCR(&tcp_hwcsum_data); if (m->m_pkthdr.csum_flags & M_CSUM_NO_PSEUDOHDR) { const struct ip *ip = mtod(m, const struct ip *); hw_csum = in_cksum_phdr(ip->ip_src.s_addr, ip->ip_dst.s_addr, htons(hw_csum + tlen + off + IPPROTO_TCP)); } if ((hw_csum ^ 0xffff) != 0) goto badcsum; break; } case M_CSUM_TCPv4: /* Checksum was okay. */ TCP_CSUM_COUNTER_INCR(&tcp_hwcsum_ok); break; default: /* * Must compute it ourselves. Maybe skip checksum * on loopback interfaces. */ if (__predict_true(!(rcvif->if_flags & IFF_LOOPBACK) || tcp_do_loopback_cksum)) { TCP_CSUM_COUNTER_INCR(&tcp_swcsum); if (in4_cksum(m, IPPROTO_TCP, toff, tlen + off) != 0) goto badcsum; } break; } break; #ifdef INET6 case AF_INET6: switch (m->m_pkthdr.csum_flags & ((rcvif->if_csum_flags_rx & M_CSUM_TCPv6) | M_CSUM_TCP_UDP_BAD | M_CSUM_DATA)) { case M_CSUM_TCPv6|M_CSUM_TCP_UDP_BAD: TCP_CSUM_COUNTER_INCR(&tcp6_hwcsum_bad); goto badcsum; #if 0 /* notyet */ case M_CSUM_TCPv6|M_CSUM_DATA: #endif case M_CSUM_TCPv6: /* Checksum was okay. */ TCP_CSUM_COUNTER_INCR(&tcp6_hwcsum_ok); break; default: /* * Must compute it ourselves. Maybe skip checksum * on loopback interfaces. */ if (__predict_true((m->m_flags & M_LOOP) == 0 || tcp_do_loopback_cksum)) { TCP_CSUM_COUNTER_INCR(&tcp6_swcsum); if (in6_cksum(m, IPPROTO_TCP, toff, tlen + off) != 0) goto badcsum; } } break; #endif /* INET6 */ } m_put_rcvif(rcvif, &s); return 0; badcsum: m_put_rcvif(rcvif, &s); TCP_STATINC(TCP_STAT_RCVBADSUM); return -1; } /* * When a packet arrives addressed to a vestigial tcpbp, we * nevertheless have to respond to it per the spec. * * This code is duplicated from the one in tcp_input(). */ static void tcp_vtw_input(struct tcphdr *th, vestigial_inpcb_t *vp, struct mbuf *m, int tlen) { int tiflags; int todrop; uint32_t t_flags = 0; uint64_t *tcps; tiflags = th->th_flags; todrop = vp->rcv_nxt - th->th_seq; if (todrop > 0) { if (tiflags & TH_SYN) { tiflags &= ~TH_SYN; th->th_seq++; tcp_urp_drop(th, 1, &tiflags); todrop--; } if (todrop > tlen || (todrop == tlen && (tiflags & TH_FIN) == 0)) { /* * Any valid FIN or RST must be to the left of the * window. At this point the FIN or RST must be a * duplicate or out of sequence; drop it. */ if (tiflags & TH_RST) goto drop; tiflags &= ~(TH_FIN|TH_RST); /* * Send an ACK to resynchronize and drop any data. * But keep on processing for RST or ACK. */ t_flags |= TF_ACKNOW; todrop = tlen; tcps = TCP_STAT_GETREF(); tcps[TCP_STAT_RCVDUPPACK] += 1; tcps[TCP_STAT_RCVDUPBYTE] += todrop; TCP_STAT_PUTREF(); } else if ((tiflags & TH_RST) && th->th_seq != vp->rcv_nxt) { /* * Test for reset before adjusting the sequence * number for overlapping data. */ goto dropafterack_ratelim; } else { tcps = TCP_STAT_GETREF(); tcps[TCP_STAT_RCVPARTDUPPACK] += 1; tcps[TCP_STAT_RCVPARTDUPBYTE] += todrop; TCP_STAT_PUTREF(); } // tcp_new_dsack(tp, th->th_seq, todrop); // hdroptlen += todrop; /*drop from head afterwards*/ th->th_seq += todrop; tlen -= todrop; tcp_urp_drop(th, todrop, &tiflags); } /* * If new data are received on a connection after the * user processes are gone, then RST the other end. */ if (tlen) { TCP_STATINC(TCP_STAT_RCVAFTERCLOSE); goto dropwithreset; } /* * If segment ends after window, drop trailing data * (and PUSH and FIN); if nothing left, just ACK. */ todrop = (th->th_seq + tlen) - (vp->rcv_nxt + vp->rcv_wnd); if (todrop > 0) { TCP_STATINC(TCP_STAT_RCVPACKAFTERWIN); if (todrop >= tlen) { /* * The segment actually starts after the window. * th->th_seq + tlen - vp->rcv_nxt - vp->rcv_wnd >= tlen * th->th_seq - vp->rcv_nxt - vp->rcv_wnd >= 0 * th->th_seq >= vp->rcv_nxt + vp->rcv_wnd */ TCP_STATADD(TCP_STAT_RCVBYTEAFTERWIN, tlen); /* * If a new connection request is received * while in TIME_WAIT, drop the old connection * and start over if the sequence numbers * are above the previous ones. */ if ((tiflags & TH_SYN) && SEQ_GT(th->th_seq, vp->rcv_nxt)) { /* * We only support this in the !NOFDREF case, which * is to say: not here. */ goto dropwithreset; } /* * If window is closed can only take segments at * window edge, and have to drop data and PUSH from * incoming segments. Continue processing, but * remember to ack. Otherwise, drop segment * and (if not RST) ack. */ if (vp->rcv_wnd == 0 && th->th_seq == vp->rcv_nxt) { t_flags |= TF_ACKNOW; TCP_STATINC(TCP_STAT_RCVWINPROBE); } else { goto dropafterack; } } else { TCP_STATADD(TCP_STAT_RCVBYTEAFTERWIN, todrop); } m_adj(m, -todrop); tlen -= todrop; tiflags &= ~(TH_PUSH|TH_FIN); } if (tiflags & TH_RST) { if (th->th_seq != vp->rcv_nxt) goto dropafterack_ratelim; vtw_del(vp->ctl, vp->vtw); goto drop; } /* * If the ACK bit is off we drop the segment and return. */ if ((tiflags & TH_ACK) == 0) { if (t_flags & TF_ACKNOW) goto dropafterack; goto drop; } /* * In TIME_WAIT state the only thing that should arrive * is a retransmission of the remote FIN. Acknowledge * it and restart the finack timer. */ vtw_restart(vp); goto dropafterack; dropafterack: /* * Generate an ACK dropping incoming segment if it occupies * sequence space, where the ACK reflects our state. */ if (tiflags & TH_RST) goto drop; goto dropafterack2; dropafterack_ratelim: /* * We may want to rate-limit ACKs against SYN/RST attack. */ if (ppsratecheck(&tcp_ackdrop_ppslim_last, &tcp_ackdrop_ppslim_count, tcp_ackdrop_ppslim) == 0) { /* XXX stat */ goto drop; } /* ...fall into dropafterack2... */ dropafterack2: (void)tcp_respond(0, m, m, th, th->th_seq + tlen, th->th_ack, TH_ACK); return; dropwithreset: /* * Generate a RST, dropping incoming segment. * Make ACK acceptable to originator of segment. */ if (tiflags & TH_RST) goto drop; if (tiflags & TH_ACK) { tcp_respond(0, m, m, th, (tcp_seq)0, th->th_ack, TH_RST); } else { if (tiflags & TH_SYN) ++tlen; (void)tcp_respond(0, m, m, th, th->th_seq + tlen, (tcp_seq)0, TH_RST|TH_ACK); } return; drop: m_freem(m); } /* * TCP input routine, follows pages 65-76 of RFC 793 very closely. */ void tcp_input(struct mbuf *m, int off, int proto) { struct tcphdr *th; struct ip *ip; struct inpcb *inp; #ifdef INET6 struct ip6_hdr *ip6; struct in6pcb *in6p; #endif u_int8_t *optp = NULL; int optlen = 0; int len, tlen, hdroptlen = 0; struct tcpcb *tp = NULL; int tiflags; struct socket *so = NULL; int todrop, acked, ourfinisacked, needoutput = 0; bool dupseg; #ifdef TCP_DEBUG short ostate = 0; #endif u_long tiwin; struct tcp_opt_info opti; int thlen, iphlen; int af; /* af on the wire */ struct mbuf *tcp_saveti = NULL; uint32_t ts_rtt; uint8_t iptos; uint64_t *tcps; vestigial_inpcb_t vestige; vestige.valid = 0; MCLAIM(m, &tcp_rx_mowner); TCP_STATINC(TCP_STAT_RCVTOTAL); memset(&opti, 0, sizeof(opti)); opti.ts_present = 0; opti.maxseg = 0; /* * RFC1122 4.2.3.10, p. 104: discard bcast/mcast SYN. * * TCP is, by definition, unicast, so we reject all * multicast outright. * * Note, there are additional src/dst address checks in * the AF-specific code below. */ if (m->m_flags & (M_BCAST|M_MCAST)) { /* XXX stat */ goto drop; } #ifdef INET6 if (m->m_flags & M_ANYCAST6) { /* XXX stat */ goto drop; } #endif M_REGION_GET(th, struct tcphdr *, m, off, sizeof(struct tcphdr)); if (th == NULL) { TCP_STATINC(TCP_STAT_RCVSHORT); return; } /* * Enforce alignment requirements that are violated in * some cases, see kern/50766 for details. */ if (ACCESSIBLE_POINTER(th, struct tcphdr) == 0) { m = m_copyup(m, off + sizeof(struct tcphdr), 0); if (m == NULL) { TCP_STATINC(TCP_STAT_RCVSHORT); return; } th = (struct tcphdr *)(mtod(m, char *) + off); } KASSERT(ACCESSIBLE_POINTER(th, struct tcphdr)); /* * Get IP and TCP header. * Note: IP leaves IP header in first mbuf. */ ip = mtod(m, struct ip *); #ifdef INET6 ip6 = mtod(m, struct ip6_hdr *); #endif switch (ip->ip_v) { case 4: af = AF_INET; iphlen = sizeof(struct ip); if (IN_MULTICAST(ip->ip_dst.s_addr) || in_broadcast(ip->ip_dst, m_get_rcvif_NOMPSAFE(m))) goto drop; /* We do the checksum after PCB lookup... */ len = ntohs(ip->ip_len); tlen = len - off; iptos = ip->ip_tos; break; #ifdef INET6 case 6: iphlen = sizeof(struct ip6_hdr); af = AF_INET6; /* * Be proactive about unspecified IPv6 address in source. * As we use all-zero to indicate unbounded/unconnected pcb, * unspecified IPv6 address can be used to confuse us. * * Note that packets with unspecified IPv6 destination is * already dropped in ip6_input. */ if (IN6_IS_ADDR_UNSPECIFIED(&ip6->ip6_src)) { /* XXX stat */ goto drop; } /* * Make sure destination address is not multicast. * Source address checked in ip6_input(). */ if (IN6_IS_ADDR_MULTICAST(&ip6->ip6_dst)) { /* XXX stat */ goto drop; } /* We do the checksum after PCB lookup... */ len = m->m_pkthdr.len; tlen = len - off; iptos = (ntohl(ip6->ip6_flow) >> 20) & 0xff; break; #endif default: m_freem(m); return; } /* * Check that TCP offset makes sense, pull out TCP options and * adjust length. */ thlen = th->th_off << 2; if (thlen < sizeof(struct tcphdr) || thlen > tlen) { TCP_STATINC(TCP_STAT_RCVBADOFF); goto drop; } tlen -= thlen; if (thlen > sizeof(struct tcphdr)) { M_REGION_GET(th, struct tcphdr *, m, off, thlen); if (th == NULL) { TCP_STATINC(TCP_STAT_RCVSHORT); return; } KASSERT(ACCESSIBLE_POINTER(th, struct tcphdr)); optlen = thlen - sizeof(struct tcphdr); optp = ((u_int8_t *)th) + sizeof(struct tcphdr); /* * Do quick retrieval of timestamp options. * * If timestamp is the only option and it's formatted as * recommended in RFC 1323 appendix A, we quickly get the * values now and don't bother calling tcp_dooptions(), * etc. */ if ((optlen == TCPOLEN_TSTAMP_APPA || (optlen > TCPOLEN_TSTAMP_APPA && optp[TCPOLEN_TSTAMP_APPA] == TCPOPT_EOL)) && be32dec(optp) == TCPOPT_TSTAMP_HDR && (th->th_flags & TH_SYN) == 0) { opti.ts_present = 1; opti.ts_val = be32dec(optp + 4); opti.ts_ecr = be32dec(optp + 8); optp = NULL; /* we've parsed the options */ } } tiflags = th->th_flags; /* * Checksum extended TCP header and data */ if (tcp_input_checksum(af, m, th, off, thlen, tlen)) goto badcsum; /* * Locate pcb for segment. */ findpcb: inp = NULL; #ifdef INET6 in6p = NULL; #endif switch (af) { case AF_INET: inp = in_pcblookup_connect(&tcbtable, ip->ip_src, th->th_sport, ip->ip_dst, th->th_dport, &vestige); if (inp == NULL && !vestige.valid) { TCP_STATINC(TCP_STAT_PCBHASHMISS); inp = in_pcblookup_bind(&tcbtable, ip->ip_dst, th->th_dport); } #ifdef INET6 if (inp == NULL && !vestige.valid) { struct in6_addr s, d; /* mapped addr case */ in6_in_2_v4mapin6(&ip->ip_src, &s); in6_in_2_v4mapin6(&ip->ip_dst, &d); in6p = in6_pcblookup_connect(&tcbtable, &s, th->th_sport, &d, th->th_dport, 0, &vestige); if (in6p == 0 && !vestige.valid) { TCP_STATINC(TCP_STAT_PCBHASHMISS); in6p = in6_pcblookup_bind(&tcbtable, &d, th->th_dport, 0); } } #endif #ifndef INET6 if (inp == NULL && !vestige.valid) #else if (inp == NULL && in6p == NULL && !vestige.valid) #endif { TCP_STATINC(TCP_STAT_NOPORT); if (tcp_log_refused && (tiflags & (TH_RST|TH_ACK|TH_SYN)) == TH_SYN) { tcp4_log_refused(ip, th); } tcp_fields_to_host(th); goto dropwithreset_ratelim; } #if defined(IPSEC) if (ipsec_used) { if (inp && ipsec_in_reject(m, inp)) { goto drop; } #ifdef INET6 else if (in6p && ipsec_in_reject(m, in6p)) { goto drop; } #endif } #endif /*IPSEC*/ break; #ifdef INET6 case AF_INET6: { int faith; #if defined(NFAITH) && NFAITH > 0 faith = faithprefix(&ip6->ip6_dst); #else faith = 0; #endif in6p = in6_pcblookup_connect(&tcbtable, &ip6->ip6_src, th->th_sport, &ip6->ip6_dst, th->th_dport, faith, &vestige); if (!in6p && !vestige.valid) { TCP_STATINC(TCP_STAT_PCBHASHMISS); in6p = in6_pcblookup_bind(&tcbtable, &ip6->ip6_dst, th->th_dport, faith); } if (!in6p && !vestige.valid) { TCP_STATINC(TCP_STAT_NOPORT); if (tcp_log_refused && (tiflags & (TH_RST|TH_ACK|TH_SYN)) == TH_SYN) { tcp6_log_refused(ip6, th); } tcp_fields_to_host(th); goto dropwithreset_ratelim; } #if defined(IPSEC) if (ipsec_used && in6p && ipsec_in_reject(m, in6p)) { goto drop; } #endif break; } #endif } tcp_fields_to_host(th); /* * If the state is CLOSED (i.e., TCB does not exist) then * all data in the incoming segment is discarded. * If the TCB exists but is in CLOSED state, it is embryonic, * but should either do a listen or a connect soon. */ tp = NULL; so = NULL; if (inp) { /* Check the minimum TTL for socket. */ if (ip->ip_ttl < inp->inp_ip_minttl) goto drop; tp = intotcpcb(inp); so = inp->inp_socket; } #ifdef INET6 else if (in6p) { tp = in6totcpcb(in6p); so = in6p->in6p_socket; } #endif else if (vestige.valid) { /* We do not support the resurrection of vtw tcpcps. */ tcp_vtw_input(th, &vestige, m, tlen); m = NULL; goto drop; } if (tp == NULL) goto dropwithreset_ratelim; if (tp->t_state == TCPS_CLOSED) goto drop; KASSERT(so->so_lock == softnet_lock); KASSERT(solocked(so)); /* Unscale the window into a 32-bit value. */ if ((tiflags & TH_SYN) == 0) tiwin = th->th_win << tp->snd_scale; else tiwin = th->th_win; #ifdef INET6 /* save packet options if user wanted */ if (in6p && (in6p->in6p_flags & IN6P_CONTROLOPTS)) { if (in6p->in6p_options) { m_freem(in6p->in6p_options); in6p->in6p_options = NULL; } ip6_savecontrol(in6p, &in6p->in6p_options, ip6, m); } #endif if (so->so_options & SO_DEBUG) { #ifdef TCP_DEBUG ostate = tp->t_state; #endif tcp_saveti = NULL; if (iphlen + sizeof(struct tcphdr) > MHLEN) goto nosave; if (m->m_len > iphlen && (m->m_flags & M_EXT) == 0) { tcp_saveti = m_copym(m, 0, iphlen, M_DONTWAIT); if (tcp_saveti == NULL) goto nosave; } else { MGETHDR(tcp_saveti, M_DONTWAIT, MT_HEADER); if (tcp_saveti == NULL) goto nosave; MCLAIM(m, &tcp_mowner); tcp_saveti->m_len = iphlen; m_copydata(m, 0, iphlen, mtod(tcp_saveti, void *)); } if (M_TRAILINGSPACE(tcp_saveti) < sizeof(struct tcphdr)) { m_freem(tcp_saveti); tcp_saveti = NULL; } else { tcp_saveti->m_len += sizeof(struct tcphdr); memcpy(mtod(tcp_saveti, char *) + iphlen, th, sizeof(struct tcphdr)); } nosave:; } if (so->so_options & SO_ACCEPTCONN) { union syn_cache_sa src; union syn_cache_sa dst; KASSERT(tp->t_state == TCPS_LISTEN); memset(&src, 0, sizeof(src)); memset(&dst, 0, sizeof(dst)); switch (af) { case AF_INET: src.sin.sin_len = sizeof(struct sockaddr_in); src.sin.sin_family = AF_INET; src.sin.sin_addr = ip->ip_src; src.sin.sin_port = th->th_sport; dst.sin.sin_len = sizeof(struct sockaddr_in); dst.sin.sin_family = AF_INET; dst.sin.sin_addr = ip->ip_dst; dst.sin.sin_port = th->th_dport; break; #ifdef INET6 case AF_INET6: src.sin6.sin6_len = sizeof(struct sockaddr_in6); src.sin6.sin6_family = AF_INET6; src.sin6.sin6_addr = ip6->ip6_src; src.sin6.sin6_port = th->th_sport; dst.sin6.sin6_len = sizeof(struct sockaddr_in6); dst.sin6.sin6_family = AF_INET6; dst.sin6.sin6_addr = ip6->ip6_dst; dst.sin6.sin6_port = th->th_dport; break; #endif } if ((tiflags & (TH_RST|TH_ACK|TH_SYN)) != TH_SYN) { if (tiflags & TH_RST) { syn_cache_reset(&src.sa, &dst.sa, th); } else if ((tiflags & (TH_ACK|TH_SYN)) == (TH_ACK|TH_SYN)) { /* * Received a SYN,ACK. This should never * happen while we are in LISTEN. Send an RST. */ goto badsyn; } else if (tiflags & TH_ACK) { so = syn_cache_get(&src.sa, &dst.sa, th, so, m); if (so == NULL) { /* * We don't have a SYN for this ACK; * send an RST. */ goto badsyn; } else if (so == (struct socket *)(-1)) { /* * We were unable to create the * connection. If the 3-way handshake * was completed, and RST has been * sent to the peer. Since the mbuf * might be in use for the reply, do * not free it. */ m = NULL; } else { /* * We have created a full-blown * connection. */ tp = NULL; inp = NULL; #ifdef INET6 in6p = NULL; #endif switch (so->so_proto->pr_domain->dom_family) { case AF_INET: inp = sotoinpcb(so); tp = intotcpcb(inp); break; #ifdef INET6 case AF_INET6: in6p = sotoin6pcb(so); tp = in6totcpcb(in6p); break; #endif } if (tp == NULL) goto badsyn; /*XXX*/ tiwin <<= tp->snd_scale; goto after_listen; } } else { /* * None of RST, SYN or ACK was set. * This is an invalid packet for a * TCB in LISTEN state. Send a RST. */ goto badsyn; } } else { /* * Received a SYN. */ #ifdef INET6 /* * If deprecated address is forbidden, we do * not accept SYN to deprecated interface * address to prevent any new inbound * connection from getting established. * When we do not accept SYN, we send a TCP * RST, with deprecated source address (instead * of dropping it). We compromise it as it is * much better for peer to send a RST, and * RST will be the final packet for the * exchange. * * If we do not forbid deprecated addresses, we * accept the SYN packet. RFC2462 does not * suggest dropping SYN in this case. * If we decipher RFC2462 5.5.4, it says like * this: * 1. use of deprecated addr with existing * communication is okay - "SHOULD continue * to be used" * 2. use of it with new communication: * (2a) "SHOULD NOT be used if alternate * address with sufficient scope is * available" * (2b) nothing mentioned otherwise. * Here we fall into (2b) case as we have no * choice in our source address selection - we * must obey the peer. * * The wording in RFC2462 is confusing, and * there are multiple description text for * deprecated address handling - worse, they * are not exactly the same. I believe 5.5.4 * is the best one, so we follow 5.5.4. */ if (af == AF_INET6 && !ip6_use_deprecated) { struct in6_ifaddr *ia6; int s; struct ifnet *rcvif = m_get_rcvif(m, &s); if (rcvif == NULL) goto dropwithreset; /* XXX */ if ((ia6 = in6ifa_ifpwithaddr(rcvif, &ip6->ip6_dst)) && (ia6->ia6_flags & IN6_IFF_DEPRECATED)) { tp = NULL; m_put_rcvif(rcvif, &s); goto dropwithreset; } m_put_rcvif(rcvif, &s); } #endif /* * LISTEN socket received a SYN from itself? This * can't possibly be valid; drop the packet. */ if (th->th_sport == th->th_dport) { int eq = 0; switch (af) { case AF_INET: eq = in_hosteq(ip->ip_src, ip->ip_dst); break; #ifdef INET6 case AF_INET6: eq = IN6_ARE_ADDR_EQUAL(&ip6->ip6_src, &ip6->ip6_dst); break; #endif } if (eq) { TCP_STATINC(TCP_STAT_BADSYN); goto drop; } } /* * SYN looks ok; create compressed TCP * state for it. */ if (so->so_qlen <= so->so_qlimit && syn_cache_add(&src.sa, &dst.sa, th, off, so, m, optp, optlen, &opti)) m = NULL; } goto drop; } after_listen: /* * From here on, we're dealing with !LISTEN. */ KASSERT(tp->t_state != TCPS_LISTEN); /* * Segment received on connection. * Reset idle time and keep-alive timer. */ tp->t_rcvtime = tcp_now; if (TCPS_HAVEESTABLISHED(tp->t_state)) TCP_TIMER_ARM(tp, TCPT_KEEP, tp->t_keepidle); /* * Process options. */ #ifdef TCP_SIGNATURE if (optp || (tp->t_flags & TF_SIGNATURE)) #else if (optp) #endif if (tcp_dooptions(tp, optp, optlen, th, m, off, &opti) < 0) goto drop; if (TCP_SACK_ENABLED(tp)) { tcp_del_sackholes(tp, th); } if (TCP_ECN_ALLOWED(tp)) { if (tiflags & TH_CWR) { tp->t_flags &= ~TF_ECN_SND_ECE; } switch (iptos & IPTOS_ECN_MASK) { case IPTOS_ECN_CE: tp->t_flags |= TF_ECN_SND_ECE; TCP_STATINC(TCP_STAT_ECN_CE); break; case IPTOS_ECN_ECT0: TCP_STATINC(TCP_STAT_ECN_ECT); break; case IPTOS_ECN_ECT1: /* XXX: ignore for now -- rpaulo */ break; } /* * Congestion experienced. * Ignore if we are already trying to recover. */ if ((tiflags & TH_ECE) && SEQ_GEQ(tp->snd_una, tp->snd_recover)) tp->t_congctl->cong_exp(tp); } if (opti.ts_present && opti.ts_ecr) { /* * Calculate the RTT from the returned time stamp and the * connection's time base. If the time stamp is later than * the current time, or is extremely old, fall back to non-1323 * RTT calculation. Since ts_rtt is unsigned, we can test both * at the same time. * * Note that ts_rtt is in units of slow ticks (500 * ms). Since most earthbound RTTs are < 500 ms, * observed values will have large quantization noise. * Our smoothed RTT is then the fraction of observed * samples that are 1 tick instead of 0 (times 500 * ms). * * ts_rtt is increased by 1 to denote a valid sample, * with 0 indicating an invalid measurement. This * extra 1 must be removed when ts_rtt is used, or * else an erroneous extra 500 ms will result. */ ts_rtt = TCP_TIMESTAMP(tp) - opti.ts_ecr + 1; if (ts_rtt > TCP_PAWS_IDLE) ts_rtt = 0; } else { ts_rtt = 0; } /* * Fast path: check for the two common cases of a uni-directional * data transfer. If: * o We are in the ESTABLISHED state, and * o The packet has no control flags, and * o The packet is in-sequence, and * o The window didn't change, and * o We are not retransmitting * It's a candidate. * * If the length (tlen) is zero and the ack moved forward, we're * the sender side of the transfer. Just free the data acked and * wake any higher level process that was blocked waiting for * space. * * If the length is non-zero and the ack didn't move, we're the * receiver side. If we're getting packets in-order (the reassembly * queue is empty), add the data to the socket buffer and note * that we need a delayed ack. */ if (tp->t_state == TCPS_ESTABLISHED && (tiflags & (TH_SYN|TH_FIN|TH_RST|TH_URG|TH_ECE|TH_CWR|TH_ACK)) == TH_ACK && (!opti.ts_present || TSTMP_GEQ(opti.ts_val, tp->ts_recent)) && th->th_seq == tp->rcv_nxt && tiwin && tiwin == tp->snd_wnd && tp->snd_nxt == tp->snd_max) { /* * If last ACK falls within this segment's sequence numbers, * record the timestamp. * NOTE that the test is modified according to the latest * proposal of the tcplw@cray.com list (Braden 1993/04/26). * * note that we already know * TSTMP_GEQ(opti.ts_val, tp->ts_recent) */ if (opti.ts_present && SEQ_LEQ(th->th_seq, tp->last_ack_sent)) { tp->ts_recent_age = tcp_now; tp->ts_recent = opti.ts_val; } if (tlen == 0) { /* Ack prediction. */ if (SEQ_GT(th->th_ack, tp->snd_una) && SEQ_LEQ(th->th_ack, tp->snd_max) && tp->snd_cwnd >= tp->snd_wnd && tp->t_partialacks < 0) { /* * this is a pure ack for outstanding data. */ if (ts_rtt) tcp_xmit_timer(tp, ts_rtt - 1); else if (tp->t_rtttime && SEQ_GT(th->th_ack, tp->t_rtseq)) tcp_xmit_timer(tp, tcp_now - tp->t_rtttime); acked = th->th_ack - tp->snd_una; tcps = TCP_STAT_GETREF(); tcps[TCP_STAT_PREDACK]++; tcps[TCP_STAT_RCVACKPACK]++; tcps[TCP_STAT_RCVACKBYTE] += acked; TCP_STAT_PUTREF(); nd_hint(tp); if (acked > (tp->t_lastoff - tp->t_inoff)) tp->t_lastm = NULL; sbdrop(&so->so_snd, acked); tp->t_lastoff -= acked; icmp_check(tp, th, acked); tp->snd_una = th->th_ack; tp->snd_fack = tp->snd_una; if (SEQ_LT(tp->snd_high, tp->snd_una)) tp->snd_high = tp->snd_una; /* * drag snd_wl2 along so only newer * ACKs can update the window size. * also avoids the state where snd_wl2 * is eventually larger than th_ack and thus * blocking the window update mechanism and * the connection gets stuck for a loooong * time in the zero sized send window state. * * see PR/kern 55567 */ tp->snd_wl2 = tp->snd_una; m_freem(m); /* * If all outstanding data are acked, stop * retransmit timer, otherwise restart timer * using current (possibly backed-off) value. * If process is waiting for space, * wakeup/selnotify/signal. If data * are ready to send, let tcp_output * decide between more output or persist. */ if (tp->snd_una == tp->snd_max) TCP_TIMER_DISARM(tp, TCPT_REXMT); else if (TCP_TIMER_ISARMED(tp, TCPT_PERSIST) == 0) TCP_TIMER_ARM(tp, TCPT_REXMT, tp->t_rxtcur); sowwakeup(so); if (so->so_snd.sb_cc) { KERNEL_LOCK(1, NULL); (void)tcp_output(tp); KERNEL_UNLOCK_ONE(NULL); } if (tcp_saveti) m_freem(tcp_saveti); return; } } else if (th->th_ack == tp->snd_una && TAILQ_FIRST(&tp->segq) == NULL && tlen <= sbspace(&so->so_rcv)) { int newsize = 0; /* * this is a pure, in-sequence data packet * with nothing on the reassembly queue and * we have enough buffer space to take it. */ tp->rcv_nxt += tlen; /* * Pull rcv_up up to prevent seq wrap relative to * rcv_nxt. */ tp->rcv_up = tp->rcv_nxt; /* * Pull snd_wl1 up to prevent seq wrap relative to * th_seq. */ tp->snd_wl1 = th->th_seq; tcps = TCP_STAT_GETREF(); tcps[TCP_STAT_PREDDAT]++; tcps[TCP_STAT_RCVPACK]++; tcps[TCP_STAT_RCVBYTE] += tlen; TCP_STAT_PUTREF(); nd_hint(tp); /* * Automatic sizing enables the performance of large buffers * and most of the efficiency of small ones by only allocating * space when it is needed. * * On the receive side the socket buffer memory is only rarely * used to any significant extent. This allows us to be much * more aggressive in scaling the receive socket buffer. For * the case that the buffer space is actually used to a large * extent and we run out of kernel memory we can simply drop * the new segments; TCP on the sender will just retransmit it * later. Setting the buffer size too big may only consume too * much kernel memory if the application doesn't read() from * the socket or packet loss or reordering makes use of the * reassembly queue. * * The criteria to step up the receive buffer one notch are: * 1. the number of bytes received during the time it takes * one timestamp to be reflected back to us (the RTT); * 2. received bytes per RTT is within seven eighth of the * current socket buffer size; * 3. receive buffer size has not hit maximal automatic size; * * This algorithm does one step per RTT at most and only if * we receive a bulk stream w/o packet losses or reorderings. * Shrinking the buffer during idle times is not necessary as * it doesn't consume any memory when idle. * * TODO: Only step up if the application is actually serving * the buffer to better manage the socket buffer resources. */ if (tcp_do_autorcvbuf && opti.ts_ecr && (so->so_rcv.sb_flags & SB_AUTOSIZE)) { if (opti.ts_ecr > tp->rfbuf_ts && opti.ts_ecr - tp->rfbuf_ts < PR_SLOWHZ) { if (tp->rfbuf_cnt > (so->so_rcv.sb_hiwat / 8 * 7) && so->so_rcv.sb_hiwat < tcp_autorcvbuf_max) { newsize = uimin(so->so_rcv.sb_hiwat + tcp_autorcvbuf_inc, tcp_autorcvbuf_max); } /* Start over with next RTT. */ tp->rfbuf_ts = 0; tp->rfbuf_cnt = 0; } else tp->rfbuf_cnt += tlen; /* add up */ } /* * Drop TCP, IP headers and TCP options then add data * to socket buffer. */ if (so->so_state & SS_CANTRCVMORE) { m_freem(m); } else { /* * Set new socket buffer size. * Give up when limit is reached. */ if (newsize) if (!sbreserve(&so->so_rcv, newsize, so)) so->so_rcv.sb_flags &= ~SB_AUTOSIZE; m_adj(m, off + thlen); sbappendstream(&so->so_rcv, m); } sorwakeup(so); tcp_setup_ack(tp, th); if (tp->t_flags & TF_ACKNOW) { KERNEL_LOCK(1, NULL); (void)tcp_output(tp); KERNEL_UNLOCK_ONE(NULL); } if (tcp_saveti) m_freem(tcp_saveti); return; } } /* * Compute mbuf offset to TCP data segment. */ hdroptlen = off + thlen; /* * Calculate amount of space in receive window. Receive window is * amount of space in rcv queue, but not less than advertised * window. */ { int win; win = sbspace(&so->so_rcv); if (win < 0) win = 0; tp->rcv_wnd = imax(win, (int)(tp->rcv_adv - tp->rcv_nxt)); } /* Reset receive buffer auto scaling when not in bulk receive mode. */ tp->rfbuf_ts = 0; tp->rfbuf_cnt = 0; switch (tp->t_state) { /* * If the state is SYN_SENT: * if seg contains an ACK, but not for our SYN, drop the input. * if seg contains a RST, then drop the connection. * if seg does not contain SYN, then drop it. * Otherwise this is an acceptable SYN segment * initialize tp->rcv_nxt and tp->irs * if seg contains ack then advance tp->snd_una * if seg contains a ECE and ECN support is enabled, the stream * is ECN capable. * if SYN has been acked change to ESTABLISHED else SYN_RCVD state * arrange for segment to be acked (eventually) * continue processing rest of data/controls, beginning with URG */ case TCPS_SYN_SENT: if ((tiflags & TH_ACK) && (SEQ_LEQ(th->th_ack, tp->iss) || SEQ_GT(th->th_ack, tp->snd_max))) goto dropwithreset; if (tiflags & TH_RST) { if (tiflags & TH_ACK) tp = tcp_drop(tp, ECONNREFUSED); goto drop; } if ((tiflags & TH_SYN) == 0) goto drop; if (tiflags & TH_ACK) { tp->snd_una = th->th_ack; if (SEQ_LT(tp->snd_nxt, tp->snd_una)) tp->snd_nxt = tp->snd_una; if (SEQ_LT(tp->snd_high, tp->snd_una)) tp->snd_high = tp->snd_una; TCP_TIMER_DISARM(tp, TCPT_REXMT); if ((tiflags & TH_ECE) && tcp_do_ecn) { tp->t_flags |= TF_ECN_PERMIT; TCP_STATINC(TCP_STAT_ECN_SHS); } } tp->irs = th->th_seq; tcp_rcvseqinit(tp); tp->t_flags |= TF_ACKNOW; tcp_mss_from_peer(tp, opti.maxseg); /* * Initialize the initial congestion window. If we * had to retransmit the SYN, we must initialize cwnd * to 1 segment (i.e. the Loss Window). */ if (tp->t_flags & TF_SYN_REXMT) tp->snd_cwnd = tp->t_peermss; else { int ss = tcp_init_win; if (inp != NULL && in_localaddr(inp->inp_faddr)) ss = tcp_init_win_local; #ifdef INET6 if (in6p != NULL && in6_localaddr(&in6p->in6p_faddr)) ss = tcp_init_win_local; #endif tp->snd_cwnd = TCP_INITIAL_WINDOW(ss, tp->t_peermss); } tcp_rmx_rtt(tp); if (tiflags & TH_ACK) { TCP_STATINC(TCP_STAT_CONNECTS); /* * move tcp_established before soisconnected * because upcall handler can drive tcp_output * functionality. * XXX we might call soisconnected at the end of * all processing */ tcp_established(tp); soisconnected(so); /* Do window scaling on this connection? */ if ((tp->t_flags & (TF_RCVD_SCALE|TF_REQ_SCALE)) == (TF_RCVD_SCALE|TF_REQ_SCALE)) { tp->snd_scale = tp->requested_s_scale; tp->rcv_scale = tp->request_r_scale; } TCP_REASS_LOCK(tp); (void)tcp_reass(tp, NULL, NULL, tlen); /* * if we didn't have to retransmit the SYN, * use its rtt as our initial srtt & rtt var. */ if (tp->t_rtttime) tcp_xmit_timer(tp, tcp_now - tp->t_rtttime); } else { tp->t_state = TCPS_SYN_RECEIVED; } /* * Advance th->th_seq to correspond to first data byte. * If data, trim to stay within window, * dropping FIN if necessary. */ th->th_seq++; if (tlen > tp->rcv_wnd) { todrop = tlen - tp->rcv_wnd; m_adj(m, -todrop); tlen = tp->rcv_wnd; tiflags &= ~TH_FIN; tcps = TCP_STAT_GETREF(); tcps[TCP_STAT_RCVPACKAFTERWIN]++; tcps[TCP_STAT_RCVBYTEAFTERWIN] += todrop; TCP_STAT_PUTREF(); } tp->snd_wl1 = th->th_seq - 1; tp->rcv_up = th->th_seq; goto step6; /* * If the state is SYN_RECEIVED: * If seg contains an ACK, but not for our SYN, drop the input * and generate an RST. See page 36, rfc793 */ case TCPS_SYN_RECEIVED: if ((tiflags & TH_ACK) && (SEQ_LEQ(th->th_ack, tp->iss) || SEQ_GT(th->th_ack, tp->snd_max))) goto dropwithreset; break; } /* * From here on, we're dealing with !LISTEN and !SYN_SENT. */ KASSERT(tp->t_state != TCPS_LISTEN && tp->t_state != TCPS_SYN_SENT); /* * RFC1323 PAWS: if we have a timestamp reply on this segment and * it's less than ts_recent, drop it. */ if (opti.ts_present && (tiflags & TH_RST) == 0 && tp->ts_recent && TSTMP_LT(opti.ts_val, tp->ts_recent)) { /* Check to see if ts_recent is over 24 days old. */ if (tcp_now - tp->ts_recent_age > TCP_PAWS_IDLE) { /* * Invalidate ts_recent. If this segment updates * ts_recent, the age will be reset later and ts_recent * will get a valid value. If it does not, setting * ts_recent to zero will at least satisfy the * requirement that zero be placed in the timestamp * echo reply when ts_recent isn't valid. The * age isn't reset until we get a valid ts_recent * because we don't want out-of-order segments to be * dropped when ts_recent is old. */ tp->ts_recent = 0; } else { tcps = TCP_STAT_GETREF(); tcps[TCP_STAT_RCVDUPPACK]++; tcps[TCP_STAT_RCVDUPBYTE] += tlen; tcps[TCP_STAT_PAWSDROP]++; TCP_STAT_PUTREF(); tcp_new_dsack(tp, th->th_seq, tlen); goto dropafterack; } } /* * Check that at least some bytes of the segment are within the * receive window. If segment begins before rcv_nxt, drop leading * data (and SYN); if nothing left, just ack. */ todrop = tp->rcv_nxt - th->th_seq; dupseg = false; if (todrop > 0) { if (tiflags & TH_SYN) { tiflags &= ~TH_SYN; th->th_seq++; tcp_urp_drop(th, 1, &tiflags); todrop--; } if (todrop > tlen || (todrop == tlen && (tiflags & TH_FIN) == 0)) { /* * Any valid FIN or RST must be to the left of the * window. At this point the FIN or RST must be a * duplicate or out of sequence; drop it. */ if (tiflags & TH_RST) goto drop; tiflags &= ~(TH_FIN|TH_RST); /* * Send an ACK to resynchronize and drop any data. * But keep on processing for RST or ACK. */ tp->t_flags |= TF_ACKNOW; todrop = tlen; dupseg = true; tcps = TCP_STAT_GETREF(); tcps[TCP_STAT_RCVDUPPACK]++; tcps[TCP_STAT_RCVDUPBYTE] += todrop; TCP_STAT_PUTREF(); } else if ((tiflags & TH_RST) && th->th_seq != tp->rcv_nxt) { /* * Test for reset before adjusting the sequence * number for overlapping data. */ goto dropafterack_ratelim; } else { tcps = TCP_STAT_GETREF(); tcps[TCP_STAT_RCVPARTDUPPACK]++; tcps[TCP_STAT_RCVPARTDUPBYTE] += todrop; TCP_STAT_PUTREF(); } tcp_new_dsack(tp, th->th_seq, todrop); hdroptlen += todrop; /* drop from head afterwards (m_adj) */ th->th_seq += todrop; tlen -= todrop; tcp_urp_drop(th, todrop, &tiflags); } /* * If new data is received on a connection after the user processes * are gone, then RST the other end. */ if ((so->so_state & SS_NOFDREF) && tp->t_state > TCPS_CLOSE_WAIT && tlen) { tp = tcp_close(tp); TCP_STATINC(TCP_STAT_RCVAFTERCLOSE); goto dropwithreset; } /* * If the segment ends after the window, drop trailing data (and * PUSH and FIN); if nothing left, just ACK. */ todrop = (th->th_seq + tlen) - (tp->rcv_nxt + tp->rcv_wnd); if (todrop > 0) { TCP_STATINC(TCP_STAT_RCVPACKAFTERWIN); if (todrop >= tlen) { /* * The segment actually starts after the window. * th->th_seq + tlen - tp->rcv_nxt - tp->rcv_wnd >= tlen * th->th_seq - tp->rcv_nxt - tp->rcv_wnd >= 0 * th->th_seq >= tp->rcv_nxt + tp->rcv_wnd */ TCP_STATADD(TCP_STAT_RCVBYTEAFTERWIN, tlen); /* * If a new connection request is received while in * TIME_WAIT, drop the old connection and start over * if the sequence numbers are above the previous * ones. * * NOTE: We need to put the header fields back into * network order. */ if ((tiflags & TH_SYN) && tp->t_state == TCPS_TIME_WAIT && SEQ_GT(th->th_seq, tp->rcv_nxt)) { tp = tcp_close(tp); tcp_fields_to_net(th); m_freem(tcp_saveti); tcp_saveti = NULL; goto findpcb; } /* * If window is closed can only take segments at * window edge, and have to drop data and PUSH from * incoming segments. Continue processing, but * remember to ack. Otherwise, drop segment * and (if not RST) ack. */ if (tp->rcv_wnd == 0 && th->th_seq == tp->rcv_nxt) { KASSERT(todrop == tlen); tp->t_flags |= TF_ACKNOW; TCP_STATINC(TCP_STAT_RCVWINPROBE); } else { goto dropafterack; } } else { TCP_STATADD(TCP_STAT_RCVBYTEAFTERWIN, todrop); } m_adj(m, -todrop); tlen -= todrop; tiflags &= ~(TH_PUSH|TH_FIN); } /* * If last ACK falls within this segment's sequence numbers, * record the timestamp. * NOTE: * 1) That the test incorporates suggestions from the latest * proposal of the tcplw@cray.com list (Braden 1993/04/26). * 2) That updating only on newer timestamps interferes with * our earlier PAWS tests, so this check should be solely * predicated on the sequence space of this segment. * 3) That we modify the segment boundary check to be * Last.ACK.Sent <= SEG.SEQ + SEG.Len * instead of RFC1323's * Last.ACK.Sent < SEG.SEQ + SEG.Len, * This modified check allows us to overcome RFC1323's * limitations as described in Stevens TCP/IP Illustrated * Vol. 2 p.869. In such cases, we can still calculate the * RTT correctly when RCV.NXT == Last.ACK.Sent. */ if (opti.ts_present && SEQ_LEQ(th->th_seq, tp->last_ack_sent) && SEQ_LEQ(tp->last_ack_sent, th->th_seq + tlen + ((tiflags & (TH_SYN|TH_FIN)) != 0))) { tp->ts_recent_age = tcp_now; tp->ts_recent = opti.ts_val; } /* * If the RST bit is set examine the state: * RECEIVED state: * If passive open, return to LISTEN state. * If active open, inform user that connection was refused. * ESTABLISHED, FIN_WAIT_1, FIN_WAIT2, CLOSE_WAIT states: * Inform user that connection was reset, and close tcb. * CLOSING, LAST_ACK, TIME_WAIT states: * Close the tcb. */ if (tiflags & TH_RST) { if (th->th_seq != tp->rcv_nxt) goto dropafterack_ratelim; switch (tp->t_state) { case TCPS_SYN_RECEIVED: so->so_error = ECONNREFUSED; goto close; case TCPS_ESTABLISHED: case TCPS_FIN_WAIT_1: case TCPS_FIN_WAIT_2: case TCPS_CLOSE_WAIT: so->so_error = ECONNRESET; close: tp->t_state = TCPS_CLOSED; TCP_STATINC(TCP_STAT_DROPS); tp = tcp_close(tp); goto drop; case TCPS_CLOSING: case TCPS_LAST_ACK: case TCPS_TIME_WAIT: tp = tcp_close(tp); goto drop; } } /* * Since we've covered the SYN-SENT and SYN-RECEIVED states above * we must be in a synchronized state. RFC793 states (under Reset * Generation) that any unacceptable segment (an out-of-order SYN * qualifies) received in a synchronized state must elicit only an * empty acknowledgment segment ... and the connection remains in * the same state. */ if (tiflags & TH_SYN) { if (tp->rcv_nxt == th->th_seq) { tcp_respond(tp, m, m, th, (tcp_seq)0, th->th_ack - 1, TH_ACK); if (tcp_saveti) m_freem(tcp_saveti); return; } goto dropafterack_ratelim; } /* * If the ACK bit is off we drop the segment and return. */ if ((tiflags & TH_ACK) == 0) { if (tp->t_flags & TF_ACKNOW) goto dropafterack; goto drop; } /* * From here on, we're doing ACK processing. */ switch (tp->t_state) { /* * In SYN_RECEIVED state if the ack ACKs our SYN then enter * ESTABLISHED state and continue processing, otherwise * send an RST. */ case TCPS_SYN_RECEIVED: if (SEQ_GT(tp->snd_una, th->th_ack) || SEQ_GT(th->th_ack, tp->snd_max)) goto dropwithreset; TCP_STATINC(TCP_STAT_CONNECTS); soisconnected(so); tcp_established(tp); /* Do window scaling? */ if ((tp->t_flags & (TF_RCVD_SCALE|TF_REQ_SCALE)) == (TF_RCVD_SCALE|TF_REQ_SCALE)) { tp->snd_scale = tp->requested_s_scale; tp->rcv_scale = tp->request_r_scale; } TCP_REASS_LOCK(tp); (void)tcp_reass(tp, NULL, NULL, tlen); tp->snd_wl1 = th->th_seq - 1; /* FALLTHROUGH */ /* * In ESTABLISHED state: drop duplicate ACKs; ACK out of range * ACKs. If the ack is in the range * tp->snd_una < th->th_ack <= tp->snd_max * then advance tp->snd_una to th->th_ack and drop * data from the retransmission queue. If this ACK reflects * more up to date window information we update our window information. */ case TCPS_ESTABLISHED: case TCPS_FIN_WAIT_1: case TCPS_FIN_WAIT_2: case TCPS_CLOSE_WAIT: case TCPS_CLOSING: case TCPS_LAST_ACK: case TCPS_TIME_WAIT: if (SEQ_LEQ(th->th_ack, tp->snd_una)) { if (tlen == 0 && !dupseg && tiwin == tp->snd_wnd) { TCP_STATINC(TCP_STAT_RCVDUPACK); /* * If we have outstanding data (other than * a window probe), this is a completely * duplicate ack (ie, window info didn't * change), the ack is the biggest we've * seen and we've seen exactly our rexmt * threshold of them, assume a packet * has been dropped and retransmit it. * Kludge snd_nxt & the congestion * window so we send only this one * packet. */ if (TCP_TIMER_ISARMED(tp, TCPT_REXMT) == 0 || th->th_ack != tp->snd_una) tp->t_dupacks = 0; else if (tp->t_partialacks < 0 && (++tp->t_dupacks == tcprexmtthresh || TCP_FACK_FASTRECOV(tp))) { /* * Do the fast retransmit, and adjust * congestion control parameters. */ if (tp->t_congctl->fast_retransmit(tp, th)) { /* False fast retransmit */ break; } goto drop; } else if (tp->t_dupacks > tcprexmtthresh) { tp->snd_cwnd += tp->t_segsz; KERNEL_LOCK(1, NULL); (void)tcp_output(tp); KERNEL_UNLOCK_ONE(NULL); goto drop; } } else { /* * If the ack appears to be very old, only * allow data that is in-sequence. This * makes it somewhat more difficult to insert * forged data by guessing sequence numbers. * Sent an ack to try to update the send * sequence number on the other side. */ if (tlen && th->th_seq != tp->rcv_nxt && SEQ_LT(th->th_ack, tp->snd_una - tp->max_sndwnd)) goto dropafterack; } break; } /* * If the congestion window was inflated to account * for the other side's cached packets, retract it. */ tp->t_congctl->fast_retransmit_newack(tp, th); if (SEQ_GT(th->th_ack, tp->snd_max)) { TCP_STATINC(TCP_STAT_RCVACKTOOMUCH); goto dropafterack; } acked = th->th_ack - tp->snd_una; tcps = TCP_STAT_GETREF(); tcps[TCP_STAT_RCVACKPACK]++; tcps[TCP_STAT_RCVACKBYTE] += acked; TCP_STAT_PUTREF(); /* * If we have a timestamp reply, update smoothed * round trip time. If no timestamp is present but * transmit timer is running and timed sequence * number was acked, update smoothed round trip time. * Since we now have an rtt measurement, cancel the * timer backoff (cf., Phil Karn's retransmit alg.). * Recompute the initial retransmit timer. */ if (ts_rtt) tcp_xmit_timer(tp, ts_rtt - 1); else if (tp->t_rtttime && SEQ_GT(th->th_ack, tp->t_rtseq)) tcp_xmit_timer(tp, tcp_now - tp->t_rtttime); /* * If all outstanding data is acked, stop retransmit * timer and remember to restart (more output or persist). * If there is more data to be acked, restart retransmit * timer, using current (possibly backed-off) value. */ if (th->th_ack == tp->snd_max) { TCP_TIMER_DISARM(tp, TCPT_REXMT); needoutput = 1; } else if (TCP_TIMER_ISARMED(tp, TCPT_PERSIST) == 0) TCP_TIMER_ARM(tp, TCPT_REXMT, tp->t_rxtcur); /* * New data has been acked, adjust the congestion window. */ tp->t_congctl->newack(tp, th); nd_hint(tp); if (acked > so->so_snd.sb_cc) { tp->snd_wnd -= so->so_snd.sb_cc; sbdrop(&so->so_snd, (int)so->so_snd.sb_cc); ourfinisacked = 1; } else { if (acked > (tp->t_lastoff - tp->t_inoff)) tp->t_lastm = NULL; sbdrop(&so->so_snd, acked); tp->t_lastoff -= acked; if (tp->snd_wnd > acked) tp->snd_wnd -= acked; else tp->snd_wnd = 0; ourfinisacked = 0; } sowwakeup(so); icmp_check(tp, th, acked); tp->snd_una = th->th_ack; if (SEQ_GT(tp->snd_una, tp->snd_fack)) tp->snd_fack = tp->snd_una; if (SEQ_LT(tp->snd_nxt, tp->snd_una)) tp->snd_nxt = tp->snd_una; if (SEQ_LT(tp->snd_high, tp->snd_una)) tp->snd_high = tp->snd_una; switch (tp->t_state) { /* * In FIN_WAIT_1 STATE in addition to the processing * for the ESTABLISHED state if our FIN is now acknowledged * then enter FIN_WAIT_2. */ case TCPS_FIN_WAIT_1: if (ourfinisacked) { /* * If we can't receive any more * data, then closing user can proceed. * Starting the timer is contrary to the * specification, but if we don't get a FIN * we'll hang forever. */ if (so->so_state & SS_CANTRCVMORE) { soisdisconnected(so); if (tp->t_maxidle > 0) TCP_TIMER_ARM(tp, TCPT_2MSL, tp->t_maxidle); } tp->t_state = TCPS_FIN_WAIT_2; } break; /* * In CLOSING STATE in addition to the processing for * the ESTABLISHED state if the ACK acknowledges our FIN * then enter the TIME-WAIT state, otherwise ignore * the segment. */ case TCPS_CLOSING: if (ourfinisacked) { tp->t_state = TCPS_TIME_WAIT; tcp_canceltimers(tp); TCP_TIMER_ARM(tp, TCPT_2MSL, 2 * tp->t_msl); soisdisconnected(so); } break; /* * In LAST_ACK, we may still be waiting for data to drain * and/or to be acked, as well as for the ack of our FIN. * If our FIN is now acknowledged, delete the TCB, * enter the closed state and return. */ case TCPS_LAST_ACK: if (ourfinisacked) { tp = tcp_close(tp); goto drop; } break; /* * In TIME_WAIT state the only thing that should arrive * is a retransmission of the remote FIN. Acknowledge * it and restart the finack timer. */ case TCPS_TIME_WAIT: TCP_TIMER_ARM(tp, TCPT_2MSL, 2 * tp->t_msl); goto dropafterack; } } step6: /* * Update window information. * Don't look at window if no ACK: TAC's send garbage on first SYN. */ if ((tiflags & TH_ACK) && (SEQ_LT(tp->snd_wl1, th->th_seq) || (tp->snd_wl1 == th->th_seq && (SEQ_LT(tp->snd_wl2, th->th_ack) || (tp->snd_wl2 == th->th_ack && tiwin > tp->snd_wnd))))) { /* keep track of pure window updates */ if (tlen == 0 && tp->snd_wl2 == th->th_ack && tiwin > tp->snd_wnd) TCP_STATINC(TCP_STAT_RCVWINUPD); tp->snd_wnd = tiwin; tp->snd_wl1 = th->th_seq; tp->snd_wl2 = th->th_ack; if (tp->snd_wnd > tp->max_sndwnd) tp->max_sndwnd = tp->snd_wnd; needoutput = 1; } /* * Process segments with URG. */ if ((tiflags & TH_URG) && th->th_urp && TCPS_HAVERCVDFIN(tp->t_state) == 0) { /* * This is a kludge, but if we receive and accept * random urgent pointers, we'll crash in * soreceive. It's hard to imagine someone * actually wanting to send this much urgent data. */ if (th->th_urp + so->so_rcv.sb_cc > sb_max) { th->th_urp = 0; /* XXX */ tiflags &= ~TH_URG; /* XXX */ goto dodata; /* XXX */ } /* * If this segment advances the known urgent pointer, * then mark the data stream. This should not happen * in CLOSE_WAIT, CLOSING, LAST_ACK or TIME_WAIT STATES since * a FIN has been received from the remote side. * In these states we ignore the URG. * * According to RFC961 (Assigned Protocols), * the urgent pointer points to the last octet * of urgent data. We continue, however, * to consider it to indicate the first octet * of data past the urgent section as the original * spec states (in one of two places). */ if (SEQ_GT(th->th_seq+th->th_urp, tp->rcv_up)) { tp->rcv_up = th->th_seq + th->th_urp; so->so_oobmark = so->so_rcv.sb_cc + (tp->rcv_up - tp->rcv_nxt) - 1; if (so->so_oobmark == 0) so->so_state |= SS_RCVATMARK; sohasoutofband(so); tp->t_oobflags &= ~(TCPOOB_HAVEDATA | TCPOOB_HADDATA); } /* * Remove out of band data so doesn't get presented to user. * This can happen independent of advancing the URG pointer, * but if two URG's are pending at once, some out-of-band * data may creep in... ick. */ if (th->th_urp <= (u_int16_t)tlen && (so->so_options & SO_OOBINLINE) == 0) tcp_pulloutofband(so, th, m, hdroptlen); } else { /* * If no out of band data is expected, * pull receive urgent pointer along * with the receive window. */ if (SEQ_GT(tp->rcv_nxt, tp->rcv_up)) tp->rcv_up = tp->rcv_nxt; } dodata: /* * Process the segment text, merging it into the TCP sequencing queue, * and arranging for acknowledgement of receipt if necessary. * This process logically involves adjusting tp->rcv_wnd as data * is presented to the user (this happens in tcp_usrreq.c, * tcp_rcvd()). If a FIN has already been received on this * connection then we just ignore the text. */ if ((tlen || (tiflags & TH_FIN)) && TCPS_HAVERCVDFIN(tp->t_state) == 0) { /* * Handle the common case: * o Segment is the next to be received, and * o The queue is empty, and * o The connection is established * In this case, we avoid calling tcp_reass. * * tcp_setup_ack: set DELACK for segments received in order, * but ack immediately when segments are out of order (so that * fast retransmit can work). */ TCP_REASS_LOCK(tp); if (th->th_seq == tp->rcv_nxt && TAILQ_FIRST(&tp->segq) == NULL && tp->t_state == TCPS_ESTABLISHED) { tcp_setup_ack(tp, th); tp->rcv_nxt += tlen; tiflags = th->th_flags & TH_FIN; tcps = TCP_STAT_GETREF(); tcps[TCP_STAT_RCVPACK]++; tcps[TCP_STAT_RCVBYTE] += tlen; TCP_STAT_PUTREF(); nd_hint(tp); if (so->so_state & SS_CANTRCVMORE) { m_freem(m); } else { m_adj(m, hdroptlen); sbappendstream(&(so)->so_rcv, m); } TCP_REASS_UNLOCK(tp); sorwakeup(so); } else { m_adj(m, hdroptlen); tiflags = tcp_reass(tp, th, m, tlen); tp->t_flags |= TF_ACKNOW; } /* * Note the amount of data that peer has sent into * our window, in order to estimate the sender's * buffer size. */ len = so->so_rcv.sb_hiwat - (tp->rcv_adv - tp->rcv_nxt); } else { m_freem(m); m = NULL; tiflags &= ~TH_FIN; } /* * If FIN is received ACK the FIN and let the user know * that the connection is closing. Ignore a FIN received before * the connection is fully established. */ if ((tiflags & TH_FIN) && TCPS_HAVEESTABLISHED(tp->t_state)) { if (TCPS_HAVERCVDFIN(tp->t_state) == 0) { socantrcvmore(so); tp->t_flags |= TF_ACKNOW; tp->rcv_nxt++; } switch (tp->t_state) { /* * In ESTABLISHED STATE enter the CLOSE_WAIT state. */ case TCPS_ESTABLISHED: tp->t_state = TCPS_CLOSE_WAIT; break; /* * If still in FIN_WAIT_1 STATE FIN has not been acked so * enter the CLOSING state. */ case TCPS_FIN_WAIT_1: tp->t_state = TCPS_CLOSING; break; /* * In FIN_WAIT_2 state enter the TIME_WAIT state, * starting the time-wait timer, turning off the other * standard timers. */ case TCPS_FIN_WAIT_2: tp->t_state = TCPS_TIME_WAIT; tcp_canceltimers(tp); TCP_TIMER_ARM(tp, TCPT_2MSL, 2 * tp->t_msl); soisdisconnected(so); break; /* * In TIME_WAIT state restart the 2 MSL time_wait timer. */ case TCPS_TIME_WAIT: TCP_TIMER_ARM(tp, TCPT_2MSL, 2 * tp->t_msl); break; } } #ifdef TCP_DEBUG if (so->so_options & SO_DEBUG) tcp_trace(TA_INPUT, ostate, tp, tcp_saveti, 0); #endif /* * Return any desired output. */ if (needoutput || (tp->t_flags & TF_ACKNOW)) { KERNEL_LOCK(1, NULL); (void)tcp_output(tp); KERNEL_UNLOCK_ONE(NULL); } if (tcp_saveti) m_freem(tcp_saveti); if (tp->t_state == TCPS_TIME_WAIT && (so->so_state & SS_NOFDREF) && (tp->t_inpcb || af != AF_INET) && (tp->t_in6pcb || af != AF_INET6) && ((af == AF_INET ? tcp4_vtw_enable : tcp6_vtw_enable) & 1) != 0 && TAILQ_EMPTY(&tp->segq) && vtw_add(af, tp)) { ; } return; badsyn: /* * Received a bad SYN. Increment counters and dropwithreset. */ TCP_STATINC(TCP_STAT_BADSYN); tp = NULL; goto dropwithreset; dropafterack: /* * Generate an ACK dropping incoming segment if it occupies * sequence space, where the ACK reflects our state. */ if (tiflags & TH_RST) goto drop; goto dropafterack2; dropafterack_ratelim: /* * We may want to rate-limit ACKs against SYN/RST attack. */ if (ppsratecheck(&tcp_ackdrop_ppslim_last, &tcp_ackdrop_ppslim_count, tcp_ackdrop_ppslim) == 0) { /* XXX stat */ goto drop; } dropafterack2: m_freem(m); tp->t_flags |= TF_ACKNOW; KERNEL_LOCK(1, NULL); (void)tcp_output(tp); KERNEL_UNLOCK_ONE(NULL); if (tcp_saveti) m_freem(tcp_saveti); return; dropwithreset_ratelim: /* * We may want to rate-limit RSTs in certain situations, * particularly if we are sending an RST in response to * an attempt to connect to or otherwise communicate with * a port for which we have no socket. */ if (ppsratecheck(&tcp_rst_ppslim_last, &tcp_rst_ppslim_count, tcp_rst_ppslim) == 0) { /* XXX stat */ goto drop; } dropwithreset: /* * Generate a RST, dropping incoming segment. * Make ACK acceptable to originator of segment. */ if (tiflags & TH_RST) goto drop; if (tiflags & TH_ACK) { (void)tcp_respond(tp, m, m, th, (tcp_seq)0, th->th_ack, TH_RST); } else { if (tiflags & TH_SYN) tlen++; (void)tcp_respond(tp, m, m, th, th->th_seq + tlen, (tcp_seq)0, TH_RST|TH_ACK); } if (tcp_saveti) m_freem(tcp_saveti); return; badcsum: drop: /* * Drop space held by incoming segment and return. */ if (tp) { if (tp->t_inpcb) so = tp->t_inpcb->inp_socket; #ifdef INET6 else if (tp->t_in6pcb) so = tp->t_in6pcb->in6p_socket; #endif else so = NULL; #ifdef TCP_DEBUG if (so && (so->so_options & SO_DEBUG) != 0) tcp_trace(TA_DROP, ostate, tp, tcp_saveti, 0); #endif } if (tcp_saveti) m_freem(tcp_saveti); m_freem(m); return; } #ifdef TCP_SIGNATURE int tcp_signature_apply(void *fstate, void *data, u_int len) { MD5Update(fstate, (u_char *)data, len); return (0); } struct secasvar * tcp_signature_getsav(struct mbuf *m) { struct ip *ip; struct ip6_hdr *ip6; ip = mtod(m, struct ip *); switch (ip->ip_v) { case 4: ip = mtod(m, struct ip *); ip6 = NULL; break; case 6: ip = NULL; ip6 = mtod(m, struct ip6_hdr *); break; default: return (NULL); } #ifdef IPSEC union sockaddr_union dst; /* Extract the destination from the IP header in the mbuf. */ memset(&dst, 0, sizeof(union sockaddr_union)); if (ip != NULL) { dst.sa.sa_len = sizeof(struct sockaddr_in); dst.sa.sa_family = AF_INET; dst.sin.sin_addr = ip->ip_dst; } else { dst.sa.sa_len = sizeof(struct sockaddr_in6); dst.sa.sa_family = AF_INET6; dst.sin6.sin6_addr = ip6->ip6_dst; } /* * Look up an SADB entry which matches the address of the peer. */ return KEY_LOOKUP_SA(&dst, IPPROTO_TCP, htonl(TCP_SIG_SPI), 0, 0); #else return NULL; #endif } int tcp_signature(struct mbuf *m, struct tcphdr *th, int thoff, struct secasvar *sav, char *sig) { MD5_CTX ctx; struct ip *ip; struct ipovly *ipovly; #ifdef INET6 struct ip6_hdr *ip6; struct ip6_hdr_pseudo ip6pseudo; #endif struct ippseudo ippseudo; struct tcphdr th0; int l, tcphdrlen; if (sav == NULL) return (-1); tcphdrlen = th->th_off * 4; switch (mtod(m, struct ip *)->ip_v) { case 4: MD5Init(&ctx); ip = mtod(m, struct ip *); memset(&ippseudo, 0, sizeof(ippseudo)); ipovly = (struct ipovly *)ip; ippseudo.ippseudo_src = ipovly->ih_src; ippseudo.ippseudo_dst = ipovly->ih_dst; ippseudo.ippseudo_pad = 0; ippseudo.ippseudo_p = IPPROTO_TCP; ippseudo.ippseudo_len = htons(m->m_pkthdr.len - thoff); MD5Update(&ctx, (char *)&ippseudo, sizeof(ippseudo)); break; #if INET6 case 6: MD5Init(&ctx); ip6 = mtod(m, struct ip6_hdr *); memset(&ip6pseudo, 0, sizeof(ip6pseudo)); ip6pseudo.ip6ph_src = ip6->ip6_src; in6_clearscope(&ip6pseudo.ip6ph_src); ip6pseudo.ip6ph_dst = ip6->ip6_dst; in6_clearscope(&ip6pseudo.ip6ph_dst); ip6pseudo.ip6ph_len = htons(m->m_pkthdr.len - thoff); ip6pseudo.ip6ph_nxt = IPPROTO_TCP; MD5Update(&ctx, (char *)&ip6pseudo, sizeof(ip6pseudo)); break; #endif default: return (-1); } th0 = *th; th0.th_sum = 0; MD5Update(&ctx, (char *)&th0, sizeof(th0)); l = m->m_pkthdr.len - thoff - tcphdrlen; if (l > 0) m_apply(m, thoff + tcphdrlen, m->m_pkthdr.len - thoff - tcphdrlen, tcp_signature_apply, &ctx); MD5Update(&ctx, _KEYBUF(sav->key_auth), _KEYLEN(sav->key_auth)); MD5Final(sig, &ctx); return (0); } #endif /* * Parse and process tcp options. * * Returns -1 if this segment should be dropped. (eg. wrong signature) * Otherwise returns 0. */ static int tcp_dooptions(struct tcpcb *tp, const u_char *cp, int cnt, struct tcphdr *th, struct mbuf *m, int toff, struct tcp_opt_info *oi) { u_int16_t mss; int opt, optlen = 0; #ifdef TCP_SIGNATURE void *sigp = NULL; char sigbuf[TCP_SIGLEN]; struct secasvar *sav = NULL; #endif for (; cp && cnt > 0; cnt -= optlen, cp += optlen) { opt = cp[0]; if (opt == TCPOPT_EOL) break; if (opt == TCPOPT_NOP) optlen = 1; else { if (cnt < 2) break; optlen = cp[1]; if (optlen < 2 || optlen > cnt) break; } switch (opt) { default: continue; case TCPOPT_MAXSEG: if (optlen != TCPOLEN_MAXSEG) continue; if (!(th->th_flags & TH_SYN)) continue; if (TCPS_HAVERCVDSYN(tp->t_state)) continue; memcpy(&mss, cp + 2, sizeof(mss)); oi->maxseg = ntohs(mss); break; case TCPOPT_WINDOW: if (optlen != TCPOLEN_WINDOW) continue; if (!(th->th_flags & TH_SYN)) continue; if (TCPS_HAVERCVDSYN(tp->t_state)) continue; tp->t_flags |= TF_RCVD_SCALE; tp->requested_s_scale = cp[2]; if (tp->requested_s_scale > TCP_MAX_WINSHIFT) { char buf[INET6_ADDRSTRLEN]; struct ip *ip = mtod(m, struct ip *); #ifdef INET6 struct ip6_hdr *ip6 = mtod(m, struct ip6_hdr *); #endif switch (ip->ip_v) { case 4: in_print(buf, sizeof(buf), &ip->ip_src); break; #ifdef INET6 case 6: in6_print(buf, sizeof(buf), &ip6->ip6_src); break; #endif default: strlcpy(buf, "(unknown)", sizeof(buf)); break; } log(LOG_ERR, "TCP: invalid wscale %d from %s, " "assuming %d\n", tp->requested_s_scale, buf, TCP_MAX_WINSHIFT); tp->requested_s_scale = TCP_MAX_WINSHIFT; } break; case TCPOPT_TIMESTAMP: if (optlen != TCPOLEN_TIMESTAMP) continue; oi->ts_present = 1; memcpy(&oi->ts_val, cp + 2, sizeof(oi->ts_val)); NTOHL(oi->ts_val); memcpy(&oi->ts_ecr, cp + 6, sizeof(oi->ts_ecr)); NTOHL(oi->ts_ecr); if (!(th->th_flags & TH_SYN)) continue; if (TCPS_HAVERCVDSYN(tp->t_state)) continue; /* * A timestamp received in a SYN makes * it ok to send timestamp requests and replies. */ tp->t_flags |= TF_RCVD_TSTMP; tp->ts_recent = oi->ts_val; tp->ts_recent_age = tcp_now; break; case TCPOPT_SACK_PERMITTED: if (optlen != TCPOLEN_SACK_PERMITTED) continue; if (!(th->th_flags & TH_SYN)) continue; if (TCPS_HAVERCVDSYN(tp->t_state)) continue; if (tcp_do_sack) { tp->t_flags |= TF_SACK_PERMIT; tp->t_flags |= TF_WILL_SACK; } break; case TCPOPT_SACK: tcp_sack_option(tp, th, cp, optlen); break; #ifdef TCP_SIGNATURE case TCPOPT_SIGNATURE: if (optlen != TCPOLEN_SIGNATURE) continue; if (sigp && !consttime_memequal(sigp, cp + 2, TCP_SIGLEN)) return (-1); sigp = sigbuf; memcpy(sigbuf, cp + 2, TCP_SIGLEN); tp->t_flags |= TF_SIGNATURE; break; #endif } } #ifndef TCP_SIGNATURE return 0; #else if (tp->t_flags & TF_SIGNATURE) { sav = tcp_signature_getsav(m); if (sav == NULL && tp->t_state == TCPS_LISTEN) return (-1); } if ((sigp ? TF_SIGNATURE : 0) ^ (tp->t_flags & TF_SIGNATURE)) goto out; if (sigp) { char sig[TCP_SIGLEN]; tcp_fields_to_net(th); if (tcp_signature(m, th, toff, sav, sig) < 0) { tcp_fields_to_host(th); goto out; } tcp_fields_to_host(th); if (!consttime_memequal(sig, sigp, TCP_SIGLEN)) { TCP_STATINC(TCP_STAT_BADSIG); goto out; } else TCP_STATINC(TCP_STAT_GOODSIG); key_sa_recordxfer(sav, m); KEY_SA_UNREF(&sav); } return 0; out: if (sav != NULL) KEY_SA_UNREF(&sav); return -1; #endif } /* * Pull out of band byte out of a segment so * it doesn't appear in the user's data queue. * It is still reflected in the segment length for * sequencing purposes. */ void tcp_pulloutofband(struct socket *so, struct tcphdr *th, struct mbuf *m, int off) { int cnt = off + th->th_urp - 1; while (cnt >= 0) { if (m->m_len > cnt) { char *cp = mtod(m, char *) + cnt; struct tcpcb *tp = sototcpcb(so); tp->t_iobc = *cp; tp->t_oobflags |= TCPOOB_HAVEDATA; memmove(cp, cp + 1, (unsigned)(m->m_len - cnt - 1)); m->m_len--; return; } cnt -= m->m_len; m = m->m_next; if (m == NULL) break; } panic("tcp_pulloutofband"); } /* * Collect new round-trip time estimate * and update averages and current timeout. * * rtt is in units of slow ticks (typically 500 ms) -- essentially the * difference of two timestamps. */ void tcp_xmit_timer(struct tcpcb *tp, uint32_t rtt) { int32_t delta; TCP_STATINC(TCP_STAT_RTTUPDATED); if (tp->t_srtt != 0) { /* * Compute the amount to add to srtt for smoothing, * *alpha, or 2^(-TCP_RTT_SHIFT). Because * srtt is stored in 1/32 slow ticks, we conceptually * shift left 5 bits, subtract srtt to get the * difference, and then shift right by TCP_RTT_SHIFT * (3) to obtain 1/8 of the difference. */ delta = (rtt << 2) - (tp->t_srtt >> TCP_RTT_SHIFT); /* * This can never happen, because delta's lowest * possible value is 1/8 of t_srtt. But if it does, * set srtt to some reasonable value, here chosen * as 1/8 tick. */ if ((tp->t_srtt += delta) <= 0) tp->t_srtt = 1 << 2; /* * RFC2988 requires that rttvar be updated first. * This code is compliant because "delta" is the old * srtt minus the new observation (scaled). * * RFC2988 says: * rttvar = (1-beta) * rttvar + beta * |srtt-observed| * * delta is in units of 1/32 ticks, and has then been * divided by 8. This is equivalent to being in 1/16s * units and divided by 4. Subtract from it 1/4 of * the existing rttvar to form the (signed) amount to * adjust. */ if (delta < 0) delta = -delta; delta -= (tp->t_rttvar >> TCP_RTTVAR_SHIFT); /* * As with srtt, this should never happen. There is * no support in RFC2988 for this operation. But 1/4s * as rttvar when faced with something arguably wrong * is ok. */ if ((tp->t_rttvar += delta) <= 0) tp->t_rttvar = 1 << 2; /* * If srtt exceeds .01 second, ensure we use the 'remote' MSL * Problem is: it doesn't work. Disabled by defaulting * tcp_rttlocal to 0; see corresponding code in * tcp_subr that selects local vs remote in a different way. * * The static branch prediction hint here should be removed * when the rtt estimator is fixed and the rtt_enable code * is turned back on. */ if (__predict_false(tcp_rttlocal) && tcp_msl_enable && tp->t_srtt > tcp_msl_remote_threshold && tp->t_msl < tcp_msl_remote) { tp->t_msl = MIN(tcp_msl_remote, TCP_MAXMSL); } } else { /* * This is the first measurement. Per RFC2988, 2.2, * set rtt=R and srtt=R/2. * For srtt, storage representation is 1/32 ticks, * so shift left by 5. * For rttvar, storage representation is 1/16 ticks, * So shift left by 4, but then right by 1 to halve. */ tp->t_srtt = rtt << (TCP_RTT_SHIFT + 2); tp->t_rttvar = rtt << (TCP_RTTVAR_SHIFT + 2 - 1); } tp->t_rtttime = 0; tp->t_rxtshift = 0; /* * the retransmit should happen at rtt + 4 * rttvar. * Because of the way we do the smoothing, srtt and rttvar * will each average +1/2 tick of bias. When we compute * the retransmit timer, we want 1/2 tick of rounding and * 1 extra tick because of +-1/2 tick uncertainty in the * firing of the timer. The bias will give us exactly the * 1.5 tick we need. But, because the bias is * statistical, we have to test that we don't drop below * the minimum feasible timer (which is 2 ticks). */ TCPT_RANGESET(tp->t_rxtcur, TCP_REXMTVAL(tp), uimax(tp->t_rttmin, rtt + 2), TCPTV_REXMTMAX); /* * We received an ack for a packet that wasn't retransmitted; * it is probably safe to discard any error indications we've * received recently. This isn't quite right, but close enough * for now (a route might have failed after we sent a segment, * and the return path might not be symmetrical). */ tp->t_softerror = 0; } /* * TCP compressed state engine. Currently used to hold compressed * state for SYN_RECEIVED. */ u_long syn_cache_count; u_int32_t syn_hash1, syn_hash2; #define SYN_HASH(sa, sp, dp) \ ((((sa)->s_addr^syn_hash1)*(((((u_int32_t)(dp))<<16) + \ ((u_int32_t)(sp)))^syn_hash2))) #ifndef INET6 #define SYN_HASHALL(hash, src, dst) \ do { \ hash = SYN_HASH(&((const struct sockaddr_in *)(src))->sin_addr, \ ((const struct sockaddr_in *)(src))->sin_port, \ ((const struct sockaddr_in *)(dst))->sin_port); \ } while (/*CONSTCOND*/ 0) #else #define SYN_HASH6(sa, sp, dp) \ ((((sa)->s6_addr32[0] ^ (sa)->s6_addr32[3] ^ syn_hash1) * \ (((((u_int32_t)(dp))<<16) + ((u_int32_t)(sp)))^syn_hash2)) \ & 0x7fffffff) #define SYN_HASHALL(hash, src, dst) \ do { \ switch ((src)->sa_family) { \ case AF_INET: \ hash = SYN_HASH(&((const struct sockaddr_in *)(src))->sin_addr, \ ((const struct sockaddr_in *)(src))->sin_port, \ ((const struct sockaddr_in *)(dst))->sin_port); \ break; \ case AF_INET6: \ hash = SYN_HASH6(&((const struct sockaddr_in6 *)(src))->sin6_addr, \ ((const struct sockaddr_in6 *)(src))->sin6_port, \ ((const struct sockaddr_in6 *)(dst))->sin6_port); \ break; \ default: \ hash = 0; \ } \ } while (/*CONSTCOND*/0) #endif /* INET6 */ static struct pool syn_cache_pool; /* * We don't estimate RTT with SYNs, so each packet starts with the default * RTT and each timer step has a fixed timeout value. */ static inline void syn_cache_timer_arm(struct syn_cache *sc) { TCPT_RANGESET(sc->sc_rxtcur, TCPTV_SRTTDFLT * tcp_backoff[sc->sc_rxtshift], TCPTV_MIN, TCPTV_REXMTMAX); callout_reset(&sc->sc_timer, sc->sc_rxtcur * (hz / PR_SLOWHZ), syn_cache_timer, sc); } #define SYN_CACHE_TIMESTAMP(sc) (tcp_now - (sc)->sc_timebase) static inline void syn_cache_rm(struct syn_cache *sc) { TAILQ_REMOVE(&tcp_syn_cache[sc->sc_bucketidx].sch_bucket, sc, sc_bucketq); sc->sc_tp = NULL; LIST_REMOVE(sc, sc_tpq); tcp_syn_cache[sc->sc_bucketidx].sch_length--; callout_stop(&sc->sc_timer); syn_cache_count--; } static inline void syn_cache_put(struct syn_cache *sc) { if (sc->sc_ipopts) (void) m_free(sc->sc_ipopts); rtcache_free(&sc->sc_route); sc->sc_flags |= SCF_DEAD; if (!callout_invoking(&sc->sc_timer)) callout_schedule(&(sc)->sc_timer, 1); } void syn_cache_init(void) { int i; pool_init(&syn_cache_pool, sizeof(struct syn_cache), 0, 0, 0, "synpl", NULL, IPL_SOFTNET); /* Initialize the hash buckets. */ for (i = 0; i < tcp_syn_cache_size; i++) TAILQ_INIT(&tcp_syn_cache[i].sch_bucket); } void syn_cache_insert(struct syn_cache *sc, struct tcpcb *tp) { struct syn_cache_head *scp; struct syn_cache *sc2; int s; /* * If there are no entries in the hash table, reinitialize * the hash secrets. */ if (syn_cache_count == 0) { syn_hash1 = cprng_fast32(); syn_hash2 = cprng_fast32(); } SYN_HASHALL(sc->sc_hash, &sc->sc_src.sa, &sc->sc_dst.sa); sc->sc_bucketidx = sc->sc_hash % tcp_syn_cache_size; scp = &tcp_syn_cache[sc->sc_bucketidx]; /* * Make sure that we don't overflow the per-bucket * limit or the total cache size limit. */ s = splsoftnet(); if (scp->sch_length >= tcp_syn_bucket_limit) { TCP_STATINC(TCP_STAT_SC_BUCKETOVERFLOW); /* * The bucket is full. Toss the oldest element in the * bucket. This will be the first entry in the bucket. */ sc2 = TAILQ_FIRST(&scp->sch_bucket); #ifdef DIAGNOSTIC /* * This should never happen; we should always find an * entry in our bucket. */ if (sc2 == NULL) panic("syn_cache_insert: bucketoverflow: impossible"); #endif syn_cache_rm(sc2); syn_cache_put(sc2); /* calls pool_put but see spl above */ } else if (syn_cache_count >= tcp_syn_cache_limit) { struct syn_cache_head *scp2, *sce; TCP_STATINC(TCP_STAT_SC_OVERFLOWED); /* * The cache is full. Toss the oldest entry in the * first non-empty bucket we can find. * * XXX We would really like to toss the oldest * entry in the cache, but we hope that this * condition doesn't happen very often. */ scp2 = scp; if (TAILQ_EMPTY(&scp2->sch_bucket)) { sce = &tcp_syn_cache[tcp_syn_cache_size]; for (++scp2; scp2 != scp; scp2++) { if (scp2 >= sce) scp2 = &tcp_syn_cache[0]; if (! TAILQ_EMPTY(&scp2->sch_bucket)) break; } #ifdef DIAGNOSTIC /* * This should never happen; we should always find a * non-empty bucket. */ if (scp2 == scp) panic("syn_cache_insert: cacheoverflow: " "impossible"); #endif } sc2 = TAILQ_FIRST(&scp2->sch_bucket); syn_cache_rm(sc2); syn_cache_put(sc2); /* calls pool_put but see spl above */ } /* * Initialize the entry's timer. */ sc->sc_rxttot = 0; sc->sc_rxtshift = 0; syn_cache_timer_arm(sc); /* Link it from tcpcb entry */ LIST_INSERT_HEAD(&tp->t_sc, sc, sc_tpq); /* Put it into the bucket. */ TAILQ_INSERT_TAIL(&scp->sch_bucket, sc, sc_bucketq); scp->sch_length++; syn_cache_count++; TCP_STATINC(TCP_STAT_SC_ADDED); splx(s); } /* * Walk the timer queues, looking for SYN,ACKs that need to be retransmitted. * If we have retransmitted an entry the maximum number of times, expire * that entry. */ static void syn_cache_timer(void *arg) { struct syn_cache *sc = arg; mutex_enter(softnet_lock); KERNEL_LOCK(1, NULL); callout_ack(&sc->sc_timer); if (__predict_false(sc->sc_flags & SCF_DEAD)) { TCP_STATINC(TCP_STAT_SC_DELAYED_FREE); goto free; } if (__predict_false(sc->sc_rxtshift == TCP_MAXRXTSHIFT)) { /* Drop it -- too many retransmissions. */ goto dropit; } /* * Compute the total amount of time this entry has * been on a queue. If this entry has been on longer * than the keep alive timer would allow, expire it. */ sc->sc_rxttot += sc->sc_rxtcur; if (sc->sc_rxttot >= MIN(tcp_keepinit, TCP_TIMER_MAXTICKS)) goto dropit; TCP_STATINC(TCP_STAT_SC_RETRANSMITTED); (void)syn_cache_respond(sc); /* Advance the timer back-off. */ sc->sc_rxtshift++; syn_cache_timer_arm(sc); goto out; dropit: TCP_STATINC(TCP_STAT_SC_TIMED_OUT); syn_cache_rm(sc); if (sc->sc_ipopts) (void) m_free(sc->sc_ipopts); rtcache_free(&sc->sc_route); free: callout_destroy(&sc->sc_timer); pool_put(&syn_cache_pool, sc); out: KERNEL_UNLOCK_ONE(NULL); mutex_exit(softnet_lock); } /* * Remove syn cache created by the specified tcb entry, * because this does not make sense to keep them * (if there's no tcb entry, syn cache entry will never be used) */ void syn_cache_cleanup(struct tcpcb *tp) { struct syn_cache *sc, *nsc; int s; s = splsoftnet(); for (sc = LIST_FIRST(&tp->t_sc); sc != NULL; sc = nsc) { nsc = LIST_NEXT(sc, sc_tpq); #ifdef DIAGNOSTIC if (sc->sc_tp != tp) panic("invalid sc_tp in syn_cache_cleanup"); #endif syn_cache_rm(sc); syn_cache_put(sc); /* calls pool_put but see spl above */ } /* just for safety */ LIST_INIT(&tp->t_sc); splx(s); } /* * Find an entry in the syn cache. */ struct syn_cache * syn_cache_lookup(const struct sockaddr *src, const struct sockaddr *dst, struct syn_cache_head **headp) { struct syn_cache *sc; struct syn_cache_head *scp; u_int32_t hash; int s; SYN_HASHALL(hash, src, dst); scp = &tcp_syn_cache[hash % tcp_syn_cache_size]; *headp = scp; s = splsoftnet(); for (sc = TAILQ_FIRST(&scp->sch_bucket); sc != NULL; sc = TAILQ_NEXT(sc, sc_bucketq)) { if (sc->sc_hash != hash) continue; if (!memcmp(&sc->sc_src, src, src->sa_len) && !memcmp(&sc->sc_dst, dst, dst->sa_len)) { splx(s); return (sc); } } splx(s); return (NULL); } /* * This function gets called when we receive an ACK for a socket in the * LISTEN state. We look up the connection in the syn cache, and if it's * there, we pull it out of the cache and turn it into a full-blown * connection in the SYN-RECEIVED state. * * The return values may not be immediately obvious, and their effects * can be subtle, so here they are: * * NULL SYN was not found in cache; caller should drop the * packet and send an RST. * * -1 We were unable to create the new connection, and are * aborting it. An ACK,RST is being sent to the peer * (unless we got screwey sequence numbers; see below), * because the 3-way handshake has been completed. Caller * should not free the mbuf, since we may be using it. If * we are not, we will free it. * * Otherwise, the return value is a pointer to the new socket * associated with the connection. */ struct socket * syn_cache_get(struct sockaddr *src, struct sockaddr *dst, struct tcphdr *th, struct socket *so, struct mbuf *m) { struct syn_cache *sc; struct syn_cache_head *scp; struct inpcb *inp = NULL; #ifdef INET6 struct in6pcb *in6p = NULL; #endif struct tcpcb *tp; int s; struct socket *oso; s = splsoftnet(); if ((sc = syn_cache_lookup(src, dst, &scp)) == NULL) { splx(s); return NULL; } /* * Verify the sequence and ack numbers. Try getting the correct * response again. */ if ((th->th_ack != sc->sc_iss + 1) || SEQ_LEQ(th->th_seq, sc->sc_irs) || SEQ_GT(th->th_seq, sc->sc_irs + 1 + sc->sc_win)) { m_freem(m); (void)syn_cache_respond(sc); splx(s); return ((struct socket *)(-1)); } /* Remove this cache entry */ syn_cache_rm(sc); splx(s); /* * Ok, create the full blown connection, and set things up * as they would have been set up if we had created the * connection when the SYN arrived. If we can't create * the connection, abort it. */ /* * inp still has the OLD in_pcb stuff, set the * v6-related flags on the new guy, too. This is * done particularly for the case where an AF_INET6 * socket is bound only to a port, and a v4 connection * comes in on that port. * we also copy the flowinfo from the original pcb * to the new one. */ oso = so; so = sonewconn(so, true); if (so == NULL) goto resetandabort; switch (so->so_proto->pr_domain->dom_family) { case AF_INET: inp = sotoinpcb(so); break; #ifdef INET6 case AF_INET6: in6p = sotoin6pcb(so); break; #endif } switch (src->sa_family) { case AF_INET: if (inp) { inp->inp_laddr = ((struct sockaddr_in *)dst)->sin_addr; inp->inp_lport = ((struct sockaddr_in *)dst)->sin_port; inp->inp_options = ip_srcroute(m); in_pcbstate(inp, INP_BOUND); if (inp->inp_options == NULL) { inp->inp_options = sc->sc_ipopts; sc->sc_ipopts = NULL; } } #ifdef INET6 else if (in6p) { /* IPv4 packet to AF_INET6 socket */ memset(&in6p->in6p_laddr, 0, sizeof(in6p->in6p_laddr)); in6p->in6p_laddr.s6_addr16[5] = htons(0xffff); bcopy(&((struct sockaddr_in *)dst)->sin_addr, &in6p->in6p_laddr.s6_addr32[3], sizeof(((struct sockaddr_in *)dst)->sin_addr)); in6p->in6p_lport = ((struct sockaddr_in *)dst)->sin_port; in6totcpcb(in6p)->t_family = AF_INET; if (sotoin6pcb(oso)->in6p_flags & IN6P_IPV6_V6ONLY) in6p->in6p_flags |= IN6P_IPV6_V6ONLY; else in6p->in6p_flags &= ~IN6P_IPV6_V6ONLY; in6_pcbstate(in6p, IN6P_BOUND); } #endif break; #ifdef INET6 case AF_INET6: if (in6p) { in6p->in6p_laddr = ((struct sockaddr_in6 *)dst)->sin6_addr; in6p->in6p_lport = ((struct sockaddr_in6 *)dst)->sin6_port; in6_pcbstate(in6p, IN6P_BOUND); } break; #endif } #ifdef INET6 if (in6p && in6totcpcb(in6p)->t_family == AF_INET6 && sotoinpcb(oso)) { struct in6pcb *oin6p = sotoin6pcb(oso); /* inherit socket options from the listening socket */ in6p->in6p_flags |= (oin6p->in6p_flags & IN6P_CONTROLOPTS); if (in6p->in6p_flags & IN6P_CONTROLOPTS) { m_freem(in6p->in6p_options); in6p->in6p_options = NULL; } ip6_savecontrol(in6p, &in6p->in6p_options, mtod(m, struct ip6_hdr *), m); } #endif /* * Give the new socket our cached route reference. */ if (inp) { rtcache_copy(&inp->inp_route, &sc->sc_route); rtcache_free(&sc->sc_route); } #ifdef INET6 else { rtcache_copy(&in6p->in6p_route, &sc->sc_route); rtcache_free(&sc->sc_route); } #endif if (inp) { struct sockaddr_in sin; memcpy(&sin, src, src->sa_len); if (in_pcbconnect(inp, &sin, &lwp0)) { goto resetandabort; } } #ifdef INET6 else if (in6p) { struct sockaddr_in6 sin6; memcpy(&sin6, src, src->sa_len); if (src->sa_family == AF_INET) { /* IPv4 packet to AF_INET6 socket */ in6_sin_2_v4mapsin6((struct sockaddr_in *)src, &sin6); } if (in6_pcbconnect(in6p, &sin6, NULL)) { goto resetandabort; } } #endif else { goto resetandabort; } if (inp) tp = intotcpcb(inp); #ifdef INET6 else if (in6p) tp = in6totcpcb(in6p); #endif else tp = NULL; tp->t_flags = sototcpcb(oso)->t_flags & TF_NODELAY; if (sc->sc_request_r_scale != 15) { tp->requested_s_scale = sc->sc_requested_s_scale; tp->request_r_scale = sc->sc_request_r_scale; tp->snd_scale = sc->sc_requested_s_scale; tp->rcv_scale = sc->sc_request_r_scale; tp->t_flags |= TF_REQ_SCALE|TF_RCVD_SCALE; } if (sc->sc_flags & SCF_TIMESTAMP) tp->t_flags |= TF_REQ_TSTMP|TF_RCVD_TSTMP; tp->ts_timebase = sc->sc_timebase; tp->t_template = tcp_template(tp); if (tp->t_template == 0) { tp = tcp_drop(tp, ENOBUFS); /* destroys socket */ so = NULL; m_freem(m); goto abort; } tp->iss = sc->sc_iss; tp->irs = sc->sc_irs; tcp_sendseqinit(tp); tcp_rcvseqinit(tp); tp->t_state = TCPS_SYN_RECEIVED; TCP_TIMER_ARM(tp, TCPT_KEEP, tp->t_keepinit); TCP_STATINC(TCP_STAT_ACCEPTS); if ((sc->sc_flags & SCF_SACK_PERMIT) && tcp_do_sack) tp->t_flags |= TF_WILL_SACK; if ((sc->sc_flags & SCF_ECN_PERMIT) && tcp_do_ecn) tp->t_flags |= TF_ECN_PERMIT; #ifdef TCP_SIGNATURE if (sc->sc_flags & SCF_SIGNATURE) tp->t_flags |= TF_SIGNATURE; #endif /* Initialize tp->t_ourmss before we deal with the peer's! */ tp->t_ourmss = sc->sc_ourmaxseg; tcp_mss_from_peer(tp, sc->sc_peermaxseg); /* * Initialize the initial congestion window. If we * had to retransmit the SYN,ACK, we must initialize cwnd * to 1 segment (i.e. the Loss Window). */ if (sc->sc_rxtshift) tp->snd_cwnd = tp->t_peermss; else { int ss = tcp_init_win; if (inp != NULL && in_localaddr(inp->inp_faddr)) ss = tcp_init_win_local; #ifdef INET6 if (in6p != NULL && in6_localaddr(&in6p->in6p_faddr)) ss = tcp_init_win_local; #endif tp->snd_cwnd = TCP_INITIAL_WINDOW(ss, tp->t_peermss); } tcp_rmx_rtt(tp); tp->snd_wl1 = sc->sc_irs; tp->rcv_up = sc->sc_irs + 1; /* * This is what would have happened in tcp_output() when * the SYN,ACK was sent. */ tp->snd_up = tp->snd_una; tp->snd_max = tp->snd_nxt = tp->iss+1; TCP_TIMER_ARM(tp, TCPT_REXMT, tp->t_rxtcur); if (sc->sc_win > 0 && SEQ_GT(tp->rcv_nxt + sc->sc_win, tp->rcv_adv)) tp->rcv_adv = tp->rcv_nxt + sc->sc_win; tp->last_ack_sent = tp->rcv_nxt; tp->t_partialacks = -1; tp->t_dupacks = 0; TCP_STATINC(TCP_STAT_SC_COMPLETED); s = splsoftnet(); syn_cache_put(sc); splx(s); return so; resetandabort: (void)tcp_respond(NULL, m, m, th, (tcp_seq)0, th->th_ack, TH_RST); abort: if (so != NULL) { (void) soqremque(so, 1); (void) soabort(so); mutex_enter(softnet_lock); } s = splsoftnet(); syn_cache_put(sc); splx(s); TCP_STATINC(TCP_STAT_SC_ABORTED); return ((struct socket *)(-1)); } /* * This function is called when we get a RST for a * non-existent connection, so that we can see if the * connection is in the syn cache. If it is, zap it. */ void syn_cache_reset(struct sockaddr *src, struct sockaddr *dst, struct tcphdr *th) { struct syn_cache *sc; struct syn_cache_head *scp; int s = splsoftnet(); if ((sc = syn_cache_lookup(src, dst, &scp)) == NULL) { splx(s); return; } if (SEQ_LT(th->th_seq, sc->sc_irs) || SEQ_GT(th->th_seq, sc->sc_irs+1)) { splx(s); return; } syn_cache_rm(sc); TCP_STATINC(TCP_STAT_SC_RESET); syn_cache_put(sc); /* calls pool_put but see spl above */ splx(s); } void syn_cache_unreach(const struct sockaddr *src, const struct sockaddr *dst, struct tcphdr *th) { struct syn_cache *sc; struct syn_cache_head *scp; int s; s = splsoftnet(); if ((sc = syn_cache_lookup(src, dst, &scp)) == NULL) { splx(s); return; } /* If the sequence number != sc_iss, then it's a bogus ICMP msg */ if (ntohl(th->th_seq) != sc->sc_iss) { splx(s); return; } /* * If we've retransmitted 3 times and this is our second error, * we remove the entry. Otherwise, we allow it to continue on. * This prevents us from incorrectly nuking an entry during a * spurious network outage. * * See tcp_notify(). */ if ((sc->sc_flags & SCF_UNREACH) == 0 || sc->sc_rxtshift < 3) { sc->sc_flags |= SCF_UNREACH; splx(s); return; } syn_cache_rm(sc); TCP_STATINC(TCP_STAT_SC_UNREACH); syn_cache_put(sc); /* calls pool_put but see spl above */ splx(s); } /* * Given a LISTEN socket and an inbound SYN request, add this to the syn * cache, and send back a segment: * <SEQ=ISS><ACK=RCV_NXT><CTL=SYN,ACK> * to the source. * * IMPORTANT NOTE: We do _NOT_ ACK data that might accompany the SYN. * Doing so would require that we hold onto the data and deliver it * to the application. However, if we are the target of a SYN-flood * DoS attack, an attacker could send data which would eventually * consume all available buffer space if it were ACKed. By not ACKing * the data, we avoid this DoS scenario. */ int syn_cache_add(struct sockaddr *src, struct sockaddr *dst, struct tcphdr *th, unsigned int toff, struct socket *so, struct mbuf *m, u_char *optp, int optlen, struct tcp_opt_info *oi) { struct tcpcb tb, *tp; long win; struct syn_cache *sc; struct syn_cache_head *scp; struct mbuf *ipopts; int s; tp = sototcpcb(so); /* * Initialize some local state. */ win = sbspace(&so->so_rcv); if (win > TCP_MAXWIN) win = TCP_MAXWIN; #ifdef TCP_SIGNATURE if (optp || (tp->t_flags & TF_SIGNATURE)) #else if (optp) #endif { tb.t_flags = tcp_do_rfc1323 ? (TF_REQ_SCALE|TF_REQ_TSTMP) : 0; #ifdef TCP_SIGNATURE tb.t_flags |= (tp->t_flags & TF_SIGNATURE); #endif tb.t_state = TCPS_LISTEN; if (tcp_dooptions(&tb, optp, optlen, th, m, toff, oi) < 0) return 0; } else tb.t_flags = 0; switch (src->sa_family) { case AF_INET: /* Remember the IP options, if any. */ ipopts = ip_srcroute(m); break; default: ipopts = NULL; } /* * See if we already have an entry for this connection. * If we do, resend the SYN,ACK. We do not count this * as a retransmission (XXX though maybe we should). */ if ((sc = syn_cache_lookup(src, dst, &scp)) != NULL) { TCP_STATINC(TCP_STAT_SC_DUPESYN); if (ipopts) { /* * If we were remembering a previous source route, * forget it and use the new one we've been given. */ if (sc->sc_ipopts) (void)m_free(sc->sc_ipopts); sc->sc_ipopts = ipopts; } sc->sc_timestamp = tb.ts_recent; m_freem(m); if (syn_cache_respond(sc) == 0) { uint64_t *tcps = TCP_STAT_GETREF(); tcps[TCP_STAT_SNDACKS]++; tcps[TCP_STAT_SNDTOTAL]++; TCP_STAT_PUTREF(); } return 1; } s = splsoftnet(); sc = pool_get(&syn_cache_pool, PR_NOWAIT); splx(s); if (sc == NULL) { if (ipopts) (void)m_free(ipopts); return 0; } /* * Fill in the cache, and put the necessary IP and TCP * options into the reply. */ memset(sc, 0, sizeof(struct syn_cache)); callout_init(&sc->sc_timer, CALLOUT_MPSAFE); memcpy(&sc->sc_src, src, src->sa_len); memcpy(&sc->sc_dst, dst, dst->sa_len); sc->sc_flags = 0; sc->sc_ipopts = ipopts; sc->sc_irs = th->th_seq; switch (src->sa_family) { case AF_INET: { struct sockaddr_in *srcin = (void *)src; struct sockaddr_in *dstin = (void *)dst; sc->sc_iss = tcp_new_iss1(&dstin->sin_addr, &srcin->sin_addr, dstin->sin_port, srcin->sin_port, sizeof(dstin->sin_addr)); break; } #ifdef INET6 case AF_INET6: { struct sockaddr_in6 *srcin6 = (void *)src; struct sockaddr_in6 *dstin6 = (void *)dst; sc->sc_iss = tcp_new_iss1(&dstin6->sin6_addr, &srcin6->sin6_addr, dstin6->sin6_port, srcin6->sin6_port, sizeof(dstin6->sin6_addr)); break; } #endif } sc->sc_peermaxseg = oi->maxseg; sc->sc_ourmaxseg = tcp_mss_to_advertise(m->m_flags & M_PKTHDR ? m_get_rcvif_NOMPSAFE(m) : NULL, sc->sc_src.sa.sa_family); sc->sc_win = win; sc->sc_timebase = tcp_now - 1; /* see tcp_newtcpcb() */ sc->sc_timestamp = tb.ts_recent; if ((tb.t_flags & (TF_REQ_TSTMP|TF_RCVD_TSTMP)) == (TF_REQ_TSTMP|TF_RCVD_TSTMP)) sc->sc_flags |= SCF_TIMESTAMP; if ((tb.t_flags & (TF_RCVD_SCALE|TF_REQ_SCALE)) == (TF_RCVD_SCALE|TF_REQ_SCALE)) { sc->sc_requested_s_scale = tb.requested_s_scale; sc->sc_request_r_scale = 0; /* * Pick the smallest possible scaling factor that * will still allow us to scale up to sb_max. * * We do this because there are broken firewalls that * will corrupt the window scale option, leading to * the other endpoint believing that our advertised * window is unscaled. At scale factors larger than * 5 the unscaled window will drop below 1500 bytes, * leading to serious problems when traversing these * broken firewalls. * * With the default sbmax of 256K, a scale factor * of 3 will be chosen by this algorithm. Those who * choose a larger sbmax should watch out * for the compatibility problems mentioned above. * * RFC1323: The Window field in a SYN (i.e., a <SYN> * or <SYN,ACK>) segment itself is never scaled. */ while (sc->sc_request_r_scale < TCP_MAX_WINSHIFT && (TCP_MAXWIN << sc->sc_request_r_scale) < sb_max) sc->sc_request_r_scale++; } else { sc->sc_requested_s_scale = 15; sc->sc_request_r_scale = 15; } if ((tb.t_flags & TF_SACK_PERMIT) && tcp_do_sack) sc->sc_flags |= SCF_SACK_PERMIT; /* * ECN setup packet received. */ if ((th->th_flags & (TH_ECE|TH_CWR)) && tcp_do_ecn) sc->sc_flags |= SCF_ECN_PERMIT; #ifdef TCP_SIGNATURE if (tb.t_flags & TF_SIGNATURE) sc->sc_flags |= SCF_SIGNATURE; #endif sc->sc_tp = tp; m_freem(m); if (syn_cache_respond(sc) == 0) { uint64_t *tcps = TCP_STAT_GETREF(); tcps[TCP_STAT_SNDACKS]++; tcps[TCP_STAT_SNDTOTAL]++; TCP_STAT_PUTREF(); syn_cache_insert(sc, tp); } else { s = splsoftnet(); /* * syn_cache_put() will try to schedule the timer, so * we need to initialize it */ syn_cache_timer_arm(sc); syn_cache_put(sc); splx(s); TCP_STATINC(TCP_STAT_SC_DROPPED); } return 1; } /* * syn_cache_respond: (re)send SYN+ACK. * * Returns 0 on success. */ int syn_cache_respond(struct syn_cache *sc) { #ifdef INET6 struct rtentry *rt = NULL; #endif struct route *ro; u_int8_t *optp; int optlen, error; u_int16_t tlen; struct ip *ip = NULL; #ifdef INET6 struct ip6_hdr *ip6 = NULL; #endif struct tcpcb *tp; struct tcphdr *th; struct mbuf *m; u_int hlen; #ifdef TCP_SIGNATURE struct secasvar *sav = NULL; u_int8_t *sigp = NULL; #endif ro = &sc->sc_route; switch (sc->sc_src.sa.sa_family) { case AF_INET: hlen = sizeof(struct ip); break; #ifdef INET6 case AF_INET6: hlen = sizeof(struct ip6_hdr); break; #endif default: return EAFNOSUPPORT; } /* Worst case scenario, since we don't know the option size yet. */ tlen = hlen + sizeof(struct tcphdr) + MAX_TCPOPTLEN; KASSERT(max_linkhdr + tlen <= MCLBYTES); /* * Create the IP+TCP header from scratch. */ MGETHDR(m, M_DONTWAIT, MT_DATA); if (m && (max_linkhdr + tlen) > MHLEN) { MCLGET(m, M_DONTWAIT); if ((m->m_flags & M_EXT) == 0) { m_freem(m); m = NULL; } } if (m == NULL) return ENOBUFS; MCLAIM(m, &tcp_tx_mowner); tp = sc->sc_tp; /* Fixup the mbuf. */ m->m_data += max_linkhdr; m_reset_rcvif(m); memset(mtod(m, void *), 0, tlen); switch (sc->sc_src.sa.sa_family) { case AF_INET: ip = mtod(m, struct ip *); ip->ip_v = 4; ip->ip_dst = sc->sc_src.sin.sin_addr; ip->ip_src = sc->sc_dst.sin.sin_addr; ip->ip_p = IPPROTO_TCP; th = (struct tcphdr *)(ip + 1); th->th_dport = sc->sc_src.sin.sin_port; th->th_sport = sc->sc_dst.sin.sin_port; break; #ifdef INET6 case AF_INET6: ip6 = mtod(m, struct ip6_hdr *); ip6->ip6_vfc = IPV6_VERSION; ip6->ip6_dst = sc->sc_src.sin6.sin6_addr; ip6->ip6_src = sc->sc_dst.sin6.sin6_addr; ip6->ip6_nxt = IPPROTO_TCP; /* ip6_plen will be updated in ip6_output() */ th = (struct tcphdr *)(ip6 + 1); th->th_dport = sc->sc_src.sin6.sin6_port; th->th_sport = sc->sc_dst.sin6.sin6_port; break; #endif default: panic("%s: impossible (1)", __func__); } th->th_seq = htonl(sc->sc_iss); th->th_ack = htonl(sc->sc_irs + 1); th->th_flags = TH_SYN|TH_ACK; th->th_win = htons(sc->sc_win); /* th_x2, th_sum, th_urp already 0 from memset */ /* Tack on the TCP options. */ optp = (u_int8_t *)(th + 1); optlen = 0; *optp++ = TCPOPT_MAXSEG; *optp++ = TCPOLEN_MAXSEG; *optp++ = (sc->sc_ourmaxseg >> 8) & 0xff; *optp++ = sc->sc_ourmaxseg & 0xff; optlen += TCPOLEN_MAXSEG; if (sc->sc_request_r_scale != 15) { *((u_int32_t *)optp) = htonl(TCPOPT_NOP << 24 | TCPOPT_WINDOW << 16 | TCPOLEN_WINDOW << 8 | sc->sc_request_r_scale); optp += TCPOLEN_WINDOW + TCPOLEN_NOP; optlen += TCPOLEN_WINDOW + TCPOLEN_NOP; } if (sc->sc_flags & SCF_SACK_PERMIT) { /* Let the peer know that we will SACK. */ *optp++ = TCPOPT_SACK_PERMITTED; *optp++ = TCPOLEN_SACK_PERMITTED; optlen += TCPOLEN_SACK_PERMITTED; } if (sc->sc_flags & SCF_TIMESTAMP) { while (optlen % 4 != 2) { optlen += TCPOLEN_NOP; *optp++ = TCPOPT_NOP; } *optp++ = TCPOPT_TIMESTAMP; *optp++ = TCPOLEN_TIMESTAMP; u_int32_t *lp = (u_int32_t *)(optp); /* Form timestamp option as shown in appendix A of RFC 1323. */ *lp++ = htonl(SYN_CACHE_TIMESTAMP(sc)); *lp = htonl(sc->sc_timestamp); optp += TCPOLEN_TIMESTAMP - 2; optlen += TCPOLEN_TIMESTAMP; } #ifdef TCP_SIGNATURE if (sc->sc_flags & SCF_SIGNATURE) { sav = tcp_signature_getsav(m); if (sav == NULL) { m_freem(m); return EPERM; } *optp++ = TCPOPT_SIGNATURE; *optp++ = TCPOLEN_SIGNATURE; sigp = optp; memset(optp, 0, TCP_SIGLEN); optp += TCP_SIGLEN; optlen += TCPOLEN_SIGNATURE; } #endif /* * Terminate and pad TCP options to a 4 byte boundary. * * According to RFC793: "The content of the header beyond the * End-of-Option option must be header padding (i.e., zero)." * And later: "The padding is composed of zeros." */ if (optlen % 4) { optlen += TCPOLEN_EOL; *optp++ = TCPOPT_EOL; } while (optlen % 4) { optlen += TCPOLEN_PAD; *optp++ = TCPOPT_PAD; } /* Compute the actual values now that we've added the options. */ tlen = hlen + sizeof(struct tcphdr) + optlen; m->m_len = m->m_pkthdr.len = tlen; th->th_off = (sizeof(struct tcphdr) + optlen) >> 2; #ifdef TCP_SIGNATURE if (sav) { (void)tcp_signature(m, th, hlen, sav, sigp); key_sa_recordxfer(sav, m); KEY_SA_UNREF(&sav); } #endif /* * Send ECN SYN-ACK setup packet. * Routes can be asymmetric, so, even if we receive a packet * with ECE and CWR set, we must not assume no one will block * the ECE packet we are about to send. */ if ((sc->sc_flags & SCF_ECN_PERMIT) && tp && SEQ_GEQ(tp->snd_nxt, tp->snd_max)) { th->th_flags |= TH_ECE; TCP_STATINC(TCP_STAT_ECN_SHS); /* * draft-ietf-tcpm-ecnsyn-00.txt * * "[...] a TCP node MAY respond to an ECN-setup * SYN packet by setting ECT in the responding * ECN-setup SYN/ACK packet, indicating to routers * that the SYN/ACK packet is ECN-Capable. * This allows a congested router along the path * to mark the packet instead of dropping the * packet as an indication of congestion." * * "[...] There can be a great benefit in setting * an ECN-capable codepoint in SYN/ACK packets [...] * Congestion is most likely to occur in * the server-to-client direction. As a result, * setting an ECN-capable codepoint in SYN/ACK * packets can reduce the occurrence of three-second * retransmit timeouts resulting from the drop * of SYN/ACK packets." * * Page 4 and 6, January 2006. */ switch (sc->sc_src.sa.sa_family) { case AF_INET: ip->ip_tos |= IPTOS_ECN_ECT0; break; #ifdef INET6 case AF_INET6: ip6->ip6_flow |= htonl(IPTOS_ECN_ECT0 << 20); break; #endif } TCP_STATINC(TCP_STAT_ECN_ECT); } /* * Compute the packet's checksum. * * Fill in some straggling IP bits. Note the stack expects * ip_len to be in host order, for convenience. */ switch (sc->sc_src.sa.sa_family) { case AF_INET: ip->ip_len = htons(tlen - hlen); th->th_sum = 0; th->th_sum = in4_cksum(m, IPPROTO_TCP, hlen, tlen - hlen); ip->ip_len = htons(tlen); ip->ip_ttl = ip_defttl; /* XXX tos? */ break; #ifdef INET6 case AF_INET6: ip6->ip6_plen = htons(tlen - hlen); th->th_sum = 0; th->th_sum = in6_cksum(m, IPPROTO_TCP, hlen, tlen - hlen); ip6->ip6_vfc &= ~IPV6_VERSION_MASK; ip6->ip6_vfc |= IPV6_VERSION; ip6->ip6_plen = htons(tlen - hlen); /* ip6_hlim will be initialized afterwards */ /* XXX flowlabel? */ break; #endif } /* XXX use IPsec policy on listening socket, on SYN ACK */ tp = sc->sc_tp; switch (sc->sc_src.sa.sa_family) { case AF_INET: error = ip_output(m, sc->sc_ipopts, ro, (ip_mtudisc ? IP_MTUDISC : 0), NULL, tp ? tp->t_inpcb : NULL); break; #ifdef INET6 case AF_INET6: ip6->ip6_hlim = in6_selecthlim(NULL, (rt = rtcache_validate(ro)) != NULL ? rt->rt_ifp : NULL); rtcache_unref(rt, ro); error = ip6_output(m, NULL /*XXX*/, ro, 0, NULL, tp ? tp->t_in6pcb : NULL, NULL); break; #endif default: panic("%s: impossible (2)", __func__); } return error; } |
| 23 23 23 23 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 197 198 199 200 201 202 203 204 205 206 207 208 209 210 211 212 213 214 215 216 217 218 219 220 221 222 223 224 225 226 227 228 229 230 231 232 233 234 235 236 237 238 239 240 241 242 243 244 245 246 247 248 249 250 251 252 253 254 255 256 257 258 259 260 261 262 263 264 265 266 267 268 269 270 271 272 273 274 275 276 277 278 279 280 281 282 283 284 285 286 287 288 289 290 291 292 293 294 295 296 297 298 299 300 301 302 303 304 305 306 307 308 309 310 311 312 313 314 315 316 317 318 319 320 321 322 323 324 325 326 327 328 329 330 331 332 333 334 335 336 337 338 339 340 341 342 343 344 345 346 347 348 349 350 351 352 353 354 355 356 357 358 359 360 361 362 363 364 365 366 367 368 369 370 371 372 373 374 375 376 377 378 379 380 381 382 383 384 385 386 387 388 389 390 391 392 393 394 395 396 397 398 399 400 401 402 403 404 405 406 | /* $NetBSD: isa_machdep.c,v 1.52 2022/04/15 17:53:44 jmcneill Exp $ */ /*- * Copyright (c) 1996, 1997, 1998 The NetBSD Foundation, Inc. * All rights reserved. * * This code is derived from software contributed to The NetBSD Foundation * by Charles M. Hannum and by Jason R. Thorpe of the Numerical Aerospace * Simulation Facility, NASA Ames Research Center. * * Redistribution and use in source and binary forms, with or without * modification, are permitted provided that the following conditions * are met: * 1. Redistributions of source code must retain the above copyright * notice, this list of conditions and the following disclaimer. * 2. Redistributions in binary form must reproduce the above copyright * notice, this list of conditions and the following disclaimer in the * documentation and/or other materials provided with the distribution. * * THIS SOFTWARE IS PROVIDED BY THE NETBSD FOUNDATION, INC. AND CONTRIBUTORS * ``AS IS'' AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED * TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR * PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE FOUNDATION OR CONTRIBUTORS * BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR * CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF * SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS * INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN * CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) * ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE * POSSIBILITY OF SUCH DAMAGE. */ /*- * Copyright (c) 1991 The Regents of the University of California. * All rights reserved. * * This code is derived from software contributed to Berkeley by * William Jolitz. * * Redistribution and use in source and binary forms, with or without * modification, are permitted provided that the following conditions * are met: * 1. Redistributions of source code must retain the above copyright * notice, this list of conditions and the following disclaimer. * 2. Redistributions in binary form must reproduce the above copyright * notice, this list of conditions and the following disclaimer in the * documentation and/or other materials provided with the distribution. * 3. Neither the name of the University nor the names of its contributors * may be used to endorse or promote products derived from this software * without specific prior written permission. * * THIS SOFTWARE IS PROVIDED BY THE REGENTS AND CONTRIBUTORS ``AS IS'' AND * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE * ARE DISCLAIMED. IN NO EVENT SHALL THE REGENTS OR CONTRIBUTORS BE LIABLE * FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL * DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS * OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) * HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT * LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY * OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF * SUCH DAMAGE. * * @(#)isa.c 7.2 (Berkeley) 5/13/91 */ #include <sys/cdefs.h> __KERNEL_RCSID(0, "$NetBSD: isa_machdep.c,v 1.52 2022/04/15 17:53:44 jmcneill Exp $"); #include <sys/param.h> #include <sys/systm.h> #include <sys/kernel.h> #include <sys/syslog.h> #include <sys/device.h> #include <sys/proc.h> #include <sys/bus.h> #include <sys/cpu.h> #include <machine/bus_private.h> #include <machine/pio.h> #include <machine/cpufunc.h> #include <machine/autoconf.h> #include <machine/bootinfo.h> #include <dev/isa/isareg.h> #include <dev/isa/isavar.h> #include <uvm/uvm_extern.h> #include "acpica.h" #include "opt_acpi.h" #include "ioapic.h" #if NIOAPIC > 0 #include <machine/i82093var.h> #include <machine/mpbiosvar.h> #endif #if NACPICA > 0 #include <dev/acpi/acpivar.h> #endif static int _isa_dma_may_bounce(bus_dma_tag_t, bus_dmamap_t, int, int *); struct x86_bus_dma_tag isa_bus_dma_tag = { ._tag_needs_free = 0, ._bounce_thresh = ISA_DMA_BOUNCE_THRESHOLD, ._bounce_alloc_lo = 0, ._bounce_alloc_hi = ISA_DMA_BOUNCE_THRESHOLD, ._may_bounce = _isa_dma_may_bounce, }; #define IDTVEC(name) __CONCAT(X,name) typedef void (vector)(void); extern vector *IDTVEC(intr)[]; #define LEGAL_IRQ(x) ((x) >= 0 && (x) < NUM_LEGACY_IRQS && (x) != 2) int isa_intr_alloc(isa_chipset_tag_t ic, int mask, int type, int *irq) { int i, tmp, bestirq, count; struct intrhand **p, *q; struct intrsource *isp; struct cpu_info *ci; if (type == IST_NONE) panic("intr_alloc: bogus type"); ci = &cpu_info_primary; bestirq = -1; count = -1; /* some interrupts should never be dynamically allocated */ mask &= 0xdef8; /* * XXX some interrupts will be used later (6 for fdc, 12 for pms). * the right answer is to do "breadth-first" searching of devices. */ mask &= 0xefbf; mutex_enter(&cpu_lock); for (i = 0; i < NUM_LEGACY_IRQS; i++) { if (LEGAL_IRQ(i) == 0 || (mask & (1<<i)) == 0) continue; isp = ci->ci_isources[i]; if (isp == NULL) { /* if nothing's using the irq, just return it */ *irq = i; mutex_exit(&cpu_lock); return 0; } switch(isp->is_type) { case IST_EDGE: case IST_LEVEL: if (type != isp->is_type) continue; /* * if the irq is shareable, count the number of other * handlers, and if it's smaller than the last irq like * this, remember it * * XXX We should probably also consider the * interrupt level and stick IPL_TTY with other * IPL_TTY, etc. */ for (p = &isp->is_handlers, tmp = 0; (q = *p) != NULL; p = &q->ih_next, tmp++) ; if ((bestirq == -1) || (count > tmp)) { bestirq = i; count = tmp; } break; case IST_PULSE: /* this just isn't shareable */ continue; } } mutex_exit(&cpu_lock); if (bestirq == -1) return 1; *irq = bestirq; return 0; } const struct evcnt * isa_intr_evcnt(isa_chipset_tag_t ic, int irq) { /* XXX for now, no evcnt parent reported */ return NULL; } void * isa_intr_establish(isa_chipset_tag_t ic, int irq, int type, int level, int (*ih_fun)(void *), void *ih_arg) { return isa_intr_establish_xname(ic, irq, type, level, ih_fun, ih_arg, "unknown"); } void * isa_intr_establish_xname(isa_chipset_tag_t ic, int irq, int type, int level, int (*ih_fun)(void *), void *ih_arg, const char *xname) { struct pic *pic; int pin; #if NIOAPIC > 0 intr_handle_t mpih = 0; struct ioapic_softc *ioapic = NULL; #endif pin = irq; pic = &i8259_pic; #if NIOAPIC > 0 if (mp_busses != NULL) { if (intr_find_mpmapping(mp_isa_bus, irq, &mpih) == 0 || intr_find_mpmapping(mp_eisa_bus, irq, &mpih) == 0) { if (!APIC_IRQ_ISLEGACY(mpih)) { pin = APIC_IRQ_PIN(mpih); ioapic = ioapic_find(APIC_IRQ_APIC(mpih)); if (ioapic == NULL) { printf("isa_intr_establish: " "unknown apic %d\n", APIC_IRQ_APIC(mpih)); return NULL; } pic = &ioapic->sc_pic; } } else printf("isa_intr_establish: no MP mapping found\n"); } #endif return intr_establish_xname(irq, pic, pin, type, level, ih_fun, ih_arg, false, xname); } /* Deregister an interrupt handler. */ void isa_intr_disestablish(isa_chipset_tag_t ic, void *arg) { #if !defined(XENPV) struct intrhand *ih = arg; if (!LEGAL_IRQ(ih->ih_pin)) panic("intr_disestablish: bogus irq"); intr_disestablish(ih); #endif } void isa_attach_hook(device_t parent, device_t self, struct isabus_attach_args *iba) { extern struct x86_isa_chipset x86_isa_chipset; extern int isa_has_been_seen; /* * Notify others that might need to know that the ISA bus * has now been attached. */ if (isa_has_been_seen) panic("isaattach: ISA bus already seen!"); isa_has_been_seen = 1; /* * Since we can only have one ISA bus, we just use a single * statically allocated ISA chipset structure. Pass it up * now. */ iba->iba_ic = &x86_isa_chipset; } void isa_detach_hook(isa_chipset_tag_t ic, device_t self) { extern int isa_has_been_seen; isa_has_been_seen = 0; } int isa_mem_alloc(bus_space_tag_t t, bus_size_t size, bus_size_t align, bus_addr_t boundary, int flags, bus_addr_t *addrp, bus_space_handle_t *bshp) { /* Allocate physical address space in the ISA hole. */ return bus_space_alloc(t, IOM_BEGIN, IOM_END - 1, size, align, boundary, flags, addrp, bshp); } void isa_mem_free(bus_space_tag_t t, bus_space_handle_t bsh, bus_size_t size) { bus_space_free(t, bsh, size); } /* * ISA only has 24-bits of address space. This means * we can't DMA to pages over 16M. In order to DMA to * arbitrary buffers, we use "bounce buffers" - pages * in memory below the 16M boundary. On DMA reads, * DMA happens to the bounce buffers, and is copied into * the caller's buffer. On writes, data is copied into * the bounce buffer, and the DMA happens from those * pages. To software using the DMA mapping interface, * this looks simply like a data cache. * * If we have more than 16M of RAM in the system, we may * need bounce buffers. We check and remember that here. * * There are exceptions, however. VLB devices can do * 32-bit DMA, and indicate that here. * * ...or, there is an opposite case. The most segments * a transfer will require is (maxxfer / PAGE_SIZE) + 1. If * the caller can't handle that many segments (e.g. the * ISA DMA controller), we may have to bounce it as well. */ static int _isa_dma_may_bounce(bus_dma_tag_t t, bus_dmamap_t map, int flags, int *cookieflagsp) { if ((flags & ISABUS_DMA_32BIT) != 0) map->_dm_bounce_thresh = 0; if (((map->_dm_size / PAGE_SIZE) + 1) > map->_dm_segcnt) *cookieflagsp |= X86_DMA_MIGHT_NEED_BOUNCE; return 0; } device_t device_isa_register(device_t dev, void *aux) { /* * Handle network interfaces here, the attachment information is * not available driver-independently later. * * For disks, there is nothing useful available at attach time. */ if (device_class(dev) == DV_IFNET) { struct btinfo_netif *bin = lookup_bootinfo(BTINFO_NETIF); if (bin == NULL) return NULL; /* * We don't check the driver name against the device name * passed by the boot ROM. The ROM should stay usable if * the driver becomes obsolete. The physical attachment * information (checked below) must be sufficient to * identify the device. */ if (bin->bus == BI_BUS_ISA && device_is_a(device_parent(dev), "isa")) { struct isa_attach_args *iaa = aux; /* Compare IO base address */ /* XXXJRT What about multiple IO addrs? */ if (iaa->ia_nio > 0 && bin->addr.iobase == iaa->ia_io[0].ir_addr) return dev; } } if (vm_guest == VM_GUEST_XENPVH) prop_dictionary_set_bool(device_properties(dev), "no-legacy-devices", true); #if NACPICA > 0 #if notyet /* * The following code block is technically correct, but unfortunately * it breaks things like being able to use lm(4) on platforms that * have no other means of exposing temperature, fan, and voltage * sensors. */ if (device_is_a(dev, "isa") && acpi_active) { /* * For FACP >= 2, the LEGACY_DEVICES flag indicates that * the motherboard supports user-visible devices on the LPC * or ISA bus. If clear, assume that no such devices are * present and we can enumerate everything we need using * ACPI tables. */ if (AcpiGbl_FADT.Header.Revision >= 2 && !(AcpiGbl_FADT.BootFlags & ACPI_FADT_LEGACY_DEVICES)) { prop_dictionary_set_bool(device_properties(dev), "no-legacy-devices", true); } } #endif if (vm_guest == VM_GUEST_VMWARE && device_is_a(dev, "isa") && acpi_active) { prop_dictionary_set_bool(device_properties(dev), "no-legacy-devices", true); } #endif /* NACPICA > 0 */ return NULL; } |
| 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 | /* $NetBSD: bufq_impl.h,v 1.10 2016/11/16 00:46:46 pgoyette Exp $ */ /* NetBSD: bufq.h,v 1.3 2005/03/31 11:28:53 yamt Exp */ /* NetBSD: buf.h,v 1.75 2004/09/18 16:40:11 yamt Exp */ /*- * Copyright (c) 1999, 2000 The NetBSD Foundation, Inc. * All rights reserved. * * This code is derived from software contributed to The NetBSD Foundation * by Jason R. Thorpe of the Numerical Aerospace Simulation Facility, * NASA Ames Research Center. * * Redistribution and use in source and binary forms, with or without * modification, are permitted provided that the following conditions * are met: * 1. Redistributions of source code must retain the above copyright * notice, this list of conditions and the following disclaimer. * 2. Redistributions in binary form must reproduce the above copyright * notice, this list of conditions and the following disclaimer in the * documentation and/or other materials provided with the distribution. * * THIS SOFTWARE IS PROVIDED BY THE NETBSD FOUNDATION, INC. AND CONTRIBUTORS * ``AS IS'' AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED * TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR * PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE FOUNDATION OR CONTRIBUTORS * BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR * CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF * SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS * INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN * CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) * ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE * POSSIBILITY OF SUCH DAMAGE. */ /* * Copyright (c) 1982, 1986, 1989, 1993 * The Regents of the University of California. All rights reserved. * (c) UNIX System Laboratories, Inc. * All or some portions of this file are derived from material licensed * to the University of California by American Telephone and Telegraph * Co. or Unix System Laboratories, Inc. and are reproduced herein with * the permission of UNIX System Laboratories, Inc. * * Redistribution and use in source and binary forms, with or without * modification, are permitted provided that the following conditions * are met: * 1. Redistributions of source code must retain the above copyright * notice, this list of conditions and the following disclaimer. * 2. Redistributions in binary form must reproduce the above copyright * notice, this list of conditions and the following disclaimer in the * documentation and/or other materials provided with the distribution. * 3. Neither the name of the University nor the names of its contributors * may be used to endorse or promote products derived from this software * without specific prior written permission. * * THIS SOFTWARE IS PROVIDED BY THE REGENTS AND CONTRIBUTORS ``AS IS'' AND * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE * ARE DISCLAIMED. IN NO EVENT SHALL THE REGENTS OR CONTRIBUTORS BE LIABLE * FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL * DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS * OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) * HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT * LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY * OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF * SUCH DAMAGE. * * @(#)buf.h 8.9 (Berkeley) 3/30/95 */ #if !defined(_KERNEL) #error not supposed to be exposed to userland. #endif struct bufq_strat; /* * Device driver buffer queue. */ struct bufq_state { void (*bq_put)(struct bufq_state *, struct buf *); struct buf *(*bq_get)(struct bufq_state *, int); struct buf *(*bq_cancel)(struct bufq_state *, struct buf *); void (*bq_fini)(struct bufq_state *); void *bq_private; int bq_flags; /* Flags from bufq_alloc() */ struct bufq_strat *bq_strat; }; static __inline void *bufq_private(const struct bufq_state *) __unused; static __inline bool buf_inorder(const struct buf *, const struct buf *, int) __unused; #include <sys/null.h> /* for NULL */ static __inline void * bufq_private(const struct bufq_state *bufq) { return bufq->bq_private; } /* * Check if two buf's are in ascending order. * * this function consider a NULL buf is after any non-NULL buf. * * this function returns false if two are "same". */ static __inline bool buf_inorder(const struct buf *bp, const struct buf *bq, int sortby) { KASSERT(bp != NULL || bq != NULL); if (bp == NULL || bq == NULL) return (bq == NULL); if (sortby == BUFQ_SORT_CYLINDER) { if (bp->b_cylinder != bq->b_cylinder) return bp->b_cylinder < bq->b_cylinder; else return bp->b_rawblkno < bq->b_rawblkno; } else return bp->b_rawblkno < bq->b_rawblkno; } struct bufq_strat { const char *bs_name; void (*bs_initfn)(struct bufq_state *); int bs_prio; int bs_refcnt; SLIST_ENTRY(bufq_strat) bs_next; }; #define BUFQ_DEFINE(name, prio, initfn) \ static struct bufq_strat bufq_strat_##name = { \ .bs_name = #name, \ .bs_prio = prio, \ .bs_initfn = initfn, \ .bs_refcnt = 0 \ }; int bufq_register(struct bufq_strat *); int bufq_unregister(struct bufq_strat *); |
| 7 3 2 2 6 5 3 11 11 5 6 7 2 5 6 2 1 4 12 9 1 11 2 2 7 8 4 5 1 23 23 9 15 9 1 12 18 8 3 1 5 1 5 4 4 4 2 9 15 3 1 6 5 3 2 3 3 10 3 7 7 2 1 6 1 8 3 6 1 6 2 5 1 4 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 197 198 199 200 201 202 203 204 205 206 207 208 209 210 211 212 213 214 215 216 217 218 219 220 221 222 223 224 225 226 227 228 229 230 231 232 233 234 235 236 237 238 239 240 241 242 243 244 245 246 247 248 249 250 251 252 253 254 255 256 257 258 259 260 261 262 263 264 265 266 267 268 269 270 271 272 273 274 275 276 277 278 279 280 281 282 283 284 285 286 287 288 289 290 291 292 293 294 295 296 297 298 299 300 301 302 303 304 305 306 307 308 309 310 311 312 313 314 315 316 317 318 319 320 321 322 323 324 325 326 327 328 329 330 331 332 333 334 335 336 337 338 339 340 341 342 343 344 345 346 347 348 349 350 351 352 353 354 355 356 357 358 359 360 361 362 363 364 365 366 367 368 369 370 371 372 373 374 375 376 377 378 379 380 381 382 383 384 385 386 387 388 389 390 391 392 393 394 395 396 397 398 399 400 401 402 403 404 405 406 407 408 409 410 411 412 413 414 415 416 417 418 419 420 421 422 423 424 425 426 427 428 429 430 431 432 433 434 435 436 437 438 439 440 441 442 443 444 445 446 447 448 449 450 451 452 453 454 455 456 457 458 459 460 461 462 463 464 465 466 467 468 469 470 471 472 473 474 475 476 477 478 479 480 481 482 483 484 485 486 487 488 489 490 491 492 493 494 495 496 497 498 499 500 501 502 503 504 505 506 507 508 509 510 511 512 513 514 515 516 517 518 519 520 521 522 523 524 525 526 527 528 529 530 531 532 533 534 535 536 537 538 539 540 541 542 543 544 545 546 547 548 549 550 551 552 553 554 555 556 557 558 559 560 561 562 563 564 565 566 567 568 569 570 571 572 573 574 575 576 577 578 579 580 581 582 583 584 585 586 587 588 589 590 591 592 593 594 595 596 597 598 599 600 601 602 603 604 605 606 607 608 609 610 611 612 613 614 615 616 617 618 619 620 621 622 623 624 625 626 627 628 629 630 631 632 633 634 635 636 637 638 639 640 641 642 643 644 645 646 647 648 649 650 651 652 653 654 655 656 657 658 659 660 661 662 663 664 665 666 667 668 669 670 671 672 673 674 675 676 677 678 679 680 681 682 683 684 685 686 687 688 689 690 691 692 693 694 695 696 697 698 699 700 701 702 703 704 705 706 707 708 709 710 711 712 713 714 715 716 717 718 719 720 721 722 723 724 725 726 727 728 729 730 731 732 733 734 735 736 737 738 739 740 741 742 743 744 745 746 747 748 749 750 751 752 753 754 755 756 757 758 | /* $NetBSD: sys_lwp.c,v 1.83 2022/06/29 22:27:01 riastradh Exp $ */ /*- * Copyright (c) 2001, 2006, 2007, 2008, 2019, 2020 The NetBSD Foundation, Inc. * All rights reserved. * * This code is derived from software contributed to The NetBSD Foundation * by Nathan J. Williams, and Andrew Doran. * * Redistribution and use in source and binary forms, with or without * modification, are permitted provided that the following conditions * are met: * 1. Redistributions of source code must retain the above copyright * notice, this list of conditions and the following disclaimer. * 2. Redistributions in binary form must reproduce the above copyright * notice, this list of conditions and the following disclaimer in the * documentation and/or other materials provided with the distribution. * * THIS SOFTWARE IS PROVIDED BY THE NETBSD FOUNDATION, INC. AND CONTRIBUTORS * ``AS IS'' AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED * TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR * PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE FOUNDATION OR CONTRIBUTORS * BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR * CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF * SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS * INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN * CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) * ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE * POSSIBILITY OF SUCH DAMAGE. */ /* * Lightweight process (LWP) system calls. See kern_lwp.c for a description * of LWPs. */ #include <sys/cdefs.h> __KERNEL_RCSID(0, "$NetBSD: sys_lwp.c,v 1.83 2022/06/29 22:27:01 riastradh Exp $"); #include <sys/param.h> #include <sys/systm.h> #include <sys/pool.h> #include <sys/proc.h> #include <sys/types.h> #include <sys/syscallargs.h> #include <sys/kauth.h> #include <sys/kmem.h> #include <sys/ptrace.h> #include <sys/sleepq.h> #include <sys/lwpctl.h> #include <sys/cpu.h> #include <sys/pserialize.h> #include <uvm/uvm_extern.h> #define LWP_UNPARK_MAX 1024 static const stack_t lwp_ss_init = SS_INIT; syncobj_t lwp_park_syncobj = { .sobj_flag = SOBJ_SLEEPQ_NULL, .sobj_unsleep = sleepq_unsleep, .sobj_changepri = sleepq_changepri, .sobj_lendpri = sleepq_lendpri, .sobj_owner = syncobj_noowner, }; static void mi_startlwp(void *arg) { struct lwp *l = curlwp; struct proc *p = l->l_proc; (p->p_emul->e_startlwp)(arg); /* If the process is traced, report lwp creation to a debugger */ if ((p->p_slflag & (PSL_TRACED|PSL_TRACELWP_CREATE)) == (PSL_TRACED|PSL_TRACELWP_CREATE)) { /* Paranoid check */ mutex_enter(&proc_lock); if ((p->p_slflag & (PSL_TRACED|PSL_TRACELWP_CREATE)) != (PSL_TRACED|PSL_TRACELWP_CREATE)) { mutex_exit(&proc_lock); return; } mutex_enter(p->p_lock); eventswitch(TRAP_LWP, PTRACE_LWP_CREATE, l->l_lid); } } int do_lwp_create(lwp_t *l, void *arg, u_long flags, lwp_t **l2, const sigset_t *sigmask, const stack_t *sigstk) { struct proc *p = l->l_proc; vaddr_t uaddr; int error; /* XXX check against resource limits */ uaddr = uvm_uarea_alloc(); if (__predict_false(uaddr == 0)) return ENOMEM; error = lwp_create(l, p, uaddr, flags & LWP_DETACHED, NULL, 0, mi_startlwp, arg, l2, l->l_class, sigmask, &lwp_ss_init); if (__predict_false(error)) { uvm_uarea_free(uaddr); return error; } return 0; } int sys__lwp_create(struct lwp *l, const struct sys__lwp_create_args *uap, register_t *retval) { /* { syscallarg(const ucontext_t *) ucp; syscallarg(u_long) flags; syscallarg(lwpid_t *) new_lwp; } */ struct proc *p = l->l_proc; ucontext_t *newuc; lwp_t *l2; int error; newuc = kmem_alloc(sizeof(ucontext_t), KM_SLEEP); error = copyin(SCARG(uap, ucp), newuc, p->p_emul->e_ucsize); if (error) goto fail; /* validate the ucontext */ if ((newuc->uc_flags & _UC_CPU) == 0) { error = EINVAL; goto fail; } error = cpu_mcontext_validate(l, &newuc->uc_mcontext); if (error) goto fail; const sigset_t *sigmask = newuc->uc_flags & _UC_SIGMASK ? &newuc->uc_sigmask : &l->l_sigmask; error = do_lwp_create(l, newuc, SCARG(uap, flags), &l2, sigmask, &SS_INIT); if (error) goto fail; error = copyout(&l2->l_lid, SCARG(uap, new_lwp), sizeof(l2->l_lid)); if (error == 0) { lwp_start(l2, SCARG(uap, flags)); return 0; } lwp_exit(l2); fail: kmem_free(newuc, sizeof(ucontext_t)); return error; } int sys__lwp_exit(struct lwp *l, const void *v, register_t *retval) { lwp_exit(l); return 0; } int sys__lwp_self(struct lwp *l, const void *v, register_t *retval) { *retval = l->l_lid; return 0; } int sys__lwp_getprivate(struct lwp *l, const void *v, register_t *retval) { *retval = (uintptr_t)l->l_private; return 0; } int sys__lwp_setprivate(struct lwp *l, const struct sys__lwp_setprivate_args *uap, register_t *retval) { /* { syscallarg(void *) ptr; } */ return lwp_setprivate(l, SCARG(uap, ptr)); } int sys__lwp_suspend(struct lwp *l, const struct sys__lwp_suspend_args *uap, register_t *retval) { /* { syscallarg(lwpid_t) target; } */ struct proc *p = l->l_proc; struct lwp *t; int error; mutex_enter(p->p_lock); if ((t = lwp_find(p, SCARG(uap, target))) == NULL) { mutex_exit(p->p_lock); return ESRCH; } /* * Check for deadlock, which is only possible when we're suspending * ourself. XXX There is a short race here, as p_nrlwps is only * incremented when an LWP suspends itself on the kernel/user * boundary. It's still possible to kill -9 the process so we * don't bother checking further. */ lwp_lock(t); if ((t == l && p->p_nrlwps == 1) || (l->l_flag & (LW_WCORE | LW_WEXIT)) != 0) { lwp_unlock(t); mutex_exit(p->p_lock); return EDEADLK; } /* * Suspend the LWP. XXX If it's on a different CPU, we should wait * for it to be preempted, where it will put itself to sleep. * * Suspension of the current LWP will happen on return to userspace. */ error = lwp_suspend(l, t); if (error) { mutex_exit(p->p_lock); return error; } /* * Wait for: * o process exiting * o target LWP suspended * o target LWP not suspended and L_WSUSPEND clear * o target LWP exited */ for (;;) { error = cv_wait_sig(&p->p_lwpcv, p->p_lock); if (error) { error = ERESTART; break; } if (lwp_find(p, SCARG(uap, target)) == NULL) { error = ESRCH; break; } if ((l->l_flag | t->l_flag) & (LW_WCORE | LW_WEXIT)) { error = ERESTART; break; } if (t->l_stat == LSSUSPENDED || (t->l_flag & LW_WSUSPEND) == 0) break; } mutex_exit(p->p_lock); return error; } int sys__lwp_continue(struct lwp *l, const struct sys__lwp_continue_args *uap, register_t *retval) { /* { syscallarg(lwpid_t) target; } */ int error; struct proc *p = l->l_proc; struct lwp *t; error = 0; mutex_enter(p->p_lock); if ((t = lwp_find(p, SCARG(uap, target))) == NULL) { mutex_exit(p->p_lock); return ESRCH; } lwp_lock(t); lwp_continue(t); mutex_exit(p->p_lock); return error; } int sys__lwp_wakeup(struct lwp *l, const struct sys__lwp_wakeup_args *uap, register_t *retval) { /* { syscallarg(lwpid_t) target; } */ struct lwp *t; struct proc *p; int error; p = l->l_proc; mutex_enter(p->p_lock); if ((t = lwp_find(p, SCARG(uap, target))) == NULL) { mutex_exit(p->p_lock); return ESRCH; } lwp_lock(t); t->l_flag |= (LW_CANCELLED | LW_UNPARKED); if (t->l_stat != LSSLEEP) { lwp_unlock(t); error = ENODEV; } else if ((t->l_flag & LW_SINTR) == 0) { lwp_unlock(t); error = EBUSY; } else { /* Wake it up. lwp_unsleep() will release the LWP lock. */ lwp_unsleep(t, true); error = 0; } mutex_exit(p->p_lock); return error; } int sys__lwp_wait(struct lwp *l, const struct sys__lwp_wait_args *uap, register_t *retval) { /* { syscallarg(lwpid_t) wait_for; syscallarg(lwpid_t *) departed; } */ struct proc *p = l->l_proc; int error; lwpid_t dep; mutex_enter(p->p_lock); error = lwp_wait(l, SCARG(uap, wait_for), &dep, false); mutex_exit(p->p_lock); if (!error && SCARG(uap, departed)) { error = copyout(&dep, SCARG(uap, departed), sizeof(dep)); } return error; } int sys__lwp_kill(struct lwp *l, const struct sys__lwp_kill_args *uap, register_t *retval) { /* { syscallarg(lwpid_t) target; syscallarg(int) signo; } */ struct proc *p = l->l_proc; struct lwp *t; ksiginfo_t ksi; int signo = SCARG(uap, signo); int error = 0; if ((u_int)signo >= NSIG) return EINVAL; KSI_INIT(&ksi); ksi.ksi_signo = signo; ksi.ksi_code = SI_LWP; ksi.ksi_pid = p->p_pid; ksi.ksi_uid = kauth_cred_geteuid(l->l_cred); ksi.ksi_lid = SCARG(uap, target); mutex_enter(&proc_lock); mutex_enter(p->p_lock); if ((t = lwp_find(p, ksi.ksi_lid)) == NULL) error = ESRCH; else if (signo != 0) kpsignal2(p, &ksi); mutex_exit(p->p_lock); mutex_exit(&proc_lock); return error; } int sys__lwp_detach(struct lwp *l, const struct sys__lwp_detach_args *uap, register_t *retval) { /* { syscallarg(lwpid_t) target; } */ struct proc *p; struct lwp *t; lwpid_t target; int error; target = SCARG(uap, target); p = l->l_proc; mutex_enter(p->p_lock); if (l->l_lid == target) t = l; else { /* * We can't use lwp_find() here because the target might * be a zombie. */ t = proc_find_lwp(p, target); KASSERT(t == NULL || t->l_lid == target); } /* * If the LWP is already detached, there's nothing to do. * If it's a zombie, we need to clean up after it. LSZOMB * is visible with the proc mutex held. * * After we have detached or released the LWP, kick any * other LWPs that may be sitting in _lwp_wait(), waiting * for the target LWP to exit. */ if (t != NULL && t->l_stat != LSIDL) { if ((t->l_prflag & LPR_DETACHED) == 0) { p->p_ndlwps++; t->l_prflag |= LPR_DETACHED; if (t->l_stat == LSZOMB) { /* Releases proc mutex. */ lwp_free(t, false, false); return 0; } error = 0; /* * Have any LWPs sleeping in lwp_wait() recheck * for deadlock. */ cv_broadcast(&p->p_lwpcv); } else error = EINVAL; } else error = ESRCH; mutex_exit(p->p_lock); return error; } int lwp_unpark(const lwpid_t *tp, const u_int ntargets) { u_int target; int error, s; proc_t *p; lwp_t *t; p = curproc; error = 0; s = pserialize_read_enter(); for (target = 0; target < ntargets; target++) { t = proc_find_lwp_unlocked(p, tp[target]); if (__predict_false(t == NULL)) { error = ESRCH; continue; } KASSERT(lwp_locked(t, NULL)); if (__predict_true(t->l_syncobj == &lwp_park_syncobj)) { /* * As expected it's parked, so wake it up. * lwp_unsleep() will release the LWP lock. */ lwp_unsleep(t, true); } else if (__predict_false(t->l_stat == LSZOMB)) { lwp_unlock(t); error = ESRCH; } else { /* * It hasn't parked yet because the wakeup side won * the race, or something else has happened to make * the thread not park. Why doesn't really matter. * Set the operation pending, so that the next call * to _lwp_park() in the LWP returns early. If it * turns out to be a spurious wakeup, no harm done. */ t->l_flag |= LW_UNPARKED; lwp_unlock(t); } } pserialize_read_exit(s); return error; } int lwp_park(clockid_t clock_id, int flags, struct timespec *ts) { int timo, error; struct timespec start; lwp_t *l; bool timeremain = !(flags & TIMER_ABSTIME) && ts; if (ts != NULL) { if ((error = ts2timo(clock_id, flags, ts, &timo, timeremain ? &start : NULL)) != 0) return error; KASSERT(timo != 0); } else { timo = 0; } /* * Before going the full route and blocking, check to see if an * unpark op is pending. */ l = curlwp; lwp_lock(l); if ((l->l_flag & (LW_CANCELLED | LW_UNPARKED)) != 0) { l->l_flag &= ~(LW_CANCELLED | LW_UNPARKED); lwp_unlock(l); return EALREADY; } l->l_biglocks = 0; sleepq_enqueue(NULL, l, "parked", &lwp_park_syncobj, true); error = sleepq_block(timo, true, &lwp_park_syncobj); switch (error) { case EWOULDBLOCK: error = ETIMEDOUT; if (timeremain) memset(ts, 0, sizeof(*ts)); break; case ERESTART: error = EINTR; /*FALLTHROUGH*/ default: if (timeremain) clock_timeleft(clock_id, ts, &start); break; } return error; } /* * 'park' an LWP waiting on a user-level synchronisation object. The LWP * will remain parked until another LWP in the same process calls in and * requests that it be unparked. */ int sys____lwp_park60(struct lwp *l, const struct sys____lwp_park60_args *uap, register_t *retval) { /* { syscallarg(clockid_t) clock_id; syscallarg(int) flags; syscallarg(struct timespec *) ts; syscallarg(lwpid_t) unpark; syscallarg(const void *) hint; syscallarg(const void *) unparkhint; } */ struct timespec ts, *tsp; int error; if (SCARG(uap, ts) == NULL) tsp = NULL; else { error = copyin(SCARG(uap, ts), &ts, sizeof(ts)); if (error != 0) return error; tsp = &ts; } if (SCARG(uap, unpark) != 0) { error = lwp_unpark(&SCARG(uap, unpark), 1); if (error != 0) return error; } error = lwp_park(SCARG(uap, clock_id), SCARG(uap, flags), tsp); if (SCARG(uap, ts) != NULL && (SCARG(uap, flags) & TIMER_ABSTIME) == 0) (void)copyout(tsp, SCARG(uap, ts), sizeof(*tsp)); return error; } int sys__lwp_unpark(struct lwp *l, const struct sys__lwp_unpark_args *uap, register_t *retval) { /* { syscallarg(lwpid_t) target; syscallarg(const void *) hint; } */ return lwp_unpark(&SCARG(uap, target), 1); } int sys__lwp_unpark_all(struct lwp *l, const struct sys__lwp_unpark_all_args *uap, register_t *retval) { /* { syscallarg(const lwpid_t *) targets; syscallarg(size_t) ntargets; syscallarg(const void *) hint; } */ lwpid_t targets[32], *tp; int error; u_int ntargets; size_t sz; ntargets = SCARG(uap, ntargets); if (SCARG(uap, targets) == NULL) { /* * Let the caller know how much we are willing to do, and * let it unpark the LWPs in blocks. */ *retval = LWP_UNPARK_MAX; return 0; } if (ntargets > LWP_UNPARK_MAX || ntargets == 0) return EINVAL; /* * Copy in the target array. If it's a small number of LWPs, then * place the numbers on the stack. */ sz = sizeof(lwpid_t) * ntargets; if (sz <= sizeof(targets)) tp = targets; else tp = kmem_alloc(sz, KM_SLEEP); error = copyin(SCARG(uap, targets), tp, sz); if (error != 0) { if (tp != targets) { kmem_free(tp, sz); } return error; } error = lwp_unpark(tp, ntargets); if (tp != targets) kmem_free(tp, sz); return error; } int sys__lwp_setname(struct lwp *l, const struct sys__lwp_setname_args *uap, register_t *retval) { /* { syscallarg(lwpid_t) target; syscallarg(const char *) name; } */ char *name, *oname; lwpid_t target; proc_t *p; lwp_t *t; int error; if ((target = SCARG(uap, target)) == 0) target = l->l_lid; name = kmem_alloc(MAXCOMLEN, KM_SLEEP); error = copyinstr(SCARG(uap, name), name, MAXCOMLEN, NULL); switch (error) { case ENAMETOOLONG: case 0: name[MAXCOMLEN - 1] = '\0'; break; default: kmem_free(name, MAXCOMLEN); return error; } p = curproc; mutex_enter(p->p_lock); if ((t = lwp_find(p, target)) == NULL) { mutex_exit(p->p_lock); kmem_free(name, MAXCOMLEN); return ESRCH; } lwp_lock(t); oname = t->l_name; t->l_name = name; lwp_unlock(t); mutex_exit(p->p_lock); if (oname != NULL) kmem_free(oname, MAXCOMLEN); return 0; } int sys__lwp_getname(struct lwp *l, const struct sys__lwp_getname_args *uap, register_t *retval) { /* { syscallarg(lwpid_t) target; syscallarg(char *) name; syscallarg(size_t) len; } */ char name[MAXCOMLEN]; lwpid_t target; size_t len; proc_t *p; lwp_t *t; if ((target = SCARG(uap, target)) == 0) target = l->l_lid; p = curproc; mutex_enter(p->p_lock); if ((t = lwp_find(p, target)) == NULL) { mutex_exit(p->p_lock); return ESRCH; } lwp_lock(t); if (t->l_name == NULL) name[0] = '\0'; else strlcpy(name, t->l_name, sizeof(name)); lwp_unlock(t); mutex_exit(p->p_lock); len = uimin(SCARG(uap, len), sizeof(name)); return copyoutstr(name, SCARG(uap, name), len, NULL); } int sys__lwp_ctl(struct lwp *l, const struct sys__lwp_ctl_args *uap, register_t *retval) { /* { syscallarg(int) features; syscallarg(struct lwpctl **) address; } */ int error, features; vaddr_t vaddr; features = SCARG(uap, features); features &= ~(LWPCTL_FEATURE_CURCPU | LWPCTL_FEATURE_PCTR); if (features != 0) return ENODEV; if ((error = lwp_ctl_alloc(&vaddr)) != 0) return error; return copyout(&vaddr, SCARG(uap, address), sizeof(void *)); } |
| 2 2 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 | /* $NetBSD: tty_tty.c,v 1.41 2020/05/23 23:42:43 ad Exp $ */ /*- * Copyright (c) 1982, 1986, 1991, 1993, 1995 * The Regents of the University of California. All rights reserved. * * Redistribution and use in source and binary forms, with or without * modification, are permitted provided that the following conditions * are met: * 1. Redistributions of source code must retain the above copyright * notice, this list of conditions and the following disclaimer. * 2. Redistributions in binary form must reproduce the above copyright * notice, this list of conditions and the following disclaimer in the * documentation and/or other materials provided with the distribution. * 3. Neither the name of the University nor the names of its contributors * may be used to endorse or promote products derived from this software * without specific prior written permission. * * THIS SOFTWARE IS PROVIDED BY THE REGENTS AND CONTRIBUTORS ``AS IS'' AND * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE * ARE DISCLAIMED. IN NO EVENT SHALL THE REGENTS OR CONTRIBUTORS BE LIABLE * FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL * DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS * OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) * HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT * LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY * OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF * SUCH DAMAGE. * * @(#)tty_tty.c 8.2 (Berkeley) 9/23/93 */ /* * Indirect driver for controlling tty. */ #include <sys/cdefs.h> __KERNEL_RCSID(0, "$NetBSD: tty_tty.c,v 1.41 2020/05/23 23:42:43 ad Exp $"); #include <sys/param.h> #include <sys/systm.h> #include <sys/ioctl.h> #include <sys/proc.h> #include <sys/tty.h> #include <sys/vnode.h> #include <sys/file.h> #include <sys/conf.h> #include <sys/kauth.h> /* XXXSMP */ #define cttyvp(p) ((p)->p_lflag & PL_CONTROLT ? (p)->p_session->s_ttyvp : NULL) /*ARGSUSED*/ static int cttyopen(dev_t dev, int flag, int mode, struct lwp *l) { struct vnode *ttyvp = cttyvp(l->l_proc); int error; if (ttyvp == NULL) return (ENXIO); vn_lock(ttyvp, LK_EXCLUSIVE | LK_RETRY); #ifdef PARANOID /* * Since group is tty and mode is 620 on most terminal lines * and since sessions protect terminals from processes outside * your session, this check is probably no longer necessary. * Since it inhibits setuid root programs that later switch * to another user from accessing /dev/tty, we have decided * to delete this test. (mckusick 5/93) */ error = VOP_ACCESS(ttyvp, (flag&FREAD ? VREAD : 0) | (flag&FWRITE ? VWRITE : 0), l->l_cred, l); if (!error) #endif /* PARANOID */ error = VOP_OPEN(ttyvp, flag, NOCRED); VOP_UNLOCK(ttyvp); return (error); } /*ARGSUSED*/ static int cttyread(dev_t dev, struct uio *uio, int flag) { struct vnode *ttyvp = cttyvp(curproc); int error; if (ttyvp == NULL) return (EIO); vn_lock(ttyvp, LK_EXCLUSIVE | LK_RETRY); error = VOP_READ(ttyvp, uio, flag, NOCRED); VOP_UNLOCK(ttyvp); return (error); } /*ARGSUSED*/ static int cttywrite(dev_t dev, struct uio *uio, int flag) { struct vnode *ttyvp = cttyvp(curproc); int error; if (ttyvp == NULL) return (EIO); vn_lock(ttyvp, LK_EXCLUSIVE | LK_RETRY); error = VOP_WRITE(ttyvp, uio, flag, NOCRED); VOP_UNLOCK(ttyvp); return (error); } /*ARGSUSED*/ static int cttyioctl(dev_t dev, u_long cmd, void *addr, int flag, struct lwp *l) { struct vnode *ttyvp = cttyvp(l->l_proc); int rv; if (ttyvp == NULL) return (EIO); if (cmd == TIOCSCTTY) /* XXX */ return (EINVAL); if (cmd == TIOCNOTTY) { mutex_enter(&proc_lock); if (!SESS_LEADER(l->l_proc)) { l->l_proc->p_lflag &= ~PL_CONTROLT; rv = 0; } else rv = EINVAL; mutex_exit(&proc_lock); return (rv); } return (VOP_IOCTL(ttyvp, cmd, addr, flag, NOCRED)); } /*ARGSUSED*/ static int cttypoll(dev_t dev, int events, struct lwp *l) { struct vnode *ttyvp = cttyvp(l->l_proc); if (ttyvp == NULL) return (seltrue(dev, events, l)); return (VOP_POLL(ttyvp, events)); } static int cttykqfilter(dev_t dev, struct knote *kn) { /* This is called from filt_fileattach() by the attaching process. */ struct proc *p = curproc; struct vnode *ttyvp = cttyvp(p); if (ttyvp == NULL) return (1); return (VOP_KQFILTER(ttyvp, kn)); } const struct cdevsw ctty_cdevsw = { .d_open = cttyopen, .d_close = nullclose, .d_read = cttyread, .d_write = cttywrite, .d_ioctl = cttyioctl, .d_stop = nullstop, .d_tty = notty, .d_poll = cttypoll, .d_mmap = nommap, .d_kqfilter = cttykqfilter, .d_discard = nodiscard, .d_flag = D_TTY }; |
| 13 22 22 1 21 21 20 1 20 4 4 4 1 3 3 630 1 630 630 630 1 630 636 7 50 98 629 634 624 42 43 43 7 28 14 9 1 1 1 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 197 198 199 200 201 202 203 204 205 206 207 208 209 210 211 212 213 214 215 216 217 218 219 220 221 222 223 224 225 226 227 228 229 230 231 232 233 234 235 236 237 238 239 240 241 242 243 244 245 246 247 248 249 250 251 252 253 254 255 256 257 258 259 260 261 262 263 264 265 266 267 268 269 270 271 272 273 274 275 276 277 278 279 280 281 282 283 284 285 286 287 288 289 290 291 292 293 294 295 296 297 298 299 300 301 302 303 304 305 306 307 308 309 310 311 312 313 314 315 316 317 318 319 320 321 322 323 324 325 326 327 328 329 330 331 332 333 334 335 336 337 338 339 340 341 342 343 344 345 346 347 348 349 350 351 352 353 354 355 356 357 358 359 360 361 362 363 364 365 366 367 368 369 370 371 372 373 374 375 376 377 378 379 380 381 382 383 384 385 386 387 388 389 390 391 392 393 394 395 396 397 398 399 400 401 402 403 404 405 406 407 408 409 410 411 412 413 414 415 416 417 418 419 420 421 422 423 424 425 426 427 428 429 430 431 432 433 434 435 436 437 438 439 440 441 442 443 444 445 446 447 448 449 450 451 452 453 454 455 456 457 458 459 460 461 462 463 464 465 466 467 468 469 470 471 472 473 474 475 476 477 478 479 480 481 482 483 484 485 486 487 488 489 490 491 492 493 494 495 496 497 498 499 500 501 502 503 504 505 506 507 508 509 510 511 512 513 514 515 516 517 518 519 520 521 522 523 524 525 526 527 528 529 530 531 532 533 534 535 536 537 538 539 540 541 542 543 544 545 546 547 548 549 550 551 552 553 554 555 556 557 558 559 560 561 562 563 564 565 566 567 568 569 570 571 572 573 574 575 576 577 578 579 580 581 582 583 584 585 586 587 588 589 590 591 592 593 594 595 596 597 598 599 600 601 602 603 604 605 606 607 608 609 610 611 612 613 614 615 616 617 618 619 620 621 622 623 624 625 626 627 628 629 630 631 632 633 634 635 636 637 638 639 640 641 642 643 644 645 646 647 648 649 650 651 652 653 654 655 656 657 658 659 660 661 662 663 664 665 666 667 668 669 670 671 672 673 674 675 676 677 678 679 680 681 682 683 684 685 686 687 688 689 690 691 692 693 694 695 696 697 698 699 700 701 702 703 704 705 706 707 708 709 710 711 712 713 714 715 716 717 718 719 720 721 722 723 724 725 726 727 728 729 730 731 732 733 734 735 736 737 738 739 740 741 742 743 744 745 746 747 748 749 750 751 752 753 754 755 756 757 758 759 760 761 762 763 764 765 766 767 768 769 770 771 772 773 774 775 776 777 778 779 780 781 782 783 784 785 786 787 788 789 790 791 792 793 794 795 796 797 798 799 800 801 802 803 804 805 806 807 808 809 810 811 812 813 814 815 816 817 818 819 820 821 822 823 824 825 826 827 828 829 830 831 832 833 834 835 836 837 838 839 840 841 842 843 844 845 846 847 848 849 850 851 852 853 854 855 856 857 858 859 860 861 862 863 864 865 866 867 868 869 870 871 872 873 874 875 876 877 878 879 880 881 882 883 884 885 886 887 888 889 890 891 892 893 894 895 896 897 898 899 900 901 902 903 904 905 906 907 908 909 910 911 912 913 914 915 916 917 918 919 920 921 922 923 924 925 926 927 928 929 930 931 932 933 934 935 936 937 938 939 940 941 942 943 944 945 946 947 948 949 950 951 952 953 954 955 956 957 958 959 960 961 962 963 964 965 966 967 968 969 970 971 972 973 974 975 976 977 978 979 980 981 982 983 984 985 986 987 988 989 990 991 992 993 994 995 996 997 998 999 1000 1001 1002 1003 1004 1005 1006 1007 1008 1009 1010 1011 1012 1013 1014 1015 1016 1017 1018 1019 1020 1021 | /* $NetBSD: dksubr.c,v 1.113 2021/04/15 00:32:50 rin Exp $ */ /*- * Copyright (c) 1996, 1997, 1998, 1999, 2002, 2008 The NetBSD Foundation, Inc. * All rights reserved. * * This code is derived from software contributed to The NetBSD Foundation * by Jason R. Thorpe and Roland C. Dowdeswell. * * Redistribution and use in source and binary forms, with or without * modification, are permitted provided that the following conditions * are met: * 1. Redistributions of source code must retain the above copyright * notice, this list of conditions and the following disclaimer. * 2. Redistributions in binary form must reproduce the above copyright * notice, this list of conditions and the following disclaimer in the * documentation and/or other materials provided with the distribution. * * THIS SOFTWARE IS PROVIDED BY THE NETBSD FOUNDATION, INC. AND CONTRIBUTORS * ``AS IS'' AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED * TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR * PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE FOUNDATION OR CONTRIBUTORS * BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR * CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF * SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS * INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN * CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) * ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE * POSSIBILITY OF SUCH DAMAGE. */ #include <sys/cdefs.h> __KERNEL_RCSID(0, "$NetBSD: dksubr.c,v 1.113 2021/04/15 00:32:50 rin Exp $"); #include <sys/param.h> #include <sys/systm.h> #include <sys/stat.h> #include <sys/proc.h> #include <sys/ioctl.h> #include <sys/device.h> #include <sys/disk.h> #include <sys/disklabel.h> #include <sys/buf.h> #include <sys/bufq.h> #include <sys/vnode.h> #include <sys/fcntl.h> #include <sys/namei.h> #include <sys/module.h> #include <sys/syslog.h> #include <dev/dkvar.h> #include <miscfs/specfs/specdev.h> /* for v_rdev */ int dkdebug = 0; #ifdef DEBUG #define DKDB_FOLLOW 0x1 #define DKDB_INIT 0x2 #define DKDB_VNODE 0x4 #define DKDB_DUMP 0x8 #define IFDEBUG(x,y) if (dkdebug & (x)) y #define DPRINTF(x,y) IFDEBUG(x, printf y) #define DPRINTF_FOLLOW(y) DPRINTF(DKDB_FOLLOW, y) #else #define IFDEBUG(x,y) #define DPRINTF(x,y) #define DPRINTF_FOLLOW(y) #endif #define DKF_READYFORDUMP (DKF_INITED|DKF_TAKEDUMP) static int dk_subr_modcmd(modcmd_t, void *); #define DKLABELDEV(dev) \ (MAKEDISKDEV(major((dev)), DISKUNIT((dev)), RAW_PART)) static void dk_makedisklabel(struct dk_softc *); static int dk_translate(struct dk_softc *, struct buf *); static void dk_done1(struct dk_softc *, struct buf *, bool); void dk_init(struct dk_softc *dksc, device_t dev, int dtype) { memset(dksc, 0x0, sizeof(*dksc)); dksc->sc_dtype = dtype; dksc->sc_dev = dev; strlcpy(dksc->sc_xname, device_xname(dev), DK_XNAME_SIZE); dksc->sc_dkdev.dk_name = dksc->sc_xname; } void dk_attach(struct dk_softc *dksc) { KASSERT(dksc->sc_dev != NULL); mutex_init(&dksc->sc_iolock, MUTEX_DEFAULT, IPL_VM); dksc->sc_flags |= DKF_READYFORDUMP; #ifdef DIAGNOSTIC dksc->sc_flags |= DKF_WARNLABEL | DKF_LABELSANITY; #endif if ((dksc->sc_flags & DKF_NO_RND) == 0) { /* Attach the device into the rnd source list. */ rnd_attach_source(&dksc->sc_rnd_source, dksc->sc_xname, RND_TYPE_DISK, RND_FLAG_DEFAULT); } } void dk_detach(struct dk_softc *dksc) { if ((dksc->sc_flags & DKF_NO_RND) == 0) { /* Unhook the entropy source. */ rnd_detach_source(&dksc->sc_rnd_source); } dksc->sc_flags &= ~DKF_READYFORDUMP; mutex_destroy(&dksc->sc_iolock); } /* ARGSUSED */ int dk_open(struct dk_softc *dksc, dev_t dev, int flags, int fmt, struct lwp *l) { const struct dkdriver *dkd = dksc->sc_dkdev.dk_driver; struct disklabel *lp = dksc->sc_dkdev.dk_label; int part = DISKPART(dev); int pmask = 1 << part; int ret = 0; struct disk *dk = &dksc->sc_dkdev; DPRINTF_FOLLOW(("%s(%s, %p, 0x%"PRIx64", 0x%x)\n", __func__, dksc->sc_xname, dksc, dev, flags)); mutex_enter(&dk->dk_openlock); /* * If there are wedges, and this is not RAW_PART, then we * need to fail. */ if (dk->dk_nwedges != 0 && part != RAW_PART) { ret = EBUSY; goto done; } /* If no dkdriver attached, bail */ if (dkd == NULL) { ret = ENXIO; goto done; } /* * initialize driver for the first opener */ if (dk->dk_openmask == 0 && dkd->d_firstopen != NULL) { ret = (*dkd->d_firstopen)(dksc->sc_dev, dev, flags, fmt); if (ret) goto done; } /* * If we're init'ed and there are no other open partitions then * update the in-core disklabel. */ if ((dksc->sc_flags & DKF_INITED)) { if ((dksc->sc_flags & DKF_VLABEL) == 0) { dksc->sc_flags |= DKF_VLABEL; dk_getdisklabel(dksc, dev); } } /* Fail if we can't find the partition. */ if (part != RAW_PART && ((dksc->sc_flags & DKF_VLABEL) == 0 || part >= lp->d_npartitions || lp->d_partitions[part].p_fstype == FS_UNUSED)) { ret = ENXIO; goto done; } /* Mark our unit as open. */ switch (fmt) { case S_IFCHR: dk->dk_copenmask |= pmask; break; case S_IFBLK: dk->dk_bopenmask |= pmask; break; } dk->dk_openmask = dk->dk_copenmask | dk->dk_bopenmask; done: mutex_exit(&dk->dk_openlock); return ret; } /* ARGSUSED */ int dk_close(struct dk_softc *dksc, dev_t dev, int flags, int fmt, struct lwp *l) { const struct dkdriver *dkd = dksc->sc_dkdev.dk_driver; int part = DISKPART(dev); int pmask = 1 << part; struct disk *dk = &dksc->sc_dkdev; DPRINTF_FOLLOW(("%s(%s, %p, 0x%"PRIx64", 0x%x)\n", __func__, dksc->sc_xname, dksc, dev, flags)); mutex_enter(&dk->dk_openlock); switch (fmt) { case S_IFCHR: dk->dk_copenmask &= ~pmask; break; case S_IFBLK: dk->dk_bopenmask &= ~pmask; break; } dk->dk_openmask = dk->dk_copenmask | dk->dk_bopenmask; if (dk->dk_openmask == 0) { if (dkd->d_lastclose != NULL) (*dkd->d_lastclose)(dksc->sc_dev); if ((dksc->sc_flags & DKF_KLABEL) == 0) dksc->sc_flags &= ~DKF_VLABEL; } mutex_exit(&dk->dk_openlock); return 0; } static int dk_translate(struct dk_softc *dksc, struct buf *bp) { int part; int wlabel; daddr_t blkno; struct disklabel *lp; struct disk *dk; uint64_t numsecs; unsigned secsize; lp = dksc->sc_dkdev.dk_label; dk = &dksc->sc_dkdev; part = DISKPART(bp->b_dev); numsecs = dk->dk_geom.dg_secperunit; secsize = dk->dk_geom.dg_secsize; /* * The transfer must be a whole number of blocks and the offset must * not be negative. */ if ((bp->b_bcount % secsize) != 0 || bp->b_blkno < 0) { bp->b_error = EINVAL; goto done; } /* If there is nothing to do, then we are done */ if (bp->b_bcount == 0) goto done; wlabel = dksc->sc_flags & (DKF_WLABEL|DKF_LABELLING); if (part == RAW_PART) { uint64_t numblocks = btodb(numsecs * secsize); if (bounds_check_with_mediasize(bp, DEV_BSIZE, numblocks) <= 0) goto done; } else { if (bounds_check_with_label(&dksc->sc_dkdev, bp, wlabel) <= 0) goto done; } /* * Convert the block number to absolute and put it in terms * of the device's logical block size. */ if (secsize >= DEV_BSIZE) blkno = bp->b_blkno / (secsize / DEV_BSIZE); else blkno = bp->b_blkno * (DEV_BSIZE / secsize); if (part != RAW_PART) blkno += lp->d_partitions[DISKPART(bp->b_dev)].p_offset; bp->b_rawblkno = blkno; return -1; done: bp->b_resid = bp->b_bcount; return bp->b_error; } static int dk_strategy1(struct dk_softc *dksc, struct buf *bp) { int error; DPRINTF_FOLLOW(("%s(%s, %p, %p)\n", __func__, dksc->sc_xname, dksc, bp)); if (!(dksc->sc_flags & DKF_INITED)) { DPRINTF_FOLLOW(("%s: not inited\n", __func__)); bp->b_error = ENXIO; bp->b_resid = bp->b_bcount; biodone(bp); return 1; } error = dk_translate(dksc, bp); if (error >= 0) { biodone(bp); return 1; } return 0; } void dk_strategy(struct dk_softc *dksc, struct buf *bp) { int error; error = dk_strategy1(dksc, bp); if (error) return; /* * Queue buffer and start unit */ dk_start(dksc, bp); } int dk_strategy_defer(struct dk_softc *dksc, struct buf *bp) { int error; error = dk_strategy1(dksc, bp); if (error) return error; /* * Queue buffer only */ mutex_enter(&dksc->sc_iolock); disk_wait(&dksc->sc_dkdev); bufq_put(dksc->sc_bufq, bp); mutex_exit(&dksc->sc_iolock); return 0; } int dk_strategy_pending(struct dk_softc *dksc) { struct buf *bp; if (!(dksc->sc_flags & DKF_INITED)) { DPRINTF_FOLLOW(("%s: not inited\n", __func__)); return 0; } mutex_enter(&dksc->sc_iolock); bp = bufq_peek(dksc->sc_bufq); mutex_exit(&dksc->sc_iolock); return bp != NULL; } void dk_start(struct dk_softc *dksc, struct buf *bp) { const struct dkdriver *dkd = dksc->sc_dkdev.dk_driver; int error; if (!(dksc->sc_flags & DKF_INITED)) { DPRINTF_FOLLOW(("%s: not inited\n", __func__)); return; } mutex_enter(&dksc->sc_iolock); if (bp != NULL) { bp->b_ci = curcpu(); disk_wait(&dksc->sc_dkdev); bufq_put(dksc->sc_bufq, bp); } /* * If another thread is running the queue, increment * busy counter to 2 so that the queue is retried, * because the driver may now accept additional * requests. */ if (dksc->sc_busy < 2) dksc->sc_busy++; if (dksc->sc_busy > 1) goto done; /* * Peeking at the buffer queue and committing the operation * only after success isn't atomic. * * So when a diskstart fails, the buffer is saved * and tried again before the next buffer is fetched. * dk_drain() handles flushing of a saved buffer. * * This keeps order of I/O operations, unlike bufq_put. */ while (dksc->sc_busy > 0) { bp = dksc->sc_deferred; dksc->sc_deferred = NULL; if (bp == NULL) bp = bufq_get(dksc->sc_bufq); while (bp != NULL) { disk_busy(&dksc->sc_dkdev); mutex_exit(&dksc->sc_iolock); error = dkd->d_diskstart(dksc->sc_dev, bp); mutex_enter(&dksc->sc_iolock); if (error == EAGAIN || error == ENOMEM) { /* * Not a disk error. Retry later. */ KASSERT(dksc->sc_deferred == NULL); dksc->sc_deferred = bp; disk_unbusy(&dksc->sc_dkdev, 0, (bp->b_flags & B_READ)); disk_wait(&dksc->sc_dkdev); break; } if (error != 0) { bp->b_error = error; bp->b_resid = bp->b_bcount; dk_done1(dksc, bp, false); } bp = bufq_get(dksc->sc_bufq); } dksc->sc_busy--; } done: mutex_exit(&dksc->sc_iolock); } static void dk_done1(struct dk_softc *dksc, struct buf *bp, bool lock) { struct disk *dk = &dksc->sc_dkdev; if (bp->b_error != 0) { struct cfdriver *cd = device_cfdriver(dksc->sc_dev); diskerr(bp, cd->cd_name, "error", LOG_PRINTF, 0, dk->dk_label); printf("\n"); } if (lock) mutex_enter(&dksc->sc_iolock); disk_unbusy(dk, bp->b_bcount - bp->b_resid, (bp->b_flags & B_READ)); if ((dksc->sc_flags & DKF_NO_RND) == 0) rnd_add_uint32(&dksc->sc_rnd_source, bp->b_rawblkno); if (lock) mutex_exit(&dksc->sc_iolock); biodone(bp); } void dk_done(struct dk_softc *dksc, struct buf *bp) { dk_done1(dksc, bp, true); } void dk_drain(struct dk_softc *dksc) { struct buf *bp; mutex_enter(&dksc->sc_iolock); bp = dksc->sc_deferred; dksc->sc_deferred = NULL; if (bp != NULL) { bp->b_error = EIO; bp->b_resid = bp->b_bcount; biodone(bp); } bufq_drain(dksc->sc_bufq); mutex_exit(&dksc->sc_iolock); } int dk_discard(struct dk_softc *dksc, dev_t dev, off_t pos, off_t len) { const struct dkdriver *dkd = dksc->sc_dkdev.dk_driver; unsigned secsize = dksc->sc_dkdev.dk_geom.dg_secsize; struct buf tmp, *bp = &tmp; int maxsz; int error = 0; KASSERT(len >= 0); DPRINTF_FOLLOW(("%s(%s, %p, 0x"PRIx64", %jd, %jd)\n", __func__, dksc->sc_xname, dksc, (intmax_t)pos, (intmax_t)len)); if (!(dksc->sc_flags & DKF_INITED)) { DPRINTF_FOLLOW(("%s: not inited\n", __func__)); return ENXIO; } if (secsize == 0 || (pos % secsize) != 0 || (len % secsize) != 0) return EINVAL; /* largest value that b_bcount can store */ maxsz = rounddown(INT_MAX, secsize); while (len > 0) { /* enough data to please the bounds checking code */ bp->b_dev = dev; bp->b_blkno = (daddr_t)(pos / secsize); bp->b_bcount = uimin(len, maxsz); bp->b_flags = B_WRITE; error = dk_translate(dksc, bp); if (error >= 0) break; error = dkd->d_discard(dksc->sc_dev, (off_t)bp->b_rawblkno * secsize, (off_t)bp->b_bcount); if (error) break; pos += bp->b_bcount; len -= bp->b_bcount; } return error; } int dk_size(struct dk_softc *dksc, dev_t dev) { const struct dkdriver *dkd = dksc->sc_dkdev.dk_driver; struct disklabel *lp; int is_open; int part; int size; if ((dksc->sc_flags & DKF_INITED) == 0) return -1; part = DISKPART(dev); is_open = dksc->sc_dkdev.dk_openmask & (1 << part); if (!is_open && dkd->d_open(dev, 0, S_IFBLK, curlwp)) return -1; lp = dksc->sc_dkdev.dk_label; if (lp->d_partitions[part].p_fstype != FS_SWAP) size = -1; else size = lp->d_partitions[part].p_size * (lp->d_secsize / DEV_BSIZE); if (!is_open && dkd->d_close(dev, 0, S_IFBLK, curlwp)) return -1; return size; } int dk_ioctl(struct dk_softc *dksc, dev_t dev, u_long cmd, void *data, int flag, struct lwp *l) { const struct dkdriver *dkd = dksc->sc_dkdev.dk_driver; struct disklabel *lp; struct disk *dk = &dksc->sc_dkdev; #ifdef __HAVE_OLD_DISKLABEL struct disklabel newlabel; #endif int error; DPRINTF_FOLLOW(("%s(%s, %p, 0x%"PRIx64", 0x%lx)\n", __func__, dksc->sc_xname, dksc, dev, cmd)); /* ensure that the pseudo disk is open for writes for these commands */ switch (cmd) { case DIOCSDINFO: case DIOCWDINFO: #ifdef __HAVE_OLD_DISKLABEL case ODIOCSDINFO: case ODIOCWDINFO: #endif case DIOCKLABEL: case DIOCWLABEL: case DIOCAWEDGE: case DIOCDWEDGE: case DIOCSSTRATEGY: if ((flag & FWRITE) == 0) return EBADF; } /* ensure that the pseudo-disk is initialized for these */ switch (cmd) { case DIOCGDINFO: case DIOCSDINFO: case DIOCWDINFO: case DIOCGPARTINFO: case DIOCKLABEL: case DIOCWLABEL: case DIOCGDEFLABEL: case DIOCAWEDGE: case DIOCDWEDGE: case DIOCLWEDGES: case DIOCMWEDGES: case DIOCRMWEDGES: case DIOCCACHESYNC: #ifdef __HAVE_OLD_DISKLABEL case ODIOCGDINFO: case ODIOCSDINFO: case ODIOCWDINFO: case ODIOCGDEFLABEL: #endif if ((dksc->sc_flags & DKF_INITED) == 0) return ENXIO; } error = disk_ioctl(dk, dev, cmd, data, flag, l); if (error != EPASSTHROUGH) return error; else error = 0; switch (cmd) { case DIOCWDINFO: case DIOCSDINFO: #ifdef __HAVE_OLD_DISKLABEL case ODIOCWDINFO: case ODIOCSDINFO: #endif #ifdef __HAVE_OLD_DISKLABEL if (cmd == ODIOCSDINFO || cmd == ODIOCWDINFO) { memset(&newlabel, 0, sizeof newlabel); memcpy(&newlabel, data, sizeof (struct olddisklabel)); lp = &newlabel; } else #endif lp = (struct disklabel *)data; mutex_enter(&dk->dk_openlock); dksc->sc_flags |= DKF_LABELLING; error = setdisklabel(dksc->sc_dkdev.dk_label, lp, 0, dksc->sc_dkdev.dk_cpulabel); if (error == 0) { if (cmd == DIOCWDINFO #ifdef __HAVE_OLD_DISKLABEL || cmd == ODIOCWDINFO #endif ) error = writedisklabel(DKLABELDEV(dev), dkd->d_strategy, dksc->sc_dkdev.dk_label, dksc->sc_dkdev.dk_cpulabel); } dksc->sc_flags &= ~DKF_LABELLING; mutex_exit(&dk->dk_openlock); break; case DIOCKLABEL: if (*(int *)data != 0) dksc->sc_flags |= DKF_KLABEL; else dksc->sc_flags &= ~DKF_KLABEL; break; case DIOCWLABEL: if (*(int *)data != 0) dksc->sc_flags |= DKF_WLABEL; else dksc->sc_flags &= ~DKF_WLABEL; break; case DIOCGDEFLABEL: dk_getdefaultlabel(dksc, (struct disklabel *)data); break; #ifdef __HAVE_OLD_DISKLABEL case ODIOCGDEFLABEL: dk_getdefaultlabel(dksc, &newlabel); if (newlabel.d_npartitions > OLDMAXPARTITIONS) return ENOTTY; memcpy(data, &newlabel, sizeof (struct olddisklabel)); break; #endif case DIOCGSTRATEGY: { struct disk_strategy *dks = (void *)data; mutex_enter(&dksc->sc_iolock); if (dksc->sc_bufq != NULL) strlcpy(dks->dks_name, bufq_getstrategyname(dksc->sc_bufq), sizeof(dks->dks_name)); else error = EINVAL; mutex_exit(&dksc->sc_iolock); dks->dks_paramlen = 0; break; } case DIOCSSTRATEGY: { struct disk_strategy *dks = (void *)data; struct bufq_state *new; struct bufq_state *old; if (dks->dks_param != NULL) { return EINVAL; } dks->dks_name[sizeof(dks->dks_name) - 1] = 0; /* ensure term */ error = bufq_alloc(&new, dks->dks_name, BUFQ_EXACT|BUFQ_SORT_RAWBLOCK); if (error) { return error; } mutex_enter(&dksc->sc_iolock); old = dksc->sc_bufq; if (old) bufq_move(new, old); dksc->sc_bufq = new; mutex_exit(&dksc->sc_iolock); if (old) bufq_free(old); break; } default: error = ENOTTY; } return error; } /* * dk_dump dumps all of physical memory into the partition specified. * This requires substantially more framework than {s,w}ddump, and hence * is probably much more fragile. * */ #define DKFF_READYFORDUMP(x) (((x) & DKF_READYFORDUMP) == DKF_READYFORDUMP) static volatile int dk_dumping = 0; /* ARGSUSED */ int dk_dump(struct dk_softc *dksc, dev_t dev, daddr_t blkno, void *vav, size_t size, int flags) { const struct dkdriver *dkd = dksc->sc_dkdev.dk_driver; struct disk_geom *dg = &dksc->sc_dkdev.dk_geom; char *va = vav; struct disklabel *lp; struct partition *p; int part, towrt, maxblkcnt, nblk; int maxxfer, rv = 0; /* * ensure that we consider this device to be safe for dumping, * and that the device is configured. */ if (!DKFF_READYFORDUMP(dksc->sc_flags)) { DPRINTF(DKDB_DUMP, ("%s: bad dump flags 0x%x\n", __func__, dksc->sc_flags)); return ENXIO; } /* ensure that we are not already dumping */ if (dk_dumping) return EFAULT; if ((flags & DK_DUMP_RECURSIVE) == 0) dk_dumping = 1; if (dkd->d_dumpblocks == NULL) { DPRINTF(DKDB_DUMP, ("%s: no dumpblocks\n", __func__)); return ENXIO; } /* device specific max transfer size */ maxxfer = MAXPHYS; if (dkd->d_iosize != NULL) (*dkd->d_iosize)(dksc->sc_dev, &maxxfer); /* Convert to disk sectors. Request must be a multiple of size. */ part = DISKPART(dev); lp = dksc->sc_dkdev.dk_label; if ((size % lp->d_secsize) != 0) { DPRINTF(DKDB_DUMP, ("%s: odd size %zu\n", __func__, size)); return EFAULT; } towrt = size / lp->d_secsize; blkno = dbtob(blkno) / lp->d_secsize; /* blkno in secsize units */ p = &lp->d_partitions[part]; if (part == RAW_PART) { if (p->p_fstype != FS_UNUSED) { DPRINTF(DKDB_DUMP, ("%s: bad fstype %d\n", __func__, p->p_fstype)); return ENXIO; } /* Check whether dump goes to a wedge */ if (dksc->sc_dkdev.dk_nwedges == 0) { DPRINTF(DKDB_DUMP, ("%s: dump to raw\n", __func__)); return ENXIO; } /* Check transfer bounds against media size */ if (blkno < 0 || (blkno + towrt) > dg->dg_secperunit) { DPRINTF(DKDB_DUMP, ("%s: out of bounds blkno=%jd, towrt=%d, " "nsects=%jd\n", __func__, (intmax_t)blkno, towrt, dg->dg_secperunit)); return EINVAL; } } else { int nsects, sectoff; if (p->p_fstype != FS_SWAP) { DPRINTF(DKDB_DUMP, ("%s: bad fstype %d\n", __func__, p->p_fstype)); return ENXIO; } nsects = p->p_size; sectoff = p->p_offset; /* Check transfer bounds against partition size. */ if ((blkno < 0) || ((blkno + towrt) > nsects)) { DPRINTF(DKDB_DUMP, ("%s: out of bounds blkno=%jd, towrt=%d, " "nsects=%d\n", __func__, (intmax_t)blkno, towrt, nsects)); return EINVAL; } /* Offset block number to start of partition. */ blkno += sectoff; } /* Start dumping and return when done. */ maxblkcnt = howmany(maxxfer, lp->d_secsize); while (towrt > 0) { nblk = uimin(maxblkcnt, towrt); if ((rv = (*dkd->d_dumpblocks)(dksc->sc_dev, va, blkno, nblk)) != 0) { DPRINTF(DKDB_DUMP, ("%s: dumpblocks %d\n", __func__, rv)); return rv; } towrt -= nblk; blkno += nblk; va += nblk * lp->d_secsize; } if ((flags & DK_DUMP_RECURSIVE) == 0) dk_dumping = 0; return 0; } /* ARGSUSED */ void dk_getdefaultlabel(struct dk_softc *dksc, struct disklabel *lp) { const struct dkdriver *dkd = dksc->sc_dkdev.dk_driver; struct disk_geom *dg = &dksc->sc_dkdev.dk_geom; memset(lp, 0, sizeof(*lp)); if (dg->dg_secperunit > UINT32_MAX) lp->d_secperunit = UINT32_MAX; else lp->d_secperunit = dg->dg_secperunit; lp->d_secsize = dg->dg_secsize; lp->d_nsectors = dg->dg_nsectors; lp->d_ntracks = dg->dg_ntracks; lp->d_ncylinders = dg->dg_ncylinders; lp->d_secpercyl = lp->d_ntracks * lp->d_nsectors; strlcpy(lp->d_typename, dksc->sc_xname, sizeof(lp->d_typename)); lp->d_type = dksc->sc_dtype; strlcpy(lp->d_packname, "fictitious", sizeof(lp->d_packname)); lp->d_rpm = 3600; lp->d_interleave = 1; lp->d_flags = 0; lp->d_partitions[RAW_PART].p_offset = 0; lp->d_partitions[RAW_PART].p_size = lp->d_secperunit; lp->d_partitions[RAW_PART].p_fstype = FS_UNUSED; lp->d_npartitions = RAW_PART + 1; lp->d_magic = DISKMAGIC; lp->d_magic2 = DISKMAGIC; if (dkd->d_label) dkd->d_label(dksc->sc_dev, lp); lp->d_checksum = dkcksum(lp); } /* ARGSUSED */ void dk_getdisklabel(struct dk_softc *dksc, dev_t dev) { const struct dkdriver *dkd = dksc->sc_dkdev.dk_driver; struct disklabel *lp = dksc->sc_dkdev.dk_label; struct cpu_disklabel *clp = dksc->sc_dkdev.dk_cpulabel; struct disk_geom *dg = &dksc->sc_dkdev.dk_geom; struct partition *pp; int i, lpratio, dgratio; const char *errstring; memset(clp, 0x0, sizeof(*clp)); dk_getdefaultlabel(dksc, lp); errstring = readdisklabel(DKLABELDEV(dev), dkd->d_strategy, dksc->sc_dkdev.dk_label, dksc->sc_dkdev.dk_cpulabel); if (errstring) { dk_makedisklabel(dksc); if (dksc->sc_flags & DKF_WARNLABEL) printf("%s: %s\n", dksc->sc_xname, errstring); return; } if ((dksc->sc_flags & DKF_LABELSANITY) == 0) return; /* Convert sector counts to multiple of DEV_BSIZE for comparison */ lpratio = dgratio = 1; if (lp->d_secsize > DEV_BSIZE) lpratio = lp->d_secsize / DEV_BSIZE; if (dg->dg_secsize > DEV_BSIZE) dgratio = dg->dg_secsize / DEV_BSIZE; /* Sanity check */ if ((uint64_t)lp->d_secperunit * lpratio > dg->dg_secperunit * dgratio) printf("WARNING: %s: " "total unit size in disklabel (%" PRIu64 ") " "!= the size of %s (%" PRIu64 ")\n", dksc->sc_xname, (uint64_t)lp->d_secperunit * lpratio, dksc->sc_xname, dg->dg_secperunit * dgratio); else if (lp->d_secperunit < UINT32_MAX && (uint64_t)lp->d_secperunit * lpratio < dg->dg_secperunit * dgratio) printf("%s: %" PRIu64 " trailing sectors not covered" " by disklabel\n", dksc->sc_xname, (dg->dg_secperunit * dgratio) - (lp->d_secperunit * lpratio)); for (i=0; i < lp->d_npartitions; i++) { uint64_t pend; pp = &lp->d_partitions[i]; pend = pp->p_offset + pp->p_size; if (pend * lpratio > dg->dg_secperunit * dgratio) printf("WARNING: %s: end of partition `%c' exceeds " "the size of %s (%" PRIu64 ")\n", dksc->sc_xname, 'a' + i, dksc->sc_xname, dg->dg_secperunit * dgratio); } } /* * Heuristic to conjure a disklabel if reading a disklabel failed. * * This is to allow the raw partition to be used for a filesystem * without caring about the write protected label sector. * * If the driver provides it's own callback, use that instead. */ /* ARGSUSED */ static void dk_makedisklabel(struct dk_softc *dksc) { const struct dkdriver *dkd = dksc->sc_dkdev.dk_driver; struct disklabel *lp = dksc->sc_dkdev.dk_label; strlcpy(lp->d_packname, "default label", sizeof(lp->d_packname)); if (dkd->d_label) dkd->d_label(dksc->sc_dev, lp); else lp->d_partitions[RAW_PART].p_fstype = FS_BSDFFS; lp->d_checksum = dkcksum(lp); } MODULE(MODULE_CLASS_MISC, dk_subr, NULL); static int dk_subr_modcmd(modcmd_t cmd, void *arg) { switch (cmd) { case MODULE_CMD_INIT: case MODULE_CMD_FINI: return 0; case MODULE_CMD_STAT: case MODULE_CMD_AUTOUNLOAD: default: return ENOTTY; } } |
| 633 633 3 3 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 197 198 199 200 201 202 203 204 205 206 207 208 209 210 211 212 213 214 215 216 217 218 219 220 221 222 223 224 225 226 227 228 229 230 231 232 233 234 235 236 237 238 239 240 241 242 243 244 245 246 247 248 249 250 251 252 253 254 255 256 257 258 259 260 261 262 263 264 265 266 267 268 269 270 271 272 273 274 275 276 277 278 279 | /* $NetBSD: scsi_base.c,v 1.93 2019/05/03 16:06:56 mlelstv Exp $ */ /*- * Copyright (c) 1998, 2004 The NetBSD Foundation, Inc. * All rights reserved. * * This code is derived from software contributed to The NetBSD Foundation * by Charles M. Hannum. * * Redistribution and use in source and binary forms, with or without * modification, are permitted provided that the following conditions * are met: * 1. Redistributions of source code must retain the above copyright * notice, this list of conditions and the following disclaimer. * 2. Redistributions in binary form must reproduce the above copyright * notice, this list of conditions and the following disclaimer in the * documentation and/or other materials provided with the distribution. * * THIS SOFTWARE IS PROVIDED BY THE NETBSD FOUNDATION, INC. AND CONTRIBUTORS * ``AS IS'' AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED * TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR * PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE FOUNDATION OR CONTRIBUTORS * BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR * CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF * SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS * INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN * CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) * ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE * POSSIBILITY OF SUCH DAMAGE. */ #include <sys/cdefs.h> __KERNEL_RCSID(0, "$NetBSD: scsi_base.c,v 1.93 2019/05/03 16:06:56 mlelstv Exp $"); #include <sys/param.h> #include <sys/systm.h> #include <sys/kernel.h> #include <sys/buf.h> #include <sys/uio.h> #include <sys/malloc.h> #include <sys/errno.h> #include <sys/device.h> #include <sys/proc.h> #include <dev/scsipi/scsipi_all.h> #include <dev/scsipi/scsi_all.h> #include <dev/scsipi/scsi_disk.h> #include <dev/scsipi/scsiconf.h> #include <dev/scsipi/scsipi_base.h> static void scsi_print_xfer_mode(struct scsipi_periph *); /* * Do a scsi operation, asking a device to run as SCSI-II if it can. */ int scsi_change_def(struct scsipi_periph *periph, int flags) { struct scsi_changedef cmd; memset(&cmd, 0, sizeof(cmd)); cmd.opcode = SCSI_CHANGE_DEFINITION; cmd.how = SC_SCSI_2; return (scsipi_command(periph, (void *)&cmd, sizeof(cmd), 0, 0, SCSIPIRETRIES, 100000, NULL, flags)); } /* * ask the scsi driver to perform a command for us. * tell it where to read/write the data, and how * long the data is supposed to be. If we have a buf * to associate with the transfer, we need that too. */ void scsi_scsipi_cmd(struct scsipi_xfer *xs) { struct scsipi_periph *periph = xs->xs_periph; SC_DEBUG(periph, SCSIPI_DB2, ("scsi_scsipi_cmd\n")); /* * Set the LUN in the CDB if we have an older device. We also * set it for more modern SCSI-2 devices "just in case". */ if (periph->periph_version <= 2) xs->cmd->bytes[0] |= ((periph->periph_lun << SCSI_CMD_LUN_SHIFT) & SCSI_CMD_LUN_MASK); } /* * Utility routines often used in SCSI stuff */ /* * Print out the periph's address info. */ void scsi_print_addr(struct scsipi_periph *periph) { struct scsipi_channel *chan = periph->periph_channel; struct scsipi_adapter *adapt = chan->chan_adapter; printf("%s(%s:%d:%d:%d): ", periph->periph_dev != NULL ? device_xname(periph->periph_dev) : "probe", device_xname(adapt->adapt_dev), chan->chan_channel, periph->periph_target, periph->periph_lun); } /* * Kill off all pending xfers for a periph. * * Must be called with channel lock held */ void scsi_kill_pending(struct scsipi_periph *periph) { struct scsipi_xfer *xs; TAILQ_FOREACH(xs, &periph->periph_xferq, device_q) { callout_stop(&xs->xs_callout); scsi_print_addr(periph); printf("killed "); scsipi_print_cdb(xs->cmd); xs->error = XS_DRIVER_STUFFUP; scsipi_done(xs); } } /* * scsi_print_xfer_mode: * * Print a parallel SCSI periph's capabilities. */ static void scsi_print_xfer_mode(struct scsipi_periph *periph) { struct scsipi_channel *chan = periph->periph_channel; struct scsipi_adapter *adapt = chan->chan_adapter; int period, freq, speed, mbs; if (periph->periph_dev) aprint_normal_dev(periph->periph_dev, ""); else aprint_normal("probe(%s:%d:%d:%d): ", device_xname(adapt->adapt_dev), chan->chan_channel, periph->periph_target, periph->periph_lun); if (periph->periph_mode & (PERIPH_CAP_SYNC | PERIPH_CAP_DT)) { period = scsipi_sync_factor_to_period(periph->periph_period); aprint_normal("sync (%d.%02dns offset %d)", period / 100, period % 100, periph->periph_offset); } else aprint_normal("async"); if (periph->periph_mode & PERIPH_CAP_WIDE32) aprint_normal(", 32-bit"); else if (periph->periph_mode & (PERIPH_CAP_WIDE16 | PERIPH_CAP_DT)) aprint_normal(", 16-bit"); else aprint_normal(", 8-bit"); if (periph->periph_mode & (PERIPH_CAP_SYNC | PERIPH_CAP_DT)) { freq = scsipi_sync_factor_to_freq(periph->periph_period); speed = freq; if (periph->periph_mode & PERIPH_CAP_WIDE32) speed *= 4; else if (periph->periph_mode & (PERIPH_CAP_WIDE16 | PERIPH_CAP_DT)) speed *= 2; mbs = speed / 1000; if (mbs > 0) { aprint_normal(" (%d.%03dMB/s)", mbs, speed % 1000); } else aprint_normal(" (%dKB/s)", speed % 1000); } aprint_normal(" transfers"); if (periph->periph_mode & PERIPH_CAP_TQING) aprint_normal(", tagged queueing"); aprint_normal("\n"); } /* * scsi_async_event_xfer_mode: * * Update the xfer mode for all parallel SCSI periphs sharing the * specified I_T Nexus. */ void scsi_async_event_xfer_mode(struct scsipi_channel *chan, void *arg) { struct scsipi_xfer_mode *xm = arg; struct scsipi_periph *periph; int lun, announce, mode, period, offset; for (lun = 0; lun < chan->chan_nluns; lun++) { periph = scsipi_lookup_periph_locked(chan, xm->xm_target, lun); if (periph == NULL) continue; announce = 0; /* * Clamp the xfer mode down to this periph's capabilities. */ mode = xm->xm_mode & periph->periph_cap; if (mode & PERIPH_CAP_SYNC) { period = xm->xm_period; offset = xm->xm_offset; } else { period = 0; offset = 0; } /* * If we do not have a valid xfer mode yet, or the parameters * are different, announce them. */ if ((periph->periph_flags & PERIPH_MODE_VALID) == 0 || periph->periph_mode != mode || periph->periph_period != period || periph->periph_offset != offset) announce = 1; periph->periph_mode = mode; periph->periph_period = period; periph->periph_offset = offset; periph->periph_flags |= PERIPH_MODE_VALID; if (announce) scsi_print_xfer_mode(periph); } } /* * scsipi_async_event_xfer_mode: * * Update the xfer mode for all SAS/FC periphs sharing the * specified I_T Nexus. */ void scsi_fc_sas_async_event_xfer_mode(struct scsipi_channel *chan, void *arg) { struct scsipi_xfer_mode *xm = arg; struct scsipi_periph *periph; int lun, announce, mode; for (lun = 0; lun < chan->chan_nluns; lun++) { periph = scsipi_lookup_periph_locked(chan, xm->xm_target, lun); if (periph == NULL) continue; announce = 0; /* * Clamp the xfer mode down to this periph's capabilities. */ mode = xm->xm_mode & periph->periph_cap; /* * If we do not have a valid xfer mode yet, or the parameters * are different, announce them. */ if ((periph->periph_flags & PERIPH_MODE_VALID) == 0 || periph->periph_mode != mode) announce = 1; periph->periph_mode = mode; periph->periph_flags |= PERIPH_MODE_VALID; if (announce && (periph->periph_mode & PERIPH_CAP_TQING) != 0) { aprint_normal_dev(periph->periph_dev, "tagged queueing\n"); } } } |
| 8 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 197 198 199 200 201 202 203 204 205 206 207 208 209 210 211 212 213 214 215 216 217 218 219 220 221 222 223 224 225 226 227 228 229 230 231 232 233 234 235 236 237 238 239 240 241 242 243 244 245 246 247 248 249 250 251 252 253 254 255 256 257 258 259 260 261 262 263 264 265 266 267 268 269 270 271 272 273 274 275 276 277 278 279 280 281 282 283 284 285 286 287 288 289 290 291 292 293 294 295 296 297 298 299 300 301 302 303 304 305 306 307 308 309 310 311 312 313 314 315 316 317 318 319 320 321 322 323 324 325 326 327 328 329 330 331 332 333 334 335 336 337 338 339 340 341 342 343 344 345 346 347 348 349 350 351 352 353 354 355 356 357 358 359 360 361 362 363 364 365 366 367 368 369 370 371 372 373 374 375 376 377 378 379 380 381 382 383 384 385 386 387 388 389 390 391 392 393 394 395 396 397 398 399 400 401 402 403 404 405 406 407 408 409 410 411 412 413 414 415 416 417 418 419 420 421 422 423 424 425 426 427 428 429 430 431 432 433 434 435 436 437 438 439 440 441 442 443 444 445 446 447 448 449 450 451 452 453 454 455 456 457 458 459 460 461 462 463 464 465 466 467 468 469 470 471 472 473 474 475 476 477 478 479 480 481 482 483 484 485 486 487 488 489 490 491 492 493 494 495 496 497 498 499 500 501 502 503 504 505 506 507 508 509 510 511 512 513 514 515 516 517 518 519 520 521 522 523 524 525 526 527 528 529 530 531 532 533 534 535 536 537 538 539 540 541 542 543 544 545 546 547 548 549 550 551 552 553 554 555 556 557 558 559 560 561 562 563 564 565 566 567 568 569 570 571 572 573 574 575 576 577 578 579 580 581 582 583 584 585 586 587 588 589 590 591 592 593 594 595 596 597 598 599 600 601 602 603 604 605 606 607 608 609 610 611 612 613 614 615 616 617 618 619 620 621 622 623 624 625 626 627 628 629 630 631 632 633 634 635 636 637 638 639 640 641 642 643 644 645 646 647 648 649 650 651 652 653 654 655 656 657 658 659 660 661 662 663 664 665 666 667 668 669 670 671 672 673 674 675 676 677 678 679 680 681 682 683 684 685 686 687 688 689 690 691 692 693 694 695 696 697 698 699 700 701 702 703 704 705 706 707 708 709 710 711 712 713 714 715 716 717 718 719 720 721 722 723 724 725 726 727 728 729 730 731 732 733 734 735 736 737 738 739 740 741 742 743 744 745 746 747 748 749 750 751 752 753 754 755 756 757 758 759 760 761 762 763 764 765 766 767 768 769 770 771 772 773 774 775 776 777 778 779 780 781 782 783 784 785 786 787 788 789 790 791 792 793 794 795 796 797 798 799 800 801 802 803 804 805 806 807 808 809 810 811 812 813 814 815 816 817 818 819 820 821 822 823 824 825 826 827 828 829 830 831 832 833 834 835 836 837 838 839 840 841 842 843 844 845 846 847 848 849 850 851 852 853 854 855 856 857 858 859 860 861 862 863 864 865 866 867 868 869 870 871 872 873 874 875 876 877 878 879 880 881 882 883 884 885 886 887 888 889 890 891 892 893 | /* $NetBSD: if_bwfm_usb.c,v 1.15 2020/07/22 17:18:10 riastradh Exp $ */ /* $OpenBSD: if_bwfm_usb.c,v 1.2 2017/10/15 14:55:13 patrick Exp $ */ /* * Copyright (c) 2010-2016 Broadcom Corporation * Copyright (c) 2016,2017 Patrick Wildt <patrick@blueri.se> * * Permission to use, copy, modify, and/or distribute this software for any * purpose with or without fee is hereby granted, provided that the above * copyright notice and this permission notice appear in all copies. * * THE SOFTWARE IS PROVIDED "AS IS" AND THE AUTHOR DISCLAIMS ALL WARRANTIES * WITH REGARD TO THIS SOFTWARE INCLUDING ALL IMPLIED WARRANTIES OF * MERCHANTABILITY AND FITNESS. IN NO EVENT SHALL THE AUTHOR BE LIABLE FOR * ANY SPECIAL, DIRECT, INDIRECT, OR CONSEQUENTIAL DAMAGES OR ANY DAMAGES * WHATSOEVER RESULTING FROM LOSS OF USE, DATA OR PROFITS, WHETHER IN AN * ACTION OF CONTRACT, NEGLIGENCE OR OTHER TORTIOUS ACTION, ARISING OUT OF * OR IN CONNECTION WITH THE USE OR PERFORMANCE OF THIS SOFTWARE. */ #include <sys/cdefs.h> __KERNEL_RCSID(0, "$NetBSD: if_bwfm_usb.c,v 1.15 2020/07/22 17:18:10 riastradh Exp $"); #include <sys/param.h> #include <sys/types.h> #include <sys/buf.h> #include <sys/device.h> #include <sys/kernel.h> #include <sys/malloc.h> #include <sys/mutex.h> #include <sys/queue.h> #include <sys/socket.h> #include <sys/systm.h> #include <sys/workqueue.h> #include <net/bpf.h> #include <net/if.h> #include <net/if_dl.h> #include <net/if_ether.h> #include <net/if_media.h> #include <netinet/in.h> #include <net80211/ieee80211_var.h> #include <dev/usb/usb.h> #include <dev/usb/usbdevs.h> #include <dev/usb/usbdi.h> #include <dev/usb/usbdi_util.h> #include <dev/usb/usbdivar.h> #include <dev/ic/bwfmreg.h> #include <dev/ic/bwfmvar.h> static const struct bwfm_firmware_selector bwfm_usb_fwtab[] = { BWFM_FW_ENTRY(BRCM_CC_43143_CHIP_ID, BWFM_FWSEL_ALLREVS, "brcmfmac43143"), BWFM_FW_ENTRY(BRCM_CC_43235_CHIP_ID, BWFM_FWSEL_REV_EQ(3), "brcmfmac43236b"), BWFM_FW_ENTRY(BRCM_CC_43236_CHIP_ID, BWFM_FWSEL_REV_EQ(3), "brcmfmac43236b"), BWFM_FW_ENTRY(BRCM_CC_43238_CHIP_ID, BWFM_FWSEL_REV_EQ(3), "brcmfmac43236b"), BWFM_FW_ENTRY(BRCM_CC_43242_CHIP_ID, BWFM_FWSEL_ALLREVS, "brcmfmac43242a"), BWFM_FW_ENTRY(BRCM_CC_43566_CHIP_ID, BWFM_FWSEL_ALLREVS, "brcmfmac43569"), BWFM_FW_ENTRY(BRCM_CC_43569_CHIP_ID, BWFM_FWSEL_ALLREVS, "brcmfmac43569"), BWFM_FW_ENTRY(CY_CC_4373_CHIP_ID, BWFM_FWSEL_ALLREVS, "brcmfmac4373"), BWFM_FW_ENTRY_END }; /* * Various supported device vendors/products. */ static const struct usb_devno bwfm_usbdevs[] = { { USB_VENDOR_BROADCOM, USB_PRODUCT_BROADCOM_BCM43143 }, { USB_VENDOR_BROADCOM, USB_PRODUCT_BROADCOM_BCM43236 }, { USB_VENDOR_BROADCOM, USB_PRODUCT_BROADCOM_BCM43242 }, { USB_VENDOR_BROADCOM, USB_PRODUCT_BROADCOM_BCM43569 }, { USB_VENDOR_BROADCOM, USB_PRODUCT_BROADCOM_BCMFW }, }; #ifdef BWFM_DEBUG #define DPRINTF(x) do { if (bwfm_debug > 0) printf x; } while (0) #define DPRINTFN(n, x) do { if (bwfm_debug >= (n)) printf x; } while (0) static int bwfm_debug = 2; #else #define DPRINTF(x) do { ; } while (0) #define DPRINTFN(n, x) do { ; } while (0) #endif #define DEVNAME(sc) device_xname((sc)->sc_sc.sc_dev) #define BRCMF_POSTBOOT_ID 0xA123 /* ID to detect if dongle * has boot up */ #define TRX_MAGIC 0x30524448 /* "HDR0" */ #define TRX_MAX_OFFSET 3 /* Max number of file offsets */ #define TRX_UNCOMP_IMAGE 0x20 /* Trx holds uncompressed img */ #define TRX_RDL_CHUNK 1500 /* size of each dl transfer */ #define TRX_OFFSETS_DLFWLEN_IDX 0 /* Control messages: bRequest values */ #define DL_GETSTATE 0 /* returns the rdl_state_t struct */ #define DL_CHECK_CRC 1 /* currently unused */ #define DL_GO 2 /* execute downloaded image */ #define DL_START 3 /* initialize dl state */ #define DL_REBOOT 4 /* reboot the device in 2 seconds */ #define DL_GETVER 5 /* returns the bootrom_id_t struct */ #define DL_GO_PROTECTED 6 /* execute the downloaded code and set reset * event to occur in 2 seconds. It is the * responsibility of the downloaded code to * clear this event */ #define DL_EXEC 7 /* jump to a supplied address */ #define DL_RESETCFG 8 /* To support single enum on dongle * - Not used by bootloader */ #define DL_DEFER_RESP_OK 9 /* Potentially defer the response to setup * if resp unavailable */ /* states */ #define DL_WAITING 0 /* waiting to rx first pkt */ #define DL_READY 1 /* hdr was good, waiting for more of the * compressed image */ #define DL_BAD_HDR 2 /* hdr was corrupted */ #define DL_BAD_CRC 3 /* compressed image was corrupted */ #define DL_RUNNABLE 4 /* download was successful,waiting for go cmd */ #define DL_START_FAIL 5 /* failed to initialize correctly */ #define DL_NVRAM_TOOBIG 6 /* host specified nvram data exceeds DL_NVRAM * value */ #define DL_IMAGE_TOOBIG 7 /* firmware image too big */ struct trx_header { uint32_t magic; /* "HDR0" */ uint32_t len; /* Length of file including header */ uint32_t crc32; /* CRC from flag_version to end of file */ uint32_t flag_version; /* 0:15 flags, 16:31 version */ uint32_t offsets[TRX_MAX_OFFSET];/* Offsets of partitions from start of * header */ }; struct rdl_state { uint32_t state; uint32_t bytes; }; struct bootrom_id { uint32_t chip; /* Chip id */ uint32_t chiprev; /* Chip rev */ uint32_t ramsize; /* Size of RAM */ uint32_t remapbase; /* Current remap base address */ uint32_t boardtype; /* Type of board */ uint32_t boardrev; /* Board revision */ }; struct bwfm_usb_rx_data { struct bwfm_usb_softc *sc; struct usbd_xfer *xfer; uint8_t *buf; }; struct bwfm_usb_tx_data { struct bwfm_usb_softc *sc; struct usbd_xfer *xfer; uint8_t *buf; struct mbuf *mbuf; TAILQ_ENTRY(bwfm_usb_tx_data) next; }; #define BWFM_RX_LIST_COUNT 50 #define BWFM_TX_LIST_COUNT 50 #define BWFM_RXBUFSZ 1600 #define BWFM_TXBUFSZ 1600 struct bwfm_usb_softc { struct bwfm_softc sc_sc; struct usbd_device *sc_udev; struct usbd_interface *sc_iface; uint8_t sc_ifaceno; uint16_t sc_vendor; uint16_t sc_product; uint32_t sc_chip; uint32_t sc_chiprev; int sc_rx_no; int sc_tx_no; struct usbd_pipe *sc_rx_pipeh; struct usbd_pipe *sc_tx_pipeh; struct bwfm_usb_rx_data sc_rx_data[BWFM_RX_LIST_COUNT]; struct bwfm_usb_tx_data sc_tx_data[BWFM_TX_LIST_COUNT]; TAILQ_HEAD(, bwfm_usb_tx_data) sc_tx_free_list; kmutex_t sc_rx_lock; kmutex_t sc_tx_lock; }; int bwfm_usb_match(device_t, cfdata_t, void *); void bwfm_usb_attachhook(device_t); void bwfm_usb_attach(device_t, device_t, void *); int bwfm_usb_detach(device_t, int); int bwfm_usb_dl_cmd(struct bwfm_usb_softc *, uint8_t, void *, int); int bwfm_usb_load_microcode(struct bwfm_usb_softc *, const u_char *, size_t); int bwfm_usb_alloc_rx_list(struct bwfm_usb_softc *); void bwfm_usb_free_rx_list(struct bwfm_usb_softc *); int bwfm_usb_alloc_tx_list(struct bwfm_usb_softc *); void bwfm_usb_free_tx_list(struct bwfm_usb_softc *); int bwfm_usb_txcheck(struct bwfm_softc *); int bwfm_usb_txdata(struct bwfm_softc *, struct mbuf **); int bwfm_usb_txctl(struct bwfm_softc *, char *, size_t); int bwfm_usb_rxctl(struct bwfm_softc *, char *, size_t *); struct mbuf * bwfm_usb_newbuf(void); void bwfm_usb_rxeof(struct usbd_xfer *, void *, usbd_status); void bwfm_usb_txeof(struct usbd_xfer *, void *, usbd_status); static const struct bwfm_bus_ops bwfm_usb_bus_ops = { .bs_init = NULL, .bs_stop = NULL, .bs_txcheck = bwfm_usb_txcheck, .bs_txdata = bwfm_usb_txdata, .bs_txctl = bwfm_usb_txctl, .bs_rxctl = bwfm_usb_rxctl, }; CFATTACH_DECL_NEW(bwfm_usb, sizeof(struct bwfm_usb_softc), bwfm_usb_match, bwfm_usb_attach, bwfm_usb_detach, NULL); int bwfm_usb_match(device_t parent, cfdata_t match, void *aux) { struct usb_attach_arg *uaa = aux; return (usb_lookup(bwfm_usbdevs, uaa->uaa_vendor, uaa->uaa_product) != NULL) ? UMATCH_VENDOR_PRODUCT_CONF_IFACE : UMATCH_NONE; } void bwfm_usb_attach(device_t parent, device_t self, void *aux) { struct bwfm_usb_softc *sc = device_private(self); struct usb_attach_arg *uaa = aux; usb_device_descriptor_t *dd; usb_interface_descriptor_t *id; usb_endpoint_descriptor_t *ed; char *devinfop; int i; sc->sc_sc.sc_dev = self; sc->sc_udev = uaa->uaa_device; mutex_init(&sc->sc_rx_lock, MUTEX_DEFAULT, IPL_NET); mutex_init(&sc->sc_tx_lock, MUTEX_DEFAULT, IPL_NET); aprint_naive("\n"); devinfop = usbd_devinfo_alloc(sc->sc_udev, 0); aprint_normal(": %s\n", devinfop); usbd_devinfo_free(devinfop); if (usbd_set_config_no(sc->sc_udev, 1, 1) != 0) { aprint_error_dev(self, "failed to set configuration\n"); return; } if (usbd_device2interface_handle(sc->sc_udev, 0, &sc->sc_iface) != 0) { aprint_error_dev(self, "failed to get interface handle\n"); return; } sc->sc_ifaceno = 0; sc->sc_vendor = uaa->uaa_vendor; sc->sc_product = uaa->uaa_product; sc->sc_sc.sc_bus_ops = &bwfm_usb_bus_ops; sc->sc_sc.sc_proto_ops = &bwfm_proto_bcdc_ops; /* Check number of configurations. */ dd = usbd_get_device_descriptor(sc->sc_udev); if (dd->bNumConfigurations != 1) { printf("%s: number of configurations not supported\n", DEVNAME(sc)); return; } /* Get endpoints. */ id = usbd_get_interface_descriptor(sc->sc_iface); sc->sc_rx_no = sc->sc_tx_no = -1; for (i = 0; i < id->bNumEndpoints; i++) { ed = usbd_interface2endpoint_descriptor(sc->sc_iface, i); if (ed == NULL) { printf("%s: no endpoint descriptor for iface %d\n", DEVNAME(sc), i); return; } if (UE_GET_DIR(ed->bEndpointAddress) == UE_DIR_IN && UE_GET_XFERTYPE(ed->bmAttributes) == UE_BULK && sc->sc_rx_no == -1) sc->sc_rx_no = ed->bEndpointAddress; else if (UE_GET_DIR(ed->bEndpointAddress) == UE_DIR_OUT && UE_GET_XFERTYPE(ed->bmAttributes) == UE_BULK && sc->sc_tx_no == -1) sc->sc_tx_no = ed->bEndpointAddress; } if (sc->sc_rx_no == -1 || sc->sc_tx_no == -1) { printf("%s: missing endpoint\n", DEVNAME(sc)); return; } config_mountroot(self, bwfm_usb_attachhook); } void bwfm_usb_attachhook(device_t self) { struct bwfm_usb_softc *sc = device_private(self); struct bwfm_softc *bwfm = &sc->sc_sc; struct bwfm_usb_rx_data *data; struct bootrom_id brom; struct bwfm_firmware_context fwctx; usbd_status error; u_char *ucode; size_t ucsize; int i; /* Read chip id and chip rev to check the firmware. */ memset(&brom, 0, sizeof(brom)); bwfm_usb_dl_cmd(sc, DL_GETVER, &brom, sizeof(brom)); sc->sc_chip = le32toh(brom.chip); sc->sc_chiprev = le32toh(brom.chiprev); /* Setup data pipes */ error = usbd_open_pipe(sc->sc_iface, sc->sc_rx_no, USBD_EXCLUSIVE_USE, &sc->sc_rx_pipeh); if (error != 0) { aprint_error_dev(bwfm->sc_dev, "could not open rx pipe: %s\n", usbd_errstr(error)); return; } error = usbd_open_pipe(sc->sc_iface, sc->sc_tx_no, USBD_EXCLUSIVE_USE, &sc->sc_tx_pipeh); if (error != 0) { aprint_error_dev(bwfm->sc_dev, "could not open tx pipe: %s\n", usbd_errstr(error)); return; } /* Firmware not yet loaded? */ if (sc->sc_chip != BRCMF_POSTBOOT_ID) { bwfm_firmware_context_init(&fwctx, sc->sc_chip, sc->sc_chiprev, NULL, BWFM_FWREQ(BWFM_FILETYPE_UCODE)); if (!bwfm_firmware_open(bwfm, bwfm_usb_fwtab, &fwctx)) { /* Error message already displayed. */ return; } ucode = bwfm_firmware_data(&fwctx, BWFM_FILETYPE_UCODE, &ucsize); KASSERT(ucode != NULL); if (bwfm_usb_load_microcode(sc, ucode, ucsize) != 0) { aprint_error_dev(bwfm->sc_dev, "could not load microcode\n"); bwfm_firmware_close(&fwctx); return; } bwfm_firmware_close(&fwctx); for (i = 0; i < 10; i++) { delay(100 * 1000); memset(&brom, 0, sizeof(brom)); bwfm_usb_dl_cmd(sc, DL_GETVER, &brom, sizeof(brom)); if (le32toh(brom.chip) == BRCMF_POSTBOOT_ID) break; } if (le32toh(brom.chip) != BRCMF_POSTBOOT_ID) { aprint_error_dev(bwfm->sc_dev, "firmware did not start up\n"); return; } sc->sc_chip = le32toh(brom.chip); sc->sc_chiprev = le32toh(brom.chiprev); } bwfm_usb_dl_cmd(sc, DL_RESETCFG, &brom, sizeof(brom)); if (bwfm_usb_alloc_rx_list(sc) || bwfm_usb_alloc_tx_list(sc)) { printf("%s: cannot allocate rx/tx lists\n", DEVNAME(sc)); return; } bwfm_attach(&sc->sc_sc); for (i = 0; i < BWFM_RX_LIST_COUNT; i++) { data = &sc->sc_rx_data[i]; usbd_setup_xfer(data->xfer, data, data->buf, BWFM_RXBUFSZ, USBD_SHORT_XFER_OK, USBD_NO_TIMEOUT, bwfm_usb_rxeof); error = usbd_transfer(data->xfer); if (error != 0 && error != USBD_IN_PROGRESS) aprint_error_dev(bwfm->sc_dev, "could not set up new transfer: %s\n", usbd_errstr(error)); } } struct mbuf * bwfm_usb_newbuf(void) { struct mbuf *m; MGETHDR(m, M_DONTWAIT, MT_DATA); if (m == NULL) return (NULL); MCLGET(m, M_DONTWAIT); if (!(m->m_flags & M_EXT)) { m_freem(m); return (NULL); } m->m_len = m->m_pkthdr.len = MCLBYTES; return (m); } void bwfm_usb_rxeof(struct usbd_xfer *xfer, void *priv, usbd_status status) { struct bwfm_usb_rx_data *data = priv; struct bwfm_usb_softc *sc = data->sc; struct bwfm_proto_bcdc_hdr *hdr; usbd_status error; uint32_t len, off; struct mbuf *m; DPRINTFN(2, ("%s: %s status %s\n", DEVNAME(sc), __func__, usbd_errstr(status))); if (__predict_false(status != USBD_NORMAL_COMPLETION)) { usbd_clear_endpoint_stall_async(sc->sc_rx_pipeh); if (status != USBD_CANCELLED) goto resubmit; return; } usbd_get_xfer_status(xfer, NULL, NULL, &len, NULL); off = 0; hdr = (void *)data->buf; if (len < sizeof(*hdr)) goto resubmit; len -= sizeof(*hdr); off += sizeof(*hdr); if (len <= hdr->data_offset << 2) goto resubmit; len -= hdr->data_offset << 2; off += hdr->data_offset << 2; m = bwfm_usb_newbuf(); if (m == NULL) goto resubmit; memcpy(mtod(m, char *), data->buf + off, len); m->m_len = m->m_pkthdr.len = len; mutex_enter(&sc->sc_rx_lock); /* XXX */ bwfm_rx(&sc->sc_sc, m); mutex_exit(&sc->sc_rx_lock); resubmit: usbd_setup_xfer(data->xfer, data, data->buf, BWFM_RXBUFSZ, USBD_SHORT_XFER_OK, USBD_NO_TIMEOUT, bwfm_usb_rxeof); error = usbd_transfer(data->xfer); if (error != 0 && error != USBD_IN_PROGRESS) printf("%s: could not set up new transfer: %s\n", DEVNAME(sc), usbd_errstr(error)); } int bwfm_usb_alloc_rx_list(struct bwfm_usb_softc *sc) { struct bwfm_usb_rx_data *data; int i, error = 0; for (i = 0; i < BWFM_RX_LIST_COUNT; i++) { data = &sc->sc_rx_data[i]; data->sc = sc; /* Backpointer for callbacks. */ if (usbd_create_xfer(sc->sc_rx_pipeh, BWFM_RXBUFSZ, 0, 0, &data->xfer) != 0) { printf("%s: could not create xfer\n", DEVNAME(sc)); error = ENOMEM; break; } data->buf = usbd_get_buffer(data->xfer); } if (error != 0) bwfm_usb_free_rx_list(sc); return (error); } void bwfm_usb_free_rx_list(struct bwfm_usb_softc *sc) { int i; /* NB: Caller must abort pipe first. */ for (i = 0; i < BWFM_RX_LIST_COUNT; i++) { if (sc->sc_rx_data[i].xfer != NULL) usbd_destroy_xfer(sc->sc_rx_data[i].xfer); sc->sc_rx_data[i].xfer = NULL; } } int bwfm_usb_alloc_tx_list(struct bwfm_usb_softc *sc) { struct bwfm_usb_tx_data *data; int i, error = 0; TAILQ_INIT(&sc->sc_tx_free_list); for (i = 0; i < BWFM_TX_LIST_COUNT; i++) { data = &sc->sc_tx_data[i]; data->sc = sc; /* Backpointer for callbacks. */ if (usbd_create_xfer(sc->sc_tx_pipeh, BWFM_TXBUFSZ, USBD_FORCE_SHORT_XFER, 0, &data->xfer) != 0) { printf("%s: could not create xfer\n", DEVNAME(sc)); error = ENOMEM; break; } data->buf = usbd_get_buffer(data->xfer); /* Append this Tx buffer to our free list. */ TAILQ_INSERT_TAIL(&sc->sc_tx_free_list, data, next); } if (error != 0) bwfm_usb_free_tx_list(sc); return (error); } void bwfm_usb_free_tx_list(struct bwfm_usb_softc *sc) { int i; /* NB: Caller must abort pipe first. */ for (i = 0; i < BWFM_TX_LIST_COUNT; i++) { if (sc->sc_tx_data[i].xfer != NULL) usbd_destroy_xfer(sc->sc_tx_data[i].xfer); sc->sc_tx_data[i].xfer = NULL; } } void bwfm_usb_txeof(struct usbd_xfer *xfer, void *priv, usbd_status status) { struct bwfm_usb_tx_data *data = priv; struct bwfm_usb_softc *sc = data->sc; struct ifnet *ifp = sc->sc_sc.sc_ic.ic_ifp; int s; DPRINTFN(2, ("%s: %s status %s\n", DEVNAME(sc), __func__, usbd_errstr(status))); m_freem(data->mbuf); data->mbuf = NULL; mutex_enter(&sc->sc_tx_lock); /* Put this Tx buffer back to our free list. */ TAILQ_INSERT_TAIL(&sc->sc_tx_free_list, data, next); mutex_exit(&sc->sc_tx_lock); s = splnet(); if (__predict_false(status != USBD_NORMAL_COMPLETION)) { if (status == USBD_CANCELLED) usbd_clear_endpoint_stall_async(sc->sc_tx_pipeh); if_statinc(ifp, if_oerrors); splx(s); return; } if_statinc(ifp, if_opackets); /* We just released a Tx buffer, notify Tx. */ if ((ifp->if_flags & IFF_OACTIVE) != 0) { ifp->if_flags &= ~IFF_OACTIVE; if_schedule_deferred_start(ifp); } splx(s); } int bwfm_usb_detach(device_t self, int flags) { struct bwfm_usb_softc *sc = device_private(self); bwfm_detach(&sc->sc_sc, flags); if (sc->sc_rx_pipeh != NULL) { usbd_abort_pipe(sc->sc_rx_pipeh); usbd_close_pipe(sc->sc_rx_pipeh); } if (sc->sc_tx_pipeh != NULL) { usbd_abort_pipe(sc->sc_tx_pipeh); usbd_close_pipe(sc->sc_tx_pipeh); } bwfm_usb_free_rx_list(sc); bwfm_usb_free_tx_list(sc); mutex_destroy(&sc->sc_rx_lock); mutex_destroy(&sc->sc_tx_lock); return 0; } int bwfm_usb_dl_cmd(struct bwfm_usb_softc *sc, uByte cmd, void *buf, int len) { usb_device_request_t req; usbd_status error; req.bmRequestType = UT_READ_VENDOR_INTERFACE; req.bRequest = cmd; USETW(req.wValue, 0); USETW(req.wIndex, sc->sc_ifaceno); USETW(req.wLength, len); error = usbd_do_request(sc->sc_udev, &req, buf); if (error != 0) { printf("%s: could not read register: %s\n", DEVNAME(sc), usbd_errstr(error)); } return error; } int bwfm_usb_load_microcode(struct bwfm_usb_softc *sc, const u_char *ucode, size_t size) { const struct trx_header *trx = (const struct trx_header *)ucode; struct rdl_state state; uint32_t rdlstate, rdlbytes, sent = 0, sendlen = 0; struct usbd_xfer *xfer; usbd_status error; char *buf; if (le32toh(trx->magic) != TRX_MAGIC || (le32toh(trx->flag_version) & TRX_UNCOMP_IMAGE) == 0) { printf("%s: invalid firmware\n", DEVNAME(sc)); return 1; } bwfm_usb_dl_cmd(sc, DL_START, &state, sizeof(state)); rdlstate = le32toh(state.state); rdlbytes = le32toh(state.bytes); if (rdlstate != DL_WAITING) { printf("%s: cannot start fw download\n", DEVNAME(sc)); return 1; } error = usbd_create_xfer(sc->sc_tx_pipeh, TRX_RDL_CHUNK, 0, 0, &xfer); if (error != 0) { printf("%s: cannot create xfer\n", DEVNAME(sc)); goto err; } buf = usbd_get_buffer(xfer); while (rdlbytes != size) { sendlen = MIN(size - sent, TRX_RDL_CHUNK); memcpy(buf, ucode + sent, sendlen); usbd_setup_xfer(xfer, NULL, buf, sendlen, USBD_SYNCHRONOUS, USBD_NO_TIMEOUT, NULL); error = usbd_transfer(xfer); if (error != 0 && error != USBD_IN_PROGRESS) { printf("%s: transfer error\n", DEVNAME(sc)); goto err; } sent += sendlen; bwfm_usb_dl_cmd(sc, DL_GETSTATE, &state, sizeof(state)); rdlstate = le32toh(state.state); rdlbytes = le32toh(state.bytes); if (rdlbytes != sent) { printf("%s: device reported different size\n", DEVNAME(sc)); goto err; } if (rdlstate == DL_BAD_HDR || rdlstate == DL_BAD_CRC) { printf("%s: device reported bad hdr/crc\n", DEVNAME(sc)); goto err; } } bwfm_usb_dl_cmd(sc, DL_GETSTATE, &state, sizeof(state)); rdlstate = le32toh(state.state); rdlbytes = le32toh(state.bytes); if (rdlstate != DL_RUNNABLE) { printf("%s: dongle not runnable\n", DEVNAME(sc)); goto err; } bwfm_usb_dl_cmd(sc, DL_GO, &state, sizeof(state)); usbd_destroy_xfer(xfer); return 0; err: if (sc->sc_tx_pipeh != NULL) { usbd_abort_pipe(sc->sc_tx_pipeh); usbd_close_pipe(sc->sc_tx_pipeh); sc->sc_tx_pipeh = NULL; } if (xfer != NULL) usbd_destroy_xfer(xfer); return 1; } int bwfm_usb_txcheck(struct bwfm_softc *bwfm) { struct bwfm_usb_softc *sc = (void *)bwfm; mutex_enter(&sc->sc_tx_lock); if (TAILQ_EMPTY(&sc->sc_tx_free_list)) { mutex_exit(&sc->sc_tx_lock); return ENOBUFS; } mutex_exit(&sc->sc_tx_lock); return 0; } int bwfm_usb_txdata(struct bwfm_softc *bwfm, struct mbuf **mp) { struct bwfm_usb_softc *sc = (void *)bwfm; struct mbuf *m = *mp; struct bwfm_proto_bcdc_hdr *hdr; struct bwfm_usb_tx_data *data; struct ether_header *eh; uint32_t len = 0; int error, ac; DPRINTFN(2, ("%s: %s\n", DEVNAME(sc), __func__)); mutex_enter(&sc->sc_tx_lock); if (TAILQ_EMPTY(&sc->sc_tx_free_list)) { mutex_exit(&sc->sc_tx_lock); return ENOBUFS; } /* No QoS for EAPOL frames. */ eh = mtod(m, struct ether_header *); ac = (eh->ether_type != htons(ETHERTYPE_PAE)) ? M_WME_GETAC(m) : WME_AC_BE; /* Grab a Tx buffer from our free list. */ data = TAILQ_FIRST(&sc->sc_tx_free_list); TAILQ_REMOVE(&sc->sc_tx_free_list, data, next); mutex_exit(&sc->sc_tx_lock); hdr = (void *)&data->buf[len]; hdr->data_offset = 0; hdr->priority = ac; hdr->flags = BWFM_BCDC_FLAG_VER(BWFM_BCDC_FLAG_PROTO_VER); hdr->flags2 = 0; len += sizeof(*hdr); m_copydata(m, 0, m->m_pkthdr.len, &data->buf[len]); len += m->m_pkthdr.len; data->mbuf = m; usbd_setup_xfer(data->xfer, data, data->buf, len, USBD_FORCE_SHORT_XFER, USBD_NO_TIMEOUT, bwfm_usb_txeof); error = usbd_transfer(data->xfer); if (error != 0 && error != USBD_IN_PROGRESS) printf("%s: could not set up new transfer: %s\n", DEVNAME(sc), usbd_errstr(error)); return 0; } int bwfm_usb_txctl(struct bwfm_softc *bwfm, char *buf, size_t len) { struct bwfm_usb_softc *sc = (void *)bwfm; usb_device_request_t req; usbd_status error; int ret = 1; DPRINTFN(2, ("%s: %s\n", DEVNAME(sc), __func__)); req.bmRequestType = UT_WRITE_CLASS_INTERFACE; req.bRequest = 0; USETW(req.wValue, 0); USETW(req.wIndex, sc->sc_ifaceno); USETW(req.wLength, len); error = usbd_do_request(sc->sc_udev, &req, buf); if (error != 0) { printf("%s: could not read ctl packet: %s\n", DEVNAME(sc), usbd_errstr(error)); goto err; } ret = 0; err: return ret; } int bwfm_usb_rxctl(struct bwfm_softc *bwfm, char *buf, size_t *len) { struct bwfm_usb_softc *sc = (void *)bwfm; usb_device_request_t req; usbd_status error; uint32_t len32; int ret = 1; DPRINTFN(2, ("%s: %s\n", DEVNAME(sc), __func__)); req.bmRequestType = UT_READ_CLASS_INTERFACE; req.bRequest = 1; USETW(req.wValue, 0); USETW(req.wIndex, sc->sc_ifaceno); USETW(req.wLength, *len); error = usbd_do_request_flags(sc->sc_udev, &req, buf, 0, &len32, USBD_DEFAULT_TIMEOUT); if (error != 0) { printf("%s: could not read ctl packet: %s\n", DEVNAME(sc), usbd_errstr(error)); goto err; } if (len32 > *len) { printf("%s: broken length\n", DEVNAME(sc)); goto err; } *len = len32; ret = 0; err: return ret; } |
| 21 17 2 3 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 | /* $NetBSD: vnd_50.c,v 1.5 2019/12/12 02:15:42 pgoyette Exp $ */ /*- * Copyright (c) 1996, 1997, 1998, 2008 The NetBSD Foundation, Inc. * All rights reserved. * * This code is derived from software contributed to The NetBSD Foundation * by Jason R. Thorpe. * * Redistribution and use in source and binary forms, with or without * modification, are permitted provided that the following conditions * are met: * 1. Redistributions of source code must retain the above copyright * notice, this list of conditions and the following disclaimer. * 2. Redistributions in binary form must reproduce the above copyright * notice, this list of conditions and the following disclaimer in the * documentation and/or other materials provided with the distribution. * * THIS SOFTWARE IS PROVIDED BY THE NETBSD FOUNDATION, INC. AND CONTRIBUTORS * ``AS IS'' AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED * TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR * PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE FOUNDATION OR CONTRIBUTORS * BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR * CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF * SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS * INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN * CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) * ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE * POSSIBILITY OF SUCH DAMAGE. */ /* * Copyright (c) 1988 University of Utah. * Copyright (c) 1990, 1993 * The Regents of the University of California. All rights reserved. * * This code is derived from software contributed to Berkeley by * the Systems Programming Group of the University of Utah Computer * Science Department. * * Redistribution and use in source and binary forms, with or without * modification, are permitted provided that the following conditions * are met: * 1. Redistributions of source code must retain the above copyright * notice, this list of conditions and the following disclaimer. * 2. Redistributions in binary form must reproduce the above copyright * notice, this list of conditions and the following disclaimer in the * documentation and/or other materials provided with the distribution. * 3. Neither the name of the University nor the names of its contributors * may be used to endorse or promote products derived from this software * without specific prior written permission. * * THIS SOFTWARE IS PROVIDED BY THE REGENTS AND CONTRIBUTORS ``AS IS'' AND * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE * ARE DISCLAIMED. IN NO EVENT SHALL THE REGENTS OR CONTRIBUTORS BE LIABLE * FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL * DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS * OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) * HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT * LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY * OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF * SUCH DAMAGE. * * from: Utah $Hdr: vn.c 1.13 94/04/02$ * * @(#)vn.c 8.9 (Berkeley) 5/14/95 */ #include <sys/cdefs.h> __KERNEL_RCSID(0, "$NetBSD: vnd_50.c,v 1.5 2019/12/12 02:15:42 pgoyette Exp $"); #if defined(_KERNEL_OPT) #include "opt_compat_netbsd.h" #endif #include <sys/param.h> #include <sys/systm.h> #include <sys/proc.h> #include <sys/errno.h> #include <sys/malloc.h> #include <sys/ioctl.h> #include <sys/device.h> #include <sys/disk.h> #include <sys/stat.h> #include <sys/vnode.h> #include <sys/file.h> #include <sys/uio.h> #include <sys/conf.h> #include <sys/compat_stub.h> #include <net/zlib.h> #include <dev/vndvar.h> #include <compat/common/compat_mod.h> static int compat_50_vndioctl(u_long, struct lwp *, void *, int, struct vattr *, int (*)(struct lwp *, void *, int, struct vattr *)); static int compat_50_vndioctl(u_long cmd, struct lwp *l, void *data, int unit, struct vattr *vattr_p, int (*get)(struct lwp *, void *, int, struct vattr *)) { struct vnd_user50 *vnu = data; int error; if (cmd != VNDIOCGET50) return EPASSTHROUGH; error = (*get)(l, data, unit, vattr_p); if (error != 0) return error; vnu->vnu_dev = vattr_p->va_fsid; vnu->vnu_ino = vattr_p->va_fileid; return 0; } void vnd_50_init(void) { MODULE_HOOK_SET(compat_vndioctl_50_hook, compat_50_vndioctl); } void vnd_50_fini(void) { MODULE_HOOK_UNSET(compat_vndioctl_50_hook); } |
| 84 31 31 8 8 113 113 113 113 3 3 3 3 3 2 1 1 53 53 53 53 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 197 198 199 200 201 202 203 204 205 206 207 208 209 210 211 212 213 214 215 216 217 218 219 220 221 222 223 224 225 226 227 228 229 230 231 232 233 234 235 236 237 238 239 240 241 242 243 244 245 246 247 248 249 250 251 252 253 254 255 256 257 258 259 260 261 262 263 264 265 266 267 268 269 270 271 272 273 274 275 276 277 278 279 280 281 282 283 284 285 286 287 288 289 290 291 292 293 294 295 296 297 298 299 300 301 302 303 304 305 306 307 308 309 310 311 312 313 314 315 316 317 318 319 320 321 322 323 324 325 326 327 328 329 330 331 332 333 334 335 336 337 338 339 340 341 342 343 344 345 346 347 348 349 350 351 352 353 354 355 356 357 358 359 360 361 362 363 364 365 366 367 368 369 370 371 372 373 374 375 376 377 378 379 380 381 382 383 384 385 386 387 388 389 390 391 392 393 394 395 396 397 398 399 400 401 402 403 404 405 406 407 408 409 410 411 412 413 414 415 416 417 418 419 420 421 422 423 424 425 426 427 428 429 430 431 432 433 434 435 436 437 438 439 440 441 442 443 444 445 446 447 448 449 450 451 452 453 454 455 456 457 458 459 460 461 462 463 464 465 466 467 468 469 470 471 472 473 474 475 476 477 478 479 480 481 482 483 484 485 486 487 488 489 490 491 492 493 494 495 496 497 498 499 500 501 502 503 504 505 506 507 508 509 510 511 512 513 514 515 516 517 518 519 520 521 522 523 524 525 526 527 528 529 530 531 532 533 534 535 536 537 538 539 540 541 542 543 544 545 546 547 548 549 550 551 552 553 554 555 556 557 558 559 560 561 562 563 564 565 566 567 568 569 570 571 572 573 574 575 576 577 578 579 580 581 582 583 584 585 586 587 588 589 590 591 592 593 594 595 596 597 598 599 600 601 602 603 604 605 606 607 608 609 610 611 612 613 614 615 616 617 618 619 620 621 622 623 624 625 626 627 628 629 630 631 632 633 634 635 636 637 638 639 640 641 642 643 644 645 646 647 648 649 650 651 652 653 654 655 656 657 658 659 660 661 662 663 664 665 666 667 668 669 670 671 672 673 674 675 676 677 678 679 680 681 682 683 684 685 686 687 688 689 690 691 692 693 694 695 696 697 698 699 700 701 702 703 704 705 706 707 708 709 710 711 712 713 714 715 716 717 718 719 720 721 722 723 724 725 726 727 728 729 730 731 732 733 734 735 736 737 738 739 740 741 742 743 744 745 746 747 748 749 750 751 752 753 754 755 756 757 758 759 760 761 762 763 764 765 766 767 768 769 770 771 772 773 774 775 776 777 778 779 780 781 782 783 784 785 786 787 788 789 790 791 792 793 794 795 796 797 798 799 800 801 802 803 804 805 806 807 808 809 810 811 812 813 814 815 816 817 818 819 820 821 822 823 824 825 826 827 828 829 830 831 832 833 834 835 836 837 838 839 840 841 842 843 844 845 846 847 848 849 850 851 852 853 854 855 856 857 858 859 860 861 862 863 864 865 866 867 868 869 870 871 872 873 874 875 876 877 878 879 880 881 882 883 884 885 886 887 888 889 890 891 892 893 894 895 896 897 898 899 900 901 902 903 904 905 906 907 908 909 910 911 912 913 914 915 916 917 918 919 920 921 922 923 924 925 926 927 928 929 930 931 932 933 934 935 936 937 938 939 940 941 942 943 944 945 946 947 948 949 950 951 952 953 954 955 956 957 958 959 960 961 962 963 964 965 966 967 968 969 970 971 972 973 974 975 976 977 978 979 980 981 982 983 984 985 986 987 988 989 990 991 992 993 994 995 996 997 998 999 1000 1001 1002 1003 1004 1005 1006 1007 1008 1009 1010 1011 1012 1013 1014 1015 1016 1017 1018 1019 1020 1021 1022 1023 1024 1025 1026 1027 1028 1029 1030 1031 1032 1033 1034 1035 1036 1037 1038 1039 1040 1041 1042 1043 1044 1045 1046 1047 1048 1049 1050 1051 1052 1053 1054 1055 1056 1057 1058 1059 1060 1061 1062 1063 1064 1065 1066 1067 1068 1069 1070 1071 1072 1073 1074 1075 1076 1077 1078 1079 1080 1081 1082 1083 1084 1085 1086 1087 1088 1089 1090 1091 1092 1093 1094 1095 1096 1097 1098 1099 1100 1101 1102 1103 1104 1105 1106 1107 1108 1109 1110 1111 1112 1113 1114 1115 1116 1117 1118 1119 1120 1121 1122 1123 1124 1125 1126 1127 1128 1129 1130 1131 1132 1133 1134 1135 1136 1137 1138 1139 1140 1141 1142 1143 1144 1145 1146 1147 1148 1149 1150 1151 1152 1153 1154 1155 1156 1157 1158 1159 1160 1161 1162 1163 1164 1165 1166 1167 1168 1169 1170 1171 1172 1173 1174 1175 1176 1177 1178 1179 1180 1181 1182 1183 1184 1185 1186 1187 1188 1189 1190 1191 1192 1193 1194 1195 1196 1197 1198 1199 1200 1201 1202 1203 1204 1205 1206 1207 1208 1209 1210 1211 1212 1213 1214 1215 1216 1217 1218 1219 1220 1221 1222 1223 1224 1225 1226 1227 1228 1229 1230 1231 1232 1233 1234 1235 1236 1237 1238 1239 1240 1241 1242 1243 1244 1245 1246 1247 1248 1249 1250 1251 1252 1253 1254 1255 1256 1257 1258 1259 1260 1261 1262 1263 1264 1265 1266 1267 1268 1269 1270 1271 1272 1273 1274 1275 1276 1277 1278 1279 1280 1281 1282 1283 1284 1285 1286 1287 1288 1289 1290 1291 1292 1293 1294 1295 1296 1297 1298 1299 1300 1301 1302 1303 1304 1305 1306 1307 1308 1309 1310 1311 1312 1313 1314 1315 1316 1317 1318 1319 1320 1321 1322 1323 1324 1325 1326 1327 1328 1329 1330 1331 1332 1333 1334 1335 1336 1337 1338 1339 1340 1341 1342 1343 1344 1345 1346 1347 1348 1349 1350 1351 1352 1353 1354 1355 1356 1357 1358 1359 1360 1361 1362 1363 1364 1365 1366 1367 1368 1369 1370 1371 1372 1373 1374 1375 1376 1377 1378 1379 1380 1381 1382 1383 1384 1385 1386 1387 1388 1389 1390 1391 1392 1393 1394 1395 1396 1397 1398 1399 1400 1401 1402 1403 1404 1405 1406 1407 1408 1409 1410 1411 1412 1413 1414 1415 1416 1417 1418 1419 1420 1421 1422 1423 1424 1425 1426 1427 1428 1429 1430 1431 1432 1433 1434 1435 1436 1437 1438 1439 1440 1441 1442 1443 1444 1445 1446 1447 1448 1449 1450 1451 1452 1453 1454 1455 1456 1457 1458 1459 1460 1461 1462 1463 1464 1465 1466 1467 1468 1469 1470 1471 1472 1473 1474 1475 1476 1477 1478 1479 1480 1481 1482 1483 1484 1485 1486 1487 1488 1489 1490 1491 1492 1493 1494 1495 1496 1497 1498 1499 1500 1501 1502 1503 1504 1505 1506 1507 1508 1509 1510 1511 1512 1513 1514 1515 1516 1517 1518 1519 1520 1521 1522 1523 1524 1525 1526 1527 1528 1529 1530 1531 1532 1533 1534 1535 1536 1537 1538 1539 1540 1541 1542 1543 1544 1545 1546 1547 1548 1549 1550 1551 1552 1553 1554 1555 1556 1557 1558 1559 1560 1561 1562 1563 1564 1565 1566 1567 1568 1569 1570 1571 1572 1573 1574 1575 1576 1577 1578 1579 1580 1581 1582 1583 1584 1585 1586 1587 1588 1589 1590 1591 1592 1593 1594 1595 1596 1597 1598 1599 1600 1601 1602 1603 1604 1605 1606 1607 1608 1609 1610 1611 1612 1613 1614 1615 1616 1617 1618 1619 1620 1621 1622 1623 1624 1625 1626 1627 1628 1629 1630 1631 1632 1633 1634 1635 1636 1637 1638 1639 1640 1641 1642 1643 1644 1645 1646 1647 1648 1649 1650 1651 1652 1653 1654 1655 1656 1657 1658 1659 1660 1661 1662 1663 1664 1665 1666 1667 1668 1669 1670 1671 1672 1673 1674 1675 1676 1677 1678 1679 1680 1681 1682 1683 1684 1685 1686 1687 1688 1689 1690 1691 1692 1693 1694 1695 1696 1697 1698 1699 1700 1701 1702 1703 1704 1705 1706 1707 1708 1709 1710 1711 1712 1713 1714 1715 1716 1717 1718 1719 1720 1721 1722 1723 1724 1725 1726 1727 1728 1729 1730 1731 1732 1733 1734 1735 1736 1737 1738 1739 1740 1741 1742 1743 1744 1745 1746 1747 1748 1749 1750 1751 1752 1753 1754 1755 1756 1757 1758 1759 1760 1761 1762 1763 1764 1765 1766 1767 1768 1769 1770 1771 1772 1773 1774 1775 1776 1777 1778 1779 1780 1781 1782 1783 1784 1785 1786 1787 1788 1789 1790 1791 1792 1793 1794 1795 1796 1797 1798 1799 1800 1801 1802 1803 1804 1805 1806 1807 1808 1809 1810 1811 1812 1813 1814 1815 1816 1817 1818 1819 1820 1821 1822 1823 1824 1825 1826 1827 1828 1829 1830 1831 1832 1833 1834 1835 1836 1837 1838 1839 1840 1841 1842 1843 1844 1845 1846 1847 1848 1849 1850 1851 1852 1853 1854 1855 1856 1857 1858 1859 1860 1861 1862 1863 1864 1865 1866 1867 1868 1869 1870 1871 1872 1873 1874 1875 1876 1877 1878 1879 1880 1881 1882 1883 1884 1885 1886 1887 1888 1889 1890 1891 1892 1893 1894 1895 1896 1897 1898 1899 1900 1901 1902 1903 1904 1905 1906 1907 1908 1909 1910 1911 1912 1913 1914 1915 1916 1917 1918 1919 1920 1921 1922 1923 1924 1925 1926 1927 1928 1929 1930 1931 1932 1933 1934 1935 1936 1937 1938 1939 1940 1941 | /* $NetBSD: ipsec.c,v 1.173 2021/12/08 20:03:26 andvar Exp $ */ /* $FreeBSD: ipsec.c,v 1.2.2.2 2003/07/01 01:38:13 sam Exp $ */ /* $KAME: ipsec.c,v 1.103 2001/05/24 07:14:18 sakane Exp $ */ /* * Copyright (C) 1995, 1996, 1997, and 1998 WIDE Project. * All rights reserved. * * Redistribution and use in source and binary forms, with or without * modification, are permitted provided that the following conditions * are met: * 1. Redistributions of source code must retain the above copyright * notice, this list of conditions and the following disclaimer. * 2. Redistributions in binary form must reproduce the above copyright * notice, this list of conditions and the following disclaimer in the * documentation and/or other materials provided with the distribution. * 3. Neither the name of the project nor the names of its contributors * may be used to endorse or promote products derived from this software * without specific prior written permission. * * THIS SOFTWARE IS PROVIDED BY THE PROJECT AND CONTRIBUTORS ``AS IS'' AND * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE * ARE DISCLAIMED. IN NO EVENT SHALL THE PROJECT OR CONTRIBUTORS BE LIABLE * FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL * DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS * OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) * HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT * LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY * OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF * SUCH DAMAGE. */ #include <sys/cdefs.h> __KERNEL_RCSID(0, "$NetBSD: ipsec.c,v 1.173 2021/12/08 20:03:26 andvar Exp $"); /* * IPsec controller part. */ #if defined(_KERNEL_OPT) #include "opt_inet.h" #include "opt_ipsec.h" #endif #include <sys/param.h> #include <sys/systm.h> #include <sys/mbuf.h> #include <sys/domain.h> #include <sys/protosw.h> #include <sys/socket.h> #include <sys/socketvar.h> #include <sys/errno.h> #include <sys/time.h> #include <sys/kernel.h> #include <sys/syslog.h> #include <sys/sysctl.h> #include <sys/proc.h> #include <sys/kauth.h> #include <sys/cpu.h> #include <sys/kmem.h> #include <sys/pserialize.h> #include <net/if.h> #include <net/route.h> #include <netinet/in.h> #include <netinet/in_systm.h> #include <netinet/ip.h> #include <netinet/ip_var.h> #include <netinet/in_var.h> #include <netinet/udp.h> #include <netinet/udp_var.h> #include <netinet/tcp.h> #include <netinet/udp.h> #include <netinet/ip_icmp.h> #include <netinet/ip_private.h> #include <netinet/ip6.h> #ifdef INET6 #include <netinet6/ip6_var.h> #endif #include <netinet/in_pcb.h> #include <netinet/in_offload.h> #ifdef INET6 #include <netinet6/in6_pcb.h> #include <netinet/icmp6.h> #endif #include <netipsec/ipsec.h> #include <netipsec/ipsec_var.h> #include <netipsec/ipsec_private.h> #ifdef INET6 #include <netipsec/ipsec6.h> #endif #include <netipsec/ah_var.h> #include <netipsec/esp_var.h> #include <netipsec/ipcomp.h> /*XXX*/ #include <netipsec/ipcomp_var.h> #include <netipsec/key.h> #include <netipsec/keydb.h> #include <netipsec/key_debug.h> #include <netipsec/xform.h> int ipsec_used = 0; int ipsec_enabled = 1; #ifdef IPSEC_DEBUG int ipsec_debug = 1; /* * When set to 1, IPsec will send packets with the same sequence number. * This allows to verify if the other side has proper replay attacks detection. */ int ipsec_replay = 0; /* * When set 1, IPsec will send packets with corrupted HMAC. * This allows to verify if the other side properly detects modified packets. */ int ipsec_integrity = 0; #else int ipsec_debug = 0; #endif percpu_t *ipsecstat_percpu; int ip4_ah_offsetmask = 0; /* maybe IP_DF? */ int ip4_ipsec_dfbit = 2; /* DF bit on encap. 0: clear 1: set 2: copy */ int ip4_esp_trans_deflev = IPSEC_LEVEL_USE; int ip4_esp_net_deflev = IPSEC_LEVEL_USE; int ip4_ah_trans_deflev = IPSEC_LEVEL_USE; int ip4_ah_net_deflev = IPSEC_LEVEL_USE; struct secpolicy ip4_def_policy; int ip4_ipsec_ecn = 0; /* ECN ignore(-1)/forbidden(0)/allowed(1) */ u_int ipsec_spdgen = 1; /* SPD generation # */ static struct secpolicy ipsec_dummy_sp __read_mostly = { .state = IPSEC_SPSTATE_ALIVE, /* If ENTRUST, the dummy SP never be used. See ipsec_getpolicybysock. */ .policy = IPSEC_POLICY_ENTRUST, }; static struct secpolicy *ipsec_checkpcbcache(struct mbuf *, struct inpcbpolicy *, int); static int ipsec_fillpcbcache(struct inpcbpolicy *, struct mbuf *, struct secpolicy *, int); static int ipsec_invalpcbcache(struct inpcbpolicy *, int); /* * Crypto support requirements: * * 1 require hardware support * -1 require software support * 0 take anything */ int crypto_support = 0; static struct secpolicy *ipsec_getpolicybysock(struct mbuf *, u_int, struct inpcb_hdr *, int *); #ifdef INET6 int ip6_esp_trans_deflev = IPSEC_LEVEL_USE; int ip6_esp_net_deflev = IPSEC_LEVEL_USE; int ip6_ah_trans_deflev = IPSEC_LEVEL_USE; int ip6_ah_net_deflev = IPSEC_LEVEL_USE; struct secpolicy ip6_def_policy; int ip6_ipsec_ecn = 0; /* ECN ignore(-1)/forbidden(0)/allowed(1) */ #endif static int ipsec_setspidx_inpcb(struct mbuf *, void *); static int ipsec_setspidx(struct mbuf *, struct secpolicyindex *, int, int); static void ipsec4_get_ulp(struct mbuf *m, struct secpolicyindex *, int); static int ipsec4_setspidx_ipaddr(struct mbuf *, struct secpolicyindex *); #ifdef INET6 static void ipsec6_get_ulp(struct mbuf *m, struct secpolicyindex *, int); static int ipsec6_setspidx_ipaddr(struct mbuf *, struct secpolicyindex *); #endif static void ipsec_delpcbpolicy(struct inpcbpolicy *); static void ipsec_destroy_policy(struct secpolicy *); static int ipsec_sp_reject(const struct secpolicy *, const struct mbuf *); static void vshiftl(unsigned char *, int, int); static size_t ipsec_sp_hdrsiz(const struct secpolicy *, const struct mbuf *); /* * Try to validate and use cached policy on a PCB. */ static struct secpolicy * ipsec_checkpcbcache(struct mbuf *m, struct inpcbpolicy *pcbsp, int dir) { struct secpolicyindex spidx; struct secpolicy *sp = NULL; int s; KASSERT(IPSEC_DIR_IS_VALID(dir)); KASSERT(pcbsp != NULL); KASSERT(dir < __arraycount(pcbsp->sp_cache)); KASSERT(inph_locked(pcbsp->sp_inph)); /* * Checking the generation and sp->state and taking a reference to an SP * must be in a critical section of pserialize. See key_unlink_sp. */ s = pserialize_read_enter(); /* SPD table change invalidate all the caches. */ if (ipsec_spdgen != pcbsp->sp_cache[dir].cachegen) { ipsec_invalpcbcache(pcbsp, dir); goto out; } sp = pcbsp->sp_cache[dir].cachesp; if (sp == NULL) goto out; if (sp->state != IPSEC_SPSTATE_ALIVE) { sp = NULL; ipsec_invalpcbcache(pcbsp, dir); goto out; } if ((pcbsp->sp_cacheflags & IPSEC_PCBSP_CONNECTED) == 0) { /* NB: assume ipsec_setspidx never sleep */ if (ipsec_setspidx(m, &spidx, dir, 1) != 0) { sp = NULL; goto out; } /* * We have to make an exact match here since the cached rule * might have lower priority than a rule that would otherwise * have matched the packet. */ if (memcmp(&pcbsp->sp_cache[dir].cacheidx, &spidx, sizeof(spidx))) { sp = NULL; goto out; } } else { /* * The pcb is connected, and the L4 code is sure that: * - outgoing side uses inp_[lf]addr * - incoming side looks up policy after inpcb lookup * and address pair is know to be stable. We do not need * to generate spidx again, nor check the address match again. * * For IPv4/v6 SOCK_STREAM sockets, this assumptions holds * and there are calls to ipsec_pcbconn() from in_pcbconnect(). */ } sp->lastused = time_second; KEY_SP_REF(sp); KEYDEBUG_PRINTF(KEYDEBUG_IPSEC_STAMP, "DP cause refcnt++:%d SP:%p\n", key_sp_refcnt(sp), pcbsp->sp_cache[dir].cachesp); out: pserialize_read_exit(s); return sp; } static int ipsec_fillpcbcache(struct inpcbpolicy *pcbsp, struct mbuf *m, struct secpolicy *sp, int dir) { KASSERT(IPSEC_DIR_IS_INOROUT(dir)); KASSERT(dir < __arraycount(pcbsp->sp_cache)); KASSERT(inph_locked(pcbsp->sp_inph)); pcbsp->sp_cache[dir].cachesp = NULL; pcbsp->sp_cache[dir].cachehint = IPSEC_PCBHINT_UNKNOWN; if (ipsec_setspidx(m, &pcbsp->sp_cache[dir].cacheidx, dir, 1) != 0) { return EINVAL; } pcbsp->sp_cache[dir].cachesp = sp; if (pcbsp->sp_cache[dir].cachesp) { /* * If the PCB is connected, we can remember a hint to * possibly short-circuit IPsec processing in other places. */ if (pcbsp->sp_cacheflags & IPSEC_PCBSP_CONNECTED) { switch (pcbsp->sp_cache[dir].cachesp->policy) { case IPSEC_POLICY_NONE: case IPSEC_POLICY_BYPASS: pcbsp->sp_cache[dir].cachehint = IPSEC_PCBHINT_NO; break; default: pcbsp->sp_cache[dir].cachehint = IPSEC_PCBHINT_YES; } } } pcbsp->sp_cache[dir].cachegen = ipsec_spdgen; return 0; } static int ipsec_invalpcbcache(struct inpcbpolicy *pcbsp, int dir) { int i; KASSERT(inph_locked(pcbsp->sp_inph)); for (i = IPSEC_DIR_INBOUND; i <= IPSEC_DIR_OUTBOUND; i++) { if (dir != IPSEC_DIR_ANY && i != dir) continue; pcbsp->sp_cache[i].cachesp = NULL; pcbsp->sp_cache[i].cachehint = IPSEC_PCBHINT_UNKNOWN; pcbsp->sp_cache[i].cachegen = 0; memset(&pcbsp->sp_cache[i].cacheidx, 0, sizeof(pcbsp->sp_cache[i].cacheidx)); } return 0; } void ipsec_pcbconn(struct inpcbpolicy *pcbsp) { KASSERT(inph_locked(pcbsp->sp_inph)); pcbsp->sp_cacheflags |= IPSEC_PCBSP_CONNECTED; ipsec_invalpcbcache(pcbsp, IPSEC_DIR_ANY); } void ipsec_pcbdisconn(struct inpcbpolicy *pcbsp) { KASSERT(inph_locked(pcbsp->sp_inph)); pcbsp->sp_cacheflags &= ~IPSEC_PCBSP_CONNECTED; ipsec_invalpcbcache(pcbsp, IPSEC_DIR_ANY); } void ipsec_invalpcbcacheall(void) { if (ipsec_spdgen == UINT_MAX) ipsec_spdgen = 1; else ipsec_spdgen++; } /* * Return a held reference to the default SP. */ static struct secpolicy * key_get_default_sp(int af, const char *where, int tag) { struct secpolicy *sp; KEYDEBUG_PRINTF(KEYDEBUG_IPSEC_STAMP, "DP from %s:%u\n", where, tag); switch(af) { case AF_INET: sp = &ip4_def_policy; break; #ifdef INET6 case AF_INET6: sp = &ip6_def_policy; break; #endif default: KEYDEBUG_PRINTF(KEYDEBUG_IPSEC_STAMP, "unexpected protocol family %u\n", af); return NULL; } if (sp->policy != IPSEC_POLICY_DISCARD && sp->policy != IPSEC_POLICY_NONE) { IPSECLOG(LOG_INFO, "fixed system default policy: %d->%d\n", sp->policy, IPSEC_POLICY_NONE); sp->policy = IPSEC_POLICY_NONE; } KEY_SP_REF(sp); KEYDEBUG_PRINTF(KEYDEBUG_IPSEC_STAMP, "DP returns SP:%p (%u)\n", sp, key_sp_refcnt(sp)); return sp; } #define KEY_GET_DEFAULT_SP(af) \ key_get_default_sp((af), __func__, __LINE__) /* * For OUTBOUND packet having a socket. Searching SPD for packet, * and return a pointer to SP. * OUT: NULL: no appropriate SP found, the following value is set to error. * 0 : bypass * EACCES : discard packet. * ENOENT : ipsec_acquire() in progress, maybe. * others : error occurred. * others: a pointer to SP * * NOTE: IPv6 mapped address concern is implemented here. */ static struct secpolicy * ipsec_getpolicybysock(struct mbuf *m, u_int dir, struct inpcb_hdr *inph, int *error) { struct inpcbpolicy *pcbsp = NULL; struct secpolicy *currsp = NULL; /* policy on socket */ struct secpolicy *sp; int af; KASSERT(m != NULL); KASSERT(inph != NULL); KASSERT(error != NULL); KASSERTMSG(IPSEC_DIR_IS_INOROUT(dir), "invalid direction %u", dir); KASSERT(inph->inph_socket != NULL); KASSERT(inph_locked(inph)); /* XXX FIXME inpcb/in6pcb vs socket*/ af = inph->inph_af; KASSERTMSG(af == AF_INET || af == AF_INET6, "unexpected protocol family %u", af); KASSERT(inph->inph_sp != NULL); /* If we have a cached entry, and if it is still valid, use it. */ IPSEC_STATINC(IPSEC_STAT_SPDCACHELOOKUP); currsp = ipsec_checkpcbcache(m, inph->inph_sp, dir); if (currsp) { *error = 0; return currsp; } IPSEC_STATINC(IPSEC_STAT_SPDCACHEMISS); switch (af) { case AF_INET: #if defined(INET6) case AF_INET6: #endif *error = ipsec_setspidx_inpcb(m, inph); pcbsp = inph->inph_sp; break; default: *error = EPFNOSUPPORT; break; } if (*error) return NULL; KASSERT(pcbsp != NULL); switch (dir) { case IPSEC_DIR_INBOUND: currsp = pcbsp->sp_in; break; case IPSEC_DIR_OUTBOUND: currsp = pcbsp->sp_out; break; } KASSERT(currsp != NULL); if (pcbsp->priv) { /* when privileged socket */ switch (currsp->policy) { case IPSEC_POLICY_BYPASS: case IPSEC_POLICY_IPSEC: KEY_SP_REF(currsp); sp = currsp; break; case IPSEC_POLICY_ENTRUST: /* look for a policy in SPD */ if (key_havesp(dir)) sp = KEY_LOOKUP_SP_BYSPIDX(&currsp->spidx, dir); else sp = NULL; if (sp == NULL) /* no SP found */ sp = KEY_GET_DEFAULT_SP(af); break; default: IPSECLOG(LOG_ERR, "Invalid policy for PCB %d\n", currsp->policy); *error = EINVAL; return NULL; } } else { /* unpriv, SPD has policy */ if (key_havesp(dir)) sp = KEY_LOOKUP_SP_BYSPIDX(&currsp->spidx, dir); else sp = NULL; if (sp == NULL) { /* no SP found */ switch (currsp->policy) { case IPSEC_POLICY_BYPASS: IPSECLOG(LOG_ERR, "Illegal policy for " "non-priviliged defined %d\n", currsp->policy); *error = EINVAL; return NULL; case IPSEC_POLICY_ENTRUST: sp = KEY_GET_DEFAULT_SP(af); break; case IPSEC_POLICY_IPSEC: KEY_SP_REF(currsp); sp = currsp; break; default: IPSECLOG(LOG_ERR, "Invalid policy for " "PCB %d\n", currsp->policy); *error = EINVAL; return NULL; } } } KASSERTMSG(sp != NULL, "null SP (priv %u policy %u", pcbsp->priv, currsp->policy); KEYDEBUG_PRINTF(KEYDEBUG_IPSEC_STAMP, "DP (priv %u policy %u) allocates SP:%p (refcnt %u)\n", pcbsp->priv, currsp->policy, sp, key_sp_refcnt(sp)); ipsec_fillpcbcache(pcbsp, m, sp, dir); return sp; } /* * For FORWARDING packet or OUTBOUND without a socket. Searching SPD for packet, * and return a pointer to SP. * OUT: positive: a pointer to the entry for security policy leaf matched. * NULL: no appropriate SP found, the following value is set to error. * 0 : bypass * EACCES : discard packet. * ENOENT : ipsec_acquire() in progress, maybe. * others : error occurred. */ static struct secpolicy * ipsec_getpolicybyaddr(struct mbuf *m, u_int dir, int flag, int *error) { struct secpolicyindex spidx; struct secpolicy *sp; KASSERT(m != NULL); KASSERT(error != NULL); KASSERTMSG(IPSEC_DIR_IS_INOROUT(dir), "invalid direction %u", dir); sp = NULL; /* Make an index to look for a policy. */ *error = ipsec_setspidx(m, &spidx, dir, (flag & IP_FORWARDING) ? 0 : 1); if (*error != 0) { IPSECLOG(LOG_DEBUG, "setpidx failed, dir %u flag %u\n", dir, flag); memset(&spidx, 0, sizeof(spidx)); return NULL; } spidx.dir = dir; if (key_havesp(dir)) { sp = KEY_LOOKUP_SP_BYSPIDX(&spidx, dir); } if (sp == NULL) { /* no SP found, use system default */ sp = KEY_GET_DEFAULT_SP(spidx.dst.sa.sa_family); } KASSERT(sp != NULL); return sp; } static struct secpolicy * ipsec_checkpolicy(struct mbuf *m, u_int dir, u_int flag, int *error, void *inp) { struct secpolicy *sp; *error = 0; if (inp == NULL) { sp = ipsec_getpolicybyaddr(m, dir, flag, error); } else { struct inpcb_hdr *inph = (struct inpcb_hdr *)inp; KASSERT(inph->inph_socket != NULL); sp = ipsec_getpolicybysock(m, dir, inph, error); } if (sp == NULL) { KASSERTMSG(*error != 0, "getpolicy failed w/o error"); IPSEC_STATINC(IPSEC_STAT_OUT_INVAL); return NULL; } KASSERTMSG(*error == 0, "sp w/ error set to %u", *error); switch (sp->policy) { case IPSEC_POLICY_ENTRUST: default: printf("%s: invalid policy %u\n", __func__, sp->policy); /* fall thru... */ case IPSEC_POLICY_DISCARD: IPSEC_STATINC(IPSEC_STAT_OUT_POLVIO); *error = -EINVAL; /* packet is discarded by caller */ break; case IPSEC_POLICY_BYPASS: case IPSEC_POLICY_NONE: KEY_SP_UNREF(&sp); sp = NULL; /* NB: force NULL result */ break; case IPSEC_POLICY_IPSEC: KASSERT(sp->req != NULL); break; } if (*error != 0) { KEY_SP_UNREF(&sp); sp = NULL; IPSECLOG(LOG_DEBUG, "done, error %d\n", *error); } return sp; } int ipsec4_output(struct mbuf *m, struct inpcb *inp, int flags, u_long *mtu, bool *natt_frag, bool *done, bool *count_drop) { struct secpolicy *sp = NULL; u_long _mtu = 0; int error, s; /* * Check the security policy (SP) for the packet and, if required, * do IPsec-related processing. There are two cases here; the first * time a packet is sent through it will be untagged and handled by * ipsec_checkpolicy(). If the packet is resubmitted to ip_output * (e.g. after AH, ESP, etc. processing), there will be a tag to * bypass the lookup and related policy checking. */ if (ipsec_outdone(m)) { return 0; } s = splsoftnet(); if (inp && ipsec_pcb_skip_ipsec(inp->inp_sp, IPSEC_DIR_OUTBOUND)) { splx(s); return 0; } sp = ipsec_checkpolicy(m, IPSEC_DIR_OUTBOUND, flags, &error, inp); /* * There are four return cases: * sp != NULL apply IPsec policy * sp == NULL, error == 0 no IPsec handling needed * sp == NULL, error == -EINVAL discard packet w/o error * sp == NULL, error != 0 discard packet, report error */ if (sp == NULL) { splx(s); if (error) { /* * Hack: -EINVAL is used to signal that a packet * should be silently discarded. This is typically * because we asked key management for an SA and * it was delayed (e.g. kicked up to IKE). */ if (error == -EINVAL) error = 0; m_freem(m); *done = true; *count_drop = true; return error; } /* No IPsec processing for this packet. */ return 0; } /* * Do delayed checksums now because we send before * this is done in the normal processing path. */ if (m->m_pkthdr.csum_flags & (M_CSUM_TCPv4|M_CSUM_UDPv4)) { in_undefer_cksum_tcpudp(m); m->m_pkthdr.csum_flags &= ~(M_CSUM_TCPv4|M_CSUM_UDPv4); } error = ipsec4_process_packet(m, sp->req, &_mtu); if (error == 0 && _mtu != 0) { /* * NAT-T ESP fragmentation: do not do IPSec processing * now, we will do it on each fragmented packet. */ *mtu = _mtu; *natt_frag = true; KEY_SP_UNREF(&sp); splx(s); return 0; } /* * Preserve KAME behaviour: ENOENT can be returned * when an SA acquire is in progress. Don't propagate * this to user-level; it confuses applications. * * XXX this will go away when the SADB is redone. */ if (error == ENOENT) error = 0; KEY_SP_UNREF(&sp); splx(s); *done = true; return error; } int ipsec_ip_input_checkpolicy(struct mbuf *m, bool forward) { struct secpolicy *sp; int error, s; s = splsoftnet(); error = ipsec_in_reject(m, NULL); splx(s); if (error) { return EINVAL; } if (!forward || !(m->m_flags & M_CANFASTFWD)) { return 0; } /* * Peek at the outbound SP for this packet to determine if * it is a Fast Forward candidate. */ s = splsoftnet(); sp = ipsec_checkpolicy(m, IPSEC_DIR_OUTBOUND, IP_FORWARDING, &error, NULL); if (sp != NULL) { m->m_flags &= ~M_CANFASTFWD; KEY_SP_UNREF(&sp); } splx(s); return 0; } /* * If the packet is routed over IPsec tunnel, tell the originator the * tunnel MTU. * tunnel MTU = if MTU - sizeof(IP) - ESP/AH hdrsiz * * XXX: Quick hack!!! * * XXX: And what if the MTU goes negative? */ void ipsec_mtu(struct mbuf *m, int *destmtu) { struct secpolicy *sp; size_t ipsechdr; int error; sp = ipsec_getpolicybyaddr(m, IPSEC_DIR_OUTBOUND, IP_FORWARDING, &error); if (sp == NULL) { return; } /* Count IPsec header size. */ ipsechdr = ipsec_sp_hdrsiz(sp, m); /* * Find the correct route for outer IP header, compute tunnel MTU. */ if (sp->req) { struct secasvar *sav; sav = ipsec_lookup_sa(sp->req, m); if (sav != NULL) { struct route *ro; struct rtentry *rt; ro = &sav->sah->sa_route; rt = rtcache_validate(ro); if (rt && rt->rt_ifp) { *destmtu = rt->rt_rmx.rmx_mtu ? rt->rt_rmx.rmx_mtu : rt->rt_ifp->if_mtu; *destmtu -= ipsechdr; } rtcache_unref(rt, ro); KEY_SA_UNREF(&sav); } } KEY_SP_UNREF(&sp); } static int ipsec_setspidx_inpcb(struct mbuf *m, void *pcb) { struct inpcb_hdr *inph = (struct inpcb_hdr *)pcb; int error; KASSERT(inph != NULL); KASSERT(inph->inph_sp != NULL); KASSERT(inph->inph_sp->sp_out != NULL); KASSERT(inph->inph_sp->sp_in != NULL); error = ipsec_setspidx(m, &inph->inph_sp->sp_in->spidx, IPSEC_DIR_INBOUND, 1); if (error == 0) { inph->inph_sp->sp_out->spidx = inph->inph_sp->sp_in->spidx; inph->inph_sp->sp_out->spidx.dir = IPSEC_DIR_OUTBOUND; } else { memset(&inph->inph_sp->sp_in->spidx, 0, sizeof(inph->inph_sp->sp_in->spidx)); memset(&inph->inph_sp->sp_out->spidx, 0, sizeof(inph->inph_sp->sp_out->spidx)); } return error; } /* * configure security policy index (src/dst/proto/sport/dport) * by looking at the content of mbuf. * the caller is responsible for error recovery (like clearing up spidx). */ static int ipsec_setspidx(struct mbuf *m, struct secpolicyindex *spidx, int dir, int needport) { struct ip *ip = NULL; struct ip ipbuf; u_int v; int error; KASSERT(m != NULL); M_VERIFY_PACKET(m); if (m->m_pkthdr.len < sizeof(struct ip)) { KEYDEBUG_PRINTF(KEYDEBUG_IPSEC_DUMP, "pkthdr.len(%d) < sizeof(struct ip), ignored.\n", m->m_pkthdr.len); return EINVAL; } memset(spidx, 0, sizeof(*spidx)); spidx->dir = dir; if (m->m_len >= sizeof(*ip)) { ip = mtod(m, struct ip *); } else { m_copydata(m, 0, sizeof(ipbuf), &ipbuf); ip = &ipbuf; } v = ip->ip_v; switch (v) { case 4: error = ipsec4_setspidx_ipaddr(m, spidx); if (error) return error; ipsec4_get_ulp(m, spidx, needport); return 0; #ifdef INET6 case 6: if (m->m_pkthdr.len < sizeof(struct ip6_hdr)) { KEYDEBUG_PRINTF(KEYDEBUG_IPSEC_DUMP, "pkthdr.len(%d) < sizeof(struct ip6_hdr), " "ignored.\n", m->m_pkthdr.len); return EINVAL; } error = ipsec6_setspidx_ipaddr(m, spidx); if (error) return error; ipsec6_get_ulp(m, spidx, needport); return 0; #endif default: KEYDEBUG_PRINTF(KEYDEBUG_IPSEC_DUMP, "unknown IP version %u, ignored.\n", v); return EINVAL; } } static void ipsec4_get_ulp(struct mbuf *m, struct secpolicyindex *spidx, int needport) { u_int8_t nxt; int off; KASSERT(m != NULL); KASSERTMSG(m->m_pkthdr.len >= sizeof(struct ip), "packet too short"); /* NB: ip_input() flips it into host endian XXX need more checking */ if (m->m_len >= sizeof(struct ip)) { struct ip *ip = mtod(m, struct ip *); if (ip->ip_off & htons(IP_MF | IP_OFFMASK)) goto done; off = ip->ip_hl << 2; nxt = ip->ip_p; } else { struct ip ih; m_copydata(m, 0, sizeof(struct ip), &ih); if (ih.ip_off & htons(IP_MF | IP_OFFMASK)) goto done; off = ih.ip_hl << 2; nxt = ih.ip_p; } while (off < m->m_pkthdr.len) { struct ip6_ext ip6e; struct tcphdr th; struct udphdr uh; struct icmp icmph; switch (nxt) { case IPPROTO_TCP: spidx->ul_proto = nxt; if (!needport) goto done_proto; if (off + sizeof(struct tcphdr) > m->m_pkthdr.len) goto done; m_copydata(m, off, sizeof(th), &th); spidx->src.sin.sin_port = th.th_sport; spidx->dst.sin.sin_port = th.th_dport; return; case IPPROTO_UDP: spidx->ul_proto = nxt; if (!needport) goto done_proto; if (off + sizeof(struct udphdr) > m->m_pkthdr.len) goto done; m_copydata(m, off, sizeof(uh), &uh); spidx->src.sin.sin_port = uh.uh_sport; spidx->dst.sin.sin_port = uh.uh_dport; return; case IPPROTO_AH: if (off + sizeof(ip6e) > m->m_pkthdr.len) goto done; /* XXX sigh, this works but is totally bogus */ m_copydata(m, off, sizeof(ip6e), &ip6e); off += (ip6e.ip6e_len + 2) << 2; nxt = ip6e.ip6e_nxt; break; case IPPROTO_ICMP: spidx->ul_proto = nxt; if (off + sizeof(struct icmp) > m->m_pkthdr.len) goto done; m_copydata(m, off, sizeof(icmph), &icmph); ((struct sockaddr_in *)&spidx->src)->sin_port = htons((uint16_t)icmph.icmp_type); ((struct sockaddr_in *)&spidx->dst)->sin_port = htons((uint16_t)icmph.icmp_code); return; default: /* XXX intermediate headers??? */ spidx->ul_proto = nxt; goto done_proto; } } done: spidx->ul_proto = IPSEC_ULPROTO_ANY; done_proto: spidx->src.sin.sin_port = IPSEC_PORT_ANY; spidx->dst.sin.sin_port = IPSEC_PORT_ANY; } static int ipsec4_setspidx_ipaddr(struct mbuf *m, struct secpolicyindex *spidx) { static const struct sockaddr_in template = { sizeof(struct sockaddr_in), AF_INET, 0, { 0 }, { 0, 0, 0, 0, 0, 0, 0, 0 } }; spidx->src.sin = template; spidx->dst.sin = template; if (m->m_len < sizeof(struct ip)) { m_copydata(m, offsetof(struct ip, ip_src), sizeof(struct in_addr), &spidx->src.sin.sin_addr); m_copydata(m, offsetof(struct ip, ip_dst), sizeof(struct in_addr), &spidx->dst.sin.sin_addr); } else { struct ip *ip = mtod(m, struct ip *); spidx->src.sin.sin_addr = ip->ip_src; spidx->dst.sin.sin_addr = ip->ip_dst; } spidx->prefs = sizeof(struct in_addr) << 3; spidx->prefd = sizeof(struct in_addr) << 3; return 0; } #ifdef INET6 static void ipsec6_get_ulp(struct mbuf *m, struct secpolicyindex *spidx, int needport) { int off, nxt; struct tcphdr th; struct udphdr uh; struct icmp6_hdr icmph; KASSERT(m != NULL); if (KEYDEBUG_ON(KEYDEBUG_IPSEC_DUMP)) { kdebug_mbuf(__func__, m); } /* set default */ spidx->ul_proto = IPSEC_ULPROTO_ANY; ((struct sockaddr_in6 *)&spidx->src)->sin6_port = IPSEC_PORT_ANY; ((struct sockaddr_in6 *)&spidx->dst)->sin6_port = IPSEC_PORT_ANY; nxt = -1; off = ip6_lasthdr(m, 0, IPPROTO_IPV6, &nxt); if (off < 0 || m->m_pkthdr.len < off) return; switch (nxt) { case IPPROTO_TCP: spidx->ul_proto = nxt; if (!needport) break; if (off + sizeof(struct tcphdr) > m->m_pkthdr.len) break; m_copydata(m, off, sizeof(th), &th); ((struct sockaddr_in6 *)&spidx->src)->sin6_port = th.th_sport; ((struct sockaddr_in6 *)&spidx->dst)->sin6_port = th.th_dport; break; case IPPROTO_UDP: spidx->ul_proto = nxt; if (!needport) break; if (off + sizeof(struct udphdr) > m->m_pkthdr.len) break; m_copydata(m, off, sizeof(uh), &uh); ((struct sockaddr_in6 *)&spidx->src)->sin6_port = uh.uh_sport; ((struct sockaddr_in6 *)&spidx->dst)->sin6_port = uh.uh_dport; break; case IPPROTO_ICMPV6: spidx->ul_proto = nxt; if (off + sizeof(struct icmp6_hdr) > m->m_pkthdr.len) break; m_copydata(m, off, sizeof(icmph), &icmph); ((struct sockaddr_in6 *)&spidx->src)->sin6_port = htons((uint16_t)icmph.icmp6_type); ((struct sockaddr_in6 *)&spidx->dst)->sin6_port = htons((uint16_t)icmph.icmp6_code); break; default: /* XXX intermediate headers??? */ spidx->ul_proto = nxt; break; } } static int ipsec6_setspidx_ipaddr(struct mbuf *m, struct secpolicyindex *spidx) { struct ip6_hdr *ip6 = NULL; struct ip6_hdr ip6buf; struct sockaddr_in6 *sin6; if (m->m_len >= sizeof(*ip6)) { ip6 = mtod(m, struct ip6_hdr *); } else { m_copydata(m, 0, sizeof(ip6buf), &ip6buf); ip6 = &ip6buf; } sin6 = (struct sockaddr_in6 *)&spidx->src; memset(sin6, 0, sizeof(*sin6)); sin6->sin6_family = AF_INET6; sin6->sin6_len = sizeof(struct sockaddr_in6); memcpy(&sin6->sin6_addr, &ip6->ip6_src, sizeof(ip6->ip6_src)); if (IN6_IS_SCOPE_LINKLOCAL(&ip6->ip6_src)) { sin6->sin6_addr.s6_addr16[1] = 0; sin6->sin6_scope_id = ntohs(ip6->ip6_src.s6_addr16[1]); } spidx->prefs = sizeof(struct in6_addr) << 3; sin6 = (struct sockaddr_in6 *)&spidx->dst; memset(sin6, 0, sizeof(*sin6)); sin6->sin6_family = AF_INET6; sin6->sin6_len = sizeof(struct sockaddr_in6); memcpy(&sin6->sin6_addr, &ip6->ip6_dst, sizeof(ip6->ip6_dst)); if (IN6_IS_SCOPE_LINKLOCAL(&ip6->ip6_dst)) { sin6->sin6_addr.s6_addr16[1] = 0; sin6->sin6_scope_id = ntohs(ip6->ip6_dst.s6_addr16[1]); } spidx->prefd = sizeof(struct in6_addr) << 3; return 0; } #endif static void ipsec_delpcbpolicy(struct inpcbpolicy *p) { kmem_intr_free(p, sizeof(*p)); } int ipsec_init_pcbpolicy(struct socket *so, struct inpcbpolicy **policy) { struct inpcbpolicy *new; KASSERT(so != NULL); KASSERT(policy != NULL); new = kmem_intr_zalloc(sizeof(*new), KM_NOSLEEP); if (new == NULL) { IPSECLOG(LOG_DEBUG, "No more memory.\n"); return ENOBUFS; } if (IPSEC_PRIVILEGED_SO(so)) new->priv = 1; else new->priv = 0; /* * Set dummy SPs. Actual SPs will be allocated later if needed. */ new->sp_in = &ipsec_dummy_sp; new->sp_out = &ipsec_dummy_sp; *policy = new; return 0; } static void ipsec_destroy_policy(struct secpolicy *sp) { if (sp == &ipsec_dummy_sp) { ; /* It's dummy. No need to free it. */ } else { /* * We cannot destroy here because it can be called in * softint. So mark the SP as DEAD and let the timer * destroy it. See key_timehandler_spd. */ sp->state = IPSEC_SPSTATE_DEAD; } } int ipsec_set_policy(void *inp, const void *request, size_t len, kauth_cred_t cred) { struct inpcb_hdr *inph = (struct inpcb_hdr *)inp; const struct sadb_x_policy *xpl; struct secpolicy *newsp, *oldsp; struct secpolicy **policy; int error; KASSERT(!cpu_softintr_p()); KASSERT(inph != NULL); KASSERT(inph_locked(inph)); KASSERT(request != NULL); if (len < sizeof(*xpl)) return EINVAL; xpl = (const struct sadb_x_policy *)request; KASSERT(inph->inph_sp != NULL); /* select direction */ switch (xpl->sadb_x_policy_dir) { case IPSEC_DIR_INBOUND: policy = &inph->inph_sp->sp_in; break; case IPSEC_DIR_OUTBOUND: policy = &inph->inph_sp->sp_out; break; default: IPSECLOG(LOG_ERR, "invalid direction=%u\n", xpl->sadb_x_policy_dir); return EINVAL; } /* sanity check. */ if (policy == NULL || *policy == NULL) return EINVAL; if (KEYDEBUG_ON(KEYDEBUG_IPSEC_DUMP)) { kdebug_sadb_xpolicy("set passed policy", request); } /* check policy type */ /* ipsec_set_policy() accepts IPSEC, ENTRUST and BYPASS. */ if (xpl->sadb_x_policy_type == IPSEC_POLICY_DISCARD || xpl->sadb_x_policy_type == IPSEC_POLICY_NONE) return EINVAL; /* check privileged socket */ if (xpl->sadb_x_policy_type == IPSEC_POLICY_BYPASS) { error = kauth_authorize_network(cred, KAUTH_NETWORK_IPSEC, KAUTH_REQ_NETWORK_IPSEC_BYPASS, NULL, NULL, NULL); if (error) return error; } /* allocation new SP entry */ if ((newsp = key_msg2sp(xpl, len, &error)) == NULL) return error; key_init_sp(newsp); newsp->created = time_uptime; /* Insert the global list for SPs for sockets */ key_socksplist_add(newsp); /* clear old SP and set new SP */ oldsp = *policy; *policy = newsp; ipsec_destroy_policy(oldsp); if (KEYDEBUG_ON(KEYDEBUG_IPSEC_DUMP)) { printf("%s: new policy\n", __func__); kdebug_secpolicy(newsp); } return 0; } int ipsec_get_policy(void *inp, const void *request, size_t len, struct mbuf **mp) { struct inpcb_hdr *inph = (struct inpcb_hdr *)inp; const struct sadb_x_policy *xpl; struct secpolicy *policy; /* sanity check. */ if (inph == NULL || request == NULL || mp == NULL) return EINVAL; KASSERT(inph->inph_sp != NULL); if (len < sizeof(*xpl)) return EINVAL; xpl = (const struct sadb_x_policy *)request; /* select direction */ switch (xpl->sadb_x_policy_dir) { case IPSEC_DIR_INBOUND: policy = inph->inph_sp->sp_in; break; case IPSEC_DIR_OUTBOUND: policy = inph->inph_sp->sp_out; break; default: IPSECLOG(LOG_ERR, "invalid direction=%u\n", xpl->sadb_x_policy_dir); return EINVAL; } if (policy == NULL) return EINVAL; *mp = key_sp2msg(policy, M_NOWAIT); if (!*mp) { IPSECLOG(LOG_DEBUG, "No more memory.\n"); return ENOBUFS; } if (KEYDEBUG_ON(KEYDEBUG_IPSEC_DUMP)) { kdebug_mbuf(__func__, *mp); } return 0; } int ipsec_delete_pcbpolicy(void *inp) { struct inpcb_hdr *inph = (struct inpcb_hdr *)inp; KASSERT(inph != NULL); if (inph->inph_sp == NULL) return 0; if (inph->inph_sp->sp_in != NULL) ipsec_destroy_policy(inph->inph_sp->sp_in); if (inph->inph_sp->sp_out != NULL) ipsec_destroy_policy(inph->inph_sp->sp_out); ipsec_invalpcbcache(inph->inph_sp, IPSEC_DIR_ANY); ipsec_delpcbpolicy(inph->inph_sp); inph->inph_sp = NULL; return 0; } /* * Return the current level (either IPSEC_LEVEL_USE or IPSEC_LEVEL_REQUIRE). */ u_int ipsec_get_reqlevel(const struct ipsecrequest *isr) { u_int level = 0; u_int esp_trans_deflev, esp_net_deflev; u_int ah_trans_deflev, ah_net_deflev; KASSERT(isr != NULL); KASSERT(isr->sp != NULL); KASSERTMSG( isr->sp->spidx.src.sa.sa_family == isr->sp->spidx.dst.sa.sa_family, "af family mismatch, src %u, dst %u", isr->sp->spidx.src.sa.sa_family, isr->sp->spidx.dst.sa.sa_family); /* XXX note that we have ipseclog() expanded here - code sync issue */ #define IPSEC_CHECK_DEFAULT(lev) \ (((lev) != IPSEC_LEVEL_USE && (lev) != IPSEC_LEVEL_REQUIRE \ && (lev) != IPSEC_LEVEL_UNIQUE) ? \ (ipsec_debug ? log(LOG_INFO, "fixed system default level " #lev \ ":%d->%d\n", (lev), IPSEC_LEVEL_REQUIRE) : (void)0), \ (lev) = IPSEC_LEVEL_REQUIRE, (lev) \ : (lev)) /* set default level */ switch (((struct sockaddr *)&isr->sp->spidx.src)->sa_family) { #ifdef INET case AF_INET: esp_trans_deflev = IPSEC_CHECK_DEFAULT(ip4_esp_trans_deflev); esp_net_deflev = IPSEC_CHECK_DEFAULT(ip4_esp_net_deflev); ah_trans_deflev = IPSEC_CHECK_DEFAULT(ip4_ah_trans_deflev); ah_net_deflev = IPSEC_CHECK_DEFAULT(ip4_ah_net_deflev); break; #endif #ifdef INET6 case AF_INET6: esp_trans_deflev = IPSEC_CHECK_DEFAULT(ip6_esp_trans_deflev); esp_net_deflev = IPSEC_CHECK_DEFAULT(ip6_esp_net_deflev); ah_trans_deflev = IPSEC_CHECK_DEFAULT(ip6_ah_trans_deflev); ah_net_deflev = IPSEC_CHECK_DEFAULT(ip6_ah_net_deflev); break; #endif default: panic("%s: unknown af %u", __func__, isr->sp->spidx.src.sa.sa_family); } #undef IPSEC_CHECK_DEFAULT /* set level */ switch (isr->level) { case IPSEC_LEVEL_DEFAULT: switch (isr->saidx.proto) { case IPPROTO_ESP: if (isr->saidx.mode == IPSEC_MODE_TUNNEL) level = esp_net_deflev; else level = esp_trans_deflev; break; case IPPROTO_AH: if (isr->saidx.mode == IPSEC_MODE_TUNNEL) level = ah_net_deflev; else level = ah_trans_deflev; break; case IPPROTO_IPCOMP: /* * we don't really care, as IPcomp document says that * we shouldn't compress small packets */ level = IPSEC_LEVEL_USE; break; default: panic("%s: Illegal protocol defined %u", __func__, isr->saidx.proto); } break; case IPSEC_LEVEL_USE: case IPSEC_LEVEL_REQUIRE: level = isr->level; break; case IPSEC_LEVEL_UNIQUE: level = IPSEC_LEVEL_REQUIRE; break; default: panic("%s: Illegal IPsec level %u", __func__, isr->level); } return level; } /* * Check security policy requirements against the actual packet contents. * * If the SP requires an IPsec packet, and the packet was neither AH nor ESP, * then kick it. */ static int ipsec_sp_reject(const struct secpolicy *sp, const struct mbuf *m) { struct ipsecrequest *isr; if (KEYDEBUG_ON(KEYDEBUG_IPSEC_DATA)) { printf("%s: using SP\n", __func__); kdebug_secpolicy(sp); } /* check policy */ switch (sp->policy) { case IPSEC_POLICY_DISCARD: return 1; case IPSEC_POLICY_BYPASS: case IPSEC_POLICY_NONE: return 0; } KASSERTMSG(sp->policy == IPSEC_POLICY_IPSEC, "invalid policy %u", sp->policy); /* XXX should compare policy against ipsec header history */ for (isr = sp->req; isr != NULL; isr = isr->next) { if (ipsec_get_reqlevel(isr) != IPSEC_LEVEL_REQUIRE) continue; switch (isr->saidx.proto) { case IPPROTO_ESP: if ((m->m_flags & M_DECRYPTED) == 0) { KEYDEBUG_PRINTF(KEYDEBUG_IPSEC_DUMP, "ESP m_flags:%x\n", m->m_flags); return 1; } break; case IPPROTO_AH: if ((m->m_flags & M_AUTHIPHDR) == 0) { KEYDEBUG_PRINTF(KEYDEBUG_IPSEC_DUMP, "AH m_flags:%x\n", m->m_flags); return 1; } break; case IPPROTO_IPCOMP: /* * We don't really care, as IPcomp document * says that we shouldn't compress small * packets, IPComp policy should always be * treated as being in "use" level. */ break; } } return 0; } /* * Check security policy requirements. */ int ipsec_in_reject(struct mbuf *m, void *inp) { struct inpcb_hdr *inph = (struct inpcb_hdr *)inp; struct secpolicy *sp; int error; int result; KASSERT(m != NULL); if (inph == NULL) sp = ipsec_getpolicybyaddr(m, IPSEC_DIR_INBOUND, IP_FORWARDING, &error); else sp = ipsec_getpolicybysock(m, IPSEC_DIR_INBOUND, inph, &error); if (sp != NULL) { result = ipsec_sp_reject(sp, m); if (result) IPSEC_STATINC(IPSEC_STAT_IN_POLVIO); KEY_SP_UNREF(&sp); } else { result = 0; } return result; } /* * Compute the byte size to be occupied by the IPsec header. If it is * tunneled, it includes the size of outer IP header. */ static size_t ipsec_sp_hdrsiz(const struct secpolicy *sp, const struct mbuf *m) { struct ipsecrequest *isr; size_t siz; if (KEYDEBUG_ON(KEYDEBUG_IPSEC_DATA)) { printf("%s: using SP\n", __func__); kdebug_secpolicy(sp); } switch (sp->policy) { case IPSEC_POLICY_DISCARD: case IPSEC_POLICY_BYPASS: case IPSEC_POLICY_NONE: return 0; } KASSERTMSG(sp->policy == IPSEC_POLICY_IPSEC, "invalid policy %u", sp->policy); siz = 0; for (isr = sp->req; isr != NULL; isr = isr->next) { size_t clen = 0; struct secasvar *sav; switch (isr->saidx.proto) { case IPPROTO_ESP: sav = ipsec_lookup_sa(isr, m); if (sav != NULL) { clen = esp_hdrsiz(sav); KEY_SA_UNREF(&sav); } else clen = esp_hdrsiz(NULL); break; case IPPROTO_AH: sav = ipsec_lookup_sa(isr, m); if (sav != NULL) { clen = ah_hdrsiz(sav); KEY_SA_UNREF(&sav); } else clen = ah_hdrsiz(NULL); break; case IPPROTO_IPCOMP: clen = sizeof(struct ipcomp); break; } if (isr->saidx.mode == IPSEC_MODE_TUNNEL) { switch (isr->saidx.dst.sa.sa_family) { case AF_INET: clen += sizeof(struct ip); break; #ifdef INET6 case AF_INET6: clen += sizeof(struct ip6_hdr); break; #endif default: IPSECLOG(LOG_ERR, "unknown AF %d in " "IPsec tunnel SA\n", ((const struct sockaddr *)&isr->saidx.dst) ->sa_family); break; } } siz += clen; } return siz; } size_t ipsec_hdrsiz(struct mbuf *m, u_int dir, void *inp) { struct inpcb_hdr *inph = (struct inpcb_hdr *)inp; struct secpolicy *sp; int error; size_t size; KASSERT(m != NULL); KASSERTMSG(inph == NULL || inph->inph_socket != NULL, "socket w/o inpcb"); if (inph == NULL) sp = ipsec_getpolicybyaddr(m, dir, IP_FORWARDING, &error); else sp = ipsec_getpolicybysock(m, dir, inph, &error); if (sp != NULL) { size = ipsec_sp_hdrsiz(sp, m); KEYDEBUG_PRINTF(KEYDEBUG_IPSEC_DATA, "size:%zu.\n", size); KEY_SP_UNREF(&sp); } else { size = 0; } return size; } /* * Check the variable replay window. * ipsec_chkreplay() performs replay check before ICV verification. * ipsec_updatereplay() updates replay bitmap. This must be called after * ICV verification (it also performs replay check, which is usually done * beforehand). * 0 (zero) is returned if packet disallowed, 1 if packet permitted. * * based on RFC 2401. */ int ipsec_chkreplay(u_int32_t seq, const struct secasvar *sav) { const struct secreplay *replay; u_int32_t diff; int fr; u_int32_t wsizeb; /* constant: bits of window size */ int frlast; /* constant: last frame */ KASSERT(sav != NULL); KASSERT(sav->replay != NULL); replay = sav->replay; if (replay->wsize == 0) return 1; /* no need to check replay. */ /* constant */ frlast = replay->wsize - 1; wsizeb = replay->wsize << 3; /* sequence number of 0 is invalid */ if (seq == 0) return 0; /* first time is always okay */ if (replay->count == 0) return 1; if (seq > replay->lastseq) { /* larger sequences are okay */ return 1; } else { /* seq is equal or less than lastseq. */ diff = replay->lastseq - seq; /* over range to check, i.e. too old or wrapped */ if (diff >= wsizeb) return 0; fr = frlast - diff / 8; /* this packet already seen ? */ if ((replay->bitmap)[fr] & (1 << (diff % 8))) return 0; /* out of order but good */ return 1; } } /* * check replay counter whether to update or not. * OUT: 0: OK * 1: NG */ int ipsec_updatereplay(u_int32_t seq, const struct secasvar *sav) { struct secreplay *replay; u_int32_t diff; int fr; u_int32_t wsizeb; /* constant: bits of window size */ int frlast; /* constant: last frame */ KASSERT(sav != NULL); KASSERT(sav->replay != NULL); replay = sav->replay; if (replay->wsize == 0) goto ok; /* no need to check replay. */ /* constant */ frlast = replay->wsize - 1; wsizeb = replay->wsize << 3; /* sequence number of 0 is invalid */ if (seq == 0) return 1; /* first time */ if (replay->count == 0) { replay->lastseq = seq; memset(replay->bitmap, 0, replay->wsize); (replay->bitmap)[frlast] = 1; goto ok; } if (seq > replay->lastseq) { /* seq is larger than lastseq. */ diff = seq - replay->lastseq; /* new larger sequence number */ if (diff < wsizeb) { /* In window */ /* set bit for this packet */ vshiftl(replay->bitmap, diff, replay->wsize); (replay->bitmap)[frlast] |= 1; } else { /* this packet has a "way larger" */ memset(replay->bitmap, 0, replay->wsize); (replay->bitmap)[frlast] = 1; } replay->lastseq = seq; /* larger is good */ } else { /* seq is equal or less than lastseq. */ diff = replay->lastseq - seq; /* over range to check, i.e. too old or wrapped */ if (diff >= wsizeb) return 1; fr = frlast - diff / 8; /* this packet already seen ? */ if ((replay->bitmap)[fr] & (1 << (diff % 8))) return 1; /* mark as seen */ (replay->bitmap)[fr] |= (1 << (diff % 8)); /* out of order but good */ } ok: if (replay->count == ~0) { char buf[IPSEC_LOGSASTRLEN]; /* set overflow flag */ replay->overflow++; /* don't increment, no more packets accepted */ if ((sav->flags & SADB_X_EXT_CYCSEQ) == 0) return 1; IPSECLOG(LOG_WARNING, "replay counter made %d cycle. %s\n", replay->overflow, ipsec_logsastr(sav, buf, sizeof(buf))); } replay->count++; return 0; } /* * shift variable length buffer to left. * IN: bitmap: pointer to the buffer * nbit: the number of to shift. * wsize: buffer size (bytes). */ static void vshiftl(unsigned char *bitmap, int nbit, int wsize) { int s, j, i; unsigned char over; for (j = 0; j < nbit; j += 8) { s = (nbit - j < 8) ? (nbit - j): 8; bitmap[0] <<= s; for (i = 1; i < wsize; i++) { over = (bitmap[i] >> (8 - s)); bitmap[i] <<= s; bitmap[i-1] |= over; } } return; } /* Return a printable string for the address. */ const char * ipsec_address(const union sockaddr_union *sa, char *buf, size_t size) { switch (sa->sa.sa_family) { case AF_INET: in_print(buf, size, &sa->sin.sin_addr); return buf; #if INET6 case AF_INET6: in6_print(buf, size, &sa->sin6.sin6_addr); return buf; #endif default: return "(unknown address family)"; } } const char * ipsec_logsastr(const struct secasvar *sav, char *buf, size_t size) { const struct secasindex *saidx = &sav->sah->saidx; char sbuf[IPSEC_ADDRSTRLEN], dbuf[IPSEC_ADDRSTRLEN]; KASSERTMSG(saidx->src.sa.sa_family == saidx->dst.sa.sa_family, "af family mismatch, src %u, dst %u", saidx->src.sa.sa_family, saidx->dst.sa.sa_family); snprintf(buf, size, "SA(SPI=%u src=%s dst=%s)", (u_int32_t)ntohl(sav->spi), ipsec_address(&saidx->src, sbuf, sizeof(sbuf)), ipsec_address(&saidx->dst, dbuf, sizeof(dbuf))); return buf; } #ifdef INET6 struct secpolicy * ipsec6_check_policy(struct mbuf *m, struct in6pcb *in6p, int flags, int *needipsecp, int *errorp) { struct secpolicy *sp = NULL; int s; int error = 0; int needipsec = 0; if (ipsec_outdone(m)) { goto skippolicycheck; } s = splsoftnet(); if (in6p && ipsec_pcb_skip_ipsec(in6p->in6p_sp, IPSEC_DIR_OUTBOUND)) { splx(s); goto skippolicycheck; } sp = ipsec_checkpolicy(m, IPSEC_DIR_OUTBOUND, flags, &error, in6p); splx(s); /* * There are four return cases: * sp != NULL apply IPsec policy * sp == NULL, error == 0 no IPsec handling needed * sp == NULL, error == -EINVAL discard packet w/o error * sp == NULL, error != 0 discard packet, report error */ if (sp == NULL) { needipsec = 0; } else { needipsec = 1; } skippolicycheck: *errorp = error; *needipsecp = needipsec; return sp; } /* * calculate UDP checksum for UDP encapsulated ESP for IPv6. * * RFC2460(Internet Protocol, Version 6 Specification) says: * * IPv6 receivers MUST discard UDP packets with a zero checksum. * * There is more relaxed specification RFC6935(IPv6 and UDP Checksums for * Tunneled Packets). The document allows zero checksum. It's too * late to publish, there are a lot of interoperability problems... */ void ipsec6_udp_cksum(struct mbuf *m) { struct ip6_hdr *ip6; uint16_t plen, uh_sum; int off; /* must called after m_pullup() */ KASSERT(m->m_len >= sizeof(struct ip6_hdr)); ip6 = mtod(m, struct ip6_hdr *); KASSERT(ip6->ip6_nxt == IPPROTO_UDP); /* ip6->ip6_plen can not be updated before ip6_output() */ plen = m->m_pkthdr.len - sizeof(*ip6); KASSERT(plen >= sizeof(struct udphdr)); uh_sum = in6_cksum(m, IPPROTO_UDP, sizeof(*ip6), plen); if (uh_sum == 0) uh_sum = 0xffff; off = sizeof(*ip6) + offsetof(struct udphdr, uh_sum); m_copyback(m, off, sizeof(uh_sum), (void *)&uh_sum); } #endif /* INET6 */ /* * ----------------------------------------------------------------------------- */ /* XXX this stuff doesn't belong here... */ static struct xformsw *xforms = NULL; /* * Register a transform; typically at system startup. */ void xform_register(struct xformsw *xsp) { xsp->xf_next = xforms; xforms = xsp; } /* * Initialize transform support in an sav. */ int xform_init(struct secasvar *sav, int xftype) { struct xformsw *xsp; if (sav->tdb_xform != NULL) /* previously initialized */ return 0; for (xsp = xforms; xsp; xsp = xsp->xf_next) if (xsp->xf_type == xftype) return (*xsp->xf_init)(sav, xsp); IPSECLOG(LOG_DEBUG, "no match for xform type %d\n", xftype); return EINVAL; } /* * XXXJRT This should be done as a protosw init call. */ void ipsec_attach(void) { ipsec_output_init(); ipsecstat_percpu = percpu_alloc(sizeof(uint64_t) * IPSEC_NSTATS); sysctl_net_inet_ipsec_setup(NULL); #ifdef INET6 sysctl_net_inet6_ipsec6_setup(NULL); #endif ah_attach(); esp_attach(); ipcomp_attach(); ipe4_attach(); #ifdef TCP_SIGNATURE tcpsignature_attach(); #endif } |
| 15 15 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 197 198 199 200 201 202 203 204 205 206 207 208 209 210 211 212 213 214 215 216 217 218 219 220 221 222 223 224 225 226 227 228 229 230 231 232 233 234 235 236 237 238 239 240 241 242 243 244 245 246 247 248 249 250 251 252 253 254 255 256 257 258 259 260 261 262 263 264 265 266 267 268 269 270 271 272 273 274 275 276 277 278 279 280 281 282 283 284 285 286 287 288 289 290 291 292 293 294 295 296 297 298 299 300 301 302 303 304 305 306 307 308 309 310 311 312 313 314 315 316 317 318 319 320 321 322 323 324 325 326 327 328 329 330 331 332 333 334 335 336 337 338 339 340 341 342 343 344 345 346 347 348 349 350 351 352 353 354 355 356 357 358 359 360 361 362 363 364 365 366 367 368 369 370 371 372 373 374 375 376 377 378 379 380 381 382 383 384 385 386 387 388 389 390 391 392 393 394 395 396 397 398 399 400 401 402 403 404 405 406 407 408 409 410 411 412 413 414 415 416 417 418 419 420 421 422 423 424 425 426 427 428 429 430 431 432 433 434 435 436 437 438 439 440 441 442 443 444 445 446 447 448 449 450 451 452 453 454 455 456 457 458 459 460 461 462 463 464 465 466 467 468 469 470 471 472 473 474 475 476 477 478 479 480 481 482 483 484 485 486 487 488 489 490 491 492 493 494 495 496 497 498 499 500 501 502 503 504 505 506 507 508 509 510 511 512 513 514 515 516 517 518 519 520 521 522 523 524 525 526 527 528 529 530 531 532 533 534 535 536 537 538 539 540 541 542 543 544 545 546 547 548 549 550 551 552 553 554 555 556 557 558 559 560 561 562 563 564 565 566 567 568 569 570 571 572 573 574 575 576 577 578 579 580 581 582 583 584 585 586 587 588 589 590 591 592 593 594 595 596 597 598 599 600 601 602 603 604 605 606 607 608 609 610 611 612 613 614 615 616 617 618 619 620 621 622 623 624 625 626 627 628 629 630 631 632 633 634 635 636 637 638 639 640 641 642 643 644 645 646 647 648 649 650 651 652 653 654 655 656 657 658 659 660 661 662 663 664 665 666 667 668 669 670 671 672 673 674 675 676 677 678 679 680 681 682 683 684 685 686 687 688 689 690 691 692 693 694 695 696 697 698 699 700 701 702 703 704 705 706 707 708 709 710 711 712 713 714 715 716 717 718 719 720 721 722 723 724 725 726 727 728 729 730 731 732 | /* $NetBSD: in6.h,v 1.101 2021/07/31 10:12:04 andvar Exp $ */ /* $KAME: in6.h,v 1.83 2001/03/29 02:55:07 jinmei Exp $ */ /* * Copyright (C) 1995, 1996, 1997, and 1998 WIDE Project. * All rights reserved. * * Redistribution and use in source and binary forms, with or without * modification, are permitted provided that the following conditions * are met: * 1. Redistributions of source code must retain the above copyright * notice, this list of conditions and the following disclaimer. * 2. Redistributions in binary form must reproduce the above copyright * notice, this list of conditions and the following disclaimer in the * documentation and/or other materials provided with the distribution. * 3. Neither the name of the project nor the names of its contributors * may be used to endorse or promote products derived from this software * without specific prior written permission. * * THIS SOFTWARE IS PROVIDED BY THE PROJECT AND CONTRIBUTORS ``AS IS'' AND * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE * ARE DISCLAIMED. IN NO EVENT SHALL THE PROJECT OR CONTRIBUTORS BE LIABLE * FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL * DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS * OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) * HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT * LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY * OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF * SUCH DAMAGE. */ /* * Copyright (c) 1982, 1986, 1990, 1993 * The Regents of the University of California. All rights reserved. * * Redistribution and use in source and binary forms, with or without * modification, are permitted provided that the following conditions * are met: * 1. Redistributions of source code must retain the above copyright * notice, this list of conditions and the following disclaimer. * 2. Redistributions in binary form must reproduce the above copyright * notice, this list of conditions and the following disclaimer in the * documentation and/or other materials provided with the distribution. * 3. Neither the name of the University nor the names of its contributors * may be used to endorse or promote products derived from this software * without specific prior written permission. * * THIS SOFTWARE IS PROVIDED BY THE REGENTS AND CONTRIBUTORS ``AS IS'' AND * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE * ARE DISCLAIMED. IN NO EVENT SHALL THE REGENTS OR CONTRIBUTORS BE LIABLE * FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL * DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS * OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) * HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT * LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY * OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF * SUCH DAMAGE. * * @(#)in.h 8.3 (Berkeley) 1/3/94 */ #ifndef _NETINET6_IN6_H_ #define _NETINET6_IN6_H_ #include <sys/featuretest.h> #ifndef __KAME_NETINET_IN_H_INCLUDED_ #error "do not include netinet6/in6.h directly, include netinet/in.h. see RFC2553" #endif #include <sys/socket.h> #include <sys/endian.h> /* ntohl */ /* * Identification of the network protocol stack * for *BSD-current/release: http://www.kame.net/dev/cvsweb.cgi/kame/COVERAGE * has the table of implementation/integration differences. */ #define __KAME__ #define __KAME_VERSION "NetBSD-current" /* * Local port number conventions: * * Ports < IPPORT_RESERVED are reserved for privileged processes (e.g. root), * unless a kernel is compiled with IPNOPRIVPORTS defined. * * When a user does a bind(2) or connect(2) with a port number of zero, * a non-conflicting local port address is chosen. * * The default range is IPPORT_ANONMIN to IPPORT_ANONMAX, although * that is settable by sysctl(3); net.inet.ip.anonportmin and * net.inet.ip.anonportmax respectively. * * A user may set the IPPROTO_IP option IP_PORTRANGE to change this * default assignment range. * * The value IP_PORTRANGE_DEFAULT causes the default behavior. * * The value IP_PORTRANGE_HIGH is the same as IP_PORTRANGE_DEFAULT, * and exists only for FreeBSD compatibility purposes. * * The value IP_PORTRANGE_LOW changes the range to the "low" are * that is (by convention) restricted to privileged processes. * This convention is based on "vouchsafe" principles only. * It is only secure if you trust the remote host to restrict these ports. * The range is IPPORT_RESERVEDMIN to IPPORT_RESERVEDMAX. */ #if defined(_NETBSD_SOURCE) #define IPV6PORT_RESERVED 1024 #define IPV6PORT_ANONMIN 49152 #define IPV6PORT_ANONMAX 65535 #define IPV6PORT_RESERVEDMIN 600 #define IPV6PORT_RESERVEDMAX (IPV6PORT_RESERVED-1) #endif /* * IPv6 address */ struct in6_addr { union { __uint8_t __u6_addr8[16]; __uint16_t __u6_addr16[8]; uint32_t __u6_addr32[4]; } __u6_addr; /* 128-bit IP6 address */ }; #define s6_addr __u6_addr.__u6_addr8 #ifdef _KERNEL /* XXX nonstandard */ #define s6_addr8 __u6_addr.__u6_addr8 #define s6_addr16 __u6_addr.__u6_addr16 #define s6_addr32 __u6_addr.__u6_addr32 #endif #define INET6_ADDRSTRLEN 46 /* * Socket address for IPv6 */ #if defined(_NETBSD_SOURCE) #define SIN6_LEN #endif struct sockaddr_in6 { uint8_t sin6_len; /* length of this struct(socklen_t)*/ sa_family_t sin6_family; /* AF_INET6 (sa_family_t) */ in_port_t sin6_port; /* Transport layer port */ uint32_t sin6_flowinfo; /* IP6 flow information */ struct in6_addr sin6_addr; /* IP6 address */ uint32_t sin6_scope_id; /* scope zone index */ }; /* * Local definition for masks */ #ifdef _KERNEL /* XXX nonstandard */ #define IN6MASK0 {{{ 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0 }}} #define IN6MASK32 {{{ 0xff, 0xff, 0xff, 0xff, 0x00, 0x00, 0x00, 0x00, \ 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00 }}} #define IN6MASK64 {{{ 0xff, 0xff, 0xff, 0xff, 0xff, 0xff, 0xff, 0xff, \ 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00 }}} #define IN6MASK96 {{{ 0xff, 0xff, 0xff, 0xff, 0xff, 0xff, 0xff, 0xff, \ 0xff, 0xff, 0xff, 0xff, 0x00, 0x00, 0x00, 0x00 }}} #define IN6MASK128 {{{ 0xff, 0xff, 0xff, 0xff, 0xff, 0xff, 0xff, 0xff, \ 0xff, 0xff, 0xff, 0xff, 0xff, 0xff, 0xff, 0xff }}} #endif #ifdef _KERNEL extern const struct sockaddr_in6 sa6_any; extern const struct in6_addr in6mask0; extern const struct in6_addr in6mask32; extern const struct in6_addr in6mask64; extern const struct in6_addr in6mask96; extern const struct in6_addr in6mask128; #endif /* _KERNEL */ /* * Macros started with IPV6_ADDR is KAME local */ #ifdef _KERNEL /* XXX nonstandard */ #if BYTE_ORDER == BIG_ENDIAN #define IPV6_ADDR_INT32_ONE 1 #define IPV6_ADDR_INT32_TWO 2 #define IPV6_ADDR_INT32_MNL 0xff010000 #define IPV6_ADDR_INT32_MLL 0xff020000 #define IPV6_ADDR_INT32_SMP 0x0000ffff #define IPV6_ADDR_INT16_ULL 0xfe80 #define IPV6_ADDR_INT16_USL 0xfec0 #define IPV6_ADDR_INT16_MLL 0xff02 #elif BYTE_ORDER == LITTLE_ENDIAN #define IPV6_ADDR_INT32_ONE 0x01000000 #define IPV6_ADDR_INT32_TWO 0x02000000 #define IPV6_ADDR_INT32_MNL 0x000001ff #define IPV6_ADDR_INT32_MLL 0x000002ff #define IPV6_ADDR_INT32_SMP 0xffff0000 #define IPV6_ADDR_INT16_ULL 0x80fe #define IPV6_ADDR_INT16_USL 0xc0fe #define IPV6_ADDR_INT16_MLL 0x02ff #endif #endif /* * Definition of some useful macros to handle IP6 addresses */ #define IN6ADDR_ANY_INIT \ {{{ 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, \ 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00 }}} #define IN6ADDR_LOOPBACK_INIT \ {{{ 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, \ 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x01 }}} #define IN6ADDR_NODELOCAL_ALLNODES_INIT \ {{{ 0xff, 0x01, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, \ 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x01 }}} #define IN6ADDR_LINKLOCAL_ALLNODES_INIT \ {{{ 0xff, 0x02, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, \ 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x01 }}} #define IN6ADDR_LINKLOCAL_ALLROUTERS_INIT \ {{{ 0xff, 0x02, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, \ 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x02 }}} extern const struct in6_addr in6addr_any; extern const struct in6_addr in6addr_loopback; extern const struct in6_addr in6addr_nodelocal_allnodes; extern const struct in6_addr in6addr_linklocal_allnodes; extern const struct in6_addr in6addr_linklocal_allrouters; #define IN6_ARE_ADDR_EQUAL(a, b) \ (memcmp(&(a)->s6_addr[0], &(b)->s6_addr[0], sizeof(struct in6_addr)) == 0) /* * Unspecified */ #define IN6_IS_ADDR_UNSPECIFIED(a) \ ((a)->__u6_addr.__u6_addr32[0] == 0 && \ (a)->__u6_addr.__u6_addr32[1] == 0 && \ (a)->__u6_addr.__u6_addr32[2] == 0 && \ (a)->__u6_addr.__u6_addr32[3] == 0) /* * Loopback */ #define IN6_IS_ADDR_LOOPBACK(a) \ ((a)->__u6_addr.__u6_addr32[0] == 0 && \ (a)->__u6_addr.__u6_addr32[1] == 0 && \ (a)->__u6_addr.__u6_addr32[2] == 0 && \ (a)->__u6_addr.__u6_addr32[3] == ntohl(1)) /* * IPv4 compatible */ #define IN6_IS_ADDR_V4COMPAT(a) \ ((a)->__u6_addr.__u6_addr32[0] == 0 && \ (a)->__u6_addr.__u6_addr32[1] == 0 && \ (a)->__u6_addr.__u6_addr32[2] == 0 && \ (a)->__u6_addr.__u6_addr32[3] != 0 && \ (a)->__u6_addr.__u6_addr32[3] != ntohl(1)) /* * Mapped */ #define IN6_IS_ADDR_V4MAPPED(a) \ ((a)->__u6_addr.__u6_addr32[0] == 0 && \ (a)->__u6_addr.__u6_addr32[1] == 0 && \ (a)->__u6_addr.__u6_addr32[2] == ntohl(0x0000ffff)) /* * KAME Scope Values */ #ifdef _KERNEL /* XXX nonstandard */ #define IPV6_ADDR_SCOPE_NODELOCAL 0x01 #define IPV6_ADDR_SCOPE_INTFACELOCAL 0x01 #define IPV6_ADDR_SCOPE_LINKLOCAL 0x02 #define IPV6_ADDR_SCOPE_SITELOCAL 0x05 #define IPV6_ADDR_SCOPE_ORGLOCAL 0x08 /* just used in this file */ #define IPV6_ADDR_SCOPE_GLOBAL 0x0e #else #define __IPV6_ADDR_SCOPE_NODELOCAL 0x01 #define __IPV6_ADDR_SCOPE_LINKLOCAL 0x02 #define __IPV6_ADDR_SCOPE_SITELOCAL 0x05 #define __IPV6_ADDR_SCOPE_ORGLOCAL 0x08 /* just used in this file */ #define __IPV6_ADDR_SCOPE_GLOBAL 0x0e #endif /* * Unicast Scope * Note that we must check topmost 10 bits only, not 16 bits (see RFC2373). */ #define IN6_IS_ADDR_LINKLOCAL(a) \ (((a)->s6_addr[0] == 0xfe) && (((a)->s6_addr[1] & 0xc0) == 0x80)) #define IN6_IS_ADDR_SITELOCAL(a) \ (((a)->s6_addr[0] == 0xfe) && (((a)->s6_addr[1] & 0xc0) == 0xc0)) /* * Multicast */ #define IN6_IS_ADDR_MULTICAST(a) ((a)->s6_addr[0] == 0xff) #ifdef _KERNEL /* XXX nonstandard */ #define IPV6_ADDR_MC_SCOPE(a) ((a)->s6_addr[1] & 0x0f) #else #define __IPV6_ADDR_MC_SCOPE(a) ((a)->s6_addr[1] & 0x0f) #endif /* * Multicast Scope */ #ifdef _KERNEL /* refers nonstandard items */ #define IN6_IS_ADDR_MC_NODELOCAL(a) \ (IN6_IS_ADDR_MULTICAST(a) && \ (IPV6_ADDR_MC_SCOPE(a) == IPV6_ADDR_SCOPE_NODELOCAL)) #define IN6_IS_ADDR_MC_INTFACELOCAL(a) \ (IN6_IS_ADDR_MULTICAST(a) && \ (IPV6_ADDR_MC_SCOPE(a) == IPV6_ADDR_SCOPE_INTFACELOCAL)) #define IN6_IS_ADDR_MC_LINKLOCAL(a) \ (IN6_IS_ADDR_MULTICAST(a) && \ (IPV6_ADDR_MC_SCOPE(a) == IPV6_ADDR_SCOPE_LINKLOCAL)) #define IN6_IS_ADDR_MC_SITELOCAL(a) \ (IN6_IS_ADDR_MULTICAST(a) && \ (IPV6_ADDR_MC_SCOPE(a) == IPV6_ADDR_SCOPE_SITELOCAL)) #define IN6_IS_ADDR_MC_ORGLOCAL(a) \ (IN6_IS_ADDR_MULTICAST(a) && \ (IPV6_ADDR_MC_SCOPE(a) == IPV6_ADDR_SCOPE_ORGLOCAL)) #define IN6_IS_ADDR_MC_GLOBAL(a) \ (IN6_IS_ADDR_MULTICAST(a) && \ (IPV6_ADDR_MC_SCOPE(a) == IPV6_ADDR_SCOPE_GLOBAL)) #else #define IN6_IS_ADDR_MC_NODELOCAL(a) \ (IN6_IS_ADDR_MULTICAST(a) && \ (__IPV6_ADDR_MC_SCOPE(a) == __IPV6_ADDR_SCOPE_NODELOCAL)) #define IN6_IS_ADDR_MC_LINKLOCAL(a) \ (IN6_IS_ADDR_MULTICAST(a) && \ (__IPV6_ADDR_MC_SCOPE(a) == __IPV6_ADDR_SCOPE_LINKLOCAL)) #define IN6_IS_ADDR_MC_SITELOCAL(a) \ (IN6_IS_ADDR_MULTICAST(a) && \ (__IPV6_ADDR_MC_SCOPE(a) == __IPV6_ADDR_SCOPE_SITELOCAL)) #define IN6_IS_ADDR_MC_ORGLOCAL(a) \ (IN6_IS_ADDR_MULTICAST(a) && \ (__IPV6_ADDR_MC_SCOPE(a) == __IPV6_ADDR_SCOPE_ORGLOCAL)) #define IN6_IS_ADDR_MC_GLOBAL(a) \ (IN6_IS_ADDR_MULTICAST(a) && \ (__IPV6_ADDR_MC_SCOPE(a) == __IPV6_ADDR_SCOPE_GLOBAL)) #endif #ifdef _KERNEL /* nonstandard */ /* * KAME Scope */ #define IN6_IS_SCOPE_LINKLOCAL(a) \ ((IN6_IS_ADDR_LINKLOCAL(a)) || \ (IN6_IS_ADDR_MC_LINKLOCAL(a))) #define IN6_IS_SCOPE_EMBEDDABLE(__a) \ (IN6_IS_SCOPE_LINKLOCAL(__a) || IN6_IS_ADDR_MC_INTFACELOCAL(__a)) #define IFA6_IS_DEPRECATED(a) \ ((a)->ia6_lifetime.ia6t_pltime != ND6_INFINITE_LIFETIME && \ (u_int32_t)((time_uptime - (a)->ia6_updatetime)) > \ (a)->ia6_lifetime.ia6t_pltime) #define IFA6_IS_INVALID(a) \ ((a)->ia6_lifetime.ia6t_vltime != ND6_INFINITE_LIFETIME && \ (u_int32_t)((time_uptime - (a)->ia6_updatetime)) > \ (a)->ia6_lifetime.ia6t_vltime) #endif /* * Options for use with [gs]etsockopt at the IPV6 level. * First word of comment is data type; bool is stored in int. */ /* no hdrincl */ #if 0 /* These are deprecated non-standard options which are no longer supported. */ #define IPV6_OPTIONS 1 /* buf/ip6_opts; set/get IP6 options */ #define IPV6_RECVOPTS 5 /* bool; receive all IP6 opts w/dgram */ #define IPV6_RECVRETOPTS 6 /* bool; receive IP6 opts for response */ #define IPV6_RECVDSTADDR 7 /* bool; receive IP6 dst addr w/dgram */ #define IPV6_RETOPTS 8 /* ip6_opts; set/get IP6 options */ #endif #define IPV6_SOCKOPT_RESERVED1 3 /* reserved for future use */ #define IPV6_UNICAST_HOPS 4 /* int; IP6 hops */ #define IPV6_MULTICAST_IF 9 /* u_int; set/get IP6 multicast i/f */ #define IPV6_MULTICAST_HOPS 10 /* int; set/get IP6 multicast hops */ #define IPV6_MULTICAST_LOOP 11 /* u_int; set/get IP6 multicast loopback */ /* The join and leave membership option numbers need to match with the v4 ones */ #define IPV6_JOIN_GROUP 12 /* ip6_mreq; join a group membership */ #define IPV6_LEAVE_GROUP 13 /* ip6_mreq; leave a group membership */ #define IPV6_PORTRANGE 14 /* int; range to choose for unspec port */ #if defined(_NETBSD_SOURCE) #define IPV6_PORTALGO 17 /* int; port selection algo (rfc6056) */ #define ICMP6_FILTER 18 /* icmp6_filter; icmp6 filter */ #endif /* RFC2292 options */ #ifdef _KERNEL #define IPV6_2292PKTINFO 19 /* bool; send/recv if, src/dst addr */ #define IPV6_2292HOPLIMIT 20 /* bool; hop limit */ #define IPV6_2292NEXTHOP 21 /* bool; next hop addr */ #define IPV6_2292HOPOPTS 22 /* bool; hop-by-hop option */ #define IPV6_2292DSTOPTS 23 /* bool; destination option */ #define IPV6_2292RTHDR 24 /* bool; routing header */ #define IPV6_2292PKTOPTIONS 25 /* buf/cmsghdr; set/get IPv6 options */ #endif #define IPV6_CHECKSUM 26 /* int; checksum offset for raw socket */ #define IPV6_V6ONLY 27 /* bool; make AF_INET6 sockets v6 only */ #define IPV6_IPSEC_POLICY 28 /* struct; get/set security policy */ #define IPV6_FAITH 29 /* bool; accept FAITH'ed connections */ /* new socket options introduced in RFC3542 */ #define IPV6_RTHDRDSTOPTS 35 /* ip6_dest; send dst option before rthdr */ #define IPV6_RECVPKTINFO 36 /* bool; recv if, dst addr */ #define IPV6_RECVHOPLIMIT 37 /* bool; recv hop limit */ #define IPV6_RECVRTHDR 38 /* bool; recv routing header */ #define IPV6_RECVHOPOPTS 39 /* bool; recv hop-by-hop option */ #define IPV6_RECVDSTOPTS 40 /* bool; recv dst option after rthdr */ #ifdef _KERNEL #define IPV6_RECVRTHDRDSTOPTS 41 /* bool; recv dst option before rthdr */ #endif #define IPV6_USE_MIN_MTU 42 /* bool; send packets at the minimum MTU */ #define IPV6_RECVPATHMTU 43 /* bool; notify an according MTU */ #define IPV6_PATHMTU 44 /* mtuinfo; get the current path MTU (sopt), 4 bytes int; MTU notification (cmsg) */ /* more new socket options introduced in RFC3542 */ #define IPV6_PKTINFO 46 /* in6_pktinfo; send if, src addr */ #define IPV6_HOPLIMIT 47 /* int; send hop limit */ #define IPV6_NEXTHOP 48 /* sockaddr; next hop addr */ #define IPV6_HOPOPTS 49 /* ip6_hbh; send hop-by-hop option */ #define IPV6_DSTOPTS 50 /* ip6_dest; send dst option before rthdr */ #define IPV6_RTHDR 51 /* ip6_rthdr; send routing header */ #define IPV6_RECVTCLASS 57 /* bool; recv traffic class values */ #ifdef _KERNEL #define IPV6_OTCLASS 58 /* u_int8_t; send traffic class value */ #endif #define IPV6_TCLASS 61 /* int; send traffic class value */ #define IPV6_DONTFRAG 62 /* bool; disable IPv6 fragmentation */ #define IPV6_PREFER_TEMPADDR 63 /* int; prefer temporary address as * the source address */ #define IPV6_BINDANY 64 /* bool: allow bind to any address */ /* to define items, should talk with KAME guys first, for *BSD compatibility */ #define IPV6_RTHDR_LOOSE 0 /* this hop need not be a neighbor. XXX old spec */ #define IPV6_RTHDR_STRICT 1 /* this hop must be a neighbor. XXX old spec */ #define IPV6_RTHDR_TYPE_0 0 /* IPv6 routing header type 0 */ /* * Defaults and limits for options */ #define IPV6_DEFAULT_MULTICAST_HOPS 1 /* normally limit m'casts to 1 hop */ #define IPV6_DEFAULT_MULTICAST_LOOP 1 /* normally hear sends if a member */ /* * Argument structure for IPV6_JOIN_GROUP and IPV6_LEAVE_GROUP. */ struct ipv6_mreq { struct in6_addr ipv6mr_multiaddr; unsigned int ipv6mr_interface; }; /* * IPV6_PKTINFO: Packet information(RFC2292 sec 5) */ struct in6_pktinfo { struct in6_addr ipi6_addr; /* src/dst IPv6 address */ unsigned int ipi6_ifindex; /* send/recv interface index */ }; /* * Control structure for IPV6_RECVPATHMTU socket option. */ struct ip6_mtuinfo { struct sockaddr_in6 ip6m_addr; /* or sockaddr_storage? */ uint32_t ip6m_mtu; }; /* * Argument for IPV6_PORTRANGE: * - which range to search when port is unspecified at bind() or connect() */ #define IPV6_PORTRANGE_DEFAULT 0 /* default range */ #define IPV6_PORTRANGE_HIGH 1 /* "high" - request firewall bypass */ #define IPV6_PORTRANGE_LOW 2 /* "low" - vouchsafe security */ #if defined(_NETBSD_SOURCE) /* * Definitions for inet6 sysctl operations. * * Third level is protocol number. * Fourth level is desired variable within that protocol. */ /* * Names for IP sysctl objects */ #define IPV6CTL_FORWARDING 1 /* act as router */ #define IPV6CTL_SENDREDIRECTS 2 /* may send redirects when forwarding*/ #define IPV6CTL_DEFHLIM 3 /* default Hop-Limit */ /* IPV6CTL_DEFMTU=4, never implemented */ #define IPV6CTL_FORWSRCRT 5 /* forward source-routed dgrams */ #define IPV6CTL_STATS 6 /* stats */ #define IPV6CTL_MRTSTATS 7 /* multicast forwarding stats */ #define IPV6CTL_MRTPROTO 8 /* multicast routing protocol */ #define IPV6CTL_MAXFRAGPACKETS 9 /* max packets reassembly queue */ #define IPV6CTL_SOURCECHECK 10 /* verify source route and intf */ #define IPV6CTL_SOURCECHECK_LOGINT 11 /* minimum logging interval */ /* 12 was IPV6CTL_ACCEPT_RTADV */ #define IPV6CTL_KEEPFAITH 13 #define IPV6CTL_LOG_INTERVAL 14 #define IPV6CTL_HDRNESTLIMIT 15 #define IPV6CTL_DAD_COUNT 16 #define IPV6CTL_AUTO_FLOWLABEL 17 #define IPV6CTL_DEFMCASTHLIM 18 #define IPV6CTL_GIF_HLIM 19 /* default HLIM for gif encap packet */ #define IPV6CTL_KAME_VERSION 20 #define IPV6CTL_USE_DEPRECATED 21 /* use deprecated addr (RFC2462 5.5.4) */ /* 22 was IPV6CTL_RR_PRUNE */ /* 23: reserved */ #define IPV6CTL_V6ONLY 24 /* 25 to 27: reserved */ #define IPV6CTL_ANONPORTMIN 28 /* minimum ephemeral port */ #define IPV6CTL_ANONPORTMAX 29 /* maximum ephemeral port */ #define IPV6CTL_LOWPORTMIN 30 /* minimum reserved port */ #define IPV6CTL_LOWPORTMAX 31 /* maximum reserved port */ /* 32 to 34: reserved */ #define IPV6CTL_AUTO_LINKLOCAL 35 /* automatic link-local addr assign */ /* 36 to 37: reserved */ #define IPV6CTL_ADDRCTLPOLICY 38 /* get/set address selection policy */ #define IPV6CTL_USE_DEFAULTZONE 39 /* use default scope zone */ /* 40: reserved */ #define IPV6CTL_MAXFRAGS 41 /* max fragments */ #define IPV6CTL_IFQ 42 /* IPv6 packet input queue */ /* 43 was IPV6CTL_RTADV_MAXROUTES */ /* 44 was IPV6CTL_RTADV_NUMROUTES */ #define IPV6CTL_GIF_PMTU 45 /* gif(4) Path MTU setting */ #define IPV6CTL_IPSEC_HLIM 46 /* default HLIM for ipsecif encap packet */ #define IPV6CTL_IPSEC_PMTU 47 /* ipsecif(4) Path MTU setting */ #endif /* _NETBSD_SOURCE */ #ifdef _KERNEL struct cmsghdr; /* * in6_cksum_phdr: * * Compute significant parts of the IPv6 checksum pseudo-header * for use in a delayed TCP/UDP checksum calculation. * * Args: * * src Source IPv6 address * dst Destination IPv6 address * len htonl(proto-hdr-len) * nxt htonl(next-proto-number) * * NOTE: We expect the src and dst addresses to be 16-bit * aligned! */ static __inline u_int16_t __unused in6_cksum_phdr(const struct in6_addr *src, const struct in6_addr *dst, u_int32_t len, u_int32_t nxt) { u_int32_t sum = 0; const u_int16_t *w; /*LINTED*/ w = (const u_int16_t *) src; sum += w[0]; if (!IN6_IS_SCOPE_LINKLOCAL(src)) sum += w[1]; sum += w[2]; sum += w[3]; sum += w[4]; sum += w[5]; sum += w[6]; sum += w[7]; /*LINTED*/ w = (const u_int16_t *) dst; sum += w[0]; if (!IN6_IS_SCOPE_LINKLOCAL(dst)) sum += w[1]; sum += w[2]; sum += w[3]; sum += w[4]; sum += w[5]; sum += w[6]; sum += w[7]; sum += (u_int16_t)(len >> 16) + (u_int16_t)(len /*& 0xffff*/); sum += (u_int16_t)(nxt >> 16) + (u_int16_t)(nxt /*& 0xffff*/); sum = (u_int16_t)(sum >> 16) + (u_int16_t)(sum /*& 0xffff*/); if (sum > 0xffff) sum -= 0xffff; return (sum); } struct mbuf; struct ifnet; int sockaddr_in6_cmp(const struct sockaddr *, const struct sockaddr *); struct sockaddr *sockaddr_in6_externalize(struct sockaddr *, socklen_t, const struct sockaddr *); int in6_cksum(struct mbuf *, u_int8_t, u_int32_t, u_int32_t); int in6_localaddr(const struct in6_addr *); int in6_addrscope(const struct in6_addr *); struct in6_ifaddr *in6_ifawithifp(struct ifnet *, struct in6_addr *); extern void in6_if_link_up(struct ifnet *); extern void in6_if_link_down(struct ifnet *); extern void in6_if_link_state_change(struct ifnet *, int); extern void in6_if_up(struct ifnet *); extern void in6_if_down(struct ifnet *); extern void addrsel_policy_init(void); extern u_char ip6_protox[]; struct ip6_hdr; int in6_tunnel_validate(const struct ip6_hdr *, const struct in6_addr *, const struct in6_addr *); #define satosin6(sa) ((struct sockaddr_in6 *)(sa)) #define satocsin6(sa) ((const struct sockaddr_in6 *)(sa)) #define sin6tosa(sin6) ((struct sockaddr *)(sin6)) #define sin6tocsa(sin6) ((const struct sockaddr *)(sin6)) #define ifatoia6(ifa) ((struct in6_ifaddr *)(ifa)) static __inline void sockaddr_in6_init1(struct sockaddr_in6 *sin6, const struct in6_addr *addr, in_port_t port, uint32_t flowinfo, uint32_t scope_id) { sin6->sin6_port = port; sin6->sin6_flowinfo = flowinfo; sin6->sin6_addr = *addr; sin6->sin6_scope_id = scope_id; } static __inline void sockaddr_in6_init(struct sockaddr_in6 *sin6, const struct in6_addr *addr, in_port_t port, uint32_t flowinfo, uint32_t scope_id) { sin6->sin6_family = AF_INET6; sin6->sin6_len = sizeof(*sin6); sockaddr_in6_init1(sin6, addr, port, flowinfo, scope_id); } static __inline struct sockaddr * sockaddr_in6_alloc(const struct in6_addr *addr, in_port_t port, uint32_t flowinfo, uint32_t scope_id, int flags) { struct sockaddr *sa; if ((sa = sockaddr_alloc(AF_INET6, sizeof(struct sockaddr_in6), flags)) == NULL) return NULL; sockaddr_in6_init1(satosin6(sa), addr, port, flowinfo, scope_id); return sa; } #endif /* _KERNEL */ #if defined(_NETBSD_SOURCE) #include <machine/ansi.h> #ifdef _BSD_SIZE_T_ typedef _BSD_SIZE_T_ size_t; #define _SIZE_T #undef _BSD_SIZE_T_ #endif #include <sys/cdefs.h> __BEGIN_DECLS struct cmsghdr; void in6_in_2_v4mapin6(const struct in_addr *, struct in6_addr *); void in6_sin6_2_sin(struct sockaddr_in *, struct sockaddr_in6 *); void in6_sin_2_v4mapsin6(const struct sockaddr_in *, struct sockaddr_in6 *); void in6_sin6_2_sin_in_sock(struct sockaddr *); void in6_sin_2_v4mapsin6_in_sock(struct sockaddr **); #define INET6_IS_ADDR_LINKLOCAL 1 #define INET6_IS_ADDR_MC_LINKLOCAL 2 #define INET6_IS_ADDR_SITELOCAL 4 void inet6_getscopeid(struct sockaddr_in6 *, int); void inet6_putscopeid(struct sockaddr_in6 *, int); extern int inet6_option_space(int); extern int inet6_option_init(void *, struct cmsghdr **, int); extern int inet6_option_append(struct cmsghdr *, const uint8_t *, int, int); extern uint8_t *inet6_option_alloc(struct cmsghdr *, int, int, int); extern int inet6_option_next(const struct cmsghdr *, uint8_t **); extern int inet6_option_find(const struct cmsghdr *, uint8_t **, int); extern size_t inet6_rthdr_space(int, int); extern struct cmsghdr *inet6_rthdr_init(void *, int); extern int inet6_rthdr_add(struct cmsghdr *, const struct in6_addr *, unsigned int); extern int inet6_rthdr_lasthop(struct cmsghdr *, unsigned int); #if 0 /* not implemented yet */ extern int inet6_rthdr_reverse(const struct cmsghdr *, struct cmsghdr *); #endif extern int inet6_rthdr_segments(const struct cmsghdr *); extern struct in6_addr *inet6_rthdr_getaddr(struct cmsghdr *, int); extern int inet6_rthdr_getflags(const struct cmsghdr *, int); extern int inet6_opt_init(void *, socklen_t); extern int inet6_opt_append(void *, socklen_t, int, uint8_t, socklen_t, uint8_t, void **); extern int inet6_opt_finish(void *, socklen_t, int); extern int inet6_opt_set_val(void *, int, void *, socklen_t); extern int inet6_opt_next(void *, socklen_t, int, uint8_t *, socklen_t *, void **); extern int inet6_opt_find(void *, socklen_t, int, uint8_t, socklen_t *, void **); extern int inet6_opt_get_val(void *, int, void *, socklen_t); extern socklen_t inet6_rth_space(int, int); extern void *inet6_rth_init(void *, socklen_t, int, int); extern int inet6_rth_add(void *, const struct in6_addr *); extern int inet6_rth_reverse(const void *, void *); extern int inet6_rth_segments(const void *); extern struct in6_addr *inet6_rth_getaddr(const void *, int); __END_DECLS #endif /* _NETBSD_SOURCE */ #if defined(_KERNEL) || defined(_TEST) int in6_print(char *, size_t, const struct in6_addr *); #define IN6_PRINT(b, a) (in6_print((b), sizeof(b), (a)), (b)) int sin6_print(char *, size_t, const void *); #endif #endif /* !_NETINET6_IN6_H_ */ |
| 95 95 149 47 149 149 149 47 105 141 141 21 43 90 90 1 41 41 9 41 3 41 9 11 29 35 8 41 47 35 14 59 59 59 59 53 58 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 197 198 199 200 201 202 203 204 205 206 207 208 209 210 211 212 213 214 215 216 217 218 219 220 221 222 223 224 225 226 227 228 229 230 231 232 233 234 235 236 237 238 239 240 241 242 243 244 245 246 247 248 249 250 251 252 253 254 255 256 257 258 259 260 261 262 263 264 265 266 267 268 269 270 271 272 273 274 275 276 277 278 279 280 281 282 283 284 285 286 287 288 289 290 291 292 293 294 295 296 297 298 299 300 301 302 303 304 305 306 307 308 309 310 311 312 313 314 315 316 317 318 319 320 321 322 323 324 325 326 327 328 329 330 331 332 333 334 335 336 337 338 339 340 341 342 343 344 345 346 347 348 349 350 351 352 353 354 355 356 357 358 359 360 361 362 363 364 365 366 367 368 369 370 371 372 373 374 375 376 377 378 379 380 381 382 383 384 385 386 387 388 389 390 391 392 393 394 395 396 397 398 399 400 401 402 403 404 | /* $NetBSD: ufs_bmap.c,v 1.53 2020/04/20 03:57:02 christos Exp $ */ /* * Copyright (c) 1989, 1991, 1993 * The Regents of the University of California. All rights reserved. * (c) UNIX System Laboratories, Inc. * All or some portions of this file are derived from material licensed * to the University of California by American Telephone and Telegraph * Co. or Unix System Laboratories, Inc. and are reproduced herein with * the permission of UNIX System Laboratories, Inc. * * Redistribution and use in source and binary forms, with or without * modification, are permitted provided that the following conditions * are met: * 1. Redistributions of source code must retain the above copyright * notice, this list of conditions and the following disclaimer. * 2. Redistributions in binary form must reproduce the above copyright * notice, this list of conditions and the following disclaimer in the * documentation and/or other materials provided with the distribution. * 3. Neither the name of the University nor the names of its contributors * may be used to endorse or promote products derived from this software * without specific prior written permission. * * THIS SOFTWARE IS PROVIDED BY THE REGENTS AND CONTRIBUTORS ``AS IS'' AND * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE * ARE DISCLAIMED. IN NO EVENT SHALL THE REGENTS OR CONTRIBUTORS BE LIABLE * FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL * DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS * OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) * HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT * LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY * OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF * SUCH DAMAGE. * * @(#)ufs_bmap.c 8.8 (Berkeley) 8/11/95 */ #include <sys/cdefs.h> __KERNEL_RCSID(0, "$NetBSD: ufs_bmap.c,v 1.53 2020/04/20 03:57:02 christos Exp $"); #include <sys/param.h> #include <sys/systm.h> #include <sys/buf.h> #include <sys/proc.h> #include <sys/vnode.h> #include <sys/mount.h> #include <sys/resourcevar.h> #include <sys/trace.h> #include <miscfs/specfs/specdev.h> #include <ufs/ufs/inode.h> #include <ufs/ufs/ufsmount.h> #include <ufs/ufs/ufs_extern.h> #include <ufs/ufs/ufs_bswap.h> static bool ufs_issequential(const struct ufsmount *ump, daddr_t daddr0, daddr_t daddr1) { /* for ufs, blocks in a hole is not 'contiguous'. */ if (daddr0 == 0) return false; return (daddr0 + ump->um_seqinc == daddr1); } /* * Bmap converts the logical block number of a file to its physical block * number on the disk. The conversion is done by using the logical block * number to index into the array of block pointers described by the dinode. */ int ufs_bmap(void *v) { struct vop_bmap_args /* { struct vnode *a_vp; daddr_t a_bn; struct vnode **a_vpp; daddr_t *a_bnp; int *a_runp; } */ *ap = v; int error; /* * Check for underlying vnode requests and ensure that logical * to physical mapping is requested. */ if (ap->a_vpp != NULL) *ap->a_vpp = VTOI(ap->a_vp)->i_devvp; if (ap->a_bnp == NULL) return (0); error = ufs_bmaparray(ap->a_vp, ap->a_bn, ap->a_bnp, NULL, NULL, ap->a_runp, ufs_issequential); return error; } /* * Indirect blocks are now on the vnode for the file. They are given negative * logical block numbers. Indirect blocks are addressed by the negative * address of the first data block to which they point. Double indirect blocks * are addressed by one less than the address of the first indirect block to * which they point. Triple indirect blocks are addressed by one less than * the address of the first double indirect block to which they point. * * ufs_bmaparray does the bmap conversion, and if requested returns the * array of logical blocks which must be traversed to get to a block. * Each entry contains the offset into that block that gets you to the * next block and the disk address of the block (if it is assigned). */ int ufs_bmaparray(struct vnode *vp, daddr_t bn, daddr_t *bnp, struct indir *ap, int *nump, int *runp, ufs_issequential_callback_t is_sequential) { struct inode *ip; struct buf *bp, *cbp; struct ufsmount *ump; struct mount *mp; struct indir a[UFS_NIADDR + 1], *xap; daddr_t daddr; daddr_t metalbn; int error, maxrun = 0, num; ip = VTOI(vp); mp = vp->v_mount; ump = ip->i_ump; KASSERTMSG(((ap == NULL) == (nump == NULL)), "ufs_bmaparray: invalid arguments: ap = %p, nump = %p", ap, nump); if (runp) { /* * XXX * If MAXBSIZE is the largest transfer the disks can handle, * we probably want maxrun to be 1 block less so that we * don't create a block larger than the device can handle. */ *runp = 0; maxrun = MAXPHYS / mp->mnt_stat.f_iosize - 1; } if (bn >= 0 && bn < UFS_NDADDR) { if (nump != NULL) *nump = 0; if (ump->um_fstype == UFS1) daddr = ufs_rw32(ip->i_ffs1_db[bn], UFS_MPNEEDSWAP(ump)); else daddr = ufs_rw64(ip->i_ffs2_db[bn], UFS_MPNEEDSWAP(ump)); *bnp = blkptrtodb(ump, daddr); /* * Since this is FFS independent code, we are out of * scope for the definitions of BLK_NOCOPY and * BLK_SNAP, but we do know that they will fall in * the range 1..um_seqinc, so we use that test and * return a request for a zeroed out buffer if attempts * are made to read a BLK_NOCOPY or BLK_SNAP block. */ if ((ip->i_flags & (SF_SNAPSHOT | SF_SNAPINVAL)) == SF_SNAPSHOT && daddr > 0 && daddr < ump->um_seqinc) { *bnp = -1; } else if (*bnp == 0) { if ((ip->i_flags & (SF_SNAPSHOT | SF_SNAPINVAL)) == SF_SNAPSHOT) { *bnp = blkptrtodb(ump, bn * ump->um_seqinc); } else { *bnp = -1; } } else if (runp) { if (ump->um_fstype == UFS1) { for (++bn; bn < UFS_NDADDR && *runp < maxrun && is_sequential(ump, ufs_rw32(ip->i_ffs1_db[bn - 1], UFS_MPNEEDSWAP(ump)), ufs_rw32(ip->i_ffs1_db[bn], UFS_MPNEEDSWAP(ump))); ++bn, ++*runp); } else { for (++bn; bn < UFS_NDADDR && *runp < maxrun && is_sequential(ump, ufs_rw64(ip->i_ffs2_db[bn - 1], UFS_MPNEEDSWAP(ump)), ufs_rw64(ip->i_ffs2_db[bn], UFS_MPNEEDSWAP(ump))); ++bn, ++*runp); } } return (0); } else if (bn < 0 && bn >= -UFS_NXADDR) { KASSERT(ump->um_fstype == UFS2); daddr = ufs_rw64(ip->i_ffs2_extb[-1 - bn], UFS_MPNEEDSWAP(ump)); *bnp = blkptrtodb(ump, daddr); if (*bnp == 0) *bnp = -1; return 0; } xap = ap == NULL ? a : ap; if (!nump) nump = # if ((error = ufs_getlbns(vp, bn, xap, nump)) != 0) return (error); num = *nump; /* Get disk address out of indirect block array */ if (ump->um_fstype == UFS1) daddr = ufs_rw32(ip->i_ffs1_ib[xap->in_off], UFS_MPNEEDSWAP(ump)); else daddr = ufs_rw64(ip->i_ffs2_ib[xap->in_off], UFS_MPNEEDSWAP(ump)); for (bp = NULL, ++xap; --num; ++xap) { /* * Exit the loop if there is no disk address assigned yet and * the indirect block isn't in the cache, or if we were * looking for an indirect block and we've found it. */ metalbn = xap->in_lbn; if (metalbn == bn) break; if (daddr == 0) { mutex_enter(&bufcache_lock); cbp = incore(vp, metalbn); mutex_exit(&bufcache_lock); if (cbp == NULL) break; } /* * If we get here, we've either got the block in the cache * or we have a disk address for it, go fetch it. */ if (bp) brelse(bp, 0); xap->in_exists = 1; bp = getblk(vp, metalbn, mp->mnt_stat.f_iosize, 0, 0); if (bp == NULL) { /* * getblk() above returns NULL only iff we are * pagedaemon. See the implementation of getblk * for detail. */ return (ENOMEM); } if (bp->b_oflags & (BO_DONE | BO_DELWRI)) { trace(TR_BREADHIT, pack(vp, size), metalbn); } else { KASSERTMSG((daddr != 0), "ufs_bmaparray: indirect block not in cache"); trace(TR_BREADMISS, pack(vp, size), metalbn); bp->b_blkno = blkptrtodb(ump, daddr); bp->b_flags |= B_READ; BIO_SETPRIO(bp, BPRIO_TIMECRITICAL); VOP_STRATEGY(vp, bp); curlwp->l_ru.ru_inblock++; /* XXX */ if ((error = biowait(bp)) != 0) { brelse(bp, 0); return (error); } } if (ump->um_fstype == UFS1) { daddr = ufs_rw32(((u_int32_t *)bp->b_data)[xap->in_off], UFS_MPNEEDSWAP(ump)); if (num == 1 && daddr && runp) { for (bn = xap->in_off + 1; bn < MNINDIR(ump) && *runp < maxrun && is_sequential(ump, ufs_rw32(((int32_t *)bp->b_data)[bn-1], UFS_MPNEEDSWAP(ump)), ufs_rw32(((int32_t *)bp->b_data)[bn], UFS_MPNEEDSWAP(ump))); ++bn, ++*runp); } } else { daddr = ufs_rw64(((u_int64_t *)bp->b_data)[xap->in_off], UFS_MPNEEDSWAP(ump)); if (num == 1 && daddr && runp) { for (bn = xap->in_off + 1; bn < MNINDIR(ump) && *runp < maxrun && is_sequential(ump, ufs_rw64(((int64_t *)bp->b_data)[bn-1], UFS_MPNEEDSWAP(ump)), ufs_rw64(((int64_t *)bp->b_data)[bn], UFS_MPNEEDSWAP(ump))); ++bn, ++*runp); } } } if (bp) brelse(bp, 0); /* * Since this is FFS independent code, we are out of scope for the * definitions of BLK_NOCOPY and BLK_SNAP, but we do know that they * will fall in the range 1..um_seqinc, so we use that test and * return a request for a zeroed out buffer if attempts are made * to read a BLK_NOCOPY or BLK_SNAP block. */ if ((ip->i_flags & (SF_SNAPSHOT | SF_SNAPINVAL)) == SF_SNAPSHOT && daddr > 0 && daddr < ump->um_seqinc) { *bnp = -1; return (0); } *bnp = blkptrtodb(ump, daddr); if (*bnp == 0) { if ((ip->i_flags & (SF_SNAPSHOT | SF_SNAPINVAL)) == SF_SNAPSHOT) { *bnp = blkptrtodb(ump, bn * ump->um_seqinc); } else { *bnp = -1; } } return (0); } /* * Create an array of logical block number/offset pairs which represent the * path of indirect blocks required to access a data block. The first "pair" * contains the logical block number of the appropriate single, double or * triple indirect block and the offset into the inode indirect block array. * Note, the logical block number of the inode single/double/triple indirect * block appears twice in the array, once with the offset into the i_ffs1_ib and * once with the offset into the page itself. */ int ufs_getlbns(struct vnode *vp, daddr_t bn, struct indir *ap, int *nump) { daddr_t metalbn, realbn; struct ufsmount *ump; int64_t blockcnt; int lbc; int i, numlevels, off; ump = VFSTOUFS(vp->v_mount); if (nump) *nump = 0; numlevels = 0; realbn = bn; if (bn < 0) bn = -bn; KASSERT(bn >= UFS_NDADDR); /* * Determine the number of levels of indirection. After this loop * is done, blockcnt indicates the number of data blocks possible * at the given level of indirection, and UFS_NIADDR - i is the number * of levels of indirection needed to locate the requested block. */ bn -= UFS_NDADDR; for (lbc = 0, i = UFS_NIADDR;; i--, bn -= blockcnt) { if (i == 0) return (EFBIG); lbc += ump->um_lognindir; blockcnt = (int64_t)1 << lbc; if (bn < blockcnt) break; } /* Calculate the address of the first meta-block. */ metalbn = -((realbn >= 0 ? realbn : -realbn) - bn + UFS_NIADDR - i); /* * At each iteration, off is the offset into the bap array which is * an array of disk addresses at the current level of indirection. * The logical block number and the offset in that block are stored * into the argument array. */ ap->in_lbn = metalbn; ap->in_off = off = UFS_NIADDR - i; ap->in_exists = 0; ap++; for (++numlevels; i <= UFS_NIADDR; i++) { /* If searching for a meta-data block, quit when found. */ if (metalbn == realbn) break; lbc -= ump->um_lognindir; off = (bn >> lbc) & (MNINDIR(ump) - 1); ++numlevels; ap->in_lbn = metalbn; ap->in_off = off; ap->in_exists = 0; ++ap; metalbn -= -1 + ((int64_t)off << lbc); } if (nump) *nump = numlevels; return (0); } |
| 60 4 2 7 7 15 15 15 14 23 23 23 9 2 5 3 1 2 11 11 1 6 7 1 2 5 1 26 26 6 1 19 5 10 19 7 30 16 14 18 3 36 36 5 7 1 8 1 6 1 1 1 6 7 6 7 1 2 23 23 16 3 2 1 13 13 12 12 5 5 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 197 198 199 200 201 202 203 204 205 206 207 208 209 210 211 212 213 214 215 216 217 218 219 220 221 222 223 224 225 226 227 228 229 230 231 232 233 234 235 236 237 238 239 240 241 242 243 244 245 246 247 248 249 250 251 252 253 254 255 256 257 258 259 260 261 262 263 264 265 266 267 268 269 270 271 272 273 274 275 276 277 278 279 280 281 282 283 284 285 286 287 288 289 290 291 292 293 294 295 296 297 298 299 300 301 302 303 304 305 306 307 308 309 310 311 312 313 314 315 316 317 318 319 320 321 322 323 324 325 326 327 328 329 330 331 332 333 334 335 336 337 338 339 340 341 342 343 344 345 346 347 348 349 350 351 352 353 354 355 356 357 358 359 360 361 362 363 364 365 366 367 368 369 370 371 372 373 374 375 376 377 378 379 380 381 382 383 384 385 386 387 388 389 390 391 392 393 394 395 396 397 398 399 400 401 402 403 404 405 406 407 408 409 410 411 412 413 414 415 416 417 418 419 420 421 422 423 424 425 426 427 428 429 430 431 432 433 434 435 436 437 438 439 440 441 442 443 444 445 446 447 448 449 450 451 452 453 454 455 456 457 458 459 460 461 462 463 464 465 466 467 468 469 470 471 472 473 474 475 476 477 478 479 480 481 482 483 484 485 486 487 488 489 490 491 492 493 494 495 496 497 498 499 500 501 502 503 504 505 506 507 508 509 510 511 512 513 514 515 516 517 518 519 520 521 522 523 524 525 526 527 528 529 530 531 532 533 534 535 536 537 538 539 540 541 542 543 544 545 546 547 548 549 550 551 552 553 554 555 556 557 558 559 560 561 562 563 564 565 566 567 568 569 570 571 572 573 574 575 576 577 578 579 580 581 582 583 584 585 586 587 588 589 590 591 592 593 594 595 596 597 598 599 600 601 602 603 604 605 606 607 608 609 610 611 612 613 614 615 616 617 618 619 620 621 622 623 624 625 626 627 628 629 630 631 632 633 634 635 636 637 638 639 640 641 642 643 644 645 646 647 648 649 650 651 652 653 654 655 656 657 658 659 660 661 662 663 664 665 666 667 668 669 670 671 672 673 674 675 676 677 678 679 680 681 682 683 684 685 686 687 688 689 690 691 692 693 694 695 696 697 698 699 700 701 702 703 704 705 706 707 708 709 710 711 712 713 714 715 716 717 718 719 720 721 722 723 724 725 726 727 728 729 730 731 732 733 734 735 736 737 738 739 740 741 742 743 744 745 746 747 748 749 750 751 752 753 754 755 756 757 758 759 760 761 762 763 764 765 766 767 768 769 770 771 772 773 774 775 776 777 778 779 780 781 782 783 784 785 786 787 788 789 790 791 792 793 794 795 796 797 798 799 800 801 802 803 804 805 806 807 808 809 810 811 812 813 814 815 816 817 818 819 820 821 822 823 824 825 826 827 828 829 830 831 832 833 834 835 836 837 838 839 840 841 842 843 844 845 846 847 848 849 850 851 852 853 854 855 856 857 858 859 860 861 862 863 864 865 866 867 868 869 870 871 872 873 874 875 876 877 878 879 880 881 882 883 884 885 886 887 888 889 890 891 892 893 894 895 896 897 898 899 900 901 902 903 904 905 906 907 908 909 910 911 912 913 914 915 916 917 918 919 920 921 922 923 924 925 926 927 928 929 930 931 932 933 934 935 936 937 938 939 940 941 942 943 944 945 946 947 948 949 950 951 952 953 954 955 956 957 958 959 960 961 962 963 964 965 966 967 968 969 970 971 972 973 974 975 976 977 978 979 980 981 982 983 984 985 986 987 988 989 990 991 992 993 994 995 996 997 998 999 1000 1001 1002 1003 1004 1005 1006 1007 1008 1009 1010 1011 1012 1013 1014 1015 1016 1017 1018 1019 1020 1021 1022 1023 1024 1025 1026 1027 1028 1029 1030 1031 1032 1033 1034 1035 1036 1037 1038 1039 1040 1041 1042 1043 1044 1045 1046 1047 1048 1049 1050 1051 1052 1053 1054 1055 1056 1057 1058 1059 1060 1061 1062 1063 1064 1065 1066 1067 1068 1069 1070 1071 1072 1073 1074 1075 1076 1077 1078 1079 1080 1081 1082 1083 1084 1085 1086 1087 1088 1089 1090 1091 1092 1093 1094 1095 1096 1097 1098 1099 1100 1101 1102 1103 1104 1105 1106 1107 1108 1109 1110 1111 1112 1113 1114 1115 1116 1117 1118 1119 1120 1121 1122 1123 1124 1125 1126 1127 1128 1129 1130 | /* $NetBSD: sysv_shm.c,v 1.141 2019/10/09 17:47:13 chs Exp $ */ /*- * Copyright (c) 1999, 2007 The NetBSD Foundation, Inc. * All rights reserved. * * This code is derived from software contributed to The NetBSD Foundation * by Jason R. Thorpe of the Numerical Aerospace Simulation Facility, * NASA Ames Research Center, and by Mindaugas Rasiukevicius. * * Redistribution and use in source and binary forms, with or without * modification, are permitted provided that the following conditions * are met: * 1. Redistributions of source code must retain the above copyright * notice, this list of conditions and the following disclaimer. * 2. Redistributions in binary form must reproduce the above copyright * notice, this list of conditions and the following disclaimer in the * documentation and/or other materials provided with the distribution. * * THIS SOFTWARE IS PROVIDED BY THE NETBSD FOUNDATION, INC. AND CONTRIBUTORS * ``AS IS'' AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED * TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR * PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE FOUNDATION OR CONTRIBUTORS * BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR * CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF * SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS * INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN * CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) * ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE * POSSIBILITY OF SUCH DAMAGE. */ /* * Copyright (c) 1994 Adam Glass and Charles M. Hannum. All rights reserved. * * Redistribution and use in source and binary forms, with or without * modification, are permitted provided that the following conditions * are met: * 1. Redistributions of source code must retain the above copyright * notice, this list of conditions and the following disclaimer. * 2. Redistributions in binary form must reproduce the above copyright * notice, this list of conditions and the following disclaimer in the * documentation and/or other materials provided with the distribution. * 3. All advertising materials mentioning features or use of this software * must display the following acknowledgement: * This product includes software developed by Adam Glass and Charles M. * Hannum. * 4. The names of the authors may not be used to endorse or promote products * derived from this software without specific prior written permission. * * THIS SOFTWARE IS PROVIDED BY THE AUTHORS ``AS IS'' AND ANY EXPRESS OR * IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES * OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. * IN NO EVENT SHALL THE AUTHORS BE LIABLE FOR ANY DIRECT, INDIRECT, * INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT * NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, * DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY * THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT * (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF * THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. */ #include <sys/cdefs.h> __KERNEL_RCSID(0, "$NetBSD: sysv_shm.c,v 1.141 2019/10/09 17:47:13 chs Exp $"); #ifdef _KERNEL_OPT #include "opt_sysv.h" #endif #include <sys/param.h> #include <sys/kernel.h> #include <sys/kmem.h> #include <sys/shm.h> #include <sys/mutex.h> #include <sys/mman.h> #include <sys/stat.h> #include <sys/sysctl.h> #include <sys/mount.h> /* XXX for <sys/syscallargs.h> */ #include <sys/syscallargs.h> #include <sys/queue.h> #include <sys/kauth.h> #include <uvm/uvm_extern.h> #include <uvm/uvm_object.h> struct shmmap_entry { SLIST_ENTRY(shmmap_entry) next; vaddr_t va; int shmid; }; int shm_nused __cacheline_aligned; struct shmid_ds * shmsegs __read_mostly; static kmutex_t shm_lock __cacheline_aligned; static kcondvar_t * shm_cv __cacheline_aligned; static int shm_last_free __cacheline_aligned; static size_t shm_committed __cacheline_aligned; static int shm_use_phys __read_mostly; static kcondvar_t shm_realloc_cv; static bool shm_realloc_state; static u_int shm_realloc_disable; struct shmmap_state { unsigned int nitems; unsigned int nrefs; SLIST_HEAD(, shmmap_entry) entries; }; extern int kern_has_sysvshm; SYSCTL_SETUP_PROTO(sysctl_ipc_shm_setup); #ifdef SHMDEBUG #define SHMPRINTF(a) printf a #else #define SHMPRINTF(a) #endif static int shmrealloc(int); /* * Find the shared memory segment permission by the index. Only used by * compat_linux to implement SHM_STAT. */ int shm_find_segment_perm_by_index(int index, struct ipc_perm *perm) { struct shmid_ds *shmseg; mutex_enter(&shm_lock); if (index < 0 || index >= shminfo.shmmni) { mutex_exit(&shm_lock); return EINVAL; } shmseg = &shmsegs[index]; memcpy(perm, &shmseg->shm_perm, sizeof(*perm)); mutex_exit(&shm_lock); return 0; } /* * Find the shared memory segment by the identifier. * => must be called with shm_lock held; */ static struct shmid_ds * shm_find_segment_by_shmid(int shmid) { int segnum; struct shmid_ds *shmseg; KASSERT(mutex_owned(&shm_lock)); segnum = IPCID_TO_IX(shmid); if (segnum < 0 || segnum >= shminfo.shmmni) return NULL; shmseg = &shmsegs[segnum]; if ((shmseg->shm_perm.mode & SHMSEG_ALLOCATED) == 0) return NULL; if ((shmseg->shm_perm.mode & (SHMSEG_REMOVED|SHMSEG_RMLINGER)) == SHMSEG_REMOVED) return NULL; if (shmseg->shm_perm._seq != IPCID_TO_SEQ(shmid)) return NULL; return shmseg; } /* * Free memory segment. * => must be called with shm_lock held; */ static void shm_free_segment(int segnum) { struct shmid_ds *shmseg; size_t size; bool wanted; KASSERT(mutex_owned(&shm_lock)); shmseg = &shmsegs[segnum]; SHMPRINTF(("shm freeing key 0x%lx seq 0x%x\n", shmseg->shm_perm._key, shmseg->shm_perm._seq)); size = (shmseg->shm_segsz + PGOFSET) & ~PGOFSET; wanted = (shmseg->shm_perm.mode & SHMSEG_WANTED); shmseg->_shm_internal = NULL; shm_committed -= btoc(size); shm_nused--; shmseg->shm_perm.mode = SHMSEG_FREE; shm_last_free = segnum; if (wanted == true) cv_broadcast(&shm_cv[segnum]); } /* * Delete entry from the shm map. * => must be called with shm_lock held; */ static struct uvm_object * shm_delete_mapping(struct shmmap_state *shmmap_s, struct shmmap_entry *shmmap_se) { struct uvm_object *uobj = NULL; struct shmid_ds *shmseg; int segnum; KASSERT(mutex_owned(&shm_lock)); segnum = IPCID_TO_IX(shmmap_se->shmid); shmseg = &shmsegs[segnum]; SLIST_REMOVE(&shmmap_s->entries, shmmap_se, shmmap_entry, next); shmmap_s->nitems--; shmseg->shm_dtime = time_second; if ((--shmseg->shm_nattch <= 0) && (shmseg->shm_perm.mode & SHMSEG_REMOVED)) { uobj = shmseg->_shm_internal; shm_free_segment(segnum); } return uobj; } /* * Get a non-shared shm map for that vmspace. Note, that memory * allocation might be performed with lock held. */ static struct shmmap_state * shmmap_getprivate(struct proc *p) { struct shmmap_state *oshmmap_s, *shmmap_s; struct shmmap_entry *oshmmap_se, *shmmap_se; KASSERT(mutex_owned(&shm_lock)); /* 1. A shm map with refcnt = 1, used by ourselves, thus return */ oshmmap_s = (struct shmmap_state *)p->p_vmspace->vm_shm; if (oshmmap_s && oshmmap_s->nrefs == 1) return oshmmap_s; /* 2. No shm map preset - create a fresh one */ shmmap_s = kmem_zalloc(sizeof(struct shmmap_state), KM_SLEEP); shmmap_s->nrefs = 1; SLIST_INIT(&shmmap_s->entries); p->p_vmspace->vm_shm = (void *)shmmap_s; if (oshmmap_s == NULL) return shmmap_s; SHMPRINTF(("shmmap_getprivate: vm %p split (%d entries), was used by %d\n", p->p_vmspace, oshmmap_s->nitems, oshmmap_s->nrefs)); /* 3. A shared shm map, copy to a fresh one and adjust refcounts */ SLIST_FOREACH(oshmmap_se, &oshmmap_s->entries, next) { shmmap_se = kmem_alloc(sizeof(struct shmmap_entry), KM_SLEEP); shmmap_se->va = oshmmap_se->va; shmmap_se->shmid = oshmmap_se->shmid; SLIST_INSERT_HEAD(&shmmap_s->entries, shmmap_se, next); } shmmap_s->nitems = oshmmap_s->nitems; oshmmap_s->nrefs--; return shmmap_s; } /* * Lock/unlock the memory. * => must be called with shm_lock held; */ static int shm_memlock(struct shmid_ds *shmseg, int shmid, int cmd) { size_t size; int error; KASSERT(mutex_owned(&shm_lock)); size = round_page(shmseg->shm_segsz); if (cmd == SHM_LOCK && (shmseg->shm_perm.mode & SHMSEG_WIRED) == 0) { /* Wire the object and map, then tag it */ error = uvm_obj_wirepages(shmseg->_shm_internal, 0, size, NULL); if (error) return EIO; shmseg->shm_perm.mode |= SHMSEG_WIRED; } else if (cmd == SHM_UNLOCK && (shmseg->shm_perm.mode & SHMSEG_WIRED) != 0) { /* Unwire the object, then untag it */ uvm_obj_unwirepages(shmseg->_shm_internal, 0, size); shmseg->shm_perm.mode &= ~SHMSEG_WIRED; } return 0; } /* * Unmap shared memory. */ int sys_shmdt(struct lwp *l, const struct sys_shmdt_args *uap, register_t *retval) { /* { syscallarg(const void *) shmaddr; } */ struct proc *p = l->l_proc; struct shmmap_state *shmmap_s1, *shmmap_s; struct shmmap_entry *shmmap_se; struct uvm_object *uobj; struct shmid_ds *shmseg; size_t size; mutex_enter(&shm_lock); /* In case of reallocation, we will wait for completion */ while (__predict_false(shm_realloc_state)) cv_wait(&shm_realloc_cv, &shm_lock); shmmap_s1 = (struct shmmap_state *)p->p_vmspace->vm_shm; if (shmmap_s1 == NULL) { mutex_exit(&shm_lock); return EINVAL; } /* Find the map entry */ SLIST_FOREACH(shmmap_se, &shmmap_s1->entries, next) if (shmmap_se->va == (vaddr_t)SCARG(uap, shmaddr)) break; if (shmmap_se == NULL) { mutex_exit(&shm_lock); return EINVAL; } shmmap_s = shmmap_getprivate(p); if (shmmap_s != shmmap_s1) { /* Map has been copied, lookup entry in new map */ SLIST_FOREACH(shmmap_se, &shmmap_s->entries, next) if (shmmap_se->va == (vaddr_t)SCARG(uap, shmaddr)) break; if (shmmap_se == NULL) { mutex_exit(&shm_lock); return EINVAL; } } SHMPRINTF(("shmdt: vm %p: remove %d @%lx\n", p->p_vmspace, shmmap_se->shmid, shmmap_se->va)); /* Delete the entry from shm map */ uobj = shm_delete_mapping(shmmap_s, shmmap_se); shmseg = &shmsegs[IPCID_TO_IX(shmmap_se->shmid)]; size = (shmseg->shm_segsz + PGOFSET) & ~PGOFSET; mutex_exit(&shm_lock); uvm_deallocate(&p->p_vmspace->vm_map, shmmap_se->va, size); if (uobj != NULL) { uao_detach(uobj); } kmem_free(shmmap_se, sizeof(struct shmmap_entry)); return 0; } /* * Map shared memory. */ int sys_shmat(struct lwp *l, const struct sys_shmat_args *uap, register_t *retval) { /* { syscallarg(int) shmid; syscallarg(const void *) shmaddr; syscallarg(int) shmflg; } */ int error, flags = 0; struct proc *p = l->l_proc; kauth_cred_t cred = l->l_cred; struct shmid_ds *shmseg; struct shmmap_state *shmmap_s; struct shmmap_entry *shmmap_se; struct uvm_object *uobj; struct vmspace *vm; vaddr_t attach_va; vm_prot_t prot; vsize_t size; /* Allocate a new map entry and set it */ shmmap_se = kmem_alloc(sizeof(struct shmmap_entry), KM_SLEEP); shmmap_se->shmid = SCARG(uap, shmid); mutex_enter(&shm_lock); /* In case of reallocation, we will wait for completion */ while (__predict_false(shm_realloc_state)) cv_wait(&shm_realloc_cv, &shm_lock); shmseg = shm_find_segment_by_shmid(SCARG(uap, shmid)); if (shmseg == NULL) { error = EINVAL; goto err; } error = ipcperm(cred, &shmseg->shm_perm, (SCARG(uap, shmflg) & SHM_RDONLY) ? IPC_R : IPC_R|IPC_W); if (error) goto err; vm = p->p_vmspace; shmmap_s = (struct shmmap_state *)vm->vm_shm; if (shmmap_s && shmmap_s->nitems >= shminfo.shmseg) { error = EMFILE; goto err; } size = (shmseg->shm_segsz + PGOFSET) & ~PGOFSET; prot = VM_PROT_READ; if ((SCARG(uap, shmflg) & SHM_RDONLY) == 0) prot |= VM_PROT_WRITE; if (SCARG(uap, shmaddr)) { flags |= UVM_FLAG_FIXED; if (SCARG(uap, shmflg) & SHM_RND) attach_va = (vaddr_t)SCARG(uap, shmaddr) & ~(SHMLBA-1); else if (((vaddr_t)SCARG(uap, shmaddr) & (SHMLBA-1)) == 0) attach_va = (vaddr_t)SCARG(uap, shmaddr); else { error = EINVAL; goto err; } } else { /* This is just a hint to uvm_map() about where to put it. */ attach_va = p->p_emul->e_vm_default_addr(p, (vaddr_t)vm->vm_daddr, size, p->p_vmspace->vm_map.flags & VM_MAP_TOPDOWN); } /* * Create a map entry, add it to the list and increase the counters. */ shmmap_s = shmmap_getprivate(p); SLIST_INSERT_HEAD(&shmmap_s->entries, shmmap_se, next); shmmap_s->nitems++; shmseg->shm_lpid = p->p_pid; shmseg->shm_nattch++; /* * Map the segment into the address space. */ uobj = shmseg->_shm_internal; uao_reference(uobj); error = uvm_map(&vm->vm_map, &attach_va, size, uobj, 0, 0, UVM_MAPFLAG(prot, prot, UVM_INH_SHARE, UVM_ADV_RANDOM, flags)); if (error) goto err_detach; /* Set the new address, and update the time */ shmmap_se->va = attach_va; shmseg->shm_atime = time_second; retval[0] = attach_va; SHMPRINTF(("shmat: vm %p: add %d @%lx\n", p->p_vmspace, shmmap_se->shmid, attach_va)); err: mutex_exit(&shm_lock); if (error && shmmap_se) { kmem_free(shmmap_se, sizeof(struct shmmap_entry)); } return error; err_detach: uao_detach(uobj); uobj = shm_delete_mapping(shmmap_s, shmmap_se); mutex_exit(&shm_lock); if (uobj != NULL) { uao_detach(uobj); } kmem_free(shmmap_se, sizeof(struct shmmap_entry)); return error; } /* * Shared memory control operations. */ int sys___shmctl50(struct lwp *l, const struct sys___shmctl50_args *uap, register_t *retval) { /* { syscallarg(int) shmid; syscallarg(int) cmd; syscallarg(struct shmid_ds *) buf; } */ struct shmid_ds shmbuf; int cmd, error; cmd = SCARG(uap, cmd); if (cmd == IPC_SET) { error = copyin(SCARG(uap, buf), &shmbuf, sizeof(shmbuf)); if (error) return error; } error = shmctl1(l, SCARG(uap, shmid), cmd, (cmd == IPC_SET || cmd == IPC_STAT) ? &shmbuf : NULL); if (error == 0 && cmd == IPC_STAT) error = copyout(&shmbuf, SCARG(uap, buf), sizeof(shmbuf)); return error; } int shmctl1(struct lwp *l, int shmid, int cmd, struct shmid_ds *shmbuf) { struct uvm_object *uobj = NULL; kauth_cred_t cred = l->l_cred; struct shmid_ds *shmseg; int error = 0; mutex_enter(&shm_lock); /* In case of reallocation, we will wait for completion */ while (__predict_false(shm_realloc_state)) cv_wait(&shm_realloc_cv, &shm_lock); shmseg = shm_find_segment_by_shmid(shmid); if (shmseg == NULL) { mutex_exit(&shm_lock); return EINVAL; } switch (cmd) { case IPC_STAT: if ((error = ipcperm(cred, &shmseg->shm_perm, IPC_R)) != 0) break; memset(shmbuf, 0, sizeof *shmbuf); shmbuf->shm_perm = shmseg->shm_perm; shmbuf->shm_perm.mode &= 0777; shmbuf->shm_segsz = shmseg->shm_segsz; shmbuf->shm_lpid = shmseg->shm_lpid; shmbuf->shm_cpid = shmseg->shm_cpid; shmbuf->shm_nattch = shmseg->shm_nattch; shmbuf->shm_atime = shmseg->shm_atime; shmbuf->shm_dtime = shmseg->shm_dtime; shmbuf->shm_ctime = shmseg->shm_ctime; break; case IPC_SET: if ((error = ipcperm(cred, &shmseg->shm_perm, IPC_M)) != 0) break; shmseg->shm_perm.uid = shmbuf->shm_perm.uid; shmseg->shm_perm.gid = shmbuf->shm_perm.gid; shmseg->shm_perm.mode = (shmseg->shm_perm.mode & ~ACCESSPERMS) | (shmbuf->shm_perm.mode & ACCESSPERMS); shmseg->shm_ctime = time_second; break; case IPC_RMID: if ((error = ipcperm(cred, &shmseg->shm_perm, IPC_M)) != 0) break; shmseg->shm_perm._key = IPC_PRIVATE; shmseg->shm_perm.mode |= SHMSEG_REMOVED; if (shmseg->shm_nattch <= 0) { uobj = shmseg->_shm_internal; shm_free_segment(IPCID_TO_IX(shmid)); } break; case SHM_LOCK: case SHM_UNLOCK: if ((error = kauth_authorize_system(cred, KAUTH_SYSTEM_SYSVIPC, (cmd == SHM_LOCK) ? KAUTH_REQ_SYSTEM_SYSVIPC_SHM_LOCK : KAUTH_REQ_SYSTEM_SYSVIPC_SHM_UNLOCK, NULL, NULL, NULL)) != 0) break; error = shm_memlock(shmseg, shmid, cmd); break; default: error = EINVAL; } mutex_exit(&shm_lock); if (uobj != NULL) uao_detach(uobj); return error; } /* * Try to take an already existing segment. * => must be called with shm_lock held; * => called from one place, thus, inline; */ static inline int shmget_existing(struct lwp *l, const struct sys_shmget_args *uap, int mode, register_t *retval) { struct shmid_ds *shmseg; kauth_cred_t cred = l->l_cred; int segnum, error; again: KASSERT(mutex_owned(&shm_lock)); /* Find segment by key */ for (segnum = 0; segnum < shminfo.shmmni; segnum++) if ((shmsegs[segnum].shm_perm.mode & SHMSEG_ALLOCATED) && shmsegs[segnum].shm_perm._key == SCARG(uap, key)) break; if (segnum == shminfo.shmmni) { /* Not found */ return -1; } shmseg = &shmsegs[segnum]; if (shmseg->shm_perm.mode & SHMSEG_REMOVED) { /* * This segment is in the process of being allocated. Wait * until it's done, and look the key up again (in case the * allocation failed or it was freed). */ shmseg->shm_perm.mode |= SHMSEG_WANTED; error = cv_wait_sig(&shm_cv[segnum], &shm_lock); if (error) return error; goto again; } /* * First check the flags, to generate a useful error when a * segment already exists. */ if ((SCARG(uap, shmflg) & (IPC_CREAT | IPC_EXCL)) == (IPC_CREAT | IPC_EXCL)) return EEXIST; /* Check the permission and segment size. */ error = ipcperm(cred, &shmseg->shm_perm, mode); if (error) return error; if (SCARG(uap, size) && SCARG(uap, size) > shmseg->shm_segsz) return EINVAL; *retval = IXSEQ_TO_IPCID(segnum, shmseg->shm_perm); return 0; } int sys_shmget(struct lwp *l, const struct sys_shmget_args *uap, register_t *retval) { /* { syscallarg(key_t) key; syscallarg(size_t) size; syscallarg(int) shmflg; } */ struct shmid_ds *shmseg; kauth_cred_t cred = l->l_cred; key_t key = SCARG(uap, key); size_t size; int error, mode, segnum; bool lockmem; mode = SCARG(uap, shmflg) & ACCESSPERMS; if (SCARG(uap, shmflg) & _SHM_RMLINGER) mode |= SHMSEG_RMLINGER; SHMPRINTF(("shmget: key 0x%lx size 0x%zx shmflg 0x%x mode 0x%x\n", SCARG(uap, key), SCARG(uap, size), SCARG(uap, shmflg), mode)); mutex_enter(&shm_lock); /* In case of reallocation, we will wait for completion */ while (__predict_false(shm_realloc_state)) cv_wait(&shm_realloc_cv, &shm_lock); if (key != IPC_PRIVATE) { error = shmget_existing(l, uap, mode, retval); if (error != -1) { mutex_exit(&shm_lock); return error; } if ((SCARG(uap, shmflg) & IPC_CREAT) == 0) { mutex_exit(&shm_lock); return ENOENT; } } error = 0; /* * Check the for the limits. */ size = SCARG(uap, size); if (size < shminfo.shmmin || size > shminfo.shmmax) { mutex_exit(&shm_lock); return EINVAL; } if (shm_nused >= shminfo.shmmni) { mutex_exit(&shm_lock); return ENOSPC; } size = round_page(size); if (shm_committed + btoc(size) > shminfo.shmall) { mutex_exit(&shm_lock); return ENOMEM; } /* Find the first available segment */ if (shm_last_free < 0) { for (segnum = 0; segnum < shminfo.shmmni; segnum++) if (shmsegs[segnum].shm_perm.mode & SHMSEG_FREE) break; KASSERT(segnum < shminfo.shmmni); } else { segnum = shm_last_free; shm_last_free = -1; } /* * Initialize the segment. * We will drop the lock while allocating the memory, thus mark the * segment present, but removed, that no other thread could take it. * Also, disable reallocation, while lock is dropped. */ shmseg = &shmsegs[segnum]; shmseg->shm_perm.mode = SHMSEG_ALLOCATED | SHMSEG_REMOVED; shm_committed += btoc(size); shm_nused++; lockmem = shm_use_phys; shm_realloc_disable++; mutex_exit(&shm_lock); /* Allocate the memory object and lock it if needed */ shmseg->_shm_internal = uao_create(size, 0); if (lockmem) { /* Wire the pages and tag it */ error = uvm_obj_wirepages(shmseg->_shm_internal, 0, size, NULL); if (error) { uao_detach(shmseg->_shm_internal); mutex_enter(&shm_lock); shm_free_segment(segnum); shm_realloc_disable--; mutex_exit(&shm_lock); return error; } } /* * Please note, while segment is marked, there are no need to hold the * lock, while setting it (except shm_perm.mode). */ shmseg->shm_perm._key = SCARG(uap, key); shmseg->shm_perm._seq = (shmseg->shm_perm._seq + 1) & 0x7fff; *retval = IXSEQ_TO_IPCID(segnum, shmseg->shm_perm); shmseg->shm_perm.cuid = shmseg->shm_perm.uid = kauth_cred_geteuid(cred); shmseg->shm_perm.cgid = shmseg->shm_perm.gid = kauth_cred_getegid(cred); shmseg->shm_segsz = SCARG(uap, size); shmseg->shm_cpid = l->l_proc->p_pid; shmseg->shm_lpid = shmseg->shm_nattch = 0; shmseg->shm_atime = shmseg->shm_dtime = 0; shmseg->shm_ctime = time_second; /* * Segment is initialized. * Enter the lock, mark as allocated, and notify waiters (if any). * Also, unmark the state of reallocation. */ mutex_enter(&shm_lock); shmseg->shm_perm.mode = (shmseg->shm_perm.mode & SHMSEG_WANTED) | (mode & (ACCESSPERMS | SHMSEG_RMLINGER)) | SHMSEG_ALLOCATED | (lockmem ? SHMSEG_WIRED : 0); if (shmseg->shm_perm.mode & SHMSEG_WANTED) { shmseg->shm_perm.mode &= ~SHMSEG_WANTED; cv_broadcast(&shm_cv[segnum]); } shm_realloc_disable--; cv_broadcast(&shm_realloc_cv); mutex_exit(&shm_lock); return error; } void shmfork(struct vmspace *vm1, struct vmspace *vm2) { struct shmmap_state *shmmap_s; struct shmmap_entry *shmmap_se; SHMPRINTF(("shmfork %p->%p\n", vm1, vm2)); mutex_enter(&shm_lock); vm2->vm_shm = vm1->vm_shm; if (vm1->vm_shm) { shmmap_s = (struct shmmap_state *)vm1->vm_shm; SLIST_FOREACH(shmmap_se, &shmmap_s->entries, next) shmsegs[IPCID_TO_IX(shmmap_se->shmid)].shm_nattch++; shmmap_s->nrefs++; } mutex_exit(&shm_lock); } void shmexit(struct vmspace *vm) { struct shmmap_state *shmmap_s; struct shmmap_entry *shmmap_se; mutex_enter(&shm_lock); shmmap_s = (struct shmmap_state *)vm->vm_shm; if (shmmap_s == NULL) { mutex_exit(&shm_lock); return; } vm->vm_shm = NULL; if (--shmmap_s->nrefs > 0) { SHMPRINTF(("shmexit: vm %p drop ref (%d entries), refs = %d\n", vm, shmmap_s->nitems, shmmap_s->nrefs)); SLIST_FOREACH(shmmap_se, &shmmap_s->entries, next) { shmsegs[IPCID_TO_IX(shmmap_se->shmid)].shm_nattch--; } mutex_exit(&shm_lock); return; } SHMPRINTF(("shmexit: vm %p cleanup (%d entries)\n", vm, shmmap_s->nitems)); if (shmmap_s->nitems == 0) { mutex_exit(&shm_lock); kmem_free(shmmap_s, sizeof(struct shmmap_state)); return; } /* * Delete the entry from shm map. */ for (;;) { struct shmid_ds *shmseg; struct uvm_object *uobj; size_t sz; shmmap_se = SLIST_FIRST(&shmmap_s->entries); KASSERT(shmmap_se != NULL); shmseg = &shmsegs[IPCID_TO_IX(shmmap_se->shmid)]; sz = (shmseg->shm_segsz + PGOFSET) & ~PGOFSET; /* shm_delete_mapping() removes from the list. */ uobj = shm_delete_mapping(shmmap_s, shmmap_se); mutex_exit(&shm_lock); uvm_deallocate(&vm->vm_map, shmmap_se->va, sz); if (uobj != NULL) { uao_detach(uobj); } kmem_free(shmmap_se, sizeof(struct shmmap_entry)); if (SLIST_EMPTY(&shmmap_s->entries)) { break; } mutex_enter(&shm_lock); KASSERT(!SLIST_EMPTY(&shmmap_s->entries)); } kmem_free(shmmap_s, sizeof(struct shmmap_state)); } static int shmrealloc(int newshmni) { vaddr_t v; struct shmid_ds *oldshmsegs, *newshmsegs; kcondvar_t *newshm_cv, *oldshm_cv; size_t sz; int i, lsegid, oldshmni; if (newshmni < 1) return EINVAL; /* Allocate new memory area */ sz = ALIGN(newshmni * sizeof(struct shmid_ds)) + ALIGN(newshmni * sizeof(kcondvar_t)); sz = round_page(sz); v = uvm_km_alloc(kernel_map, sz, 0, UVM_KMF_WIRED|UVM_KMF_ZERO); if (v == 0) return ENOMEM; mutex_enter(&shm_lock); while (shm_realloc_state || shm_realloc_disable) cv_wait(&shm_realloc_cv, &shm_lock); /* * Get the number of last segment. Fail we are trying to * reallocate less memory than we use. */ lsegid = 0; for (i = 0; i < shminfo.shmmni; i++) if ((shmsegs[i].shm_perm.mode & SHMSEG_FREE) == 0) lsegid = i; if (lsegid >= newshmni) { mutex_exit(&shm_lock); uvm_km_free(kernel_map, v, sz, UVM_KMF_WIRED); return EBUSY; } shm_realloc_state = true; newshmsegs = (void *)v; newshm_cv = (void *)((uintptr_t)newshmsegs + ALIGN(newshmni * sizeof(struct shmid_ds))); /* Copy all memory to the new area */ for (i = 0; i < shm_nused; i++) { cv_init(&newshm_cv[i], "shmwait"); (void)memcpy(&newshmsegs[i], &shmsegs[i], sizeof(newshmsegs[0])); } /* Mark as free all new segments, if there is any */ for (; i < newshmni; i++) { cv_init(&newshm_cv[i], "shmwait"); newshmsegs[i].shm_perm.mode = SHMSEG_FREE; newshmsegs[i].shm_perm._seq = 0; } oldshmsegs = shmsegs; oldshmni = shminfo.shmmni; shminfo.shmmni = newshmni; shmsegs = newshmsegs; shm_cv = newshm_cv; /* Reallocation completed - notify all waiters, if any */ shm_realloc_state = false; cv_broadcast(&shm_realloc_cv); mutex_exit(&shm_lock); /* Release now unused resources. */ oldshm_cv = (void *)((uintptr_t)oldshmsegs + ALIGN(oldshmni * sizeof(struct shmid_ds))); for (i = 0; i < oldshmni; i++) cv_destroy(&oldshm_cv[i]); sz = ALIGN(oldshmni * sizeof(struct shmid_ds)) + ALIGN(oldshmni * sizeof(kcondvar_t)); sz = round_page(sz); uvm_km_free(kernel_map, (vaddr_t)oldshmsegs, sz, UVM_KMF_WIRED); return 0; } int shminit(void) { vaddr_t v; size_t sz; int i; mutex_init(&shm_lock, MUTEX_DEFAULT, IPL_NONE); cv_init(&shm_realloc_cv, "shmrealc"); /* Allocate the wired memory for our structures */ sz = ALIGN(shminfo.shmmni * sizeof(struct shmid_ds)) + ALIGN(shminfo.shmmni * sizeof(kcondvar_t)); sz = round_page(sz); v = uvm_km_alloc(kernel_map, sz, 0, UVM_KMF_WIRED|UVM_KMF_ZERO); if (v == 0) { printf("sysv_shm: cannot allocate memory"); return ENOMEM; } shmsegs = (void *)v; shm_cv = (void *)((uintptr_t)shmsegs + ALIGN(shminfo.shmmni * sizeof(struct shmid_ds))); if (shminfo.shmmax == 0) shminfo.shmmax = uimax(physmem / 4, 1024) * PAGE_SIZE; else shminfo.shmmax *= PAGE_SIZE; shminfo.shmall = shminfo.shmmax / PAGE_SIZE; for (i = 0; i < shminfo.shmmni; i++) { cv_init(&shm_cv[i], "shmwait"); shmsegs[i].shm_perm.mode = SHMSEG_FREE; shmsegs[i].shm_perm._seq = 0; } shm_last_free = 0; shm_nused = 0; shm_committed = 0; shm_realloc_disable = 0; shm_realloc_state = false; kern_has_sysvshm = 1; /* Load the callback function pointers for the uvm subsystem */ uvm_shmexit = shmexit; uvm_shmfork = shmfork; return 0; } int shmfini(void) { size_t sz; int i; vaddr_t v = (vaddr_t)shmsegs; mutex_enter(&shm_lock); if (shm_nused) { mutex_exit(&shm_lock); return 1; } /* Clear the callback function pointers for the uvm subsystem */ uvm_shmexit = NULL; uvm_shmfork = NULL; /* Destroy all condvars */ for (i = 0; i < shminfo.shmmni; i++) cv_destroy(&shm_cv[i]); cv_destroy(&shm_realloc_cv); /* Free the allocated/wired memory */ sz = ALIGN(shminfo.shmmni * sizeof(struct shmid_ds)) + ALIGN(shminfo.shmmni * sizeof(kcondvar_t)); sz = round_page(sz); uvm_km_free(kernel_map, v, sz, UVM_KMF_WIRED); /* Release and destroy our mutex */ mutex_exit(&shm_lock); mutex_destroy(&shm_lock); kern_has_sysvshm = 0; return 0; } static int sysctl_ipc_shmmni(SYSCTLFN_ARGS) { int newsize, error; struct sysctlnode node; node = *rnode; node.sysctl_data = &newsize; newsize = shminfo.shmmni; error = sysctl_lookup(SYSCTLFN_CALL(&node)); if (error || newp == NULL) return error; sysctl_unlock(); error = shmrealloc(newsize); sysctl_relock(); return error; } static int sysctl_ipc_shmmaxpgs(SYSCTLFN_ARGS) { uint32_t newsize; int error; struct sysctlnode node; node = *rnode; node.sysctl_data = &newsize; newsize = shminfo.shmall; error = sysctl_lookup(SYSCTLFN_CALL(&node)); if (error || newp == NULL) return error; if (newsize < 1) return EINVAL; shminfo.shmall = newsize; shminfo.shmmax = (uint64_t)shminfo.shmall * PAGE_SIZE; return 0; } static int sysctl_ipc_shmmax(SYSCTLFN_ARGS) { uint64_t newsize; int error; struct sysctlnode node; node = *rnode; node.sysctl_data = &newsize; newsize = shminfo.shmmax; error = sysctl_lookup(SYSCTLFN_CALL(&node)); if (error || newp == NULL) return error; if (newsize < PAGE_SIZE) return EINVAL; shminfo.shmmax = round_page(newsize); shminfo.shmall = shminfo.shmmax >> PAGE_SHIFT; return 0; } SYSCTL_SETUP(sysctl_ipc_shm_setup, "sysctl kern.ipc subtree setup") { sysctl_createv(clog, 0, NULL, NULL, CTLFLAG_PERMANENT, CTLTYPE_NODE, "ipc", SYSCTL_DESCR("SysV IPC options"), NULL, 0, NULL, 0, CTL_KERN, KERN_SYSVIPC, CTL_EOL); sysctl_createv(clog, 0, NULL, NULL, CTLFLAG_PERMANENT | CTLFLAG_READWRITE, CTLTYPE_QUAD, "shmmax", SYSCTL_DESCR("Max shared memory segment size in bytes"), sysctl_ipc_shmmax, 0, &shminfo.shmmax, 0, CTL_KERN, KERN_SYSVIPC, KERN_SYSVIPC_SHMMAX, CTL_EOL); sysctl_createv(clog, 0, NULL, NULL, CTLFLAG_PERMANENT | CTLFLAG_READWRITE, CTLTYPE_INT, "shmmni", SYSCTL_DESCR("Max number of shared memory identifiers"), sysctl_ipc_shmmni, 0, &shminfo.shmmni, 0, CTL_KERN, KERN_SYSVIPC, KERN_SYSVIPC_SHMMNI, CTL_EOL); sysctl_createv(clog, 0, NULL, NULL, CTLFLAG_PERMANENT | CTLFLAG_READWRITE, CTLTYPE_INT, "shmseg", SYSCTL_DESCR("Max shared memory segments per process"), NULL, 0, &shminfo.shmseg, 0, CTL_KERN, KERN_SYSVIPC, KERN_SYSVIPC_SHMSEG, CTL_EOL); sysctl_createv(clog, 0, NULL, NULL, CTLFLAG_PERMANENT | CTLFLAG_READWRITE, CTLTYPE_INT, "shmmaxpgs", SYSCTL_DESCR("Max amount of shared memory in pages"), sysctl_ipc_shmmaxpgs, 0, &shminfo.shmall, 0, CTL_KERN, KERN_SYSVIPC, KERN_SYSVIPC_SHMMAXPGS, CTL_EOL); sysctl_createv(clog, 0, NULL, NULL, CTLFLAG_PERMANENT | CTLFLAG_READWRITE, CTLTYPE_INT, "shm_use_phys", SYSCTL_DESCR("Enable/disable locking of shared memory in " "physical memory"), NULL, 0, &shm_use_phys, 0, CTL_KERN, KERN_SYSVIPC, KERN_SYSVIPC_SHMUSEPHYS, CTL_EOL); } |
| 4 7 745 744 1 4 3 1 1 5 1 5 10 2 2 37 10 10 6 8 14 8 5 2 1 3 3 3 2 1 22 22 22 2 15 8 7 7 2 3 5 9 9 1 2 2 1 1 1 2 5 1 6 4 1 1 4 2 2 4 1 9 4 5 1 1 3 13 4 10 6 3 5 3 14 3 4 7 3 6 3 3 11 3 8 2 3 3 3 3 3 1 1 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 197 198 199 200 201 202 203 204 205 206 207 208 209 210 211 212 213 214 215 216 217 218 219 220 221 222 223 224 225 226 227 228 229 230 231 232 233 234 235 236 237 238 239 240 241 242 243 244 245 246 247 248 249 250 251 252 253 254 255 256 257 258 259 260 261 262 263 264 265 266 267 268 269 270 271 272 273 274 275 276 277 278 279 280 281 282 283 284 285 286 287 288 289 290 291 292 293 294 295 296 297 298 299 300 301 302 303 304 305 306 307 308 309 310 311 312 313 314 315 316 317 318 319 320 321 322 323 324 325 326 327 328 329 330 331 332 333 334 335 336 337 338 339 340 341 342 343 344 345 346 347 348 349 350 351 352 353 354 355 356 357 358 359 360 361 362 363 364 365 366 367 368 369 370 371 372 373 374 375 376 377 378 379 380 381 382 383 384 385 386 387 388 389 390 391 392 393 394 395 396 397 398 399 400 401 402 403 404 405 406 407 408 409 410 411 412 413 414 415 416 417 418 419 420 421 422 423 424 425 426 427 428 429 430 431 432 433 434 435 436 437 438 439 440 441 442 443 444 445 446 447 448 449 450 451 452 453 454 455 456 457 458 459 460 461 462 463 464 465 466 467 468 469 470 471 472 473 474 475 476 477 478 479 480 481 482 483 484 485 486 487 488 489 490 491 492 493 494 495 496 497 498 499 500 501 502 503 504 505 506 507 508 509 510 511 512 513 514 515 516 517 518 519 520 521 522 523 524 525 526 527 528 529 530 531 532 533 534 535 536 537 538 539 540 541 542 543 544 545 546 547 548 549 550 551 552 553 554 555 556 557 558 559 560 561 562 563 564 565 566 567 568 569 570 571 572 573 574 575 576 577 578 579 580 581 582 583 584 585 586 587 588 589 590 591 592 593 594 595 596 597 598 599 600 601 602 603 604 605 606 607 608 609 610 611 612 613 614 615 616 617 618 619 620 621 622 623 624 625 626 627 628 629 630 631 632 633 634 635 636 637 638 639 640 641 642 643 644 645 646 647 648 649 650 651 652 653 654 655 656 657 658 659 660 661 662 663 664 665 666 667 668 669 670 671 672 673 674 675 676 677 678 679 680 681 682 683 684 685 686 687 688 689 690 691 692 693 694 695 696 697 698 699 700 701 702 703 704 705 706 707 708 709 710 711 712 713 714 715 716 717 718 719 720 721 722 723 724 725 726 727 728 729 730 731 732 733 734 735 736 737 738 739 740 741 742 743 744 745 746 747 748 749 750 751 752 753 754 755 756 757 758 759 760 761 762 763 764 765 766 767 768 769 770 771 772 773 774 775 776 777 778 779 780 781 782 783 784 785 786 787 788 789 790 791 792 793 794 795 796 797 798 799 800 801 802 803 804 805 806 807 808 809 810 811 812 813 814 815 816 817 818 819 820 821 822 823 824 825 826 827 828 829 830 831 832 833 834 835 836 837 838 839 840 841 842 843 844 845 846 847 848 849 850 851 852 853 854 855 856 857 858 859 860 861 862 863 864 865 866 867 868 869 870 871 872 873 874 875 876 877 878 879 880 881 882 883 884 885 886 887 888 889 890 891 892 893 894 895 896 897 898 899 900 901 902 903 904 905 906 907 908 909 910 911 912 913 914 915 916 917 918 919 920 921 922 923 924 925 926 927 928 929 930 931 932 933 934 935 936 937 938 939 940 941 942 943 944 945 946 947 948 949 950 951 952 953 954 955 956 957 958 959 960 961 962 963 964 965 966 967 968 969 970 971 972 973 974 975 976 977 978 979 980 981 982 983 984 985 986 987 988 989 990 991 992 993 994 995 996 997 998 999 1000 1001 1002 1003 1004 1005 1006 1007 1008 1009 1010 1011 1012 1013 1014 1015 1016 1017 1018 1019 1020 1021 1022 1023 1024 1025 1026 1027 1028 1029 1030 1031 1032 1033 1034 1035 1036 1037 1038 1039 1040 1041 1042 1043 1044 1045 1046 1047 1048 1049 1050 1051 1052 1053 1054 1055 1056 1057 1058 1059 1060 1061 1062 1063 1064 1065 1066 1067 1068 1069 1070 1071 1072 1073 1074 1075 1076 1077 1078 1079 1080 1081 1082 1083 1084 1085 1086 1087 1088 1089 1090 | /* $NetBSD: uipc_sem.c,v 1.60 2020/12/14 23:12:12 chs Exp $ */ /*- * Copyright (c) 2011, 2019 The NetBSD Foundation, Inc. * All rights reserved. * * This code is derived from software contributed to The NetBSD Foundation * by Mindaugas Rasiukevicius and Jason R. Thorpe. * * Redistribution and use in source and binary forms, with or without * modification, are permitted provided that the following conditions * are met: * 1. Redistributions of source code must retain the above copyright * notice, this list of conditions and the following disclaimer. * 2. Redistributions in binary form must reproduce the above copyright * notice, this list of conditions and the following disclaimer in the * documentation and/or other materials provided with the distribution. * * THIS SOFTWARE IS PROVIDED BY THE NETBSD FOUNDATION, INC. AND CONTRIBUTORS * ``AS IS'' AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED * TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR * PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE FOUNDATION OR CONTRIBUTORS * BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR * CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF * SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS * INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN * CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) * ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE * POSSIBILITY OF SUCH DAMAGE. */ /* * Copyright (c) 2002 Alfred Perlstein <alfred@FreeBSD.org> * All rights reserved. * * Redistribution and use in source and binary forms, with or without * modification, are permitted provided that the following conditions * are met: * 1. Redistributions of source code must retain the above copyright * notice, this list of conditions and the following disclaimer. * 2. Redistributions in binary form must reproduce the above copyright * notice, this list of conditions and the following disclaimer in the * documentation and/or other materials provided with the distribution. * * THIS SOFTWARE IS PROVIDED BY THE AUTHOR AND CONTRIBUTORS ``AS IS'' AND * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE * ARE DISCLAIMED. IN NO EVENT SHALL THE AUTHOR OR CONTRIBUTORS BE LIABLE * FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL * DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS * OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) * HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT * LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY * OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF * SUCH DAMAGE. */ /* * Implementation of POSIX semaphore. */ #include <sys/cdefs.h> __KERNEL_RCSID(0, "$NetBSD: uipc_sem.c,v 1.60 2020/12/14 23:12:12 chs Exp $"); #include <sys/param.h> #include <sys/kernel.h> #include <sys/atomic.h> #include <sys/proc.h> #include <sys/lwp.h> #include <sys/ksem.h> #include <sys/syscall.h> #include <sys/stat.h> #include <sys/kmem.h> #include <sys/fcntl.h> #include <sys/file.h> #include <sys/filedesc.h> #include <sys/kauth.h> #include <sys/module.h> #include <sys/mount.h> #include <sys/mutex.h> #include <sys/rwlock.h> #include <sys/semaphore.h> #include <sys/syscall.h> #include <sys/syscallargs.h> #include <sys/syscallvar.h> #include <sys/sysctl.h> #include <sys/uidinfo.h> #include <sys/cprng.h> MODULE(MODULE_CLASS_MISC, ksem, NULL); #define SEM_MAX_NAMELEN NAME_MAX #define KS_UNLINKED 0x01 static kmutex_t ksem_lock __cacheline_aligned; static LIST_HEAD(,ksem) ksem_head __cacheline_aligned; static u_int nsems_total __cacheline_aligned; static u_int nsems __cacheline_aligned; static krwlock_t ksem_pshared_lock __cacheline_aligned; static LIST_HEAD(, ksem) *ksem_pshared_hashtab __cacheline_aligned; static u_long ksem_pshared_hashmask __read_mostly; #define KSEM_PSHARED_HASHSIZE 32 static kauth_listener_t ksem_listener; static int ksem_sysinit(void); static int ksem_sysfini(bool); static int ksem_modcmd(modcmd_t, void *); static void ksem_release(ksem_t *, int); static int ksem_close_fop(file_t *); static int ksem_stat_fop(file_t *, struct stat *); static int ksem_read_fop(file_t *, off_t *, struct uio *, kauth_cred_t, int); static const struct fileops semops = { .fo_name = "sem", .fo_read = ksem_read_fop, .fo_write = fbadop_write, .fo_ioctl = fbadop_ioctl, .fo_fcntl = fnullop_fcntl, .fo_poll = fnullop_poll, .fo_stat = ksem_stat_fop, .fo_close = ksem_close_fop, .fo_kqfilter = fnullop_kqfilter, .fo_restart = fnullop_restart, }; static const struct syscall_package ksem_syscalls[] = { { SYS__ksem_init, 0, (sy_call_t *)sys__ksem_init }, { SYS__ksem_open, 0, (sy_call_t *)sys__ksem_open }, { SYS__ksem_unlink, 0, (sy_call_t *)sys__ksem_unlink }, { SYS__ksem_close, 0, (sy_call_t *)sys__ksem_close }, { SYS__ksem_post, 0, (sy_call_t *)sys__ksem_post }, { SYS__ksem_wait, 0, (sy_call_t *)sys__ksem_wait }, { SYS__ksem_trywait, 0, (sy_call_t *)sys__ksem_trywait }, { SYS__ksem_getvalue, 0, (sy_call_t *)sys__ksem_getvalue }, { SYS__ksem_destroy, 0, (sy_call_t *)sys__ksem_destroy }, { SYS__ksem_timedwait, 0, (sy_call_t *)sys__ksem_timedwait }, { 0, 0, NULL }, }; struct sysctllog *ksem_clog; int ksem_max = KSEM_MAX; static int name_copyin(const char *uname, char **name) { *name = kmem_alloc(SEM_MAX_NAMELEN, KM_SLEEP); int error = copyinstr(uname, *name, SEM_MAX_NAMELEN, NULL); if (error) kmem_free(*name, SEM_MAX_NAMELEN); return error; } static void name_destroy(char **name) { if (!*name) return; kmem_free(*name, SEM_MAX_NAMELEN); *name = NULL; } static int ksem_listener_cb(kauth_cred_t cred, kauth_action_t action, void *cookie, void *arg0, void *arg1, void *arg2, void *arg3) { ksem_t *ks; mode_t mode; if (action != KAUTH_SYSTEM_SEMAPHORE) return KAUTH_RESULT_DEFER; ks = arg1; mode = ks->ks_mode; if ((kauth_cred_geteuid(cred) == ks->ks_uid && (mode & S_IWUSR) != 0) || (kauth_cred_getegid(cred) == ks->ks_gid && (mode & S_IWGRP) != 0) || (mode & S_IWOTH) != 0) return KAUTH_RESULT_ALLOW; return KAUTH_RESULT_DEFER; } static int ksem_sysinit(void) { int error; const struct sysctlnode *rnode; mutex_init(&ksem_lock, MUTEX_DEFAULT, IPL_NONE); LIST_INIT(&ksem_head); nsems_total = 0; nsems = 0; rw_init(&ksem_pshared_lock); ksem_pshared_hashtab = hashinit(KSEM_PSHARED_HASHSIZE, HASH_LIST, true, &ksem_pshared_hashmask); KASSERT(ksem_pshared_hashtab != NULL); ksem_listener = kauth_listen_scope(KAUTH_SCOPE_SYSTEM, ksem_listener_cb, NULL); /* Define module-specific sysctl tree */ ksem_clog = NULL; sysctl_createv(&ksem_clog, 0, NULL, &rnode, CTLFLAG_PERMANENT, CTLTYPE_NODE, "posix", SYSCTL_DESCR("POSIX options"), NULL, 0, NULL, 0, CTL_KERN, CTL_CREATE, CTL_EOL); sysctl_createv(&ksem_clog, 0, &rnode, NULL, CTLFLAG_PERMANENT | CTLFLAG_READWRITE, CTLTYPE_INT, "semmax", SYSCTL_DESCR("Maximal number of semaphores"), NULL, 0, &ksem_max, 0, CTL_CREATE, CTL_EOL); sysctl_createv(&ksem_clog, 0, &rnode, NULL, CTLFLAG_PERMANENT | CTLFLAG_READONLY, CTLTYPE_INT, "semcnt", SYSCTL_DESCR("Current number of semaphores"), NULL, 0, &nsems, 0, CTL_CREATE, CTL_EOL); error = syscall_establish(NULL, ksem_syscalls); if (error) { (void)ksem_sysfini(false); } return error; } static int ksem_sysfini(bool interface) { int error; if (interface) { error = syscall_disestablish(NULL, ksem_syscalls); if (error != 0) { return error; } /* * Make sure that no semaphores are in use. Note: semops * must be unused at this point. */ if (nsems_total) { error = syscall_establish(NULL, ksem_syscalls); KASSERT(error == 0); return EBUSY; } } kauth_unlisten_scope(ksem_listener); hashdone(ksem_pshared_hashtab, HASH_LIST, ksem_pshared_hashmask); rw_destroy(&ksem_pshared_lock); mutex_destroy(&ksem_lock); sysctl_teardown(&ksem_clog); return 0; } static int ksem_modcmd(modcmd_t cmd, void *arg) { switch (cmd) { case MODULE_CMD_INIT: return ksem_sysinit(); case MODULE_CMD_FINI: return ksem_sysfini(true); default: return ENOTTY; } } static ksem_t * ksem_lookup(const char *name) { ksem_t *ks; KASSERT(mutex_owned(&ksem_lock)); LIST_FOREACH(ks, &ksem_head, ks_entry) { if (strcmp(ks->ks_name, name) == 0) { mutex_enter(&ks->ks_lock); return ks; } } return NULL; } static int ksem_perm(lwp_t *l, ksem_t *ks) { kauth_cred_t uc = l->l_cred; KASSERT(mutex_owned(&ks->ks_lock)); if (kauth_authorize_system(uc, KAUTH_SYSTEM_SEMAPHORE, 0, ks, NULL, NULL) != 0) return EACCES; return 0; } /* * Bits 1..23 are random, just pluck a few of those and assume the * distribution is going to be pretty good. */ #define KSEM_PSHARED_HASH(id) (((id) >> 1) & ksem_pshared_hashmask) static void ksem_remove_pshared(ksem_t *ksem) { rw_enter(&ksem_pshared_lock, RW_WRITER); LIST_REMOVE(ksem, ks_entry); rw_exit(&ksem_pshared_lock); } static ksem_t * ksem_lookup_pshared_locked(intptr_t id) { u_long bucket = KSEM_PSHARED_HASH(id); ksem_t *ksem = NULL; /* ksem_t is locked and referenced upon return. */ LIST_FOREACH(ksem, &ksem_pshared_hashtab[bucket], ks_entry) { if (ksem->ks_pshared_id == id) { mutex_enter(&ksem->ks_lock); if (ksem->ks_pshared_proc == NULL) { /* * This entry is dead, and in the process * of being torn down; skip it. */ mutex_exit(&ksem->ks_lock); continue; } ksem->ks_ref++; KASSERT(ksem->ks_ref != 0); return ksem; } } return NULL; } static ksem_t * ksem_lookup_pshared(intptr_t id) { rw_enter(&ksem_pshared_lock, RW_READER); ksem_t *ksem = ksem_lookup_pshared_locked(id); rw_exit(&ksem_pshared_lock); return ksem; } static void ksem_alloc_pshared_id(ksem_t *ksem) { ksem_t *ksem0; uint32_t try; KASSERT(ksem->ks_pshared_proc != NULL); rw_enter(&ksem_pshared_lock, RW_WRITER); for (;;) { try = (cprng_fast32() & ~KSEM_MARKER_MASK) | KSEM_PSHARED_MARKER; if ((ksem0 = ksem_lookup_pshared_locked(try)) == NULL) { /* Got it! */ break; } ksem_release(ksem0, -1); } ksem->ks_pshared_id = try; u_long bucket = KSEM_PSHARED_HASH(ksem->ks_pshared_id); LIST_INSERT_HEAD(&ksem_pshared_hashtab[bucket], ksem, ks_entry); rw_exit(&ksem_pshared_lock); } /* * ksem_get: get the semaphore from the descriptor. * * => locks the semaphore, if found, and holds an extra reference. * => holds a reference on the file descriptor. */ static int ksem_get(intptr_t id, ksem_t **ksret, int *fdp) { ksem_t *ks; int fd; if ((id & KSEM_MARKER_MASK) == KSEM_PSHARED_MARKER) { /* * ksem_lookup_pshared() returns the ksem_t * * locked and referenced. */ ks = ksem_lookup_pshared(id); if (ks == NULL) return EINVAL; KASSERT(ks->ks_pshared_id == id); KASSERT(ks->ks_pshared_proc != NULL); fd = -1; } else if (id <= INT_MAX) { fd = (int)id; file_t *fp = fd_getfile(fd); if (__predict_false(fp == NULL)) return EINVAL; if (__predict_false(fp->f_type != DTYPE_SEM)) { fd_putfile(fd); return EINVAL; } ks = fp->f_ksem; mutex_enter(&ks->ks_lock); ks->ks_ref++; } else { return EINVAL; } *ksret = ks; *fdp = fd; return 0; } /* * ksem_create: allocate and setup a new semaphore structure. */ static int ksem_create(lwp_t *l, const char *name, ksem_t **ksret, mode_t mode, u_int val) { ksem_t *ks; kauth_cred_t uc; char *kname; size_t len; /* Pre-check for the limit. */ if (nsems >= ksem_max) { return ENFILE; } if (val > SEM_VALUE_MAX) { return EINVAL; } if (name != NULL) { len = strlen(name); if (len > SEM_MAX_NAMELEN) { return ENAMETOOLONG; } /* Name must start with a '/' but not contain one. */ if (*name != '/' || len < 2 || strchr(name + 1, '/') != NULL) { return EINVAL; } kname = kmem_alloc(++len, KM_SLEEP); strlcpy(kname, name, len); } else { kname = NULL; len = 0; } ks = kmem_zalloc(sizeof(ksem_t), KM_SLEEP); mutex_init(&ks->ks_lock, MUTEX_DEFAULT, IPL_NONE); cv_init(&ks->ks_cv, "psem"); ks->ks_name = kname; ks->ks_namelen = len; ks->ks_mode = mode; ks->ks_value = val; ks->ks_ref = 1; uc = l->l_cred; ks->ks_uid = kauth_cred_geteuid(uc); ks->ks_gid = kauth_cred_getegid(uc); chgsemcnt(ks->ks_uid, 1); atomic_inc_uint(&nsems_total); *ksret = ks; return 0; } static void ksem_free(ksem_t *ks) { KASSERT(!cv_has_waiters(&ks->ks_cv)); chgsemcnt(ks->ks_uid, -1); atomic_dec_uint(&nsems_total); if (ks->ks_pshared_id) { KASSERT(ks->ks_pshared_proc == NULL); ksem_remove_pshared(ks); } if (ks->ks_name) { KASSERT(ks->ks_namelen > 0); kmem_free(ks->ks_name, ks->ks_namelen); } mutex_destroy(&ks->ks_lock); cv_destroy(&ks->ks_cv); kmem_free(ks, sizeof(ksem_t)); } #define KSEM_ID_IS_PSHARED(id) \ (((id) & KSEM_MARKER_MASK) == KSEM_PSHARED_MARKER) static void ksem_release(ksem_t *ksem, int fd) { bool destroy = false; KASSERT(mutex_owned(&ksem->ks_lock)); KASSERT(ksem->ks_ref > 0); if (--ksem->ks_ref == 0) { /* * Destroy if the last reference and semaphore is unnamed, * or unlinked (for named semaphore). */ destroy = (ksem->ks_flags & KS_UNLINKED) || (ksem->ks_name == NULL); } mutex_exit(&ksem->ks_lock); if (destroy) { ksem_free(ksem); } if (fd != -1) { fd_putfile(fd); } } int sys__ksem_init(struct lwp *l, const struct sys__ksem_init_args *uap, register_t *retval) { /* { unsigned int value; intptr_t *idp; } */ return do_ksem_init(l, SCARG(uap, value), SCARG(uap, idp), copyin, copyout); } int do_ksem_init(lwp_t *l, u_int val, intptr_t *idp, copyin_t docopyin, copyout_t docopyout) { proc_t *p = l->l_proc; ksem_t *ks; file_t *fp; intptr_t id, arg; int fd, error; /* * Newer versions of librt / libpthread pass us 'PSRD' in *idp to * indicate that a pshared semaphore is wanted. In that case we * allocate globally unique ID and return that, rather than the * process-scoped file descriptor ID. */ error = (*docopyin)(idp, &arg, sizeof(*idp)); if (error) { return error; } error = fd_allocfile(&fp, &fd); if (error) { return error; } fp->f_type = DTYPE_SEM; fp->f_flag = FREAD | FWRITE; fp->f_ops = &semops; if (fd >= KSEM_MARKER_MIN) { /* * This is super-unlikely, but we check for it anyway * because potential collisions with the pshared marker * would be bad. */ fd_abort(p, fp, fd); return EMFILE; } /* Note the mode does not matter for anonymous semaphores. */ error = ksem_create(l, NULL, &ks, 0, val); if (error) { fd_abort(p, fp, fd); return error; } if (arg == KSEM_PSHARED) { ks->ks_pshared_proc = curproc; ks->ks_pshared_fd = fd; ksem_alloc_pshared_id(ks); id = ks->ks_pshared_id; } else { id = (intptr_t)fd; } error = (*docopyout)(&id, idp, sizeof(*idp)); if (error) { ksem_free(ks); fd_abort(p, fp, fd); return error; } fp->f_ksem = ks; fd_affix(p, fp, fd); return error; } int sys__ksem_open(struct lwp *l, const struct sys__ksem_open_args *uap, register_t *retval) { /* { const char *name; int oflag; mode_t mode; unsigned int value; intptr_t *idp; } */ return do_ksem_open(l, SCARG(uap, name), SCARG(uap, oflag), SCARG(uap, mode), SCARG(uap, value), SCARG(uap, idp), copyout); } int do_ksem_open(struct lwp *l, const char *semname, int oflag, mode_t mode, unsigned int value, intptr_t *idp, copyout_t docopyout) { char *name; proc_t *p = l->l_proc; ksem_t *ksnew = NULL, *ks; file_t *fp; intptr_t id; int fd, error; error = name_copyin(semname, &name); if (error) { return error; } error = fd_allocfile(&fp, &fd); if (error) { name_destroy(&name); return error; } fp->f_type = DTYPE_SEM; fp->f_flag = FREAD | FWRITE; fp->f_ops = &semops; if (fd >= KSEM_MARKER_MIN) { /* * This is super-unlikely, but we check for it anyway * because potential collisions with the pshared marker * would be bad. */ fd_abort(p, fp, fd); return EMFILE; } /* * The ID (file descriptor number) can be stored early. * Note that zero is a special value for libpthread. */ id = (intptr_t)fd; error = (*docopyout)(&id, idp, sizeof(*idp)); if (error) { goto err; } if (oflag & O_CREAT) { /* Create a new semaphore. */ error = ksem_create(l, name, &ksnew, mode, value); if (error) { goto err; } KASSERT(ksnew != NULL); } /* Lookup for a semaphore with such name. */ mutex_enter(&ksem_lock); ks = ksem_lookup(name); name_destroy(&name); if (ks) { KASSERT(mutex_owned(&ks->ks_lock)); mutex_exit(&ksem_lock); /* Check for exclusive create. */ if (oflag & O_EXCL) { mutex_exit(&ks->ks_lock); error = EEXIST; goto err; } /* * Verify permissions. If we can access it, * add the reference of this thread. */ error = ksem_perm(l, ks); if (error == 0) { ks->ks_ref++; } mutex_exit(&ks->ks_lock); if (error) { goto err; } } else { /* Fail if not found and not creating. */ if ((oflag & O_CREAT) == 0) { mutex_exit(&ksem_lock); KASSERT(ksnew == NULL); error = ENOENT; goto err; } /* Check for the limit locked. */ if (nsems >= ksem_max) { mutex_exit(&ksem_lock); error = ENFILE; goto err; } /* * Finally, insert semaphore into the list. * Note: it already has the initial reference. */ ks = ksnew; LIST_INSERT_HEAD(&ksem_head, ks, ks_entry); nsems++; mutex_exit(&ksem_lock); ksnew = NULL; } KASSERT(ks != NULL); fp->f_ksem = ks; fd_affix(p, fp, fd); err: name_destroy(&name); if (error) { fd_abort(p, fp, fd); } if (ksnew) { ksem_free(ksnew); } return error; } int sys__ksem_close(struct lwp *l, const struct sys__ksem_close_args *uap, register_t *retval) { /* { intptr_t id; } */ intptr_t id = SCARG(uap, id); int fd, error; ksem_t *ks; error = ksem_get(id, &ks, &fd); if (error) { return error; } /* This is only for named semaphores. */ if (ks->ks_name == NULL) { error = EINVAL; } ksem_release(ks, -1); if (error) { if (fd != -1) fd_putfile(fd); return error; } return fd_close(fd); } static int ksem_read_fop(file_t *fp, off_t *offset, struct uio *uio, kauth_cred_t cred, int flags) { size_t len; char *name; ksem_t *ks = fp->f_ksem; mutex_enter(&ks->ks_lock); len = ks->ks_namelen; name = ks->ks_name; mutex_exit(&ks->ks_lock); if (name == NULL || len == 0) return 0; return uiomove(name, len, uio); } static int ksem_stat_fop(file_t *fp, struct stat *ub) { ksem_t *ks = fp->f_ksem; mutex_enter(&ks->ks_lock); memset(ub, 0, sizeof(*ub)); ub->st_mode = ks->ks_mode | ((ks->ks_name && ks->ks_namelen) ? _S_IFLNK : _S_IFREG); ub->st_uid = ks->ks_uid; ub->st_gid = ks->ks_gid; ub->st_size = ks->ks_value; ub->st_blocks = (ub->st_size) ? 1 : 0; ub->st_nlink = ks->ks_ref; ub->st_blksize = 4096; nanotime(&ub->st_atimespec); ub->st_mtimespec = ub->st_ctimespec = ub->st_birthtimespec = ub->st_atimespec; /* * Left as 0: st_dev, st_ino, st_rdev, st_flags, st_gen. * XXX (st_dev, st_ino) should be unique. */ mutex_exit(&ks->ks_lock); return 0; } static int ksem_close_fop(file_t *fp) { ksem_t *ks = fp->f_ksem; mutex_enter(&ks->ks_lock); if (ks->ks_pshared_id) { if (ks->ks_pshared_proc != curproc) { /* Do nothing if this is not the creator. */ mutex_exit(&ks->ks_lock); return 0; } /* Mark this semaphore as dead. */ ks->ks_pshared_proc = NULL; } ksem_release(ks, -1); return 0; } int sys__ksem_unlink(struct lwp *l, const struct sys__ksem_unlink_args *uap, register_t *retval) { /* { const char *name; } */ char *name; ksem_t *ks; u_int refcnt; int error; error = name_copyin(SCARG(uap, name), &name); if (error) return error; mutex_enter(&ksem_lock); ks = ksem_lookup(name); name_destroy(&name); if (ks == NULL) { mutex_exit(&ksem_lock); return ENOENT; } KASSERT(mutex_owned(&ks->ks_lock)); /* Verify permissions. */ error = ksem_perm(l, ks); if (error) { mutex_exit(&ks->ks_lock); mutex_exit(&ksem_lock); return error; } /* Remove from the global list. */ LIST_REMOVE(ks, ks_entry); nsems--; mutex_exit(&ksem_lock); refcnt = ks->ks_ref; if (refcnt) { /* Mark as unlinked, if there are references. */ ks->ks_flags |= KS_UNLINKED; } mutex_exit(&ks->ks_lock); if (refcnt == 0) { ksem_free(ks); } return 0; } int sys__ksem_post(struct lwp *l, const struct sys__ksem_post_args *uap, register_t *retval) { /* { intptr_t id; } */ int fd, error; ksem_t *ks; error = ksem_get(SCARG(uap, id), &ks, &fd); if (error) { return error; } KASSERT(mutex_owned(&ks->ks_lock)); if (ks->ks_value == SEM_VALUE_MAX) { error = EOVERFLOW; goto out; } ks->ks_value++; if (ks->ks_waiters) { cv_broadcast(&ks->ks_cv); } out: ksem_release(ks, fd); return error; } int do_ksem_wait(lwp_t *l, intptr_t id, bool try_p, struct timespec *abstime) { int fd, error, timeo; ksem_t *ks; error = ksem_get(id, &ks, &fd); if (error) { return error; } KASSERT(mutex_owned(&ks->ks_lock)); while (ks->ks_value == 0) { ks->ks_waiters++; if (!try_p && abstime != NULL) { error = ts2timo(CLOCK_REALTIME, TIMER_ABSTIME, abstime, &timeo, NULL); if (error != 0) goto out; } else { timeo = 0; } error = try_p ? EAGAIN : cv_timedwait_sig(&ks->ks_cv, &ks->ks_lock, timeo); ks->ks_waiters--; if (error) goto out; } ks->ks_value--; out: ksem_release(ks, fd); return error; } int sys__ksem_wait(struct lwp *l, const struct sys__ksem_wait_args *uap, register_t *retval) { /* { intptr_t id; } */ return do_ksem_wait(l, SCARG(uap, id), false, NULL); } int sys__ksem_timedwait(struct lwp *l, const struct sys__ksem_timedwait_args *uap, register_t *retval) { /* { intptr_t id; const struct timespec *abstime; } */ struct timespec ts; int error; error = copyin(SCARG(uap, abstime), &ts, sizeof(ts)); if (error != 0) return error; if (ts.tv_sec < 0 || ts.tv_nsec < 0 || ts.tv_nsec >= 1000000000) return EINVAL; error = do_ksem_wait(l, SCARG(uap, id), false, &ts); if (error == EWOULDBLOCK) error = ETIMEDOUT; return error; } int sys__ksem_trywait(struct lwp *l, const struct sys__ksem_trywait_args *uap, register_t *retval) { /* { intptr_t id; } */ return do_ksem_wait(l, SCARG(uap, id), true, NULL); } int sys__ksem_getvalue(struct lwp *l, const struct sys__ksem_getvalue_args *uap, register_t *retval) { /* { intptr_t id; unsigned int *value; } */ int fd, error; ksem_t *ks; unsigned int val; error = ksem_get(SCARG(uap, id), &ks, &fd); if (error) { return error; } KASSERT(mutex_owned(&ks->ks_lock)); val = ks->ks_value; ksem_release(ks, fd); return copyout(&val, SCARG(uap, value), sizeof(val)); } int sys__ksem_destroy(struct lwp *l, const struct sys__ksem_destroy_args *uap, register_t *retval) { /* { intptr_t id; } */ int fd, error; ksem_t *ks; intptr_t id = SCARG(uap, id); error = ksem_get(id, &ks, &fd); if (error) { return error; } KASSERT(mutex_owned(&ks->ks_lock)); /* Operation is only for unnamed semaphores. */ if (ks->ks_name != NULL) { error = EINVAL; goto out; } /* Cannot destroy if there are waiters. */ if (ks->ks_waiters) { error = EBUSY; goto out; } if (KSEM_ID_IS_PSHARED(id)) { /* Cannot destroy if we did't create it. */ KASSERT(fd == -1); KASSERT(ks->ks_pshared_proc != NULL); if (ks->ks_pshared_proc != curproc) { error = EINVAL; goto out; } fd = ks->ks_pshared_fd; /* Mark it dead so subsequent lookups fail. */ ks->ks_pshared_proc = NULL; /* Do an fd_getfile() to for the benefit of fd_close(). */ file_t *fp __diagused = fd_getfile(fd); KASSERT(fp != NULL); KASSERT(fp->f_ksem == ks); } out: ksem_release(ks, -1); if (error) { if (!KSEM_ID_IS_PSHARED(id)) fd_putfile(fd); return error; } return fd_close(fd); } |
| 1 1 8 8 8 4 4 4 4 4 1 1 1 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 197 198 199 200 201 202 203 204 205 206 207 208 209 210 211 212 213 214 215 216 217 218 219 220 221 222 223 224 225 226 227 228 229 230 231 232 233 234 235 236 237 238 239 240 241 242 243 244 245 246 247 248 249 250 251 252 253 254 255 256 257 258 259 260 261 262 263 264 265 266 267 268 269 270 271 272 273 274 275 276 277 278 279 280 281 282 283 284 285 286 287 288 289 290 291 292 293 294 295 296 297 298 299 300 301 302 303 304 305 306 307 308 309 310 311 312 313 314 315 316 317 318 319 320 321 322 323 324 325 326 327 328 329 330 331 332 333 334 335 336 337 338 339 340 341 342 343 344 345 346 347 348 349 350 351 352 353 354 355 356 357 358 359 360 361 362 363 364 365 366 367 368 369 370 371 372 373 374 375 376 377 378 379 380 381 382 383 384 385 386 387 388 389 390 391 392 393 394 395 396 397 398 399 400 401 402 403 404 405 406 407 408 409 410 411 412 413 414 415 416 417 418 419 420 421 422 423 424 425 426 427 428 429 430 431 432 433 434 435 436 437 438 439 440 441 442 443 444 445 446 447 448 449 450 451 452 453 454 455 456 457 458 459 460 461 462 463 464 465 466 467 468 469 470 471 472 473 474 475 476 477 478 479 480 481 482 483 484 485 486 487 488 489 490 491 492 493 494 495 496 497 498 499 500 501 502 503 504 505 506 507 508 509 510 511 512 513 514 515 516 517 518 519 520 521 522 523 524 525 526 527 528 529 530 531 532 533 534 535 536 537 538 539 540 541 542 543 544 545 546 547 548 549 550 551 552 553 554 555 556 557 558 559 560 561 562 563 564 565 566 567 568 569 570 571 572 573 574 575 576 577 578 579 580 581 582 583 584 585 586 587 588 589 590 591 592 593 594 595 596 597 598 599 600 601 602 603 604 605 606 607 608 609 610 611 612 613 614 615 616 617 618 619 620 621 622 623 624 625 626 627 628 629 630 631 632 633 634 635 636 637 638 639 640 641 642 643 644 645 646 647 648 649 650 651 652 653 654 655 656 657 658 659 660 661 662 663 664 665 666 667 668 669 670 671 672 673 674 675 676 677 678 679 680 681 682 683 684 685 686 687 688 689 690 691 692 693 694 695 696 697 698 699 700 701 702 703 704 705 706 707 708 709 710 711 712 713 714 715 716 717 718 719 720 721 722 723 724 725 726 727 728 729 730 731 732 733 734 735 736 737 738 739 740 741 742 743 744 745 746 747 748 749 750 751 752 753 754 755 756 757 758 759 760 761 762 763 764 765 766 767 768 769 770 771 772 773 774 775 776 777 778 779 780 781 782 783 784 785 786 787 788 789 790 791 792 793 794 795 796 797 798 799 800 801 802 803 804 805 806 807 808 809 810 811 812 813 814 815 816 817 818 819 820 821 822 823 824 825 826 827 828 829 830 831 832 833 834 835 836 837 838 839 840 841 842 843 844 845 846 847 848 849 850 851 852 853 854 855 856 857 858 859 860 861 862 863 864 865 866 867 868 869 870 871 872 873 874 875 876 877 878 879 880 881 882 883 884 885 886 887 888 889 890 891 892 893 894 895 896 897 898 899 900 901 902 903 904 905 906 907 908 909 910 911 912 913 914 915 916 917 918 919 920 921 922 923 924 925 926 927 928 929 930 931 932 933 934 935 936 937 938 939 940 941 942 943 944 945 946 947 948 949 950 951 952 953 954 955 956 957 958 959 960 961 962 963 964 965 966 967 968 969 970 971 972 973 974 975 976 977 978 979 980 981 982 983 984 985 986 987 988 989 990 991 992 993 994 995 996 997 998 999 1000 1001 1002 1003 1004 1005 1006 1007 1008 1009 1010 1011 1012 1013 1014 1015 1016 1017 1018 1019 1020 1021 1022 1023 1024 1025 1026 1027 1028 1029 1030 1031 1032 1033 1034 1035 1036 1037 1038 1039 1040 1041 1042 1043 1044 1045 1046 1047 1048 1049 1050 1051 1052 1053 1054 1055 1056 1057 1058 1059 1060 1061 1062 1063 1064 1065 1066 1067 1068 1069 1070 1071 1072 1073 1074 1075 1076 1077 1078 1079 1080 1081 1082 1083 1084 1085 1086 1087 1088 1089 1090 1091 1092 1093 1094 1095 1096 1097 1098 1099 1100 1101 1102 1103 1104 1105 1106 1107 1108 1109 1110 1111 1112 1113 1114 1115 1116 1117 1118 1119 1120 1121 1122 1123 1124 1125 1126 1127 1128 1129 1130 1131 1132 1133 1134 1135 1136 1137 1138 1139 1140 1141 1142 1143 1144 1145 1146 1147 1148 1149 1150 1151 1152 1153 1154 1155 1156 1157 1158 1159 1160 1161 1162 1163 1164 1165 1166 1167 1168 1169 1170 1171 1172 1173 1174 1175 1176 1177 1178 1179 1180 1181 1182 1183 1184 1185 1186 1187 1188 1189 1190 1191 1192 1193 1194 1195 1196 1197 1198 1199 1200 1201 1202 1203 1204 1205 1206 1207 1208 1209 1210 1211 1212 1213 1214 1215 1216 1217 1218 1219 1220 1221 1222 1223 1224 1225 1226 1227 1228 1229 1230 1231 1232 1233 1234 1235 1236 1237 1238 1239 1240 1241 1242 1243 1244 1245 1246 1247 1248 1249 1250 1251 1252 1253 1254 1255 1256 1257 1258 1259 1260 1261 1262 1263 1264 1265 1266 1267 1268 1269 1270 1271 1272 1273 1274 1275 1276 1277 1278 1279 1280 1281 1282 1283 1284 1285 1286 1287 1288 1289 1290 1291 1292 1293 1294 1295 1296 1297 1298 1299 1300 1301 1302 1303 1304 1305 1306 1307 1308 1309 1310 1311 1312 1313 1314 1315 1316 1317 1318 1319 1320 1321 1322 1323 1324 1325 1326 1327 1328 1329 1330 1331 1332 1333 1334 1335 1336 1337 1338 1339 1340 1341 1342 1343 1344 1345 1346 1347 1348 1349 1350 1351 1352 1353 1354 1355 1356 1357 1358 1359 1360 1361 1362 1363 1364 1365 1366 1367 1368 1369 1370 1371 1372 1373 1374 1375 1376 1377 1378 1379 1380 1381 1382 1383 1384 1385 1386 1387 1388 1389 1390 1391 1392 1393 1394 1395 1396 1397 1398 1399 1400 1401 1402 1403 1404 1405 1406 1407 1408 1409 1410 1411 1412 1413 1414 1415 1416 1417 1418 1419 1420 1421 1422 1423 1424 1425 1426 1427 1428 1429 1430 1431 1432 1433 1434 1435 1436 1437 1438 1439 1440 1441 1442 1443 1444 1445 1446 1447 1448 1449 1450 1451 1452 1453 1454 1455 1456 1457 1458 1459 1460 1461 1462 1463 1464 1465 1466 1467 1468 1469 1470 1471 1472 1473 1474 1475 1476 1477 1478 1479 1480 1481 1482 1483 1484 1485 1486 1487 1488 1489 1490 1491 1492 1493 1494 1495 1496 1497 1498 1499 1500 1501 1502 1503 1504 1505 1506 1507 1508 1509 1510 1511 1512 1513 1514 1515 1516 1517 1518 1519 1520 1521 1522 1523 1524 1525 1526 1527 1528 1529 1530 1531 1532 1533 1534 1535 1536 1537 1538 1539 1540 1541 1542 1543 1544 1545 1546 1547 1548 1549 1550 1551 1552 1553 1554 1555 1556 1557 1558 1559 1560 1561 1562 1563 1564 1565 1566 1567 1568 1569 1570 1571 1572 1573 1574 1575 1576 1577 1578 1579 1580 1581 1582 1583 1584 1585 1586 1587 1588 1589 1590 1591 1592 1593 1594 1595 1596 1597 1598 1599 1600 1601 1602 1603 1604 1605 1606 1607 1608 1609 1610 1611 1612 1613 1614 1615 1616 1617 1618 1619 1620 1621 1622 1623 1624 1625 1626 1627 1628 1629 1630 1631 1632 1633 1634 1635 1636 1637 1638 1639 1640 1641 1642 1643 1644 1645 1646 1647 1648 1649 1650 1651 1652 1653 1654 1655 1656 1657 1658 1659 1660 1661 1662 1663 1664 1665 1666 1667 1668 1669 1670 1671 1672 1673 1674 1675 1676 1677 1678 1679 1680 1681 1682 1683 1684 1685 1686 1687 1688 1689 1690 1691 1692 1693 1694 1695 1696 1697 1698 1699 1700 1701 1702 1703 1704 1705 1706 1707 1708 1709 1710 1711 1712 1713 1714 1715 1716 1717 1718 1719 1720 1721 1722 1723 1724 1725 1726 1727 1728 1729 1730 1731 1732 1733 1734 1735 1736 1737 1738 1739 1740 1741 1742 1743 1744 1745 1746 1747 1748 1749 1750 1751 1752 1753 1754 1755 1756 1757 1758 1759 1760 1761 1762 1763 1764 1765 1766 1767 1768 1769 1770 1771 1772 1773 1774 1775 1776 1777 1778 1779 1780 1781 1782 1783 1784 1785 1786 1787 1788 1789 1790 1791 1792 1793 1794 1795 1796 1797 1798 1799 1800 1801 1802 1803 1804 1805 1806 1807 1808 1809 1810 1811 1812 1813 1814 1815 1816 1817 1818 1819 1820 1821 1822 1823 1824 1825 1826 1827 1828 1829 1830 1831 1832 1833 1834 1835 1836 1837 1838 1839 1840 1841 1842 1843 1844 1845 1846 1847 1848 1849 1850 1851 1852 1853 1854 1855 1856 1857 1858 1859 1860 1861 1862 1863 1864 1865 1866 1867 1868 1869 1870 1871 1872 1873 1874 1875 1876 1877 1878 1879 1880 1881 1882 1883 1884 1885 1886 1887 1888 1889 1890 1891 1892 1893 1894 1895 1896 1897 1898 1899 1900 1901 1902 1903 1904 1905 1906 1907 1908 1909 1910 1911 1912 1913 1914 1915 1916 1917 1918 1919 1920 1921 1922 1923 1924 1925 1926 1927 1928 1929 1930 1931 1932 1933 1934 1935 1936 1937 1938 1939 1940 1941 1942 1943 1944 1945 1946 1947 1948 1949 1950 1951 1952 1953 1954 1955 1956 1957 1958 1959 1960 1961 1962 1963 1964 1965 1966 1967 1968 1969 1970 1971 1972 1973 1974 1975 1976 1977 1978 1979 1980 1981 1982 1983 1984 1985 1986 1987 1988 1989 1990 1991 1992 1993 1994 1995 1996 1997 1998 1999 2000 2001 2002 2003 2004 2005 2006 2007 2008 2009 2010 2011 2012 2013 2014 2015 2016 2017 2018 2019 2020 2021 2022 2023 2024 2025 2026 2027 2028 2029 2030 2031 2032 2033 2034 2035 2036 2037 2038 2039 2040 2041 2042 2043 2044 2045 2046 2047 2048 2049 2050 2051 2052 2053 2054 2055 2056 2057 2058 2059 2060 2061 2062 2063 2064 2065 2066 2067 2068 2069 2070 2071 2072 2073 2074 2075 2076 2077 2078 2079 2080 2081 2082 2083 2084 2085 2086 2087 2088 2089 2090 2091 2092 2093 2094 2095 2096 2097 2098 2099 2100 2101 2102 2103 2104 2105 2106 2107 2108 2109 2110 2111 2112 2113 2114 2115 2116 2117 2118 2119 2120 2121 2122 2123 2124 2125 2126 2127 2128 2129 2130 2131 2132 2133 2134 2135 2136 2137 2138 2139 2140 2141 2142 2143 2144 2145 2146 2147 2148 2149 2150 2151 2152 2153 2154 2155 2156 2157 2158 2159 2160 2161 2162 2163 2164 2165 2166 2167 2168 2169 2170 2171 2172 2173 2174 2175 2176 2177 2178 2179 2180 2181 2182 2183 2184 2185 2186 2187 2188 2189 2190 2191 2192 2193 2194 2195 2196 2197 2198 2199 2200 2201 2202 2203 2204 2205 2206 2207 2208 2209 2210 2211 2212 2213 2214 2215 2216 2217 2218 2219 2220 2221 2222 2223 2224 2225 2226 2227 2228 2229 2230 2231 2232 2233 2234 2235 2236 2237 2238 2239 2240 2241 2242 2243 2244 2245 2246 2247 2248 2249 2250 2251 2252 2253 2254 2255 2256 2257 2258 2259 2260 2261 2262 2263 2264 2265 2266 2267 2268 2269 2270 2271 2272 2273 2274 2275 2276 2277 2278 2279 2280 2281 2282 2283 2284 2285 2286 2287 2288 2289 2290 2291 2292 2293 2294 2295 2296 2297 2298 2299 2300 2301 2302 2303 2304 2305 2306 2307 2308 2309 2310 2311 2312 2313 2314 2315 2316 2317 2318 2319 2320 2321 2322 2323 2324 2325 2326 2327 2328 2329 2330 2331 2332 2333 2334 2335 2336 2337 2338 2339 2340 2341 2342 2343 2344 2345 2346 2347 2348 2349 2350 2351 2352 2353 2354 2355 2356 | /* $NetBSD: ugen.c,v 1.168 2021/09/26 01:16:09 thorpej Exp $ */ /* * Copyright (c) 1998, 2004 The NetBSD Foundation, Inc. * All rights reserved. * * This code is derived from software contributed to The NetBSD Foundation * by Lennart Augustsson (lennart@augustsson.net) at * Carlstedt Research & Technology. * * Copyright (c) 2006 BBN Technologies Corp. All rights reserved. * Effort sponsored in part by the Defense Advanced Research Projects * Agency (DARPA) and the Department of the Interior National Business * Center under agreement number NBCHC050166. * * Redistribution and use in source and binary forms, with or without * modification, are permitted provided that the following conditions * are met: * 1. Redistributions of source code must retain the above copyright * notice, this list of conditions and the following disclaimer. * 2. Redistributions in binary form must reproduce the above copyright * notice, this list of conditions and the following disclaimer in the * documentation and/or other materials provided with the distribution. * * THIS SOFTWARE IS PROVIDED BY THE NETBSD FOUNDATION, INC. AND CONTRIBUTORS * ``AS IS'' AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED * TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR * PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE FOUNDATION OR CONTRIBUTORS * BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR * CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF * SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS * INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN * CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) * ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE * POSSIBILITY OF SUCH DAMAGE. */ #include <sys/cdefs.h> __KERNEL_RCSID(0, "$NetBSD: ugen.c,v 1.168 2021/09/26 01:16:09 thorpej Exp $"); #ifdef _KERNEL_OPT #include "opt_compat_netbsd.h" #include "opt_usb.h" #endif #include <sys/param.h> #include <sys/systm.h> #include <sys/kernel.h> #include <sys/kmem.h> #include <sys/device.h> #include <sys/ioctl.h> #include <sys/conf.h> #include <sys/tty.h> #include <sys/file.h> #include <sys/select.h> #include <sys/proc.h> #include <sys/vnode.h> #include <sys/poll.h> #include <sys/compat_stub.h> #include <sys/module.h> #include <sys/rbtree.h> #include <dev/usb/usb.h> #include <dev/usb/usbdi.h> #include <dev/usb/usbdi_util.h> #include "ioconf.h" #ifdef UGEN_DEBUG #define DPRINTF(x) if (ugendebug) printf x #define DPRINTFN(n,x) if (ugendebug>(n)) printf x int ugendebug = 0; #else #define DPRINTF(x) #define DPRINTFN(n,x) #endif #define UGEN_CHUNK 128 /* chunk size for read */ #define UGEN_IBSIZE 1020 /* buffer size */ #define UGEN_BBSIZE 1024 #define UGEN_NISOREQS 4 /* number of outstanding xfer requests */ #define UGEN_NISORFRMS 8 /* number of transactions per req */ #define UGEN_NISOFRAMES (UGEN_NISORFRMS * UGEN_NISOREQS) #define UGEN_BULK_RA_WB_BUFSIZE 16384 /* default buffer size */ #define UGEN_BULK_RA_WB_BUFMAX (1 << 20) /* maximum allowed buffer */ struct isoreq { struct ugen_endpoint *sce; struct usbd_xfer *xfer; void *dmabuf; uint16_t sizes[UGEN_NISORFRMS]; }; struct ugen_endpoint { struct ugen_softc *sc; usb_endpoint_descriptor_t *edesc; struct usbd_interface *iface; int state; #define UGEN_SHORT_OK 0x04 /* short xfers are OK */ #define UGEN_BULK_RA 0x08 /* in bulk read-ahead mode */ #define UGEN_BULK_WB 0x10 /* in bulk write-behind mode */ #define UGEN_RA_WB_STOP 0x20 /* RA/WB xfer is stopped (buffer full/empty) */ struct usbd_pipe *pipeh; struct clist q; u_char *ibuf; /* start of buffer (circular for isoc) */ u_char *fill; /* location for input (isoc) */ u_char *limit; /* end of circular buffer (isoc) */ u_char *cur; /* current read location (isoc) */ uint32_t timeout; uint32_t ra_wb_bufsize; /* requested size for RA/WB buffer */ uint32_t ra_wb_reqsize; /* requested xfer length for RA/WB */ uint32_t ra_wb_used; /* how much is in buffer */ uint32_t ra_wb_xferlen; /* current xfer length for RA/WB */ struct usbd_xfer *ra_wb_xfer; struct isoreq isoreqs[UGEN_NISOREQS]; /* Keep these last; we don't overwrite them in ugen_set_config() */ #define UGEN_ENDPOINT_NONZERO_CRUFT offsetof(struct ugen_endpoint, rsel) struct selinfo rsel; kcondvar_t cv; }; struct ugen_softc { device_t sc_dev; /* base device */ struct usbd_device *sc_udev; struct rb_node sc_node; unsigned sc_unit; kmutex_t sc_lock; kcondvar_t sc_detach_cv; char sc_is_open[USB_MAX_ENDPOINTS]; struct ugen_endpoint sc_endpoints[USB_MAX_ENDPOINTS][2]; #define OUT 0 #define IN 1 int sc_refcnt; char sc_buffer[UGEN_BBSIZE]; u_char sc_dying; u_char sc_attached; }; static struct { kmutex_t lock; rb_tree_t tree; } ugenif __cacheline_aligned; static int compare_ugen(void *cookie, const void *vsca, const void *vscb) { const struct ugen_softc *sca = vsca; const struct ugen_softc *scb = vscb; if (sca->sc_unit < scb->sc_unit) return -1; if (sca->sc_unit > scb->sc_unit) return +1; return 0; } static int compare_ugen_key(void *cookie, const void *vsc, const void *vk) { const struct ugen_softc *sc = vsc; const unsigned *k = vk; if (sc->sc_unit < *k) return -1; if (sc->sc_unit > *k) return +1; return 0; } static const rb_tree_ops_t ugenif_tree_ops = { .rbto_compare_nodes = compare_ugen, .rbto_compare_key = compare_ugen_key, .rbto_node_offset = offsetof(struct ugen_softc, sc_node), }; static void ugenif_get_unit(struct ugen_softc *sc) { struct ugen_softc *sc0; unsigned i; mutex_enter(&ugenif.lock); for (i = 0, sc0 = RB_TREE_MIN(&ugenif.tree); sc0 != NULL && i == sc0->sc_unit; i++, sc0 = RB_TREE_NEXT(&ugenif.tree, sc0)) KASSERT(i < UINT_MAX); KASSERT(rb_tree_find_node(&ugenif.tree, &i) == NULL); sc->sc_unit = i; sc0 = rb_tree_insert_node(&ugenif.tree, sc); KASSERT(sc0 == sc); KASSERT(rb_tree_find_node(&ugenif.tree, &i) == sc); mutex_exit(&ugenif.lock); } static void ugenif_put_unit(struct ugen_softc *sc) { mutex_enter(&ugenif.lock); KASSERT(rb_tree_find_node(&ugenif.tree, &sc->sc_unit) == sc); rb_tree_remove_node(&ugenif.tree, sc); sc->sc_unit = -1; mutex_exit(&ugenif.lock); } static struct ugen_softc * ugenif_acquire(unsigned unit) { struct ugen_softc *sc; mutex_enter(&ugenif.lock); sc = rb_tree_find_node(&ugenif.tree, &unit); if (sc == NULL) goto out; mutex_enter(&sc->sc_lock); if (sc->sc_dying) { mutex_exit(&sc->sc_lock); sc = NULL; goto out; } KASSERT(sc->sc_refcnt < INT_MAX); sc->sc_refcnt++; mutex_exit(&sc->sc_lock); out: mutex_exit(&ugenif.lock); return sc; } static void ugenif_release(struct ugen_softc *sc) { mutex_enter(&sc->sc_lock); if (--sc->sc_refcnt < 0) cv_broadcast(&sc->sc_detach_cv); mutex_exit(&sc->sc_lock); } static dev_type_open(ugenopen); static dev_type_close(ugenclose); static dev_type_read(ugenread); static dev_type_write(ugenwrite); static dev_type_ioctl(ugenioctl); static dev_type_poll(ugenpoll); static dev_type_kqfilter(ugenkqfilter); const struct cdevsw ugen_cdevsw = { .d_open = ugenopen, .d_close = ugenclose, .d_read = ugenread, .d_write = ugenwrite, .d_ioctl = ugenioctl, .d_stop = nostop, .d_tty = notty, .d_poll = ugenpoll, .d_mmap = nommap, .d_kqfilter = ugenkqfilter, .d_discard = nodiscard, .d_flag = D_OTHER, }; Static void ugenintr(struct usbd_xfer *, void *, usbd_status); Static void ugen_isoc_rintr(struct usbd_xfer *, void *, usbd_status); Static void ugen_bulkra_intr(struct usbd_xfer *, void *, usbd_status); Static void ugen_bulkwb_intr(struct usbd_xfer *, void *, usbd_status); Static int ugen_do_read(struct ugen_softc *, int, struct uio *, int); Static int ugen_do_write(struct ugen_softc *, int, struct uio *, int); Static int ugen_do_ioctl(struct ugen_softc *, int, u_long, void *, int, struct lwp *); Static int ugen_set_config(struct ugen_softc *, int, int); Static usb_config_descriptor_t *ugen_get_cdesc(struct ugen_softc *, int, int *); Static usbd_status ugen_set_interface(struct ugen_softc *, int, int); Static int ugen_get_alt_index(struct ugen_softc *, int); Static void ugen_clear_endpoints(struct ugen_softc *); #define UGENUNIT(n) ((minor(n) >> 4) & 0xf) #define UGENENDPOINT(n) (minor(n) & 0xf) #define UGENDEV(u, e) (makedev(0, ((u) << 4) | (e))) static int ugenif_match(device_t, cfdata_t, void *); static void ugenif_attach(device_t, device_t, void *); static int ugen_match(device_t, cfdata_t, void *); static void ugen_attach(device_t, device_t, void *); static int ugen_detach(device_t, int); static int ugen_activate(device_t, enum devact); CFATTACH_DECL_NEW(ugen, sizeof(struct ugen_softc), ugen_match, ugen_attach, ugen_detach, ugen_activate); CFATTACH_DECL_NEW(ugenif, sizeof(struct ugen_softc), ugenif_match, ugenif_attach, ugen_detach, ugen_activate); /* toggle to control attach priority. -1 means "let autoconf decide" */ int ugen_override = -1; static int ugen_match(device_t parent, cfdata_t match, void *aux) { struct usb_attach_arg *uaa = aux; int override; if (ugen_override != -1) override = ugen_override; else override = match->cf_flags & 1; if (override) return UMATCH_HIGHEST; else if (uaa->uaa_usegeneric) return UMATCH_GENERIC; else return UMATCH_NONE; } static int ugenif_match(device_t parent, cfdata_t match, void *aux) { /* Assume that they knew what they configured! (see ugenif(4)) */ return UMATCH_HIGHEST; } static void ugen_attach(device_t parent, device_t self, void *aux) { struct usb_attach_arg *uaa = aux; struct usbif_attach_arg uiaa; memset(&uiaa, 0, sizeof(uiaa)); uiaa.uiaa_port = uaa->uaa_port; uiaa.uiaa_vendor = uaa->uaa_vendor; uiaa.uiaa_product = uaa->uaa_product; uiaa.uiaa_release = uaa->uaa_release; uiaa.uiaa_device = uaa->uaa_device; uiaa.uiaa_configno = -1; uiaa.uiaa_ifaceno = -1; ugenif_attach(parent, self, &uiaa); } static void ugenif_attach(device_t parent, device_t self, void *aux) { struct ugen_softc *sc = device_private(self); struct usbif_attach_arg *uiaa = aux; struct usbd_device *udev; char *devinfop; usbd_status err; int i, dir, conf; aprint_naive("\n"); aprint_normal("\n"); mutex_init(&sc->sc_lock, MUTEX_DEFAULT, IPL_SOFTUSB); cv_init(&sc->sc_detach_cv, "ugendet"); devinfop = usbd_devinfo_alloc(uiaa->uiaa_device, 0); aprint_normal_dev(self, "%s\n", devinfop); usbd_devinfo_free(devinfop); sc->sc_dev = self; sc->sc_udev = udev = uiaa->uiaa_device; for (i = 0; i < USB_MAX_ENDPOINTS; i++) { for (dir = OUT; dir <= IN; dir++) { struct ugen_endpoint *sce; sce = &sc->sc_endpoints[i][dir]; selinit(&sce->rsel); cv_init(&sce->cv, "ugensce"); } } if (!pmf_device_register(self, NULL, NULL)) aprint_error_dev(self, "couldn't establish power handler\n"); if (uiaa->uiaa_ifaceno < 0) { /* * If we attach the whole device, * set configuration index 0, the default one. */ err = usbd_set_config_index(udev, 0, 0); if (err) { aprint_error_dev(self, "setting configuration index 0 failed\n"); return; } } /* Get current configuration */ conf = usbd_get_config_descriptor(udev)->bConfigurationValue; /* Set up all the local state for this configuration. */ err = ugen_set_config(sc, conf, uiaa->uiaa_ifaceno < 0); if (err) { aprint_error_dev(self, "setting configuration %d failed\n", conf); return; } ugenif_get_unit(sc); usbd_add_drv_event(USB_EVENT_DRIVER_ATTACH, sc->sc_udev, sc->sc_dev); sc->sc_attached = 1; } Static void ugen_clear_endpoints(struct ugen_softc *sc) { /* Clear out the old info, but leave the selinfo and cv initialised. */ for (int i = 0; i < USB_MAX_ENDPOINTS; i++) { for (int dir = OUT; dir <= IN; dir++) { struct ugen_endpoint *sce = &sc->sc_endpoints[i][dir]; memset(sce, 0, UGEN_ENDPOINT_NONZERO_CRUFT); } } } Static int ugen_set_config(struct ugen_softc *sc, int configno, int chkopen) { struct usbd_device *dev = sc->sc_udev; usb_config_descriptor_t *cdesc; struct usbd_interface *iface; usb_endpoint_descriptor_t *ed; struct ugen_endpoint *sce; uint8_t niface, nendpt; int ifaceno, endptno, endpt; usbd_status err; int dir; DPRINTFN(1,("ugen_set_config: %s to configno %d, sc=%p\n", device_xname(sc->sc_dev), configno, sc)); KASSERT(KERNEL_LOCKED_P()); /* sc_is_open */ if (chkopen) { /* * We start at 1, not 0, because we don't care whether the * control endpoint is open or not. It is always present. */ for (endptno = 1; endptno < USB_MAX_ENDPOINTS; endptno++) if (sc->sc_is_open[endptno]) { DPRINTFN(1, ("ugen_set_config: %s - endpoint %d is open\n", device_xname(sc->sc_dev), endptno)); return USBD_IN_USE; } /* Prevent opening while we're setting the config. */ for (endptno = 1; endptno < USB_MAX_ENDPOINTS; endptno++) { KASSERT(!sc->sc_is_open[endptno]); sc->sc_is_open[endptno] = 1; } } /* Avoid setting the current value. */ cdesc = usbd_get_config_descriptor(dev); if (!cdesc || cdesc->bConfigurationValue != configno) { err = usbd_set_config_no(dev, configno, 1); if (err) goto out; } ugen_clear_endpoints(sc); err = usbd_interface_count(dev, &niface); if (err) goto out; for (ifaceno = 0; ifaceno < niface; ifaceno++) { DPRINTFN(1,("ugen_set_config: ifaceno %d\n", ifaceno)); err = usbd_device2interface_handle(dev, ifaceno, &iface); if (err) goto out; err = usbd_endpoint_count(iface, &nendpt); if (err) goto out; for (endptno = 0; endptno < nendpt; endptno++) { ed = usbd_interface2endpoint_descriptor(iface,endptno); KASSERT(ed != NULL); endpt = ed->bEndpointAddress; dir = UE_GET_DIR(endpt) == UE_DIR_IN ? IN : OUT; sce = &sc->sc_endpoints[UE_GET_ADDR(endpt)][dir]; DPRINTFN(1,("ugen_set_config: endptno %d, endpt=0x%02x" "(%d,%d), sce=%p\n", endptno, endpt, UE_GET_ADDR(endpt), UE_GET_DIR(endpt), sce)); sce->sc = sc; sce->edesc = ed; sce->iface = iface; } } err = USBD_NORMAL_COMPLETION; out: if (chkopen) { /* * Allow open again now that we're done trying to set * the config. */ for (endptno = 1; endptno < USB_MAX_ENDPOINTS; endptno++) { KASSERT(sc->sc_is_open[endptno]); sc->sc_is_open[endptno] = 0; } } return err; } static int ugenopen(dev_t dev, int flag, int mode, struct lwp *l) { struct ugen_softc *sc; int unit = UGENUNIT(dev); int endpt = UGENENDPOINT(dev); usb_endpoint_descriptor_t *edesc; struct ugen_endpoint *sce; int dir, isize; usbd_status err; struct usbd_xfer *xfer; int i, j; int error; int opened; KASSERT(KERNEL_LOCKED_P()); /* sc_is_open */ if ((sc = ugenif_acquire(unit)) == NULL) return ENXIO; DPRINTFN(5, ("ugenopen: flag=%d, mode=%d, unit=%d endpt=%d\n", flag, mode, unit, endpt)); /* The control endpoint allows multiple opens. */ if (endpt == USB_CONTROL_ENDPOINT) { opened = sc->sc_is_open[USB_CONTROL_ENDPOINT] = 1; error = 0; goto out; } if (sc->sc_is_open[endpt]) { error = EBUSY; goto out; } opened = sc->sc_is_open[endpt] = 1; /* Make sure there are pipes for all directions. */ for (dir = OUT; dir <= IN; dir++) { if (flag & (dir == OUT ? FWRITE : FREAD)) { sce = &sc->sc_endpoints[endpt][dir]; if (sce->edesc == NULL) { error = ENXIO; goto out; } } } /* Actually open the pipes. */ /* XXX Should back out properly if it fails. */ for (dir = OUT; dir <= IN; dir++) { if (!(flag & (dir == OUT ? FWRITE : FREAD))) continue; sce = &sc->sc_endpoints[endpt][dir]; sce->state = 0; sce->timeout = USBD_NO_TIMEOUT; DPRINTFN(5, ("ugenopen: sc=%p, endpt=%d, dir=%d, sce=%p\n", sc, endpt, dir, sce)); edesc = sce->edesc; switch (edesc->bmAttributes & UE_XFERTYPE) { case UE_INTERRUPT: if (dir == OUT) { err = usbd_open_pipe(sce->iface, edesc->bEndpointAddress, 0, &sce->pipeh); if (err) { error = EIO; goto out; } break; } isize = UGETW(edesc->wMaxPacketSize); if (isize == 0) { /* shouldn't happen */ error = EINVAL; goto out; } sce->ibuf = kmem_alloc(isize, KM_SLEEP); DPRINTFN(5, ("ugenopen: intr endpt=%d,isize=%d\n", endpt, isize)); if (clalloc(&sce->q, UGEN_IBSIZE, 0) == -1) { kmem_free(sce->ibuf, isize); sce->ibuf = NULL; error = ENOMEM; goto out; } err = usbd_open_pipe_intr(sce->iface, edesc->bEndpointAddress, USBD_SHORT_XFER_OK, &sce->pipeh, sce, sce->ibuf, isize, ugenintr, USBD_DEFAULT_INTERVAL); if (err) { clfree(&sce->q); kmem_free(sce->ibuf, isize); sce->ibuf = NULL; error = EIO; goto out; } DPRINTFN(5, ("ugenopen: interrupt open done\n")); break; case UE_BULK: err = usbd_open_pipe(sce->iface, edesc->bEndpointAddress, 0, &sce->pipeh); if (err) { error = EIO; goto out; } sce->ra_wb_bufsize = UGEN_BULK_RA_WB_BUFSIZE; /* * Use request size for non-RA/WB transfers * as the default. */ sce->ra_wb_reqsize = UGEN_BBSIZE; break; case UE_ISOCHRONOUS: if (dir == OUT) { error = EINVAL; goto out; } isize = UGETW(edesc->wMaxPacketSize); if (isize == 0) { /* shouldn't happen */ error = EINVAL; goto out; } sce->ibuf = kmem_alloc(isize * UGEN_NISOFRAMES, KM_SLEEP); sce->cur = sce->fill = sce->ibuf; sce->limit = sce->ibuf + isize * UGEN_NISOFRAMES; DPRINTFN(5, ("ugenopen: isoc endpt=%d, isize=%d\n", endpt, isize)); err = usbd_open_pipe(sce->iface, edesc->bEndpointAddress, 0, &sce->pipeh); if (err) { kmem_free(sce->ibuf, isize * UGEN_NISOFRAMES); sce->ibuf = NULL; error = EIO; goto out; } for (i = 0; i < UGEN_NISOREQS; ++i) { sce->isoreqs[i].sce = sce; err = usbd_create_xfer(sce->pipeh, isize * UGEN_NISORFRMS, 0, UGEN_NISORFRMS, &xfer); if (err) goto bad; sce->isoreqs[i].xfer = xfer; sce->isoreqs[i].dmabuf = usbd_get_buffer(xfer); for (j = 0; j < UGEN_NISORFRMS; ++j) sce->isoreqs[i].sizes[j] = isize; usbd_setup_isoc_xfer(xfer, &sce->isoreqs[i], sce->isoreqs[i].sizes, UGEN_NISORFRMS, 0, ugen_isoc_rintr); (void)usbd_transfer(xfer); } DPRINTFN(5, ("ugenopen: isoc open done\n")); break; bad: while (--i >= 0) { /* implicit buffer free */ usbd_destroy_xfer(sce->isoreqs[i].xfer); sce->isoreqs[i].xfer = NULL; } usbd_close_pipe(sce->pipeh); sce->pipeh = NULL; kmem_free(sce->ibuf, isize * UGEN_NISOFRAMES); sce->ibuf = NULL; error = ENOMEM; goto out; case UE_CONTROL: sce->timeout = USBD_DEFAULT_TIMEOUT; error = EINVAL; goto out; } } error = 0; out: if (error && opened) sc->sc_is_open[endpt] = 0; ugenif_release(sc); return error; } static void ugen_do_close(struct ugen_softc *sc, int flag, int endpt) { struct ugen_endpoint *sce; int dir; int i; KASSERT(KERNEL_LOCKED_P()); /* sc_is_open */ if (!sc->sc_is_open[endpt]) goto out; if (endpt == USB_CONTROL_ENDPOINT) { DPRINTFN(5, ("ugenclose: close control\n")); goto out; } for (dir = OUT; dir <= IN; dir++) { if (!(flag & (dir == OUT ? FWRITE : FREAD))) continue; sce = &sc->sc_endpoints[endpt][dir]; if (sce->pipeh == NULL) continue; DPRINTFN(5, ("ugenclose: endpt=%d dir=%d sce=%p\n", endpt, dir, sce)); usbd_abort_pipe(sce->pipeh); int isize = UGETW(sce->edesc->wMaxPacketSize); int msize = 0; switch (sce->edesc->bmAttributes & UE_XFERTYPE) { case UE_INTERRUPT: ndflush(&sce->q, sce->q.c_cc); clfree(&sce->q); msize = isize; break; case UE_ISOCHRONOUS: for (i = 0; i < UGEN_NISOREQS; ++i) { usbd_destroy_xfer(sce->isoreqs[i].xfer); sce->isoreqs[i].xfer = NULL; } msize = isize * UGEN_NISOFRAMES; break; case UE_BULK: if (sce->state & (UGEN_BULK_RA | UGEN_BULK_WB)) { usbd_destroy_xfer(sce->ra_wb_xfer); sce->ra_wb_xfer = NULL; msize = sce->ra_wb_bufsize; } break; default: break; } usbd_close_pipe(sce->pipeh); sce->pipeh = NULL; if (sce->ibuf != NULL) { kmem_free(sce->ibuf, msize); sce->ibuf = NULL; } } out: sc->sc_is_open[endpt] = 0; for (dir = OUT; dir <= IN; dir++) { sce = &sc->sc_endpoints[endpt][dir]; KASSERT(sce->pipeh == NULL); KASSERT(sce->ibuf == NULL); KASSERT(sce->ra_wb_xfer == NULL); for (i = 0; i < UGEN_NISOREQS; i++) KASSERT(sce->isoreqs[i].xfer == NULL); } } static int ugenclose(dev_t dev, int flag, int mode, struct lwp *l) { int endpt = UGENENDPOINT(dev); struct ugen_softc *sc; DPRINTFN(5, ("ugenclose: flag=%d, mode=%d, unit=%d, endpt=%d\n", flag, mode, UGENUNIT(dev), endpt)); KASSERT(KERNEL_LOCKED_P()); /* ugen_do_close */ if ((sc = ugenif_acquire(UGENUNIT(dev))) == NULL) return ENXIO; KASSERT(sc->sc_is_open[endpt]); ugen_do_close(sc, flag, endpt); KASSERT(!sc->sc_is_open[endpt]); ugenif_release(sc); return 0; } Static int ugen_do_read(struct ugen_softc *sc, int endpt, struct uio *uio, int flag) { struct ugen_endpoint *sce = &sc->sc_endpoints[endpt][IN]; uint32_t n, tn; struct usbd_xfer *xfer; usbd_status err; int error = 0; DPRINTFN(5, ("%s: ugenread: %d\n", device_xname(sc->sc_dev), endpt)); if (endpt == USB_CONTROL_ENDPOINT) return ENODEV; KASSERT(sce->edesc); KASSERT(sce->pipeh); switch (sce->edesc->bmAttributes & UE_XFERTYPE) { case UE_INTERRUPT: /* Block until activity occurred. */ mutex_enter(&sc->sc_lock); while (sce->q.c_cc == 0) { if (flag & IO_NDELAY) { mutex_exit(&sc->sc_lock); return EWOULDBLOCK; } DPRINTFN(5, ("ugenread: sleep on %p\n", sce)); /* "ugenri" */ error = cv_timedwait_sig(&sce->cv, &sc->sc_lock, mstohz(sce->timeout)); DPRINTFN(5, ("ugenread: woke, error=%d\n", error)); if (sc->sc_dying) error = EIO; if (error) break; } mutex_exit(&sc->sc_lock); /* Transfer as many chunks as possible. */ while (sce->q.c_cc > 0 && uio->uio_resid > 0 && !error) { n = uimin(sce->q.c_cc, uio->uio_resid); if (n > sizeof(sc->sc_buffer)) n = sizeof(sc->sc_buffer); /* Remove a small chunk from the input queue. */ q_to_b(&sce->q, sc->sc_buffer, n); DPRINTFN(5, ("ugenread: got %d chars\n", n)); /* Copy the data to the user process. */ error = uiomove(sc->sc_buffer, n, uio); if (error) break; } break; case UE_BULK: if (sce->state & UGEN_BULK_RA) { DPRINTFN(5, ("ugenread: BULK_RA req: %zd used: %d\n", uio->uio_resid, sce->ra_wb_used)); xfer = sce->ra_wb_xfer; mutex_enter(&sc->sc_lock); if (sce->ra_wb_used == 0 && flag & IO_NDELAY) { mutex_exit(&sc->sc_lock); return EWOULDBLOCK; } while (uio->uio_resid > 0 && !error) { while (sce->ra_wb_used == 0) { DPRINTFN(5, ("ugenread: sleep on %p\n", sce)); /* "ugenrb" */ error = cv_timedwait_sig(&sce->cv, &sc->sc_lock, mstohz(sce->timeout)); DPRINTFN(5, ("ugenread: woke, error=%d\n", error)); if (sc->sc_dying) error = EIO; if (error) break; } /* Copy data to the process. */ while (uio->uio_resid > 0 && sce->ra_wb_used > 0) { n = uimin(uio->uio_resid, sce->ra_wb_used); n = uimin(n, sce->limit - sce->cur); error = uiomove(sce->cur, n, uio); if (error) break; sce->cur += n; sce->ra_wb_used -= n; if (sce->cur == sce->limit) sce->cur = sce->ibuf; } /* * If the transfers stopped because the * buffer was full, restart them. */ if (sce->state & UGEN_RA_WB_STOP && sce->ra_wb_used < sce->limit - sce->ibuf) { n = (sce->limit - sce->ibuf) - sce->ra_wb_used; usbd_setup_xfer(xfer, sce, NULL, uimin(n, sce->ra_wb_xferlen), 0, USBD_NO_TIMEOUT, ugen_bulkra_intr); sce->state &= ~UGEN_RA_WB_STOP; err = usbd_transfer(xfer); if (err != USBD_IN_PROGRESS) /* * The transfer has not been * queued. Setting STOP * will make us try * again at the next read. */ sce->state |= UGEN_RA_WB_STOP; } } mutex_exit(&sc->sc_lock); break; } error = usbd_create_xfer(sce->pipeh, UGEN_BBSIZE, 0, 0, &xfer); if (error) return error; while ((n = uimin(UGEN_BBSIZE, uio->uio_resid)) != 0) { DPRINTFN(1, ("ugenread: start transfer %d bytes\n",n)); tn = n; err = usbd_bulk_transfer(xfer, sce->pipeh, sce->state & UGEN_SHORT_OK ? USBD_SHORT_XFER_OK : 0, sce->timeout, sc->sc_buffer, &tn); if (err) { if (err == USBD_INTERRUPTED) error = EINTR; else if (err == USBD_TIMEOUT) error = ETIMEDOUT; else error = EIO; break; } DPRINTFN(1, ("ugenread: got %d bytes\n", tn)); error = uiomove(sc->sc_buffer, tn, uio); if (error || tn < n) break; } usbd_destroy_xfer(xfer); break; case UE_ISOCHRONOUS: mutex_enter(&sc->sc_lock); while (sce->cur == sce->fill) { if (flag & IO_NDELAY) { mutex_exit(&sc->sc_lock); return EWOULDBLOCK; } /* "ugenri" */ DPRINTFN(5, ("ugenread: sleep on %p\n", sce)); error = cv_timedwait_sig(&sce->cv, &sc->sc_lock, mstohz(sce->timeout)); DPRINTFN(5, ("ugenread: woke, error=%d\n", error)); if (sc->sc_dying) error = EIO; if (error) break; } while (sce->cur != sce->fill && uio->uio_resid > 0 && !error) { if(sce->fill > sce->cur) n = uimin(sce->fill - sce->cur, uio->uio_resid); else n = uimin(sce->limit - sce->cur, uio->uio_resid); DPRINTFN(5, ("ugenread: isoc got %d chars\n", n)); /* Copy the data to the user process. */ error = uiomove(sce->cur, n, uio); if (error) break; sce->cur += n; if (sce->cur >= sce->limit) sce->cur = sce->ibuf; } mutex_exit(&sc->sc_lock); break; default: return ENXIO; } return error; } static int ugenread(dev_t dev, struct uio *uio, int flag) { int endpt = UGENENDPOINT(dev); struct ugen_softc *sc; int error; if ((sc = ugenif_acquire(UGENUNIT(dev))) == NULL) return ENXIO; error = ugen_do_read(sc, endpt, uio, flag); ugenif_release(sc); return error; } Static int ugen_do_write(struct ugen_softc *sc, int endpt, struct uio *uio, int flag) { struct ugen_endpoint *sce = &sc->sc_endpoints[endpt][OUT]; uint32_t n; int error = 0; uint32_t tn; char *dbuf; struct usbd_xfer *xfer; usbd_status err; DPRINTFN(5, ("%s: ugenwrite: %d\n", device_xname(sc->sc_dev), endpt)); if (endpt == USB_CONTROL_ENDPOINT) return ENODEV; KASSERT(sce->edesc); KASSERT(sce->pipeh); switch (sce->edesc->bmAttributes & UE_XFERTYPE) { case UE_BULK: if (sce->state & UGEN_BULK_WB) { DPRINTFN(5, ("ugenwrite: BULK_WB req: %zd used: %d\n", uio->uio_resid, sce->ra_wb_used)); xfer = sce->ra_wb_xfer; mutex_enter(&sc->sc_lock); if (sce->ra_wb_used == sce->limit - sce->ibuf && flag & IO_NDELAY) { mutex_exit(&sc->sc_lock); return EWOULDBLOCK; } while (uio->uio_resid > 0 && !error) { while (sce->ra_wb_used == sce->limit - sce->ibuf) { DPRINTFN(5, ("ugenwrite: sleep on %p\n", sce)); /* "ugenwb" */ error = cv_timedwait_sig(&sce->cv, &sc->sc_lock, mstohz(sce->timeout)); DPRINTFN(5, ("ugenwrite: woke, error=%d\n", error)); if (sc->sc_dying) error = EIO; if (error) break; } /* Copy data from the process. */ while (uio->uio_resid > 0 && sce->ra_wb_used < sce->limit - sce->ibuf) { n = uimin(uio->uio_resid, (sce->limit - sce->ibuf) - sce->ra_wb_used); n = uimin(n, sce->limit - sce->fill); error = uiomove(sce->fill, n, uio); if (error) break; sce->fill += n; sce->ra_wb_used += n; if (sce->fill == sce->limit) sce->fill = sce->ibuf; } /* * If the transfers stopped because the * buffer was empty, restart them. */ if (sce->state & UGEN_RA_WB_STOP && sce->ra_wb_used > 0) { dbuf = (char *)usbd_get_buffer(xfer); n = uimin(sce->ra_wb_used, sce->ra_wb_xferlen); tn = uimin(n, sce->limit - sce->cur); memcpy(dbuf, sce->cur, tn); dbuf += tn; if (n - tn > 0) memcpy(dbuf, sce->ibuf, n - tn); usbd_setup_xfer(xfer, sce, NULL, n, 0, USBD_NO_TIMEOUT, ugen_bulkwb_intr); sce->state &= ~UGEN_RA_WB_STOP; err = usbd_transfer(xfer); if (err != USBD_IN_PROGRESS) /* * The transfer has not been * queued. Setting STOP * will make us try again * at the next read. */ sce->state |= UGEN_RA_WB_STOP; } } mutex_exit(&sc->sc_lock); break; } error = usbd_create_xfer(sce->pipeh, UGEN_BBSIZE, 0, 0, &xfer); if (error) return error; while ((n = uimin(UGEN_BBSIZE, uio->uio_resid)) != 0) { error = uiomove(sc->sc_buffer, n, uio); if (error) break; DPRINTFN(1, ("ugenwrite: transfer %d bytes\n", n)); err = usbd_bulk_transfer(xfer, sce->pipeh, 0, sce->timeout, sc->sc_buffer, &n); if (err) { if (err == USBD_INTERRUPTED) error = EINTR; else if (err == USBD_TIMEOUT) error = ETIMEDOUT; else error = EIO; break; } } usbd_destroy_xfer(xfer); break; case UE_INTERRUPT: error = usbd_create_xfer(sce->pipeh, UGETW(sce->edesc->wMaxPacketSize), 0, 0, &xfer); if (error) return error; while ((n = uimin(UGETW(sce->edesc->wMaxPacketSize), uio->uio_resid)) != 0) { error = uiomove(sc->sc_buffer, n, uio); if (error) break; DPRINTFN(1, ("ugenwrite: transfer %d bytes\n", n)); err = usbd_intr_transfer(xfer, sce->pipeh, 0, sce->timeout, sc->sc_buffer, &n); if (err) { if (err == USBD_INTERRUPTED) error = EINTR; else if (err == USBD_TIMEOUT) error = ETIMEDOUT; else error = EIO; break; } } usbd_destroy_xfer(xfer); break; default: return ENXIO; } return error; } static int ugenwrite(dev_t dev, struct uio *uio, int flag) { int endpt = UGENENDPOINT(dev); struct ugen_softc *sc; int error; if ((sc = ugenif_acquire(UGENUNIT(dev))) == NULL) return ENXIO; error = ugen_do_write(sc, endpt, uio, flag); ugenif_release(sc); return error; } static int ugen_activate(device_t self, enum devact act) { struct ugen_softc *sc = device_private(self); switch (act) { case DVACT_DEACTIVATE: sc->sc_dying = 1; return 0; default: return EOPNOTSUPP; } } static int ugen_detach(device_t self, int flags) { struct ugen_softc *sc = device_private(self); struct ugen_endpoint *sce; int i, dir; int maj, mn; DPRINTF(("ugen_detach: sc=%p flags=%d\n", sc, flags)); KASSERT(KERNEL_LOCKED_P()); /* sc_is_open */ /* * Fail if we're not forced to detach and userland has any * endpoints open. */ if ((flags & DETACH_FORCE) == 0) { for (i = 0; i < USB_MAX_ENDPOINTS; i++) { if (sc->sc_is_open[i]) return EBUSY; } } /* Prevent new users. Prevent suspend/resume. */ sc->sc_dying = 1; pmf_device_deregister(self); /* * If we never finished attaching, skip nixing endpoints and * users because there aren't any. */ if (!sc->sc_attached) goto out; /* Abort all pipes. */ for (i = 0; i < USB_MAX_ENDPOINTS; i++) { for (dir = OUT; dir <= IN; dir++) { sce = &sc->sc_endpoints[i][dir]; if (sce->pipeh) usbd_abort_pipe(sce->pipeh); } } /* * Wait for users to drain. Before this point there can be no * more I/O operations started because we set sc_dying; after * this, there can be no more I/O operations in progress, so it * will be safe to free things. */ mutex_enter(&sc->sc_lock); if (--sc->sc_refcnt >= 0) { /* Wake everyone */ for (i = 0; i < USB_MAX_ENDPOINTS; i++) { for (dir = OUT; dir <= IN; dir++) cv_broadcast(&sc->sc_endpoints[i][dir].cv); } /* Wait for processes to go away. */ do { cv_wait(&sc->sc_detach_cv, &sc->sc_lock); } while (sc->sc_refcnt >= 0); } mutex_exit(&sc->sc_lock); /* locate the major number */ maj = cdevsw_lookup_major(&ugen_cdevsw); /* * Nuke the vnodes for any open instances (calls ugenclose, but * with no effect because we already set sc_dying). */ mn = sc->sc_unit * USB_MAX_ENDPOINTS; vdevgone(maj, mn, mn + USB_MAX_ENDPOINTS - 1, VCHR); /* Actually close any lingering pipes. */ for (i = 0; i < USB_MAX_ENDPOINTS; i++) ugen_do_close(sc, FREAD|FWRITE, i); usbd_add_drv_event(USB_EVENT_DRIVER_DETACH, sc->sc_udev, sc->sc_dev); ugenif_put_unit(sc); out: for (i = 0; i < USB_MAX_ENDPOINTS; i++) { for (dir = OUT; dir <= IN; dir++) { sce = &sc->sc_endpoints[i][dir]; seldestroy(&sce->rsel); cv_destroy(&sce->cv); } } cv_destroy(&sc->sc_detach_cv); mutex_destroy(&sc->sc_lock); return 0; } Static void ugenintr(struct usbd_xfer *xfer, void *addr, usbd_status status) { struct ugen_endpoint *sce = addr; struct ugen_softc *sc = sce->sc; uint32_t count; u_char *ibuf; if (status == USBD_CANCELLED) return; if (status != USBD_NORMAL_COMPLETION) { DPRINTF(("ugenintr: status=%d\n", status)); if (status == USBD_STALLED) usbd_clear_endpoint_stall_async(sce->pipeh); return; } usbd_get_xfer_status(xfer, NULL, NULL, &count, NULL); ibuf = sce->ibuf; DPRINTFN(5, ("ugenintr: xfer=%p status=%d count=%d\n", xfer, status, count)); DPRINTFN(5, (" data = %02x %02x %02x\n", ibuf[0], ibuf[1], ibuf[2])); mutex_enter(&sc->sc_lock); (void)b_to_q(ibuf, count, &sce->q); cv_signal(&sce->cv); mutex_exit(&sc->sc_lock); selnotify(&sce->rsel, 0, 0); } Static void ugen_isoc_rintr(struct usbd_xfer *xfer, void *addr, usbd_status status) { struct isoreq *req = addr; struct ugen_endpoint *sce = req->sce; struct ugen_softc *sc = sce->sc; uint32_t count, n; int i, isize; /* Return if we are aborting. */ if (status == USBD_CANCELLED) return; usbd_get_xfer_status(xfer, NULL, NULL, &count, NULL); DPRINTFN(5,("ugen_isoc_rintr: xfer %ld, count=%d\n", (long)(req - sce->isoreqs), count)); mutex_enter(&sc->sc_lock); /* throw away oldest input if the buffer is full */ if (sce->fill < sce->cur && sce->cur <= sce->fill + count) { sce->cur += count; if (sce->cur >= sce->limit) sce->cur = sce->ibuf + (sce->limit - sce->cur); DPRINTFN(5, ("ugen_isoc_rintr: throwing away %d bytes\n", count)); } isize = UGETW(sce->edesc->wMaxPacketSize); for (i = 0; i < UGEN_NISORFRMS; i++) { uint32_t actlen = req->sizes[i]; char const *tbuf = (char const *)req->dmabuf + isize * i; /* copy data to buffer */ while (actlen > 0) { n = uimin(actlen, sce->limit - sce->fill); memcpy(sce->fill, tbuf, n); tbuf += n; actlen -= n; sce->fill += n; if (sce->fill == sce->limit) sce->fill = sce->ibuf; } /* setup size for next transfer */ req->sizes[i] = isize; } usbd_setup_isoc_xfer(xfer, req, req->sizes, UGEN_NISORFRMS, 0, ugen_isoc_rintr); (void)usbd_transfer(xfer); cv_signal(&sce->cv); mutex_exit(&sc->sc_lock); selnotify(&sce->rsel, 0, 0); } Static void ugen_bulkra_intr(struct usbd_xfer *xfer, void *addr, usbd_status status) { struct ugen_endpoint *sce = addr; struct ugen_softc *sc = sce->sc; uint32_t count, n; char const *tbuf; usbd_status err; /* Return if we are aborting. */ if (status == USBD_CANCELLED) return; if (status != USBD_NORMAL_COMPLETION) { DPRINTF(("ugen_bulkra_intr: status=%d\n", status)); sce->state |= UGEN_RA_WB_STOP; if (status == USBD_STALLED) usbd_clear_endpoint_stall_async(sce->pipeh); return; } usbd_get_xfer_status(xfer, NULL, NULL, &count, NULL); mutex_enter(&sc->sc_lock); /* Keep track of how much is in the buffer. */ sce->ra_wb_used += count; /* Copy data to buffer. */ tbuf = (char const *)usbd_get_buffer(sce->ra_wb_xfer); n = uimin(count, sce->limit - sce->fill); memcpy(sce->fill, tbuf, n); tbuf += n; count -= n; sce->fill += n; if (sce->fill == sce->limit) sce->fill = sce->ibuf; if (count > 0) { memcpy(sce->fill, tbuf, count); sce->fill += count; } /* Set up the next request if necessary. */ n = (sce->limit - sce->ibuf) - sce->ra_wb_used; if (n > 0) { usbd_setup_xfer(xfer, sce, NULL, uimin(n, sce->ra_wb_xferlen), 0, USBD_NO_TIMEOUT, ugen_bulkra_intr); err = usbd_transfer(xfer); if (err != USBD_IN_PROGRESS) { printf("usbd_bulkra_intr: error=%d\n", err); /* * The transfer has not been queued. Setting STOP * will make us try again at the next read. */ sce->state |= UGEN_RA_WB_STOP; } } else sce->state |= UGEN_RA_WB_STOP; cv_signal(&sce->cv); mutex_exit(&sc->sc_lock); selnotify(&sce->rsel, 0, 0); } Static void ugen_bulkwb_intr(struct usbd_xfer *xfer, void *addr, usbd_status status) { struct ugen_endpoint *sce = addr; struct ugen_softc *sc = sce->sc; uint32_t count, n; char *tbuf; usbd_status err; /* Return if we are aborting. */ if (status == USBD_CANCELLED) return; if (status != USBD_NORMAL_COMPLETION) { DPRINTF(("ugen_bulkwb_intr: status=%d\n", status)); sce->state |= UGEN_RA_WB_STOP; if (status == USBD_STALLED) usbd_clear_endpoint_stall_async(sce->pipeh); return; } usbd_get_xfer_status(xfer, NULL, NULL, &count, NULL); mutex_enter(&sc->sc_lock); /* Keep track of how much is in the buffer. */ sce->ra_wb_used -= count; /* Update buffer pointers. */ sce->cur += count; if (sce->cur >= sce->limit) sce->cur = sce->ibuf + (sce->cur - sce->limit); /* Set up next request if necessary. */ if (sce->ra_wb_used > 0) { /* copy data from buffer */ tbuf = (char *)usbd_get_buffer(sce->ra_wb_xfer); count = uimin(sce->ra_wb_used, sce->ra_wb_xferlen); n = uimin(count, sce->limit - sce->cur); memcpy(tbuf, sce->cur, n); tbuf += n; if (count - n > 0) memcpy(tbuf, sce->ibuf, count - n); usbd_setup_xfer(xfer, sce, NULL, count, 0, USBD_NO_TIMEOUT, ugen_bulkwb_intr); err = usbd_transfer(xfer); if (err != USBD_IN_PROGRESS) { printf("usbd_bulkwb_intr: error=%d\n", err); /* * The transfer has not been queued. Setting STOP * will make us try again at the next write. */ sce->state |= UGEN_RA_WB_STOP; } } else sce->state |= UGEN_RA_WB_STOP; cv_signal(&sce->cv); mutex_exit(&sc->sc_lock); selnotify(&sce->rsel, 0, 0); } Static usbd_status ugen_set_interface(struct ugen_softc *sc, int ifaceidx, int altno) { struct usbd_interface *iface; usb_endpoint_descriptor_t *ed; usbd_status err; struct ugen_endpoint *sce; uint8_t niface, nendpt, endptno, endpt; int dir; DPRINTFN(15, ("ugen_set_interface %d %d\n", ifaceidx, altno)); err = usbd_interface_count(sc->sc_udev, &niface); if (err) return err; if (ifaceidx < 0 || ifaceidx >= niface) return USBD_INVAL; err = usbd_device2interface_handle(sc->sc_udev, ifaceidx, &iface); if (err) return err; err = usbd_endpoint_count(iface, &nendpt); if (err) return err; /* change setting */ err = usbd_set_interface(iface, altno); if (err) return err; err = usbd_endpoint_count(iface, &nendpt); if (err) return err; ugen_clear_endpoints(sc); for (endptno = 0; endptno < nendpt; endptno++) { ed = usbd_interface2endpoint_descriptor(iface,endptno); KASSERT(ed != NULL); endpt = ed->bEndpointAddress; dir = UE_GET_DIR(endpt) == UE_DIR_IN ? IN : OUT; sce = &sc->sc_endpoints[UE_GET_ADDR(endpt)][dir]; sce->sc = sc; sce->edesc = ed; sce->iface = iface; } return 0; } /* Retrieve a complete descriptor for a certain device and index. */ Static usb_config_descriptor_t * ugen_get_cdesc(struct ugen_softc *sc, int index, int *lenp) { usb_config_descriptor_t *cdesc, *tdesc, cdescr; int len; usbd_status err; if (index == USB_CURRENT_CONFIG_INDEX) { tdesc = usbd_get_config_descriptor(sc->sc_udev); if (tdesc == NULL) return NULL; len = UGETW(tdesc->wTotalLength); if (lenp) *lenp = len; cdesc = kmem_alloc(len, KM_SLEEP); memcpy(cdesc, tdesc, len); DPRINTFN(5,("ugen_get_cdesc: current, len=%d\n", len)); } else { err = usbd_get_config_desc(sc->sc_udev, index, &cdescr); if (err) return 0; len = UGETW(cdescr.wTotalLength); DPRINTFN(5,("ugen_get_cdesc: index=%d, len=%d\n", index, len)); if (lenp) *lenp = len; cdesc = kmem_alloc(len, KM_SLEEP); err = usbd_get_config_desc_full(sc->sc_udev, index, cdesc, len); if (err) { kmem_free(cdesc, len); return 0; } } return cdesc; } Static int ugen_get_alt_index(struct ugen_softc *sc, int ifaceidx) { struct usbd_interface *iface; usbd_status err; err = usbd_device2interface_handle(sc->sc_udev, ifaceidx, &iface); if (err) return -1; return usbd_get_interface_altindex(iface); } Static int ugen_do_ioctl(struct ugen_softc *sc, int endpt, u_long cmd, void *addr, int flag, struct lwp *l) { struct ugen_endpoint *sce; usbd_status err; struct usbd_interface *iface; struct usb_config_desc *cd; usb_config_descriptor_t *cdesc; struct usb_interface_desc *id; usb_interface_descriptor_t *idesc; struct usb_endpoint_desc *ed; usb_endpoint_descriptor_t *edesc; struct usb_alt_interface *ai; struct usb_string_desc *si; uint8_t conf, alt; int cdesclen; int error; int dir; KASSERT(KERNEL_LOCKED_P()); /* ugen_set_config */ DPRINTFN(5, ("ugenioctl: cmd=%08lx\n", cmd)); switch (cmd) { case FIONBIO: /* All handled in the upper FS layer. */ return 0; case USB_SET_SHORT_XFER: if (endpt == USB_CONTROL_ENDPOINT) return EINVAL; /* This flag only affects read */ sce = &sc->sc_endpoints[endpt][IN]; if (sce == NULL || sce->pipeh == NULL) return EINVAL; if (*(int *)addr) sce->state |= UGEN_SHORT_OK; else sce->state &= ~UGEN_SHORT_OK; return 0; case USB_SET_TIMEOUT: for (dir = OUT; dir <= IN; dir++) { sce = &sc->sc_endpoints[endpt][dir]; if (sce == NULL) return EINVAL; sce->timeout = *(int *)addr; } return 0; case USB_SET_BULK_RA: if (endpt == USB_CONTROL_ENDPOINT) return EINVAL; sce = &sc->sc_endpoints[endpt][IN]; if (sce == NULL || sce->pipeh == NULL) return EINVAL; edesc = sce->edesc; if ((edesc->bmAttributes & UE_XFERTYPE) != UE_BULK) return EINVAL; if (*(int *)addr) { /* Only turn RA on if it's currently off. */ if (sce->state & UGEN_BULK_RA) return 0; KASSERT(sce->ra_wb_xfer == NULL); KASSERT(sce->ibuf == NULL); if (sce->ra_wb_bufsize == 0 || sce->ra_wb_reqsize == 0) /* shouldn't happen */ return EINVAL; error = usbd_create_xfer(sce->pipeh, sce->ra_wb_reqsize, 0, 0, &sce->ra_wb_xfer); if (error) return error; sce->ra_wb_xferlen = sce->ra_wb_reqsize; sce->ibuf = kmem_alloc(sce->ra_wb_bufsize, KM_SLEEP); sce->fill = sce->cur = sce->ibuf; sce->limit = sce->ibuf + sce->ra_wb_bufsize; sce->ra_wb_used = 0; sce->state |= UGEN_BULK_RA; sce->state &= ~UGEN_RA_WB_STOP; /* Now start reading. */ usbd_setup_xfer(sce->ra_wb_xfer, sce, NULL, uimin(sce->ra_wb_xferlen, sce->ra_wb_bufsize), 0, USBD_NO_TIMEOUT, ugen_bulkra_intr); err = usbd_transfer(sce->ra_wb_xfer); if (err != USBD_IN_PROGRESS) { sce->state &= ~UGEN_BULK_RA; kmem_free(sce->ibuf, sce->ra_wb_bufsize); sce->ibuf = NULL; usbd_destroy_xfer(sce->ra_wb_xfer); sce->ra_wb_xfer = NULL; return EIO; } } else { /* Only turn RA off if it's currently on. */ if (!(sce->state & UGEN_BULK_RA)) return 0; sce->state &= ~UGEN_BULK_RA; usbd_abort_pipe(sce->pipeh); usbd_destroy_xfer(sce->ra_wb_xfer); sce->ra_wb_xfer = NULL; /* * XXX Discard whatever's in the buffer, but we * should keep it around and drain the buffer * instead. */ kmem_free(sce->ibuf, sce->ra_wb_bufsize); sce->ibuf = NULL; } return 0; case USB_SET_BULK_WB: if (endpt == USB_CONTROL_ENDPOINT) return EINVAL; sce = &sc->sc_endpoints[endpt][OUT]; if (sce == NULL || sce->pipeh == NULL) return EINVAL; edesc = sce->edesc; if ((edesc->bmAttributes & UE_XFERTYPE) != UE_BULK) return EINVAL; if (*(int *)addr) { /* Only turn WB on if it's currently off. */ if (sce->state & UGEN_BULK_WB) return 0; KASSERT(sce->ra_wb_xfer == NULL); KASSERT(sce->ibuf == NULL); if (sce->ra_wb_bufsize == 0 || sce->ra_wb_reqsize == 0) /* shouldn't happen */ return EINVAL; error = usbd_create_xfer(sce->pipeh, sce->ra_wb_reqsize, 0, 0, &sce->ra_wb_xfer); /* XXX check error??? */ sce->ra_wb_xferlen = sce->ra_wb_reqsize; sce->ibuf = kmem_alloc(sce->ra_wb_bufsize, KM_SLEEP); sce->fill = sce->cur = sce->ibuf; sce->limit = sce->ibuf + sce->ra_wb_bufsize; sce->ra_wb_used = 0; sce->state |= UGEN_BULK_WB | UGEN_RA_WB_STOP; } else { /* Only turn WB off if it's currently on. */ if (!(sce->state & UGEN_BULK_WB)) return 0; sce->state &= ~UGEN_BULK_WB; /* * XXX Discard whatever's in the buffer, but we * should keep it around and keep writing to * drain the buffer instead. */ usbd_abort_pipe(sce->pipeh); usbd_destroy_xfer(sce->ra_wb_xfer); sce->ra_wb_xfer = NULL; kmem_free(sce->ibuf, sce->ra_wb_bufsize); sce->ibuf = NULL; } return 0; case USB_SET_BULK_RA_OPT: case USB_SET_BULK_WB_OPT: { struct usb_bulk_ra_wb_opt *opt; if (endpt == USB_CONTROL_ENDPOINT) return EINVAL; opt = (struct usb_bulk_ra_wb_opt *)addr; if (cmd == USB_SET_BULK_RA_OPT) sce = &sc->sc_endpoints[endpt][IN]; else sce = &sc->sc_endpoints[endpt][OUT]; if (sce == NULL || sce->pipeh == NULL) return EINVAL; if (opt->ra_wb_buffer_size < 1 || opt->ra_wb_buffer_size > UGEN_BULK_RA_WB_BUFMAX || opt->ra_wb_request_size < 1 || opt->ra_wb_request_size > opt->ra_wb_buffer_size) return EINVAL; /* * XXX These changes do not take effect until the * next time RA/WB mode is enabled but they ought to * take effect immediately. */ sce->ra_wb_bufsize = opt->ra_wb_buffer_size; sce->ra_wb_reqsize = opt->ra_wb_request_size; return 0; } default: break; } if (endpt != USB_CONTROL_ENDPOINT) return EINVAL; switch (cmd) { #ifdef UGEN_DEBUG case USB_SETDEBUG: ugendebug = *(int *)addr; break; #endif case USB_GET_CONFIG: err = usbd_get_config(sc->sc_udev, &conf); if (err) return EIO; *(int *)addr = conf; break; case USB_SET_CONFIG: if (!(flag & FWRITE)) return EPERM; err = ugen_set_config(sc, *(int *)addr, 1); switch (err) { case USBD_NORMAL_COMPLETION: break; case USBD_IN_USE: return EBUSY; default: return EIO; } break; case USB_GET_ALTINTERFACE: ai = (struct usb_alt_interface *)addr; err = usbd_device2interface_handle(sc->sc_udev, ai->uai_interface_index, &iface); if (err) return EINVAL; idesc = usbd_get_interface_descriptor(iface); if (idesc == NULL) return EIO; ai->uai_alt_no = idesc->bAlternateSetting; break; case USB_SET_ALTINTERFACE: if (!(flag & FWRITE)) return EPERM; ai = (struct usb_alt_interface *)addr; err = usbd_device2interface_handle(sc->sc_udev, ai->uai_interface_index, &iface); if (err) return EINVAL; err = ugen_set_interface(sc, ai->uai_interface_index, ai->uai_alt_no); if (err) return EINVAL; break; case USB_GET_NO_ALT: ai = (struct usb_alt_interface *)addr; cdesc = ugen_get_cdesc(sc, ai->uai_config_index, &cdesclen); if (cdesc == NULL) return EINVAL; idesc = usbd_find_idesc(cdesc, ai->uai_interface_index, 0); if (idesc == NULL) { kmem_free(cdesc, cdesclen); return EINVAL; } ai->uai_alt_no = usbd_get_no_alts(cdesc, idesc->bInterfaceNumber); kmem_free(cdesc, cdesclen); break; case USB_GET_DEVICE_DESC: *(usb_device_descriptor_t *)addr = *usbd_get_device_descriptor(sc->sc_udev); break; case USB_GET_CONFIG_DESC: cd = (struct usb_config_desc *)addr; cdesc = ugen_get_cdesc(sc, cd->ucd_config_index, &cdesclen); if (cdesc == NULL) return EINVAL; cd->ucd_desc = *cdesc; kmem_free(cdesc, cdesclen); break; case USB_GET_INTERFACE_DESC: id = (struct usb_interface_desc *)addr; cdesc = ugen_get_cdesc(sc, id->uid_config_index, &cdesclen); if (cdesc == NULL) return EINVAL; if (id->uid_config_index == USB_CURRENT_CONFIG_INDEX && id->uid_alt_index == USB_CURRENT_ALT_INDEX) alt = ugen_get_alt_index(sc, id->uid_interface_index); else alt = id->uid_alt_index; idesc = usbd_find_idesc(cdesc, id->uid_interface_index, alt); if (idesc == NULL) { kmem_free(cdesc, cdesclen); return EINVAL; } id->uid_desc = *idesc; kmem_free(cdesc, cdesclen); break; case USB_GET_ENDPOINT_DESC: ed = (struct usb_endpoint_desc *)addr; cdesc = ugen_get_cdesc(sc, ed->ued_config_index, &cdesclen); if (cdesc == NULL) return EINVAL; if (ed->ued_config_index == USB_CURRENT_CONFIG_INDEX && ed->ued_alt_index == USB_CURRENT_ALT_INDEX) alt = ugen_get_alt_index(sc, ed->ued_interface_index); else alt = ed->ued_alt_index; edesc = usbd_find_edesc(cdesc, ed->ued_interface_index, alt, ed->ued_endpoint_index); if (edesc == NULL) { kmem_free(cdesc, cdesclen); return EINVAL; } ed->ued_desc = *edesc; kmem_free(cdesc, cdesclen); break; case USB_GET_FULL_DESC: { int len; struct iovec iov; struct uio uio; struct usb_full_desc *fd = (struct usb_full_desc *)addr; cdesc = ugen_get_cdesc(sc, fd->ufd_config_index, &cdesclen); if (cdesc == NULL) return EINVAL; len = cdesclen; if (len > fd->ufd_size) len = fd->ufd_size; iov.iov_base = (void *)fd->ufd_data; iov.iov_len = len; uio.uio_iov = &iov; uio.uio_iovcnt = 1; uio.uio_resid = len; uio.uio_offset = 0; uio.uio_rw = UIO_READ; uio.uio_vmspace = l->l_proc->p_vmspace; error = uiomove((void *)cdesc, len, &uio); kmem_free(cdesc, cdesclen); return error; } case USB_GET_STRING_DESC: { int len; si = (struct usb_string_desc *)addr; err = usbd_get_string_desc(sc->sc_udev, si->usd_string_index, si->usd_language_id, &si->usd_desc, &len); if (err) return EINVAL; break; } case USB_DO_REQUEST: { struct usb_ctl_request *ur = (void *)addr; int len = UGETW(ur->ucr_request.wLength); struct iovec iov; struct uio uio; void *ptr = 0; usbd_status xerr; error = 0; if (!(flag & FWRITE)) return EPERM; /* Avoid requests that would damage the bus integrity. */ if ((ur->ucr_request.bmRequestType == UT_WRITE_DEVICE && ur->ucr_request.bRequest == UR_SET_ADDRESS) || (ur->ucr_request.bmRequestType == UT_WRITE_DEVICE && ur->ucr_request.bRequest == UR_SET_CONFIG) || (ur->ucr_request.bmRequestType == UT_WRITE_INTERFACE && ur->ucr_request.bRequest == UR_SET_INTERFACE)) return EINVAL; if (len < 0 || len > 32767) return EINVAL; if (len != 0) { iov.iov_base = (void *)ur->ucr_data; iov.iov_len = len; uio.uio_iov = &iov; uio.uio_iovcnt = 1; uio.uio_resid = len; uio.uio_offset = 0; uio.uio_rw = ur->ucr_request.bmRequestType & UT_READ ? UIO_READ : UIO_WRITE; uio.uio_vmspace = l->l_proc->p_vmspace; ptr = kmem_alloc(len, KM_SLEEP); if (uio.uio_rw == UIO_WRITE) { error = uiomove(ptr, len, &uio); if (error) goto ret; } } sce = &sc->sc_endpoints[endpt][IN]; xerr = usbd_do_request_flags(sc->sc_udev, &ur->ucr_request, ptr, ur->ucr_flags, &ur->ucr_actlen, sce->timeout); if (xerr) { error = EIO; goto ret; } if (len != 0) { if (uio.uio_rw == UIO_READ) { size_t alen = uimin(len, ur->ucr_actlen); error = uiomove(ptr, alen, &uio); if (error) goto ret; } } ret: if (ptr) kmem_free(ptr, len); return error; } case USB_GET_DEVICEINFO: usbd_fill_deviceinfo(sc->sc_udev, (struct usb_device_info *)addr, 0); break; case USB_GET_DEVICEINFO_OLD: { int ret; MODULE_HOOK_CALL(usb_subr_fill_30_hook, (sc->sc_udev, (struct usb_device_info_old *)addr, 0, usbd_devinfo_vp, usbd_printBCD), enosys(), ret); if (ret == 0) return 0; return EINVAL; } default: return EINVAL; } return 0; } static int ugenioctl(dev_t dev, u_long cmd, void *addr, int flag, struct lwp *l) { int endpt = UGENENDPOINT(dev); struct ugen_softc *sc; int error; if ((sc = ugenif_acquire(UGENUNIT(dev))) == 0) return ENXIO; error = ugen_do_ioctl(sc, endpt, cmd, addr, flag, l); ugenif_release(sc); return error; } static int ugenpoll(dev_t dev, int events, struct lwp *l) { struct ugen_softc *sc; struct ugen_endpoint *sce_in, *sce_out; int revents = 0; if ((sc = ugenif_acquire(UGENUNIT(dev))) == NULL) return POLLHUP; if (UGENENDPOINT(dev) == USB_CONTROL_ENDPOINT) { revents |= POLLERR; goto out; } sce_in = &sc->sc_endpoints[UGENENDPOINT(dev)][IN]; sce_out = &sc->sc_endpoints[UGENENDPOINT(dev)][OUT]; KASSERT(sce_in->edesc || sce_out->edesc); KASSERT(sce_in->pipeh || sce_out->pipeh); mutex_enter(&sc->sc_lock); if (sce_in && sce_in->pipeh && (events & (POLLIN | POLLRDNORM))) switch (sce_in->edesc->bmAttributes & UE_XFERTYPE) { case UE_INTERRUPT: if (sce_in->q.c_cc > 0) revents |= events & (POLLIN | POLLRDNORM); else selrecord(l, &sce_in->rsel); break; case UE_ISOCHRONOUS: if (sce_in->cur != sce_in->fill) revents |= events & (POLLIN | POLLRDNORM); else selrecord(l, &sce_in->rsel); break; case UE_BULK: if (sce_in->state & UGEN_BULK_RA) { if (sce_in->ra_wb_used > 0) revents |= events & (POLLIN | POLLRDNORM); else selrecord(l, &sce_in->rsel); break; } /* * We have no easy way of determining if a read will * yield any data or a write will happen. * Pretend they will. */ revents |= events & (POLLIN | POLLRDNORM); break; default: break; } if (sce_out && sce_out->pipeh && (events & (POLLOUT | POLLWRNORM))) switch (sce_out->edesc->bmAttributes & UE_XFERTYPE) { case UE_INTERRUPT: case UE_ISOCHRONOUS: /* XXX unimplemented */ break; case UE_BULK: if (sce_out->state & UGEN_BULK_WB) { if (sce_out->ra_wb_used < sce_out->limit - sce_out->ibuf) revents |= events & (POLLOUT | POLLWRNORM); else selrecord(l, &sce_out->rsel); break; } /* * We have no easy way of determining if a read will * yield any data or a write will happen. * Pretend they will. */ revents |= events & (POLLOUT | POLLWRNORM); break; default: break; } mutex_exit(&sc->sc_lock); out: ugenif_release(sc); return revents; } static void filt_ugenrdetach(struct knote *kn) { struct ugen_endpoint *sce = kn->kn_hook; struct ugen_softc *sc = sce->sc; mutex_enter(&sc->sc_lock); selremove_knote(&sce->rsel, kn); mutex_exit(&sc->sc_lock); } static int filt_ugenread_intr(struct knote *kn, long hint) { struct ugen_endpoint *sce = kn->kn_hook; struct ugen_softc *sc = sce->sc; int ret; mutex_enter(&sc->sc_lock); if (sc->sc_dying) { ret = 0; } else { kn->kn_data = sce->q.c_cc; ret = kn->kn_data > 0; } mutex_exit(&sc->sc_lock); return ret; } static int filt_ugenread_isoc(struct knote *kn, long hint) { struct ugen_endpoint *sce = kn->kn_hook; struct ugen_softc *sc = sce->sc; int ret; mutex_enter(&sc->sc_lock); if (sc->sc_dying) { ret = 0; } else if (sce->cur == sce->fill) { ret = 0; } else if (sce->cur < sce->fill) { kn->kn_data = sce->fill - sce->cur; ret = 1; } else { kn->kn_data = (sce->limit - sce->cur) + (sce->fill - sce->ibuf); ret = 1; } mutex_exit(&sc->sc_lock); return ret; } static int filt_ugenread_bulk(struct knote *kn, long hint) { struct ugen_endpoint *sce = kn->kn_hook; struct ugen_softc *sc = sce->sc; int ret; mutex_enter(&sc->sc_lock); if (sc->sc_dying) { ret = 0; } else if (!(sce->state & UGEN_BULK_RA)) { /* * We have no easy way of determining if a read will * yield any data or a write will happen. * So, emulate "seltrue". */ ret = filt_seltrue(kn, hint); } else if (sce->ra_wb_used == 0) { ret = 0; } else { kn->kn_data = sce->ra_wb_used; ret = 1; } mutex_exit(&sc->sc_lock); return ret; } static int filt_ugenwrite_bulk(struct knote *kn, long hint) { struct ugen_endpoint *sce = kn->kn_hook; struct ugen_softc *sc = sce->sc; int ret; mutex_enter(&sc->sc_lock); if (sc->sc_dying) { ret = 0; } else if (!(sce->state & UGEN_BULK_WB)) { /* * We have no easy way of determining if a read will * yield any data or a write will happen. * So, emulate "seltrue". */ ret = filt_seltrue(kn, hint); } else if (sce->ra_wb_used == sce->limit - sce->ibuf) { ret = 0; } else { kn->kn_data = (sce->limit - sce->ibuf) - sce->ra_wb_used; ret = 1; } mutex_exit(&sc->sc_lock); return ret; } static const struct filterops ugenread_intr_filtops = { .f_flags = FILTEROP_ISFD, .f_attach = NULL, .f_detach = filt_ugenrdetach, .f_event = filt_ugenread_intr, }; static const struct filterops ugenread_isoc_filtops = { .f_flags = FILTEROP_ISFD, .f_attach = NULL, .f_detach = filt_ugenrdetach, .f_event = filt_ugenread_isoc, }; static const struct filterops ugenread_bulk_filtops = { .f_flags = FILTEROP_ISFD, .f_attach = NULL, .f_detach = filt_ugenrdetach, .f_event = filt_ugenread_bulk, }; static const struct filterops ugenwrite_bulk_filtops = { .f_flags = FILTEROP_ISFD, .f_attach = NULL, .f_detach = filt_ugenrdetach, .f_event = filt_ugenwrite_bulk, }; static int ugenkqfilter(dev_t dev, struct knote *kn) { struct ugen_softc *sc; struct ugen_endpoint *sce; struct selinfo *sip; int error; if ((sc = ugenif_acquire(UGENUNIT(dev))) == NULL) return ENXIO; if (UGENENDPOINT(dev) == USB_CONTROL_ENDPOINT) { error = ENODEV; goto out; } switch (kn->kn_filter) { case EVFILT_READ: sce = &sc->sc_endpoints[UGENENDPOINT(dev)][IN]; if (sce == NULL) { error = EINVAL; goto out; } sip = &sce->rsel; switch (sce->edesc->bmAttributes & UE_XFERTYPE) { case UE_INTERRUPT: kn->kn_fop = &ugenread_intr_filtops; break; case UE_ISOCHRONOUS: kn->kn_fop = &ugenread_isoc_filtops; break; case UE_BULK: kn->kn_fop = &ugenread_bulk_filtops; break; default: error = EINVAL; goto out; } break; case EVFILT_WRITE: sce = &sc->sc_endpoints[UGENENDPOINT(dev)][OUT]; if (sce == NULL) { error = EINVAL; goto out; } sip = &sce->rsel; switch (sce->edesc->bmAttributes & UE_XFERTYPE) { case UE_INTERRUPT: case UE_ISOCHRONOUS: /* XXX poll doesn't support this */ error = EINVAL; goto out; case UE_BULK: kn->kn_fop = &ugenwrite_bulk_filtops; break; default: error = EINVAL; goto out; } break; default: error = EINVAL; goto out; } kn->kn_hook = sce; mutex_enter(&sc->sc_lock); selrecord_knote(sip, kn); mutex_exit(&sc->sc_lock); error = 0; out: ugenif_release(sc); return error; } MODULE(MODULE_CLASS_DRIVER, ugen, NULL); static int ugen_modcmd(modcmd_t cmd, void *aux) { switch (cmd) { case MODULE_CMD_INIT: mutex_init(&ugenif.lock, MUTEX_DEFAULT, IPL_NONE); rb_tree_init(&ugenif.tree, &ugenif_tree_ops); return 0; default: return ENOTTY; } } |
| 2 2 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 197 198 199 200 201 202 203 204 205 206 207 208 209 210 211 212 213 214 215 216 217 218 219 220 221 222 223 224 225 226 227 228 229 230 231 232 233 234 235 236 237 238 239 240 241 242 243 244 245 246 247 248 249 250 251 252 253 254 255 256 257 258 259 260 261 262 263 264 265 266 267 268 269 270 271 272 273 274 275 276 277 278 279 280 281 282 283 284 285 286 287 288 289 290 291 292 293 294 295 296 297 298 299 300 301 302 303 304 305 306 307 308 309 310 311 312 313 314 315 316 317 318 319 320 321 322 323 324 325 326 327 328 329 330 331 332 333 334 335 336 337 338 339 340 341 342 343 344 345 346 347 348 349 350 351 352 353 354 355 356 357 358 359 360 361 362 363 364 365 366 367 368 369 370 371 372 373 374 375 376 377 378 379 380 381 382 383 384 385 386 387 388 389 390 391 392 393 394 395 396 397 398 399 400 401 402 403 404 405 406 407 408 409 410 411 412 413 414 415 416 417 418 419 420 421 422 423 424 425 426 427 428 429 430 431 432 433 434 435 436 437 438 439 440 441 442 443 444 445 446 447 448 449 450 451 452 453 454 455 456 457 458 459 460 461 462 463 464 465 466 467 468 469 470 471 472 473 474 475 476 477 478 479 480 481 482 483 484 485 486 487 488 489 490 491 492 493 494 495 496 497 498 499 500 501 502 503 504 505 506 507 508 509 510 511 512 513 514 515 516 517 518 519 520 521 522 523 524 525 526 527 528 529 530 531 532 533 534 535 536 537 538 539 540 541 542 543 544 545 546 547 548 549 550 551 552 553 554 555 556 557 558 559 560 561 562 563 564 565 566 567 568 569 570 571 572 573 574 575 576 577 578 579 580 581 582 583 584 585 586 587 588 589 590 591 592 593 594 595 596 597 598 599 600 601 602 603 604 605 606 607 608 609 610 611 612 613 614 615 616 617 618 619 620 621 622 623 624 625 626 627 628 629 630 631 632 633 634 635 636 637 638 639 640 641 642 643 644 645 646 647 648 649 650 651 652 653 654 655 656 657 658 659 660 661 662 663 664 665 666 667 668 669 670 671 672 673 674 675 676 677 678 679 680 681 682 683 684 685 686 687 688 689 690 691 692 693 694 695 696 697 698 699 700 701 702 703 704 705 706 707 708 709 710 711 712 713 714 715 716 717 718 719 720 721 722 723 724 725 726 727 728 729 730 731 732 733 734 735 736 737 738 739 740 741 742 743 744 745 746 747 748 749 750 751 752 753 754 755 756 757 758 759 760 761 762 763 764 765 766 767 768 769 770 771 772 773 774 775 776 777 778 779 780 781 782 783 784 785 786 787 788 789 790 791 792 793 794 795 796 797 798 799 800 801 802 803 804 805 806 807 808 809 810 811 812 813 814 815 816 817 818 819 820 821 822 823 824 825 826 827 828 829 830 831 832 833 834 835 836 837 838 839 840 841 842 843 844 845 846 847 848 849 850 851 852 853 854 855 856 857 858 859 860 861 862 863 864 865 866 867 868 869 870 871 872 873 874 875 876 877 878 879 880 881 882 883 884 885 886 | /* $NetBSD: rf_map.c,v 1.51 2021/07/23 00:54:45 oster Exp $ */ /* * Copyright (c) 1995 Carnegie-Mellon University. * All rights reserved. * * Author: Mark Holland * * Permission to use, copy, modify and distribute this software and * its documentation is hereby granted, provided that both the copyright * notice and this permission notice appear in all copies of the * software, derivative works or modified versions, and any portions * thereof, and that both notices appear in supporting documentation. * * CARNEGIE MELLON ALLOWS FREE USE OF THIS SOFTWARE IN ITS "AS IS" * CONDITION. CARNEGIE MELLON DISCLAIMS ANY LIABILITY OF ANY KIND * FOR ANY DAMAGES WHATSOEVER RESULTING FROM THE USE OF THIS SOFTWARE. * * Carnegie Mellon requests users of this software to return to * * Software Distribution Coordinator or Software.Distribution@CS.CMU.EDU * School of Computer Science * Carnegie Mellon University * Pittsburgh PA 15213-3890 * * any improvements or extensions that they make and grant Carnegie the * rights to redistribute these changes. */ /************************************************************************** * * map.c -- main code for mapping RAID addresses to physical disk addresses * **************************************************************************/ #include <sys/cdefs.h> __KERNEL_RCSID(0, "$NetBSD: rf_map.c,v 1.51 2021/07/23 00:54:45 oster Exp $"); #include <dev/raidframe/raidframevar.h> #include "rf_threadstuff.h" #include "rf_raid.h" #include "rf_general.h" #include "rf_map.h" #include "rf_shutdown.h" static void rf_FreePDAList(RF_Raid_t *raidPtr, RF_PhysDiskAddr_t *pda_list); static void rf_FreeASMList(RF_Raid_t *raidPtr, RF_AccessStripeMap_t *asm_list); /*************************************************************************** * * MapAccess -- main 1st order mapping routine. Maps an access in the * RAID address space to the corresponding set of physical disk * addresses. The result is returned as a list of AccessStripeMap * structures, one per stripe accessed. Each ASM structure contains a * pointer to a list of PhysDiskAddr structures, which describe the * physical locations touched by the user access. Note that this * routine returns only static mapping information, i.e. the list of * physical addresses returned does not necessarily identify the set * of physical locations that will actually be read or written. The * routine also maps the parity. The physical disk location returned * always indicates the entire parity unit, even when only a subset of * it is being accessed. This is because an access that is not stripe * unit aligned but that spans a stripe unit boundary may require * access two distinct portions of the parity unit, and we can't yet * tell which portion(s) we'll actually need. We leave it up to the * algorithm selection code to decide what subset of the parity unit * to access. Note that addresses in the RAID address space must * always be maintained as longs, instead of ints. * * This routine returns NULL if numBlocks is 0 * * raidAddress - starting address in RAID address space * numBlocks - number of blocks in RAID address space to access * buffer - buffer to supply/receive data * remap - 1 => remap address to spare space ***************************************************************************/ RF_AccessStripeMapHeader_t * rf_MapAccess(RF_Raid_t *raidPtr, RF_RaidAddr_t raidAddress, RF_SectorCount_t numBlocks, void *buffer, int remap) { RF_RaidLayout_t *layoutPtr = &(raidPtr->Layout); RF_AccessStripeMapHeader_t *asm_hdr = NULL; RF_AccessStripeMap_t *asm_list = NULL, *asm_p = NULL; int faultsTolerated = layoutPtr->map->faultsTolerated; /* we'll change raidAddress along the way */ RF_RaidAddr_t startAddress = raidAddress; RF_RaidAddr_t endAddress = raidAddress + numBlocks; RF_RaidDisk_t *disks = raidPtr->Disks; RF_PhysDiskAddr_t *pda_p; #if (RF_INCLUDE_DECL_PQ > 0) || (RF_INCLUDE_RAID6 > 0) RF_PhysDiskAddr_t *pda_q; #endif RF_StripeCount_t numStripes = 0; RF_RaidAddr_t stripeRealEndAddress, stripeEndAddress, nextStripeUnitAddress; RF_RaidAddr_t startAddrWithinStripe, lastRaidAddr; RF_StripeCount_t totStripes; RF_StripeNum_t stripeID, lastSID, SUID, lastSUID; RF_AccessStripeMap_t *asmList, *t_asm; RF_PhysDiskAddr_t *pdaList, *t_pda; /* allocate all the ASMs and PDAs up front */ lastRaidAddr = raidAddress + numBlocks - 1; stripeID = rf_RaidAddressToStripeID(layoutPtr, raidAddress); lastSID = rf_RaidAddressToStripeID(layoutPtr, lastRaidAddr); totStripes = lastSID - stripeID + 1; SUID = rf_RaidAddressToStripeUnitID(layoutPtr, raidAddress); lastSUID = rf_RaidAddressToStripeUnitID(layoutPtr, lastRaidAddr); asmList = rf_AllocASMList(raidPtr, totStripes); /* may also need pda(s) per stripe for parity */ pdaList = rf_AllocPDAList(raidPtr, lastSUID - SUID + 1 + faultsTolerated * totStripes); if (raidAddress + numBlocks > raidPtr->totalSectors) { RF_ERRORMSG1("Unable to map access because offset (%d) was invalid\n", (int) raidAddress); return (NULL); } #if RF_DEBUG_MAP if (rf_mapDebug) rf_PrintRaidAddressInfo(raidPtr, raidAddress, numBlocks); #endif for (; raidAddress < endAddress;) { /* make the next stripe structure */ RF_ASSERT(asmList); t_asm = asmList; asmList = asmList->next; memset(t_asm, 0, sizeof(*t_asm)); if (!asm_p) asm_list = asm_p = t_asm; else { asm_p->next = t_asm; asm_p = asm_p->next; } numStripes++; /* map SUs from current location to the end of the stripe */ asm_p->stripeID = /* rf_RaidAddressToStripeID(layoutPtr, raidAddress) */ stripeID++; stripeRealEndAddress = rf_RaidAddressOfNextStripeBoundary(layoutPtr, raidAddress); stripeEndAddress = RF_MIN(endAddress, stripeRealEndAddress); asm_p->raidAddress = raidAddress; asm_p->endRaidAddress = stripeEndAddress; /* map each stripe unit in the stripe */ pda_p = NULL; /* Raid addr of start of portion of access that is within this stripe */ startAddrWithinStripe = raidAddress; for (; raidAddress < stripeEndAddress;) { RF_ASSERT(pdaList); t_pda = pdaList; pdaList = pdaList->next; memset(t_pda, 0, sizeof(*t_pda)); if (!pda_p) asm_p->physInfo = pda_p = t_pda; else { pda_p->next = t_pda; pda_p = pda_p->next; } pda_p->type = RF_PDA_TYPE_DATA; (layoutPtr->map->MapSector) (raidPtr, raidAddress, &(pda_p->col), &(pda_p->startSector), remap); /* mark any failures we find. failedPDA is * don't-care if there is more than one * failure */ /* the RAID address corresponding to this physical diskaddress */ pda_p->raidAddress = raidAddress; nextStripeUnitAddress = rf_RaidAddressOfNextStripeUnitBoundary(layoutPtr, raidAddress); pda_p->numSector = RF_MIN(endAddress, nextStripeUnitAddress) - raidAddress; RF_ASSERT(pda_p->numSector != 0); rf_ASMCheckStatus(raidPtr, pda_p, asm_p, disks, 0); pda_p->bufPtr = (char *)buffer + rf_RaidAddressToByte(raidPtr, (raidAddress - startAddress)); asm_p->totalSectorsAccessed += pda_p->numSector; asm_p->numStripeUnitsAccessed++; raidAddress = RF_MIN(endAddress, nextStripeUnitAddress); } /* Map the parity. At this stage, the startSector and * numSector fields for the parity unit are always set * to indicate the entire parity unit. We may modify * this after mapping the data portion. */ switch (faultsTolerated) { case 0: break; case 1: /* single fault tolerant */ RF_ASSERT(pdaList); t_pda = pdaList; pdaList = pdaList->next; memset(t_pda, 0, sizeof(*t_pda)); pda_p = asm_p->parityInfo = t_pda; pda_p->type = RF_PDA_TYPE_PARITY; (layoutPtr->map->MapParity) (raidPtr, rf_RaidAddressOfPrevStripeUnitBoundary(layoutPtr, startAddrWithinStripe), &(pda_p->col), &(pda_p->startSector), remap); pda_p->numSector = layoutPtr->sectorsPerStripeUnit; /* raidAddr may be needed to find unit to redirect to */ pda_p->raidAddress = rf_RaidAddressOfPrevStripeUnitBoundary(layoutPtr, startAddrWithinStripe); rf_ASMCheckStatus(raidPtr, pda_p, asm_p, disks, 1); rf_ASMParityAdjust(raidPtr, asm_p->parityInfo, startAddrWithinStripe, endAddress, layoutPtr, asm_p); break; #if (RF_INCLUDE_DECL_PQ > 0) || (RF_INCLUDE_RAID6 > 0) case 2: /* two fault tolerant */ RF_ASSERT(pdaList && pdaList->next); t_pda = pdaList; pdaList = pdaList->next; memset(t_pda, 0, sizeof(*t_pda)); pda_p = asm_p->parityInfo = t_pda; pda_p->type = RF_PDA_TYPE_PARITY; t_pda = pdaList; pdaList = pdaList->next; memset(t_pda, 0, sizeof(*t_pda)); pda_q = asm_p->qInfo = t_pda; pda_q->type = RF_PDA_TYPE_Q; (layoutPtr->map->MapParity) (raidPtr, rf_RaidAddressOfPrevStripeUnitBoundary(layoutPtr, startAddrWithinStripe), &(pda_p->col), &(pda_p->startSector), remap); (layoutPtr->map->MapQ) (raidPtr, rf_RaidAddressOfPrevStripeUnitBoundary(layoutPtr, startAddrWithinStripe), &(pda_q->col), &(pda_q->startSector), remap); pda_q->numSector = pda_p->numSector = layoutPtr->sectorsPerStripeUnit; /* raidAddr may be needed to find unit to redirect to */ pda_p->raidAddress = rf_RaidAddressOfPrevStripeUnitBoundary(layoutPtr, startAddrWithinStripe); pda_q->raidAddress = rf_RaidAddressOfPrevStripeUnitBoundary(layoutPtr, startAddrWithinStripe); /* failure mode stuff */ rf_ASMCheckStatus(raidPtr, pda_p, asm_p, disks, 1); rf_ASMCheckStatus(raidPtr, pda_q, asm_p, disks, 1); rf_ASMParityAdjust(raidPtr, asm_p->parityInfo, startAddrWithinStripe, endAddress, layoutPtr, asm_p); rf_ASMParityAdjust(raidPtr, asm_p->qInfo, startAddrWithinStripe, endAddress, layoutPtr, asm_p); break; #endif } } RF_ASSERT(asmList == NULL && pdaList == NULL); /* make the header structure */ asm_hdr = rf_AllocAccessStripeMapHeader(raidPtr); RF_ASSERT(numStripes == totStripes); asm_hdr->numStripes = numStripes; asm_hdr->stripeMap = asm_list; #if RF_DEBUG_MAP if (rf_mapDebug) rf_PrintAccessStripeMap(asm_hdr); #endif return (asm_hdr); } /*************************************************************************** * This routine walks through an ASM list and marks the PDAs that have * failed. It's called only when a disk failure causes an in-flight * DAG to fail. The parity may consist of two components, but we want * to use only one failedPDA pointer. Thus we set failedPDA to point * to the first parity component, and rely on the rest of the code to * do the right thing with this. ***************************************************************************/ void rf_MarkFailuresInASMList(RF_Raid_t *raidPtr, RF_AccessStripeMapHeader_t *asm_h) { RF_RaidDisk_t *disks = raidPtr->Disks; RF_AccessStripeMap_t *asmap; RF_PhysDiskAddr_t *pda; for (asmap = asm_h->stripeMap; asmap; asmap = asmap->next) { asmap->numDataFailed = 0; asmap->numParityFailed = 0; asmap->numQFailed = 0; asmap->numFailedPDAs = 0; memset(asmap->failedPDAs, 0, RF_MAX_FAILED_PDA * sizeof(*asmap->failedPDAs)); for (pda = asmap->physInfo; pda; pda = pda->next) { if (RF_DEAD_DISK(disks[pda->col].status)) { asmap->numDataFailed++; asmap->failedPDAs[asmap->numFailedPDAs] = pda; asmap->numFailedPDAs++; } } pda = asmap->parityInfo; if (pda && RF_DEAD_DISK(disks[pda->col].status)) { asmap->numParityFailed++; asmap->failedPDAs[asmap->numFailedPDAs] = pda; asmap->numFailedPDAs++; } pda = asmap->qInfo; if (pda && RF_DEAD_DISK(disks[pda->col].status)) { asmap->numQFailed++; asmap->failedPDAs[asmap->numFailedPDAs] = pda; asmap->numFailedPDAs++; } } } /*************************************************************************** * * routines to allocate and free list elements. All allocation * routines zero the structure before returning it. * * FreePhysDiskAddr is static. It should never be called directly, * because FreeAccessStripeMap takes care of freeing the PhysDiskAddr * list. * ***************************************************************************/ #define RF_MAX_FREE_ASMHDR 128 #define RF_MIN_FREE_ASMHDR 32 #define RF_MAX_FREE_ASM 192 #define RF_MIN_FREE_ASM 64 #define RF_MAX_FREE_PDA 192 #define RF_MIN_FREE_PDA 64 #define RF_MAX_FREE_ASMHLE 64 #define RF_MIN_FREE_ASMHLE 16 #define RF_MAX_FREE_FSS 128 #define RF_MIN_FREE_FSS 32 #define RF_MAX_FREE_VFPLE 128 #define RF_MIN_FREE_VFPLE 32 #define RF_MAX_FREE_VPLE 128 #define RF_MIN_FREE_VPLE 32 /* called at shutdown time. So far, all that is necessary is to release all the free lists */ static void rf_ShutdownMapModule(void *); static void rf_ShutdownMapModule(void *arg) { RF_Raid_t *raidPtr; raidPtr = (RF_Raid_t *) arg; pool_destroy(&raidPtr->pools.asm_hdr); pool_destroy(&raidPtr->pools.asmap); pool_destroy(&raidPtr->pools.asmhle); pool_destroy(&raidPtr->pools.pda); pool_destroy(&raidPtr->pools.fss); pool_destroy(&raidPtr->pools.vfple); pool_destroy(&raidPtr->pools.vple); } int rf_ConfigureMapModule(RF_ShutdownList_t **listp, RF_Raid_t *raidPtr, RF_Config_t *cfgPtr) { rf_pool_init(raidPtr, raidPtr->poolNames.asm_hdr, &raidPtr->pools.asm_hdr, sizeof(RF_AccessStripeMapHeader_t), "asmhdr", RF_MIN_FREE_ASMHDR, RF_MAX_FREE_ASMHDR); rf_pool_init(raidPtr, raidPtr->poolNames.asmap, &raidPtr->pools.asmap, sizeof(RF_AccessStripeMap_t), "asmap", RF_MIN_FREE_ASM, RF_MAX_FREE_ASM); rf_pool_init(raidPtr, raidPtr->poolNames.asmhle, &raidPtr->pools.asmhle, sizeof(RF_ASMHeaderListElem_t), "asmhle", RF_MIN_FREE_ASMHLE, RF_MAX_FREE_ASMHLE); rf_pool_init(raidPtr, raidPtr->poolNames.pda, &raidPtr->pools.pda, sizeof(RF_PhysDiskAddr_t), "pda", RF_MIN_FREE_PDA, RF_MAX_FREE_PDA); rf_pool_init(raidPtr, raidPtr->poolNames.fss, &raidPtr->pools.fss, sizeof(RF_FailedStripe_t), "fss", RF_MIN_FREE_FSS, RF_MAX_FREE_FSS); rf_pool_init(raidPtr, raidPtr->poolNames.vfple, &raidPtr->pools.vfple, sizeof(RF_VoidFunctionPointerListElem_t), "vfple", RF_MIN_FREE_VFPLE, RF_MAX_FREE_VFPLE); rf_pool_init(raidPtr, raidPtr->poolNames.vple, &raidPtr->pools.vple, sizeof(RF_VoidPointerListElem_t), "vple", RF_MIN_FREE_VPLE, RF_MAX_FREE_VPLE); rf_ShutdownCreate(listp, rf_ShutdownMapModule, raidPtr); return (0); } RF_AccessStripeMapHeader_t * rf_AllocAccessStripeMapHeader(RF_Raid_t *raidPtr) { return pool_get(&raidPtr->pools.asm_hdr, PR_WAITOK | PR_ZERO); } void rf_FreeAccessStripeMapHeader(RF_Raid_t *raidPtr, RF_AccessStripeMapHeader_t *p) { pool_put(&raidPtr->pools.asm_hdr, p); } RF_VoidFunctionPointerListElem_t * rf_AllocVFPListElem(RF_Raid_t *raidPtr) { return pool_get(&raidPtr->pools.vfple, PR_WAITOK | PR_ZERO); } void rf_FreeVFPListElem(RF_Raid_t *raidPtr, RF_VoidFunctionPointerListElem_t *p) { pool_put(&raidPtr->pools.vfple, p); } RF_VoidPointerListElem_t * rf_AllocVPListElem(RF_Raid_t *raidPtr) { return pool_get(&raidPtr->pools.vple, PR_WAITOK | PR_ZERO); } void rf_FreeVPListElem(RF_Raid_t *raidPtr, RF_VoidPointerListElem_t *p) { pool_put(&raidPtr->pools.vple, p); } RF_ASMHeaderListElem_t * rf_AllocASMHeaderListElem(RF_Raid_t *raidPtr) { return pool_get(&raidPtr->pools.asmhle, PR_WAITOK | PR_ZERO); } void rf_FreeASMHeaderListElem(RF_Raid_t *raidPtr, RF_ASMHeaderListElem_t *p) { pool_put(&raidPtr->pools.asmhle, p); } RF_FailedStripe_t * rf_AllocFailedStripeStruct(RF_Raid_t *raidPtr) { return pool_get(&raidPtr->pools.fss, PR_WAITOK | PR_ZERO); } void rf_FreeFailedStripeStruct(RF_Raid_t *raidPtr, RF_FailedStripe_t *p) { pool_put(&raidPtr->pools.fss, p); } RF_PhysDiskAddr_t * rf_AllocPhysDiskAddr(RF_Raid_t *raidPtr) { return pool_get(&raidPtr->pools.pda, PR_WAITOK | PR_ZERO); } /* allocates a list of PDAs, locking the free list only once when we * have to call calloc, we do it one component at a time to simplify * the process of freeing the list at program shutdown. This should * not be much of a performance hit, because it should be very * infrequently executed. */ RF_PhysDiskAddr_t * rf_AllocPDAList(RF_Raid_t *raidPtr, int count) { RF_PhysDiskAddr_t *p, *prev; int i; p = NULL; prev = NULL; for (i = 0; i < count; i++) { p = pool_get(&raidPtr->pools.pda, PR_WAITOK); p->next = prev; prev = p; } return (p); } void rf_FreePhysDiskAddr(RF_Raid_t *raidPtr, RF_PhysDiskAddr_t *p) { pool_put(&raidPtr->pools.pda, p); } static void rf_FreePDAList(RF_Raid_t *raidPtr, RF_PhysDiskAddr_t *pda_list) { RF_PhysDiskAddr_t *p, *tmp; p=pda_list; while (p) { tmp = p->next; pool_put(&raidPtr->pools.pda, p); p = tmp; } } /* this is essentially identical to AllocPDAList. I should combine * the two. when we have to call calloc, we do it one component at a * time to simplify the process of freeing the list at program * shutdown. This should not be much of a performance hit, because it * should be very infrequently executed. */ RF_AccessStripeMap_t * rf_AllocASMList(RF_Raid_t *raidPtr, int count) { RF_AccessStripeMap_t *p, *prev; int i; p = NULL; prev = NULL; for (i = 0; i < count; i++) { p = pool_get(&raidPtr->pools.asmap, PR_WAITOK); p->next = prev; prev = p; } return (p); } static void rf_FreeASMList(RF_Raid_t *raidPtr, RF_AccessStripeMap_t *asm_list) { RF_AccessStripeMap_t *p, *tmp; p=asm_list; while (p) { tmp = p->next; pool_put(&raidPtr->pools.asmap, p); p = tmp; } } void rf_FreeAccessStripeMap(RF_Raid_t *raidPtr, RF_AccessStripeMapHeader_t *hdr) { RF_AccessStripeMap_t *p; RF_PhysDiskAddr_t *pdp, *trailer, *pdaList = NULL, *pdaEnd = NULL; int count = 0, t; for (p = hdr->stripeMap; p; p = p->next) { /* link the 3 pda lists into the accumulating pda list */ if (!pdaList) pdaList = p->qInfo; else pdaEnd->next = p->qInfo; for (trailer = NULL, pdp = p->qInfo; pdp;) { trailer = pdp; pdp = pdp->next; count++; } if (trailer) pdaEnd = trailer; if (!pdaList) pdaList = p->parityInfo; else pdaEnd->next = p->parityInfo; for (trailer = NULL, pdp = p->parityInfo; pdp;) { trailer = pdp; pdp = pdp->next; count++; } if (trailer) pdaEnd = trailer; if (!pdaList) pdaList = p->physInfo; else pdaEnd->next = p->physInfo; for (trailer = NULL, pdp = p->physInfo; pdp;) { trailer = pdp; pdp = pdp->next; count++; } if (trailer) pdaEnd = trailer; } /* debug only */ for (t = 0, pdp = pdaList; pdp; pdp = pdp->next) t++; RF_ASSERT(t == count); if (pdaList) rf_FreePDAList(raidPtr, pdaList); rf_FreeASMList(raidPtr, hdr->stripeMap); rf_FreeAccessStripeMapHeader(raidPtr, hdr); } /* We can't use the large write optimization if there are any failures * in the stripe. In the declustered layout, there is no way to * immediately determine what disks constitute a stripe, so we * actually have to hunt through the stripe looking for failures. The * reason we map the parity instead of just using asm->parityInfo->col * is because the latter may have been already redirected to a spare * drive, which would mess up the computation of the stripe offset. * * ASSUMES AT MOST ONE FAILURE IN THE STRIPE. */ int rf_CheckStripeForFailures(RF_Raid_t *raidPtr, RF_AccessStripeMap_t *asmap) { RF_RowCol_t tcol, pcol, *diskids, i; RF_RaidLayout_t *layoutPtr = &raidPtr->Layout; RF_StripeCount_t stripeOffset; int numFailures; RF_RaidAddr_t sosAddr; RF_SectorNum_t diskOffset, poffset; /* quick out in the fault-free case. */ rf_lock_mutex2(raidPtr->mutex); numFailures = raidPtr->numFailures; rf_unlock_mutex2(raidPtr->mutex); if (numFailures == 0) return (0); sosAddr = rf_RaidAddressOfPrevStripeBoundary(layoutPtr, asmap->raidAddress); (layoutPtr->map->IdentifyStripe) (raidPtr, asmap->raidAddress, &diskids); (layoutPtr->map->MapParity) (raidPtr, asmap->raidAddress, &pcol, &poffset, 0); /* get pcol */ /* this need not be true if we've redirected the access to a * spare in another row RF_ASSERT(row == testrow); */ stripeOffset = 0; for (i = 0; i < layoutPtr->numDataCol + layoutPtr->numParityCol; i++) { if (diskids[i] != pcol) { if (RF_DEAD_DISK(raidPtr->Disks[diskids[i]].status)) { if (raidPtr->status != rf_rs_reconstructing) return (1); RF_ASSERT(raidPtr->reconControl->fcol == diskids[i]); layoutPtr->map->MapSector(raidPtr, sosAddr + stripeOffset * layoutPtr->sectorsPerStripeUnit, &tcol, &diskOffset, 0); RF_ASSERT(tcol == diskids[i]); if (!rf_CheckRUReconstructed(raidPtr->reconControl->reconMap, diskOffset)) return (1); asmap->flags |= RF_ASM_REDIR_LARGE_WRITE; return (0); } stripeOffset++; } } return (0); } #if (RF_INCLUDE_DECL_PQ > 0) || (RF_INCLUDE_RAID6 > 0) || (RF_INCLUDE_EVENODD >0) /* return the number of failed data units in the stripe. */ int rf_NumFailedDataUnitsInStripe(RF_Raid_t *raidPtr, RF_AccessStripeMap_t *asmap) { RF_RaidLayout_t *layoutPtr = &raidPtr->Layout; RF_RowCol_t tcol, i; RF_SectorNum_t diskOffset; RF_RaidAddr_t sosAddr; int numFailures; /* quick out in the fault-free case. */ rf_lock_mutex2(raidPtr->mutex); numFailures = raidPtr->numFailures; rf_unlock_mutex2(raidPtr->mutex); if (numFailures == 0) return (0); numFailures = 0; sosAddr = rf_RaidAddressOfPrevStripeBoundary(layoutPtr, asmap->raidAddress); for (i = 0; i < layoutPtr->numDataCol; i++) { (layoutPtr->map->MapSector) (raidPtr, sosAddr + i * layoutPtr->sectorsPerStripeUnit, &tcol, &diskOffset, 0); if (RF_DEAD_DISK(raidPtr->Disks[tcol].status)) numFailures++; } return numFailures; } #endif /**************************************************************************** * * debug routines * ***************************************************************************/ #if RF_DEBUG_MAP void rf_PrintAccessStripeMap(RF_AccessStripeMapHeader_t *asm_h) { rf_PrintFullAccessStripeMap(asm_h, 0); } #endif /* prbuf - flag to print buffer pointers */ void rf_PrintFullAccessStripeMap(RF_AccessStripeMapHeader_t *asm_h, int prbuf) { int i; RF_AccessStripeMap_t *asmap = asm_h->stripeMap; RF_PhysDiskAddr_t *p; printf("%d stripes total\n", (int) asm_h->numStripes); for (; asmap; asmap = asmap->next) { /* printf("Num failures: %d\n",asmap->numDataFailed); */ /* printf("Num sectors: * %d\n",(int)asmap->totalSectorsAccessed); */ printf("Stripe %d (%d sectors), failures: %d data, %d parity: ", (int) asmap->stripeID, (int) asmap->totalSectorsAccessed, (int) asmap->numDataFailed, (int) asmap->numParityFailed); if (asmap->parityInfo) { printf("Parity [c%d s%d-%d", asmap->parityInfo->col, (int) asmap->parityInfo->startSector, (int) (asmap->parityInfo->startSector + asmap->parityInfo->numSector - 1)); if (prbuf) printf(" b0x%lx", (unsigned long) asmap->parityInfo->bufPtr); if (asmap->parityInfo->next) { printf(", c%d s%d-%d", asmap->parityInfo->next->col, (int) asmap->parityInfo->next->startSector, (int) (asmap->parityInfo->next->startSector + asmap->parityInfo->next->numSector - 1)); if (prbuf) printf(" b0x%lx", (unsigned long) asmap->parityInfo->next->bufPtr); RF_ASSERT(asmap->parityInfo->next->next == NULL); } printf("]\n\t"); } for (i = 0, p = asmap->physInfo; p; p = p->next, i++) { printf("SU c%d s%d-%d ", p->col, (int) p->startSector, (int) (p->startSector + p->numSector - 1)); if (prbuf) printf("b0x%lx ", (unsigned long) p->bufPtr); if (i && !(i & 1)) printf("\n\t"); } printf("\n"); p = asm_h->stripeMap->failedPDAs[0]; if (asm_h->stripeMap->numDataFailed + asm_h->stripeMap->numParityFailed > 1) printf("[multiple failures]\n"); else if (asm_h->stripeMap->numDataFailed + asm_h->stripeMap->numParityFailed > 0) printf("\t[Failed PDA: c%d s%d-%d]\n", p->col, (int) p->startSector, (int) (p->startSector + p->numSector - 1)); } } #if RF_MAP_DEBUG void rf_PrintRaidAddressInfo(RF_Raid_t *raidPtr, RF_RaidAddr_t raidAddr, RF_SectorCount_t numBlocks) { RF_RaidLayout_t *layoutPtr = &raidPtr->Layout; RF_RaidAddr_t ra, sosAddr = rf_RaidAddressOfPrevStripeBoundary(layoutPtr, raidAddr); printf("Raid addrs of SU boundaries from start of stripe to end of access:\n\t"); for (ra = sosAddr; ra <= raidAddr + numBlocks; ra += layoutPtr->sectorsPerStripeUnit) { printf("%d (0x%x), ", (int) ra, (int) ra); } printf("\n"); printf("Offset into stripe unit: %d (0x%x)\n", (int) (raidAddr % layoutPtr->sectorsPerStripeUnit), (int) (raidAddr % layoutPtr->sectorsPerStripeUnit)); } #endif /* given a parity descriptor and the starting address within a stripe, * range restrict the parity descriptor to touch only the correct * stuff. */ void rf_ASMParityAdjust(RF_Raid_t *raidPtr, RF_PhysDiskAddr_t *toAdjust, RF_StripeNum_t startAddrWithinStripe, RF_SectorNum_t endAddress, RF_RaidLayout_t *layoutPtr, RF_AccessStripeMap_t *asm_p) { RF_PhysDiskAddr_t *new_pda; /* when we're accessing only a portion of one stripe unit, we * want the parity descriptor to identify only the chunk of * parity associated with the data. When the access spans * exactly one stripe unit boundary and is less than a stripe * unit in size, it uses two disjoint regions of the parity * unit. When an access spans more than one stripe unit * boundary, it uses all of the parity unit. * * To better handle the case where stripe units are small, we * may eventually want to change the 2nd case so that if the * SU size is below some threshold, we just read/write the * whole thing instead of breaking it up into two accesses. */ if (asm_p->numStripeUnitsAccessed == 1) { int x = (startAddrWithinStripe % layoutPtr->sectorsPerStripeUnit); toAdjust->startSector += x; toAdjust->raidAddress += x; toAdjust->numSector = asm_p->physInfo->numSector; RF_ASSERT(toAdjust->numSector != 0); } else if (asm_p->numStripeUnitsAccessed == 2 && asm_p->totalSectorsAccessed < layoutPtr->sectorsPerStripeUnit) { int x = (startAddrWithinStripe % layoutPtr->sectorsPerStripeUnit); /* create a second pda and copy the parity map info * into it */ RF_ASSERT(toAdjust->next == NULL); /* the following will get freed in rf_FreeAccessStripeMap() via rf_FreePDAList() */ new_pda = toAdjust->next = rf_AllocPhysDiskAddr(raidPtr); *new_pda = *toAdjust; /* structure assignment */ new_pda->next = NULL; /* adjust the start sector & number of blocks for the * first parity pda */ toAdjust->startSector += x; toAdjust->raidAddress += x; toAdjust->numSector = rf_RaidAddressOfNextStripeUnitBoundary(layoutPtr, startAddrWithinStripe) - startAddrWithinStripe; RF_ASSERT(toAdjust->numSector != 0); /* adjust the second pda */ new_pda->numSector = endAddress - rf_RaidAddressOfPrevStripeUnitBoundary(layoutPtr, endAddress); /* new_pda->raidAddress = * rf_RaidAddressOfNextStripeUnitBoundary(layoutPtr, * toAdjust->raidAddress); */ RF_ASSERT(new_pda->numSector != 0); } } /* Check if a disk has been spared or failed. If spared, redirect the * I/O. If it has been failed, record it in the asm pointer. Fifth * arg is whether data or parity. */ void rf_ASMCheckStatus(RF_Raid_t *raidPtr, RF_PhysDiskAddr_t *pda_p, RF_AccessStripeMap_t *asm_p, RF_RaidDisk_t *disks, int parity) { RF_DiskStatus_t dstatus; RF_RowCol_t fcol; dstatus = disks[pda_p->col].status; if (dstatus == rf_ds_spared) { /* if the disk has been spared, redirect access to the spare */ fcol = pda_p->col; pda_p->col = disks[fcol].spareCol; } else if (dstatus == rf_ds_dist_spared) { /* ditto if disk has been spared to dist spare space */ #if RF_DEBUG_MAP RF_RowCol_t oc = pda_p->col; RF_SectorNum_t oo = pda_p->startSector; #endif if (pda_p->type == RF_PDA_TYPE_DATA) raidPtr->Layout.map->MapSector(raidPtr, pda_p->raidAddress, &pda_p->col, &pda_p->startSector, RF_REMAP); else raidPtr->Layout.map->MapParity(raidPtr, pda_p->raidAddress, &pda_p->col, &pda_p->startSector, RF_REMAP); #if RF_DEBUG_MAP if (rf_mapDebug) { printf("Redirected c %d o %d -> c %d o %d\n", oc, (int) oo, pda_p->col, (int) pda_p->startSector); } #endif } else if (RF_DEAD_DISK(dstatus)) { /* if the disk is inaccessible, mark the * failure */ if (parity) asm_p->numParityFailed++; else { asm_p->numDataFailed++; } asm_p->failedPDAs[asm_p->numFailedPDAs] = pda_p; asm_p->numFailedPDAs++; #if 0 switch (asm_p->numParityFailed + asm_p->numDataFailed) { case 1: asm_p->failedPDAs[0] = pda_p; break; case 2: asm_p->failedPDAs[1] = pda_p; default: break; } #endif } /* the redirected access should never span a stripe unit boundary */ RF_ASSERT(rf_RaidAddressToStripeUnitID(&raidPtr->Layout, pda_p->raidAddress) == rf_RaidAddressToStripeUnitID(&raidPtr->Layout, pda_p->raidAddress + pda_p->numSector - 1)); RF_ASSERT(pda_p->col != -1); } |
| 904 94 815 201 203 131 76 153 94 59 153 4 3 3 3 3 3 3 3 3 2 2 2 2 1633 1632 119 1494 1495 732 733 92 92 1495 717 92 23 93 23 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 197 198 199 200 201 202 203 204 205 206 207 208 209 210 211 212 213 214 215 216 217 218 219 220 221 222 223 224 225 226 227 228 229 230 231 232 233 234 235 236 237 238 239 240 241 242 243 244 245 246 247 248 249 250 251 252 253 254 255 256 257 258 259 260 261 262 263 264 265 266 267 268 269 270 271 272 273 274 275 276 277 278 279 280 281 282 283 284 285 286 287 288 289 290 291 292 293 294 295 296 297 298 299 300 301 302 303 304 305 306 307 308 309 310 311 312 313 314 315 316 317 318 319 320 321 322 323 324 325 326 327 328 329 330 331 332 333 334 335 336 337 338 339 340 341 342 343 344 345 346 347 348 349 350 351 352 353 354 355 356 357 358 359 360 361 362 363 364 365 366 367 368 369 370 371 372 373 374 375 | /* $NetBSD: uvm_object.c,v 1.25 2020/08/15 07:24:09 chs Exp $ */ /* * Copyright (c) 2006, 2010, 2019 The NetBSD Foundation, Inc. * All rights reserved. * * This code is derived from software contributed to The NetBSD Foundation * by Mindaugas Rasiukevicius. * * Redistribution and use in source and binary forms, with or without * modification, are permitted provided that the following conditions * are met: * 1. Redistributions of source code must retain the above copyright * notice, this list of conditions and the following disclaimer. * 2. Redistributions in binary form must reproduce the above copyright * notice, this list of conditions and the following disclaimer in the * documentation and/or other materials provided with the distribution. * * THIS SOFTWARE IS PROVIDED BY THE NETBSD FOUNDATION, INC. AND CONTRIBUTORS * ``AS IS'' AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED * TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR * PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE FOUNDATION OR CONTRIBUTORS * BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR * CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF * SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS * INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN * CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) * ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE * POSSIBILITY OF SUCH DAMAGE. */ /* * uvm_object.c: operate with memory objects * * TODO: * 1. Support PG_RELEASED-using objects */ #include <sys/cdefs.h> __KERNEL_RCSID(0, "$NetBSD: uvm_object.c,v 1.25 2020/08/15 07:24:09 chs Exp $"); #ifdef _KERNEL_OPT #include "opt_ddb.h" #endif #include <sys/param.h> #include <sys/rwlock.h> #include <sys/queue.h> #include <uvm/uvm.h> #include <uvm/uvm_ddb.h> #include <uvm/uvm_page_array.h> /* Page count to fetch per single step. */ #define FETCH_PAGECOUNT 16 /* * uvm_obj_init: initialize UVM memory object. */ void uvm_obj_init(struct uvm_object *uo, const struct uvm_pagerops *ops, bool alock, u_int refs) { #if 0 /* notyet */ KASSERT(ops); #endif if (alock) { /* Allocate and assign a lock. */ uo->vmobjlock = rw_obj_alloc(); } else { /* The lock will need to be set via uvm_obj_setlock(). */ uo->vmobjlock = NULL; } uo->pgops = ops; LIST_INIT(&uo->uo_ubc); uo->uo_npages = 0; uo->uo_refs = refs; radix_tree_init_tree(&uo->uo_pages); } /* * uvm_obj_destroy: destroy UVM memory object. */ void uvm_obj_destroy(struct uvm_object *uo, bool dlock) { KASSERT(radix_tree_empty_tree_p(&uo->uo_pages)); /* Purge any UBC entries associated with this object. */ ubc_purge(uo); /* Destroy the lock, if requested. */ if (dlock) { rw_obj_free(uo->vmobjlock); } radix_tree_fini_tree(&uo->uo_pages); } /* * uvm_obj_setlock: assign a vmobjlock to the UVM object. * * => Caller is responsible to ensure that UVM objects is not use. * => Only dynamic lock may be previously set. We drop the reference then. */ void uvm_obj_setlock(struct uvm_object *uo, krwlock_t *lockptr) { krwlock_t *olockptr = uo->vmobjlock; if (olockptr) { /* Drop the reference on the old lock. */ rw_obj_free(olockptr); } if (lockptr == NULL) { /* If new lock is not passed - allocate default one. */ lockptr = rw_obj_alloc(); } uo->vmobjlock = lockptr; } /* * uvm_obj_wirepages: wire the pages of entire UVM object. * * => NOTE: this function should only be used for types of objects * where PG_RELEASED flag is never set (aobj objects) * => caller must pass page-aligned start and end values */ int uvm_obj_wirepages(struct uvm_object *uobj, off_t start, off_t end, struct pglist *list) { int i, npages, error; struct vm_page *pgs[FETCH_PAGECOUNT], *pg = NULL; off_t offset = start, left; left = (end - start) >> PAGE_SHIFT; rw_enter(uobj->vmobjlock, RW_WRITER); while (left) { npages = MIN(FETCH_PAGECOUNT, left); /* Get the pages */ memset(pgs, 0, sizeof(pgs)); error = (*uobj->pgops->pgo_get)(uobj, offset, pgs, &npages, 0, VM_PROT_READ | VM_PROT_WRITE, UVM_ADV_SEQUENTIAL, PGO_SYNCIO); if (error) goto error; rw_enter(uobj->vmobjlock, RW_WRITER); for (i = 0; i < npages; i++) { KASSERT(pgs[i] != NULL); KASSERT(!(pgs[i]->flags & PG_RELEASED)); /* * Loan break */ if (pgs[i]->loan_count) { while (pgs[i]->loan_count) { pg = uvm_loanbreak(pgs[i]); if (!pg) { rw_exit(uobj->vmobjlock); uvm_wait("uobjwirepg"); rw_enter(uobj->vmobjlock, RW_WRITER); continue; } } pgs[i] = pg; } if (pgs[i]->flags & PG_AOBJ) { uvm_pagemarkdirty(pgs[i], UVM_PAGE_STATUS_DIRTY); uao_dropswap(uobj, i); } } /* Wire the pages */ for (i = 0; i < npages; i++) { uvm_pagelock(pgs[i]); uvm_pagewire(pgs[i]); uvm_pageunlock(pgs[i]); if (list != NULL) TAILQ_INSERT_TAIL(list, pgs[i], pageq.queue); } /* Unbusy the pages */ uvm_page_unbusy(pgs, npages); left -= npages; offset += npages << PAGE_SHIFT; } rw_exit(uobj->vmobjlock); return 0; error: /* Unwire the pages which has been wired */ uvm_obj_unwirepages(uobj, start, offset); return error; } /* * uvm_obj_unwirepages: unwire the pages of entire UVM object. * * => NOTE: this function should only be used for types of objects * where PG_RELEASED flag is never set * => caller must pass page-aligned start and end values */ void uvm_obj_unwirepages(struct uvm_object *uobj, off_t start, off_t end) { struct vm_page *pg; off_t offset; rw_enter(uobj->vmobjlock, RW_WRITER); for (offset = start; offset < end; offset += PAGE_SIZE) { pg = uvm_pagelookup(uobj, offset); KASSERT(pg != NULL); KASSERT(!(pg->flags & PG_RELEASED)); uvm_pagelock(pg); uvm_pageunwire(pg); uvm_pageunlock(pg); } rw_exit(uobj->vmobjlock); } static inline bool uvm_obj_notag_p(struct uvm_object *uobj, int tag) { KASSERT(rw_lock_held(uobj->vmobjlock)); return radix_tree_empty_tagged_tree_p(&uobj->uo_pages, tag); } bool uvm_obj_clean_p(struct uvm_object *uobj) { return uvm_obj_notag_p(uobj, UVM_PAGE_DIRTY_TAG); } bool uvm_obj_nowriteback_p(struct uvm_object *uobj) { return uvm_obj_notag_p(uobj, UVM_PAGE_WRITEBACK_TAG); } static inline bool uvm_obj_page_tag_p(struct vm_page *pg, int tag) { struct uvm_object *uobj = pg->uobject; uint64_t pgidx = pg->offset >> PAGE_SHIFT; KASSERT(uobj != NULL); KASSERT(rw_lock_held(uobj->vmobjlock)); return radix_tree_get_tag(&uobj->uo_pages, pgidx, tag) != 0; } static inline void uvm_obj_page_set_tag(struct vm_page *pg, int tag) { struct uvm_object *uobj = pg->uobject; uint64_t pgidx = pg->offset >> PAGE_SHIFT; KASSERT(uobj != NULL); KASSERT(rw_write_held(uobj->vmobjlock)); radix_tree_set_tag(&uobj->uo_pages, pgidx, tag); } static inline void uvm_obj_page_clear_tag(struct vm_page *pg, int tag) { struct uvm_object *uobj = pg->uobject; uint64_t pgidx = pg->offset >> PAGE_SHIFT; KASSERT(uobj != NULL); KASSERT(rw_write_held(uobj->vmobjlock)); radix_tree_clear_tag(&uobj->uo_pages, pgidx, tag); } bool uvm_obj_page_dirty_p(struct vm_page *pg) { return uvm_obj_page_tag_p(pg, UVM_PAGE_DIRTY_TAG); } void uvm_obj_page_set_dirty(struct vm_page *pg) { uvm_obj_page_set_tag(pg, UVM_PAGE_DIRTY_TAG); } void uvm_obj_page_clear_dirty(struct vm_page *pg) { uvm_obj_page_clear_tag(pg, UVM_PAGE_DIRTY_TAG); } bool uvm_obj_page_writeback_p(struct vm_page *pg) { return uvm_obj_page_tag_p(pg, UVM_PAGE_WRITEBACK_TAG); } void uvm_obj_page_set_writeback(struct vm_page *pg) { uvm_obj_page_set_tag(pg, UVM_PAGE_WRITEBACK_TAG); } void uvm_obj_page_clear_writeback(struct vm_page *pg) { uvm_obj_page_clear_tag(pg, UVM_PAGE_WRITEBACK_TAG); } #if defined(DDB) || defined(DEBUGPRINT) /* * uvm_object_printit: actually prints the object */ void uvm_object_printit(struct uvm_object *uobj, bool full, void (*pr)(const char *, ...)) { struct uvm_page_array a; struct vm_page *pg; int cnt = 0; voff_t off; (*pr)("OBJECT %p: locked=%d, pgops=%p, npages=%d, ", uobj, rw_write_held(uobj->vmobjlock), uobj->pgops, uobj->uo_npages); if (UVM_OBJ_IS_KERN_OBJECT(uobj)) (*pr)("refs=<SYSTEM>\n"); else (*pr)("refs=%d\n", uobj->uo_refs); if (!full) { return; } (*pr)(" PAGES <pg,offset>:\n "); uvm_page_array_init(&a, uobj, 0); off = 0; while ((pg = uvm_page_array_fill_and_peek(&a, off, 0)) != NULL) { cnt++; (*pr)("<%p,0x%llx> ", pg, (long long)pg->offset); if ((cnt % 3) == 0) { (*pr)("\n "); } off = pg->offset + PAGE_SIZE; uvm_page_array_advance(&a); } if ((cnt % 3) != 0) { (*pr)("\n"); } uvm_page_array_fini(&a); } #endif /* DDB || DEBUGPRINT */ |
| 72 72 72 72 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 197 198 199 200 201 202 203 204 205 206 207 208 209 210 211 212 213 214 215 216 217 218 219 220 221 222 223 224 225 226 227 228 229 230 231 232 233 234 235 236 237 238 239 240 241 242 243 244 245 246 247 248 249 250 251 252 253 254 255 256 257 258 259 260 261 262 263 264 265 266 267 268 269 270 271 272 273 274 275 276 277 278 279 280 281 282 283 284 285 286 287 288 289 290 291 292 293 294 295 296 297 298 299 300 301 302 303 304 305 306 307 308 309 310 311 312 313 314 315 316 317 318 319 320 321 322 323 324 325 326 327 328 329 330 331 332 333 334 335 336 337 338 339 340 341 342 343 344 345 346 347 348 349 350 351 352 353 354 355 356 357 358 359 360 361 362 363 364 365 366 367 368 369 370 371 372 373 374 375 376 377 378 379 380 381 382 383 384 385 386 387 388 389 390 391 392 393 394 395 396 397 398 399 400 401 402 403 404 405 406 407 408 409 410 411 412 413 414 415 416 417 418 419 420 421 422 423 424 425 426 427 428 429 430 431 432 433 434 435 436 437 438 439 440 441 442 443 444 445 446 447 448 449 450 451 452 453 454 455 456 457 458 459 460 461 462 463 464 465 466 467 468 469 470 471 472 473 474 475 476 477 478 479 480 481 482 483 484 485 486 487 488 489 490 491 492 493 494 495 496 497 498 499 500 501 502 503 504 505 506 507 508 509 510 511 512 513 514 515 516 517 518 519 520 521 522 523 524 525 526 527 528 529 530 531 532 533 534 535 536 537 538 539 540 541 542 543 544 545 546 547 548 549 550 551 552 553 554 555 556 557 558 559 560 561 562 563 564 565 566 567 568 569 570 571 572 573 574 575 576 577 578 579 580 581 582 583 584 585 586 587 588 589 590 591 592 593 594 595 596 597 598 599 600 601 | /* $NetBSD: pktqueue.c,v 1.16 2021/12/21 04:09:32 knakahara Exp $ */ /*- * Copyright (c) 2014 The NetBSD Foundation, Inc. * All rights reserved. * * This code is derived from software contributed to The NetBSD Foundation * by Mindaugas Rasiukevicius. * * Redistribution and use in source and binary forms, with or without * modification, are permitted provided that the following conditions * are met: * 1. Redistributions of source code must retain the above copyright * notice, this list of conditions and the following disclaimer. * 2. Redistributions in binary form must reproduce the above copyright * notice, this list of conditions and the following disclaimer in the * documentation and/or other materials provided with the distribution. * * THIS SOFTWARE IS PROVIDED BY THE NETBSD FOUNDATION, INC. AND CONTRIBUTORS * ``AS IS'' AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED * TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR * PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE FOUNDATION OR CONTRIBUTORS * BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR * CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF * SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS * INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN * CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) * ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE * POSSIBILITY OF SUCH DAMAGE. */ /* * The packet queue (pktqueue) interface is a lockless IP input queue * which also abstracts and handles network ISR scheduling. It provides * a mechanism to enable receiver-side packet steering (RPS). */ #include <sys/cdefs.h> __KERNEL_RCSID(0, "$NetBSD: pktqueue.c,v 1.16 2021/12/21 04:09:32 knakahara Exp $"); #ifdef _KERNEL_OPT #include "opt_net_mpsafe.h" #endif #include <sys/param.h> #include <sys/types.h> #include <sys/atomic.h> #include <sys/cpu.h> #include <sys/pcq.h> #include <sys/intr.h> #include <sys/mbuf.h> #include <sys/proc.h> #include <sys/percpu.h> #include <sys/xcall.h> #include <net/pktqueue.h> #include <net/rss_config.h> #include <netinet/in.h> #include <netinet/ip.h> #include <netinet/ip6.h> struct pktqueue { /* * The lock used for a barrier mechanism. The barrier counter, * as well as the drop counter, are managed atomically though. * Ensure this group is in a separate cache line. */ union { struct { kmutex_t pq_lock; volatile u_int pq_barrier; }; uint8_t _pad[COHERENCY_UNIT]; }; /* The size of the queue, counters and the interrupt handler. */ u_int pq_maxlen; percpu_t * pq_counters; void * pq_sih; /* Finally, per-CPU queues. */ struct percpu * pq_pcq; /* struct pcq * */ }; /* The counters of the packet queue. */ #define PQCNT_ENQUEUE 0 #define PQCNT_DEQUEUE 1 #define PQCNT_DROP 2 #define PQCNT_NCOUNTERS 3 typedef struct { uint64_t count[PQCNT_NCOUNTERS]; } pktq_counters_t; /* Special marker value used by pktq_barrier() mechanism. */ #define PKTQ_MARKER ((void *)(~0ULL)) static void pktq_init_cpu(void *vqp, void *vpq, struct cpu_info *ci) { struct pcq **qp = vqp; struct pktqueue *pq = vpq; *qp = pcq_create(pq->pq_maxlen, KM_SLEEP); } static void pktq_fini_cpu(void *vqp, void *vpq, struct cpu_info *ci) { struct pcq **qp = vqp, *q = *qp; KASSERT(pcq_peek(q) == NULL); pcq_destroy(q); *qp = NULL; /* paranoia */ } static struct pcq * pktq_pcq(struct pktqueue *pq, struct cpu_info *ci) { struct pcq **qp, *q; /* * As long as preemption is disabled, the xcall to swap percpu * buffers can't complete, so it is safe to read the pointer. */ KASSERT(kpreempt_disabled()); qp = percpu_getptr_remote(pq->pq_pcq, ci); q = *qp; return q; } pktqueue_t * pktq_create(size_t maxlen, void (*intrh)(void *), void *sc) { const u_int sflags = SOFTINT_NET | SOFTINT_MPSAFE | SOFTINT_RCPU; pktqueue_t *pq; percpu_t *pc; void *sih; pc = percpu_alloc(sizeof(pktq_counters_t)); if ((sih = softint_establish(sflags, intrh, sc)) == NULL) { percpu_free(pc, sizeof(pktq_counters_t)); return NULL; } pq = kmem_zalloc(sizeof(*pq), KM_SLEEP); mutex_init(&pq->pq_lock, MUTEX_DEFAULT, IPL_NONE); pq->pq_maxlen = maxlen; pq->pq_counters = pc; pq->pq_sih = sih; pq->pq_pcq = percpu_create(sizeof(struct pcq *), pktq_init_cpu, pktq_fini_cpu, pq); return pq; } void pktq_destroy(pktqueue_t *pq) { percpu_free(pq->pq_pcq, sizeof(struct pcq *)); percpu_free(pq->pq_counters, sizeof(pktq_counters_t)); softint_disestablish(pq->pq_sih); mutex_destroy(&pq->pq_lock); kmem_free(pq, sizeof(*pq)); } /* * - pktq_inc_counter: increment the counter given an ID. * - pktq_collect_counts: handler to sum up the counts from each CPU. * - pktq_getcount: return the effective count given an ID. */ static inline void pktq_inc_count(pktqueue_t *pq, u_int i) { percpu_t *pc = pq->pq_counters; pktq_counters_t *c; c = percpu_getref(pc); c->count[i]++; percpu_putref(pc); } static void pktq_collect_counts(void *mem, void *arg, struct cpu_info *ci) { const pktq_counters_t *c = mem; pktq_counters_t *sum = arg; int s = splnet(); for (u_int i = 0; i < PQCNT_NCOUNTERS; i++) { sum->count[i] += c->count[i]; } splx(s); } uint64_t pktq_get_count(pktqueue_t *pq, pktq_count_t c) { pktq_counters_t sum; if (c != PKTQ_MAXLEN) { memset(&sum, 0, sizeof(sum)); percpu_foreach_xcall(pq->pq_counters, XC_HIGHPRI_IPL(IPL_SOFTNET), pktq_collect_counts, &sum); } switch (c) { case PKTQ_NITEMS: return sum.count[PQCNT_ENQUEUE] - sum.count[PQCNT_DEQUEUE]; case PKTQ_DROPS: return sum.count[PQCNT_DROP]; case PKTQ_MAXLEN: return pq->pq_maxlen; } return 0; } uint32_t pktq_rps_hash(pktq_rps_hash_func_t *funcp, const struct mbuf *m) { pktq_rps_hash_func_t func = atomic_load_relaxed(funcp); KASSERT(func != NULL); return (*func)(m); } static uint32_t pktq_rps_hash_zero(const struct mbuf *m __unused) { return 0; } static uint32_t pktq_rps_hash_curcpu(const struct mbuf *m __unused) { return cpu_index(curcpu()); } static uint32_t pktq_rps_hash_toeplitz(const struct mbuf *m) { struct ip *ip; /* * Disable UDP port - IP fragments aren't currently being handled * and so we end up with a mix of 2-tuple and 4-tuple * traffic. */ const u_int flag = RSS_TOEPLITZ_USE_TCP_PORT; /* glance IP version */ if ((m->m_flags & M_PKTHDR) == 0) return 0; ip = mtod(m, struct ip *); if (ip->ip_v == IPVERSION) { if (__predict_false(m->m_len < sizeof(struct ip))) return 0; return rss_toeplitz_hash_from_mbuf_ipv4(m, flag); } else if (ip->ip_v == 6) { if (__predict_false(m->m_len < sizeof(struct ip6_hdr))) return 0; return rss_toeplitz_hash_from_mbuf_ipv6(m, flag); } return 0; } /* * toeplitz without curcpu. * Generally, this has better performance than toeplitz. */ static uint32_t pktq_rps_hash_toeplitz_othercpus(const struct mbuf *m) { uint32_t hash; if (ncpu == 1) return 0; hash = pktq_rps_hash_toeplitz(m); hash %= ncpu - 1; if (hash >= cpu_index(curcpu())) return hash + 1; else return hash; } static struct pktq_rps_hash_table { const char* prh_type; pktq_rps_hash_func_t prh_func; } const pktq_rps_hash_tab[] = { { "zero", pktq_rps_hash_zero }, { "curcpu", pktq_rps_hash_curcpu }, { "toeplitz", pktq_rps_hash_toeplitz }, { "toeplitz-othercpus", pktq_rps_hash_toeplitz_othercpus }, }; const pktq_rps_hash_func_t pktq_rps_hash_default = #ifdef NET_MPSAFE pktq_rps_hash_curcpu; #else pktq_rps_hash_zero; #endif static const char * pktq_get_rps_hash_type(pktq_rps_hash_func_t func) { for (int i = 0; i < __arraycount(pktq_rps_hash_tab); i++) { if (func == pktq_rps_hash_tab[i].prh_func) { return pktq_rps_hash_tab[i].prh_type; } } return NULL; } static int pktq_set_rps_hash_type(pktq_rps_hash_func_t *func, const char *type) { if (strcmp(type, pktq_get_rps_hash_type(*func)) == 0) return 0; for (int i = 0; i < __arraycount(pktq_rps_hash_tab); i++) { if (strcmp(type, pktq_rps_hash_tab[i].prh_type) == 0) { atomic_store_relaxed(func, pktq_rps_hash_tab[i].prh_func); return 0; } } return ENOENT; } int sysctl_pktq_rps_hash_handler(SYSCTLFN_ARGS) { struct sysctlnode node; pktq_rps_hash_func_t *func; int error; char type[PKTQ_RPS_HASH_NAME_LEN]; node = *rnode; func = node.sysctl_data; strlcpy(type, pktq_get_rps_hash_type(*func), PKTQ_RPS_HASH_NAME_LEN); node.sysctl_data = &type; node.sysctl_size = sizeof(type); error = sysctl_lookup(SYSCTLFN_CALL(&node)); if (error || newp == NULL) return error; error = pktq_set_rps_hash_type(func, type); return error; } /* * pktq_enqueue: inject the packet into the end of the queue. * * => Must be called from the interrupt or with the preemption disabled. * => Consumes the packet and returns true on success. * => Returns false on failure; caller is responsible to free the packet. */ bool pktq_enqueue(pktqueue_t *pq, struct mbuf *m, const u_int hash __unused) { #if defined(_RUMPKERNEL) || defined(_RUMP_NATIVE_ABI) struct cpu_info *ci = curcpu(); #else struct cpu_info *ci = cpu_lookup(hash % ncpu); #endif KASSERT(kpreempt_disabled()); if (__predict_false(!pcq_put(pktq_pcq(pq, ci), m))) { pktq_inc_count(pq, PQCNT_DROP); return false; } softint_schedule_cpu(pq->pq_sih, ci); pktq_inc_count(pq, PQCNT_ENQUEUE); return true; } /* * pktq_dequeue: take a packet from the queue. * * => Must be called with preemption disabled. * => Must ensure there are not concurrent dequeue calls. */ struct mbuf * pktq_dequeue(pktqueue_t *pq) { struct cpu_info *ci = curcpu(); struct mbuf *m; KASSERT(kpreempt_disabled()); m = pcq_get(pktq_pcq(pq, ci)); if (__predict_false(m == PKTQ_MARKER)) { /* Note the marker entry. */ atomic_inc_uint(&pq->pq_barrier); return NULL; } if (__predict_true(m != NULL)) { pktq_inc_count(pq, PQCNT_DEQUEUE); } return m; } /* * pktq_barrier: waits for a grace period when all packets enqueued at * the moment of calling this routine will be processed. This is used * to ensure that e.g. packets referencing some interface were drained. */ void pktq_barrier(pktqueue_t *pq) { CPU_INFO_ITERATOR cii; struct cpu_info *ci; u_int pending = 0; mutex_enter(&pq->pq_lock); KASSERT(pq->pq_barrier == 0); for (CPU_INFO_FOREACH(cii, ci)) { struct pcq *q; kpreempt_disable(); q = pktq_pcq(pq, ci); kpreempt_enable(); /* If the queue is empty - nothing to do. */ if (pcq_peek(q) == NULL) { continue; } /* Otherwise, put the marker and entry. */ while (!pcq_put(q, PKTQ_MARKER)) { kpause("pktqsync", false, 1, NULL); } kpreempt_disable(); softint_schedule_cpu(pq->pq_sih, ci); kpreempt_enable(); pending++; } /* Wait for each queue to process the markers. */ while (pq->pq_barrier != pending) { kpause("pktqsync", false, 1, NULL); } pq->pq_barrier = 0; mutex_exit(&pq->pq_lock); } /* * pktq_flush: free mbufs in all queues. * * => The caller must ensure there are no concurrent writers or flush calls. */ void pktq_flush(pktqueue_t *pq) { CPU_INFO_ITERATOR cii; struct cpu_info *ci; struct mbuf *m; for (CPU_INFO_FOREACH(cii, ci)) { struct pcq *q; kpreempt_disable(); q = pktq_pcq(pq, ci); kpreempt_enable(); /* * XXX This can't be right -- if the softint is running * then pcq_get isn't safe here. */ while ((m = pcq_get(q)) != NULL) { pktq_inc_count(pq, PQCNT_DEQUEUE); m_freem(m); } } } static void pktq_set_maxlen_cpu(void *vpq, void *vqs) { struct pktqueue *pq = vpq; struct pcq **qp, *q, **qs = vqs; unsigned i = cpu_index(curcpu()); int s; s = splnet(); qp = percpu_getref(pq->pq_pcq); q = *qp; *qp = qs[i]; qs[i] = q; percpu_putref(pq->pq_pcq); splx(s); } /* * pktq_set_maxlen: create per-CPU queues using a new size and replace * the existing queues without losing any packets. * * XXX ncpu must remain stable throughout. */ int pktq_set_maxlen(pktqueue_t *pq, size_t maxlen) { const u_int slotbytes = ncpu * sizeof(pcq_t *); pcq_t **qs; if (!maxlen || maxlen > PCQ_MAXLEN) return EINVAL; if (pq->pq_maxlen == maxlen) return 0; /* First, allocate the new queues. */ qs = kmem_zalloc(slotbytes, KM_SLEEP); for (u_int i = 0; i < ncpu; i++) { qs[i] = pcq_create(maxlen, KM_SLEEP); } /* * Issue an xcall to replace the queue pointers on each CPU. * This implies all the necessary memory barriers. */ mutex_enter(&pq->pq_lock); xc_wait(xc_broadcast(XC_HIGHPRI, pktq_set_maxlen_cpu, pq, qs)); pq->pq_maxlen = maxlen; mutex_exit(&pq->pq_lock); /* * At this point, the new packets are flowing into the new * queues. However, the old queues may have some packets * present which are no longer being processed. We are going * to re-enqueue them. This may change the order of packet * arrival, but it is not considered an issue. * * There may be in-flight interrupts calling pktq_dequeue() * which reference the old queues. Issue a barrier to ensure * that we are going to be the only pcq_get() callers on the * old queues. */ pktq_barrier(pq); for (u_int i = 0; i < ncpu; i++) { struct pcq *q; struct mbuf *m; kpreempt_disable(); q = pktq_pcq(pq, cpu_lookup(i)); kpreempt_enable(); while ((m = pcq_get(qs[i])) != NULL) { while (!pcq_put(q, m)) { kpause("pktqrenq", false, 1, NULL); } } pcq_destroy(qs[i]); } /* Well, that was fun. */ kmem_free(qs, slotbytes); return 0; } int sysctl_pktq_maxlen(SYSCTLFN_ARGS, pktqueue_t *pq) { u_int nmaxlen = pktq_get_count(pq, PKTQ_MAXLEN); struct sysctlnode node = *rnode; int error; node.sysctl_data = &nmaxlen; error = sysctl_lookup(SYSCTLFN_CALL(&node)); if (error || newp == NULL) return error; return pktq_set_maxlen(pq, nmaxlen); } int sysctl_pktq_count(SYSCTLFN_ARGS, pktqueue_t *pq, u_int count_id) { uint64_t count = pktq_get_count(pq, count_id); struct sysctlnode node = *rnode; node.sysctl_data = &count; return sysctl_lookup(SYSCTLFN_CALL(&node)); } |
| 7 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 197 198 199 200 201 202 203 204 205 206 207 208 209 210 211 212 213 214 215 216 217 218 219 220 221 222 223 224 225 226 227 228 229 230 231 232 233 234 235 236 237 238 239 240 241 242 243 244 245 246 247 248 249 250 251 252 253 254 255 256 257 258 259 260 261 262 263 264 265 266 267 268 269 270 271 272 273 274 275 276 277 278 279 280 | /* $NetBSD: scsipiconf.c,v 1.45 2019/03/28 10:44:29 kardel Exp $ */ /*- * Copyright (c) 1998, 1999, 2004 The NetBSD Foundation, Inc. * All rights reserved. * * This code is derived from software contributed to The NetBSD Foundation * by Charles M. Hannum; by Jason R. Thorpe of the Numerical Aerospace * Simulation Facility, NASA Ames Research Center. * * Redistribution and use in source and binary forms, with or without * modification, are permitted provided that the following conditions * are met: * 1. Redistributions of source code must retain the above copyright * notice, this list of conditions and the following disclaimer. * 2. Redistributions in binary form must reproduce the above copyright * notice, this list of conditions and the following disclaimer in the * documentation and/or other materials provided with the distribution. * * THIS SOFTWARE IS PROVIDED BY THE NETBSD FOUNDATION, INC. AND CONTRIBUTORS * ``AS IS'' AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED * TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR * PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE FOUNDATION OR CONTRIBUTORS * BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR * CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF * SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS * INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN * CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) * ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE * POSSIBILITY OF SUCH DAMAGE. */ /* * Originally written by Julian Elischer (julian@tfs.com) * for TRW Financial Systems for use under the MACH(2.5) operating system. * * TRW Financial Systems, in accordance with their agreement with Carnegie * Mellon University, makes this software available to CMU to distribute * or use in any manner that they see fit as long as this message is kept with * the software. For this reason TFS also grants any other persons or * organisations permission to use or modify this software. * * TFS supplies this software to be publicly redistributed * on the understanding that TFS is not responsible for the correct * functioning of this software in any circumstances. * * Ported to run under 386BSD by Julian Elischer (julian@tfs.com) Sept 1992 */ #include <sys/cdefs.h> __KERNEL_RCSID(0, "$NetBSD: scsipiconf.c,v 1.45 2019/03/28 10:44:29 kardel Exp $"); #include <sys/param.h> #include <sys/systm.h> #include <sys/malloc.h> #include <sys/module.h> #include <sys/device.h> #include <sys/proc.h> #include <dev/scsipi/scsipi_all.h> #include <dev/scsipi/scsipiconf.h> #include <dev/scsipi/scsipi_base.h> /* Function pointers and stub routines for scsiverbose module */ int (*scsipi_print_sense)(struct scsipi_xfer *, int) = scsipi_print_sense_stub; void (*scsipi_print_sense_data)(struct scsi_sense_data *, int) = scsipi_print_sense_data_stub; int scsi_verbose_loaded = 0; int scsipi_print_sense_stub(struct scsipi_xfer * xs, int verbosity) { scsipi_load_verbose(); if (scsi_verbose_loaded) return scsipi_print_sense(xs, verbosity); else return 0; } void scsipi_print_sense_data_stub(struct scsi_sense_data *sense, int verbosity) { scsipi_load_verbose(); if (scsi_verbose_loaded) scsipi_print_sense_data(sense, verbosity); } int scsipi_command(struct scsipi_periph *periph, struct scsipi_generic *cmd, int cmdlen, u_char *data_addr, int datalen, int retries, int timeout, struct buf *bp, int flags) { struct scsipi_xfer *xs; int rc; /* * execute unlocked to allow waiting for memory */ xs = scsipi_make_xs_unlocked(periph, cmd, cmdlen, data_addr, datalen, retries, timeout, bp, flags); if (!xs) return (ENOMEM); mutex_enter(chan_mtx(periph->periph_channel)); rc = scsipi_execute_xs(xs); mutex_exit(chan_mtx(periph->periph_channel)); return rc; } /* * Load the scsiverbose module */ void scsipi_load_verbose(void) { if (scsi_verbose_loaded == 0) module_autoload("scsiverbose", MODULE_CLASS_MISC); } /* * allocate and init a scsipi_periph structure for a new device. */ struct scsipi_periph * scsipi_alloc_periph(int malloc_flag) { struct scsipi_periph *periph; u_int i; periph = malloc(sizeof(*periph), M_DEVBUF, malloc_flag|M_ZERO); if (periph == NULL) return NULL; periph->periph_dev = NULL; periph->periph_opcs = NULL; /* * Start with one command opening. The periph driver * will grow this if it knows it can take advantage of it. */ periph->periph_openings = 1; periph->periph_active = 0; for (i = 0; i < PERIPH_NTAGWORDS; i++) periph->periph_freetags[i] = 0xffffffff; TAILQ_INIT(&periph->periph_xferq); callout_init(&periph->periph_callout, 0); cv_init(&periph->periph_cv, "periph"); return periph; } /* * cleanup and free scsipi_periph structure */ void scsipi_free_periph(struct scsipi_periph *periph) { scsipi_free_opcodeinfo(periph); cv_destroy(&periph->periph_cv); free(periph, M_DEVBUF); } /* * Return a priority based on how much of the inquiry data matches * the patterns for the particular driver. */ const void * scsipi_inqmatch(struct scsipi_inquiry_pattern *inqbuf, const void *base, size_t nmatches, size_t matchsize, int *bestpriority) { u_int8_t type; const struct scsipi_inquiry_pattern *bestmatch; /* Include the qualifier to catch vendor-unique types. */ type = inqbuf->type; for (*bestpriority = 0, bestmatch = 0; nmatches--; base = (const char *)base + matchsize) { const struct scsipi_inquiry_pattern *match = base; int priority, len; if (type != match->type) continue; if (inqbuf->removable != match->removable) continue; priority = 2; len = strlen(match->vendor); if (memcmp(inqbuf->vendor, match->vendor, len)) continue; priority += len; len = strlen(match->product); if (memcmp(inqbuf->product, match->product, len)) continue; priority += len; len = strlen(match->revision); if (memcmp(inqbuf->revision, match->revision, len)) continue; priority += len; #ifdef SCSIPI_DEBUG printf("scsipi_inqmatch: %d/%d/%d <%s, %s, %s>\n", priority, match->type, match->removable, match->vendor, match->product, match->revision); #endif if (priority > *bestpriority) { *bestpriority = priority; bestmatch = base; } } return (bestmatch); } const char * scsipi_dtype(int type) { const char *dtype; switch (type) { case T_DIRECT: dtype = "disk"; break; case T_SEQUENTIAL: dtype = "tape"; break; case T_PRINTER: dtype = "printer"; break; case T_PROCESSOR: dtype = "processor"; break; case T_WORM: dtype = "worm"; break; case T_CDROM: dtype = "cdrom"; break; case T_SCANNER: dtype = "scanner"; break; case T_OPTICAL: dtype = "optical"; break; case T_CHANGER: dtype = "changer"; break; case T_COMM: dtype = "communication"; break; case T_IT8_1: case T_IT8_2: dtype = "graphic arts pre-press"; break; case T_STORARRAY: dtype = "storage array"; break; case T_ENCLOSURE: dtype = "enclosure services"; break; case T_SIMPLE_DIRECT: dtype = "simplified direct"; break; case T_OPTIC_CARD_RW: dtype = "optical card r/w"; break; case T_OBJECT_STORED: dtype = "object-based storage"; break; case T_NODEVICE: panic("scsipi_dtype: impossible device type"); default: dtype = "unknown"; break; } return (dtype); } |
| 1 1 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 197 198 199 200 201 202 203 204 205 206 207 208 209 210 211 212 213 214 215 216 217 218 219 220 221 222 223 224 225 226 227 228 229 230 231 232 233 234 235 236 237 238 239 240 241 242 243 244 245 246 247 248 249 250 251 252 253 254 255 256 257 258 259 260 261 262 263 264 265 266 267 268 269 270 271 272 273 274 275 276 277 278 279 280 281 282 283 284 285 286 287 288 289 290 291 292 293 294 295 296 297 298 299 300 301 302 303 304 305 306 307 308 309 310 311 312 313 314 315 316 317 318 319 320 321 322 323 324 325 326 327 328 329 330 331 332 333 334 335 336 337 338 339 340 341 342 343 344 345 346 347 348 349 350 351 352 353 354 355 356 357 358 359 360 361 362 363 364 365 366 367 368 369 370 371 372 373 374 375 376 377 378 379 380 381 382 383 384 385 386 387 388 389 390 391 392 393 394 395 396 397 398 399 400 401 402 403 404 405 406 407 408 409 410 411 412 413 414 415 416 417 418 419 420 421 422 423 424 425 426 427 428 429 430 431 432 433 434 435 436 437 438 439 440 441 442 443 | /* $NetBSD: lpt.c,v 1.82 2018/09/03 16:29:31 riastradh Exp $ */ /* * Copyright (c) 1993, 1994 Charles M. Hannum. * Copyright (c) 1990 William F. Jolitz, TeleMuse * All rights reserved. * * Redistribution and use in source and binary forms, with or without * modification, are permitted provided that the following conditions * are met: * 1. Redistributions of source code must retain the above copyright * notice, this list of conditions and the following disclaimer. * 2. Redistributions in binary form must reproduce the above copyright * notice, this list of conditions and the following disclaimer in the * documentation and/or other materials provided with the distribution. * 3. All advertising materials mentioning features or use of this software * must display the following acknowledgement: * This software is a component of "386BSD" developed by * William F. Jolitz, TeleMuse. * 4. Neither the name of the developer nor the name "386BSD" * may be used to endorse or promote products derived from this software * without specific prior written permission. * * THIS SOFTWARE IS A COMPONENT OF 386BSD DEVELOPED BY WILLIAM F. JOLITZ * AND IS INTENDED FOR RESEARCH AND EDUCATIONAL PURPOSES ONLY. THIS * SOFTWARE SHOULD NOT BE CONSIDERED TO BE A COMMERCIAL PRODUCT. * THE DEVELOPER URGES THAT USERS WHO REQUIRE A COMMERCIAL PRODUCT * NOT MAKE USE OF THIS WORK. * * FOR USERS WHO WISH TO UNDERSTAND THE 386BSD SYSTEM DEVELOPED * BY WILLIAM F. JOLITZ, WE RECOMMEND THE USER STUDY WRITTEN * REFERENCES SUCH AS THE "PORTING UNIX TO THE 386" SERIES * (BEGINNING JANUARY 1991 "DR. DOBBS JOURNAL", USA AND BEGINNING * JUNE 1991 "UNIX MAGAZIN", GERMANY) BY WILLIAM F. JOLITZ AND * LYNNE GREER JOLITZ, AS WELL AS OTHER BOOKS ON UNIX AND THE * ON-LINE 386BSD USER MANUAL BEFORE USE. A BOOK DISCUSSING THE INTERNALS * OF 386BSD ENTITLED "386BSD FROM THE INSIDE OUT" WILL BE AVAILABLE LATE 1992. * * THIS SOFTWARE IS PROVIDED BY THE DEVELOPER ``AS IS'' AND * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE * ARE DISCLAIMED. IN NO EVENT SHALL THE DEVELOPER BE LIABLE * FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL * DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS * OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) * HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT * LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY * OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF * SUCH DAMAGE. */ /* * Device Driver for AT style parallel printer port */ #include <sys/cdefs.h> __KERNEL_RCSID(0, "$NetBSD: lpt.c,v 1.82 2018/09/03 16:29:31 riastradh Exp $"); #include <sys/param.h> #include <sys/systm.h> #include <sys/proc.h> #include <sys/malloc.h> #include <sys/kernel.h> #include <sys/ioctl.h> #include <sys/uio.h> #include <sys/device.h> #include <sys/conf.h> #include <sys/syslog.h> #include <sys/intr.h> #include <sys/bus.h> #include <dev/ic/lptreg.h> #include <dev/ic/lptvar.h> #include "ioconf.h" #define TIMEOUT hz*16 /* wait up to 16 seconds for a ready */ #define STEP hz/4 #define LPTPRI (PZERO+8) #define LPT_BSIZE 1024 #define LPTDEBUG #ifndef LPTDEBUG #define LPRINTF(a) #else #define LPRINTF(a) if (lptdebug) printf a int lptdebug = 0; #endif dev_type_open(lptopen); dev_type_close(lptclose); dev_type_write(lptwrite); dev_type_ioctl(lptioctl); const struct cdevsw lpt_cdevsw = { .d_open = lptopen, .d_close = lptclose, .d_read = noread, .d_write = lptwrite, .d_ioctl = lptioctl, .d_stop = nostop, .d_tty = notty, .d_poll = nopoll, .d_mmap = nommap, .d_kqfilter = nokqfilter, .d_discard = nodiscard, .d_flag = D_OTHER }; #define LPTUNIT(s) (minor(s) & 0x1f) #define LPTFLAGS(s) (minor(s) & 0xe0) static void lptsoftintr(void *); void lpt_attach_subr(struct lpt_softc *sc) { bus_space_tag_t iot; bus_space_handle_t ioh; sc->sc_state = 0; iot = sc->sc_iot; ioh = sc->sc_ioh; bus_space_write_1(iot, ioh, lpt_control, LPC_NINIT); callout_init(&sc->sc_wakeup_ch, 0); sc->sc_sih = softint_establish(SOFTINT_SERIAL, lptsoftintr, sc); sc->sc_dev_ok = 1; } int lpt_detach_subr(device_t self, int flags) { struct lpt_softc *sc = device_private(self); sc->sc_dev_ok = 0; softint_disestablish(sc->sc_sih); callout_destroy(&sc->sc_wakeup_ch); return 0; } /* * Reset the printer, then wait until it's selected and not busy. */ int lptopen(dev_t dev, int flag, int mode, struct lwp *l) { u_char flags = LPTFLAGS(dev); struct lpt_softc *sc; bus_space_tag_t iot; bus_space_handle_t ioh; u_char control; int error; int spin; sc = device_lookup_private(&lpt_cd, LPTUNIT(dev)); if (!sc || !sc->sc_dev_ok) return ENXIO; #if 0 /* XXX what to do? */ if (sc->sc_irq == IRQUNK && (flags & LPT_NOINTR) == 0) return ENXIO; #endif #ifdef DIAGNOSTIC if (sc->sc_state) aprint_verbose_dev(sc->sc_dev, "stat=0x%x not zero\n", sc->sc_state); #endif if (sc->sc_state) return EBUSY; sc->sc_state = LPT_INIT; sc->sc_flags = flags; LPRINTF(("%s: open: flags=0x%x\n", device_xname(sc->sc_dev), (unsigned)flags)); iot = sc->sc_iot; ioh = sc->sc_ioh; if ((flags & LPT_NOPRIME) == 0) { /* assert INIT for 100 usec to start up printer */ bus_space_write_1(iot, ioh, lpt_control, LPC_SELECT); delay(100); } control = LPC_SELECT | LPC_NINIT; bus_space_write_1(iot, ioh, lpt_control, control); /* wait till ready (printer running diagnostics) */ for (spin = 0; NOT_READY_ERR(); spin += STEP) { if (spin >= TIMEOUT) { sc->sc_state = 0; return EBUSY; } /* wait 1/4 second, give up if we get a signal */ error = tsleep((void *)sc, LPTPRI | PCATCH, "lptopen", STEP); if (error != EWOULDBLOCK) { sc->sc_state = 0; return error; } } if ((flags & LPT_NOINTR) == 0) control |= LPC_IENABLE; if (flags & LPT_AUTOLF) control |= LPC_AUTOLF; sc->sc_control = control; bus_space_write_1(iot, ioh, lpt_control, control); sc->sc_inbuf = malloc(LPT_BSIZE, M_DEVBUF, M_WAITOK); sc->sc_count = 0; sc->sc_state = LPT_OPEN; if ((sc->sc_flags & LPT_NOINTR) == 0) lptwakeup(sc); LPRINTF(("%s: opened\n", device_xname(sc->sc_dev))); return 0; } int lptnotready(u_char status, struct lpt_softc *sc) { u_char new; status = (status ^ LPS_INVERT) & LPS_MASK; new = status & ~sc->sc_laststatus; sc->sc_laststatus = status; if (sc->sc_state & LPT_OPEN) { if (new & LPS_SELECT) log(LOG_NOTICE, "%s: offline\n", device_xname(sc->sc_dev)); else if (new & LPS_NOPAPER) log(LOG_NOTICE, "%s: out of paper\n", device_xname(sc->sc_dev)); else if (new & LPS_NERR) log(LOG_NOTICE, "%s: output error\n", device_xname(sc->sc_dev)); } return status; } void lptwakeup(void *arg) { struct lpt_softc *sc = arg; int s; s = splvm(); lptintr(sc); splx(s); callout_reset(&sc->sc_wakeup_ch, STEP, lptwakeup, sc); } /* * Close the device, and free the local line buffer. */ int lptclose(dev_t dev, int flag, int mode, struct lwp *l) { struct lpt_softc *sc = device_lookup_private(&lpt_cd, LPTUNIT(dev)); bus_space_tag_t iot = sc->sc_iot; bus_space_handle_t ioh = sc->sc_ioh; if (sc->sc_count) (void) lptpushbytes(sc); if ((sc->sc_flags & LPT_NOINTR) == 0) callout_stop(&sc->sc_wakeup_ch); bus_space_write_1(iot, ioh, lpt_control, LPC_NINIT); sc->sc_state = 0; bus_space_write_1(iot, ioh, lpt_control, LPC_NINIT); free(sc->sc_inbuf, M_DEVBUF); LPRINTF(("%s: closed\n", device_xname(sc->sc_dev))); return 0; } int lptpushbytes(struct lpt_softc *sc) { bus_space_tag_t iot = sc->sc_iot; bus_space_handle_t ioh = sc->sc_ioh; int error; if (sc->sc_flags & LPT_NOINTR) { int spin, tic; u_char control = sc->sc_control; while (sc->sc_count > 0) { spin = 0; while (NOT_READY()) { if (++spin < sc->sc_spinmax) continue; tic = 0; /* adapt busy-wait algorithm */ sc->sc_spinmax++; while (NOT_READY_ERR()) { /* exponential backoff */ tic = tic + tic + 1; if (tic > TIMEOUT) tic = TIMEOUT; error = tsleep((void *)sc, LPTPRI | PCATCH, "lptpsh", tic); if (error != EWOULDBLOCK) return error; } break; } bus_space_write_1(iot, ioh, lpt_data, *sc->sc_cp++); DELAY(1); bus_space_write_1(iot, ioh, lpt_control, control | LPC_STROBE); DELAY(1); sc->sc_count--; bus_space_write_1(iot, ioh, lpt_control, control); DELAY(1); /* adapt busy-wait algorithm */ if (spin*2 + 16 < sc->sc_spinmax) sc->sc_spinmax--; } } else { int s; while (sc->sc_count > 0) { /* if the printer is ready for a char, give it one */ if ((sc->sc_state & LPT_OBUSY) == 0) { LPRINTF(("%s: write %lu\n", device_xname(sc->sc_dev), (u_long)sc->sc_count)); s = splvm(); (void) lptintr(sc); splx(s); } error = tsleep((void *)sc, LPTPRI | PCATCH, "lptwrite2", 0); if (error) return error; } } return 0; } /* * Copy a line from user space to a local buffer, then call putc to get the * chars moved to the output queue. */ int lptwrite(dev_t dev, struct uio *uio, int flags) { struct lpt_softc *sc = device_lookup_private(&lpt_cd, LPTUNIT(dev)); size_t n; int error = 0; while ((n = uimin(LPT_BSIZE, uio->uio_resid)) != 0) { uiomove(sc->sc_cp = sc->sc_inbuf, n, uio); sc->sc_count = n; error = lptpushbytes(sc); if (error) { /* * Return accurate residual if interrupted or timed * out. */ uio->uio_resid += sc->sc_count; sc->sc_count = 0; return error; } } return 0; } /* * Handle printer interrupts which occur when the printer is ready to accept * another char. */ int lptintr(void *arg) { struct lpt_softc *sc = arg; bus_space_tag_t iot = sc->sc_iot; bus_space_handle_t ioh = sc->sc_ioh; #if 0 if ((sc->sc_state & LPT_OPEN) == 0) return 0; #endif /* is printer online and ready for output */ if (NOT_READY() && NOT_READY_ERR()) return 0; if (sc->sc_count) { u_char control = sc->sc_control; /* send char */ bus_space_write_1(iot, ioh, lpt_data, *sc->sc_cp++); DELAY(1); bus_space_write_1(iot, ioh, lpt_control, control | LPC_STROBE); DELAY(1); sc->sc_count--; bus_space_write_1(iot, ioh, lpt_control, control); DELAY(1); sc->sc_state |= LPT_OBUSY; } else sc->sc_state &= ~LPT_OBUSY; if (sc->sc_count == 0) { /* none, wake up the top half to get more */ softint_schedule(sc->sc_sih); } return 1; } static void lptsoftintr(void *cookie) { wakeup(cookie); } int lptioctl(dev_t dev, u_long cmd, void *data, int flag, struct lwp *l) { return ENODEV; } |
| 2 2 2 2 2 2 1 1 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 197 198 199 200 201 202 203 204 205 206 207 208 209 210 211 212 213 214 215 216 217 218 219 220 221 222 223 224 225 226 227 228 229 230 231 232 233 234 235 236 237 238 239 240 241 242 243 244 245 246 247 248 249 250 251 252 253 254 255 256 257 258 259 260 261 262 263 264 265 266 267 268 269 270 271 272 273 274 275 276 277 278 279 280 281 282 283 284 285 286 287 288 289 290 291 292 293 294 295 296 297 298 299 300 301 302 303 304 305 306 307 308 309 310 311 312 313 314 315 316 317 318 319 320 321 322 323 324 325 326 327 328 329 330 331 332 333 334 335 336 337 338 339 340 341 342 343 344 345 346 347 348 349 350 351 352 353 354 355 356 357 358 359 360 361 362 363 364 365 366 367 368 369 370 371 372 373 374 375 376 377 378 379 380 381 382 383 384 385 386 387 388 389 390 391 392 393 394 395 396 397 398 | /* $NetBSD: irframe.c,v 1.46 2014/07/25 08:10:37 dholland Exp $ */ /* * Copyright (c) 2001 The NetBSD Foundation, Inc. * All rights reserved. * * This code is derived from software contributed to The NetBSD Foundation * by Lennart Augustsson (lennart@augustsson.net). * * Redistribution and use in source and binary forms, with or without * modification, are permitted provided that the following conditions * are met: * 1. Redistributions of source code must retain the above copyright * notice, this list of conditions and the following disclaimer. * 2. Redistributions in binary form must reproduce the above copyright * notice, this list of conditions and the following disclaimer in the * documentation and/or other materials provided with the distribution. * * THIS SOFTWARE IS PROVIDED BY THE NETBSD FOUNDATION, INC. AND CONTRIBUTORS * ``AS IS'' AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED * TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR * PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE FOUNDATION OR CONTRIBUTORS * BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR * CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF * SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS * INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN * CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) * ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE * POSSIBILITY OF SUCH DAMAGE. */ #include <sys/cdefs.h> __KERNEL_RCSID(0, "$NetBSD: irframe.c,v 1.46 2014/07/25 08:10:37 dholland Exp $"); #include "irframe.h" #include <sys/param.h> #include <sys/systm.h> #include <sys/ioctl.h> #include <sys/kernel.h> #include <sys/device.h> #include <sys/conf.h> #include <sys/malloc.h> #include <sys/poll.h> #include <sys/select.h> #include <sys/vnode.h> #include <dev/ir/ir.h> #include <dev/ir/irdaio.h> #include <dev/ir/irframevar.h> #ifdef IRFRAME_DEBUG #define DPRINTF(x) if (irframedebug) printf x #define Static int irframedebug = 0; #else #define DPRINTF(x) #define Static static #endif dev_type_open(irframeopen); dev_type_close(irframeclose); dev_type_read(irframeread); dev_type_write(irframewrite); dev_type_ioctl(irframeioctl); dev_type_poll(irframepoll); dev_type_kqfilter(irframekqfilter); const struct cdevsw irframe_cdevsw = { .d_open = irframeopen, .d_close = irframeclose, .d_read = irframeread, .d_write = irframewrite, .d_ioctl = irframeioctl, .d_stop = nostop, .d_tty = notty, .d_poll = irframepoll, .d_mmap = nommap, .d_kqfilter = irframekqfilter, .d_discard = nodiscard, .d_flag = D_OTHER }; int irframe_match(device_t parent, cfdata_t match, void *aux); Static int irf_set_params(struct irframe_softc *sc, struct irda_params *p); Static int irf_reset_params(struct irframe_softc *sc); #if NIRFRAME == 0 /* In case we just have tty attachment. */ CFDRIVER_DECL(irframe, DV_DULL, NULL); #endif CFATTACH_DECL_NEW(irframe, sizeof(struct irframe_softc), irframe_match, irframe_attach, irframe_detach, NULL); extern struct cfdriver irframe_cd; #define IRFRAMEUNIT(dev) (minor(dev)) int irframe_match(device_t parent, cfdata_t match, void *aux) { struct ir_attach_args *ia = aux; return (ia->ia_type == IR_TYPE_IRFRAME); } void irframe_attach(device_t parent, device_t self, void *aux) { struct irframe_softc *sc = device_private(self); struct ir_attach_args *ia = aux; const char *delim; int speeds = 0; sc->sc_dev = self; sc->sc_methods = ia->ia_methods; sc->sc_handle = ia->ia_handle; #ifdef DIAGNOSTIC if (sc->sc_methods->im_read == NULL || sc->sc_methods->im_write == NULL || sc->sc_methods->im_poll == NULL || sc->sc_methods->im_kqfilter == NULL || sc->sc_methods->im_set_params == NULL || sc->sc_methods->im_get_speeds == NULL || sc->sc_methods->im_get_turnarounds == NULL) panic("%s: missing methods", device_xname(self)); #endif (void)sc->sc_methods->im_get_speeds(sc->sc_handle, &speeds); sc->sc_speedmask = speeds; delim = ":"; if (speeds & IRDA_SPEEDS_SIR) { printf("%s SIR", delim); delim = ","; } if (speeds & IRDA_SPEEDS_MIR) { printf("%s MIR", delim); delim = ","; } if (speeds & IRDA_SPEEDS_FIR) { printf("%s FIR", delim); delim = ","; } if (speeds & IRDA_SPEEDS_VFIR) { printf("%s VFIR", delim); delim = ","; } printf("\n"); if (!pmf_device_register(self, NULL, NULL)) aprint_error_dev(self, "couldn't establish power handler\n"); } int irframe_detach(device_t self, int flags) { /*struct irframe_softc *sc = device_private(self);*/ int maj, mn; pmf_device_deregister(self); /* XXX needs reference count */ /* locate the major number */ maj = cdevsw_lookup_major(&irframe_cdevsw); /* Nuke the vnodes for any open instances (calls close). */ mn = device_unit(self); vdevgone(maj, mn, mn, VCHR); return (0); } int irframeopen(dev_t dev, int flag, int mode, struct lwp *l) { struct irframe_softc *sc; int error; sc = device_lookup_private(&irframe_cd, IRFRAMEUNIT(dev)); if (sc == NULL) return (ENXIO); if (!device_is_active(sc->sc_dev)) return (EIO); if (sc->sc_open) return (EBUSY); if (sc->sc_methods->im_open != NULL) { error = sc->sc_methods->im_open(sc->sc_handle, flag, mode, l); if (error) return (error); } sc->sc_open = 1; #ifdef DIAGNOSTIC sc->sc_speed = IRDA_DEFAULT_SPEED; #endif (void)irf_reset_params(sc); return (0); } int irframeclose(dev_t dev, int flag, int mode, struct lwp *l) { struct irframe_softc *sc; int error; sc = device_lookup_private(&irframe_cd, IRFRAMEUNIT(dev)); if (sc == NULL) return (ENXIO); sc->sc_open = 0; if (sc->sc_methods->im_close != NULL) error = sc->sc_methods->im_close(sc->sc_handle, flag, mode, l); else error = 0; return (error); } int irframeread(dev_t dev, struct uio *uio, int flag) { struct irframe_softc *sc; sc = device_lookup_private(&irframe_cd, IRFRAMEUNIT(dev)); if (sc == NULL) return (ENXIO); if (!device_is_active(sc->sc_dev) || !sc->sc_open) return (EIO); if (uio->uio_resid < sc->sc_params.maxsize) { #ifdef DIAGNOSTIC printf("irframeread: short read %ld < %d\n", (long)uio->uio_resid, sc->sc_params.maxsize); #endif return (EINVAL); } return (sc->sc_methods->im_read(sc->sc_handle, uio, flag)); } int irframewrite(dev_t dev, struct uio *uio, int flag) { struct irframe_softc *sc; sc = device_lookup_private(&irframe_cd, IRFRAMEUNIT(dev)); if (sc == NULL) return (ENXIO); if (!device_is_active(sc->sc_dev) || !sc->sc_open) return (EIO); if (uio->uio_resid > sc->sc_params.maxsize) { #ifdef DIAGNOSTIC printf("irframeread: long write %ld > %d\n", (long)uio->uio_resid, sc->sc_params.maxsize); #endif return (EINVAL); } return (sc->sc_methods->im_write(sc->sc_handle, uio, flag)); } int irf_set_params(struct irframe_softc *sc, struct irda_params *p) { int error; DPRINTF(("irf_set_params: set params speed=%u ebofs=%u maxsize=%u " "speedmask=0x%x\n", p->speed, p->ebofs, p->maxsize, sc->sc_speedmask)); if (p->maxsize > IRDA_MAX_FRAME_SIZE) { #ifdef IRFRAME_DEBUG printf("irf_set_params: bad maxsize=%u\n", p->maxsize); #endif return (EINVAL); } if (p->ebofs > IRDA_MAX_EBOFS) { #ifdef IRFRAME_DEBUG printf("irf_set_params: bad maxsize=%u\n", p->maxsize); #endif return (EINVAL); } #define CONC(x,y) x##y #define CASE(s) case s: if (!(sc->sc_speedmask & CONC(IRDA_SPEED_,s))) return (EINVAL); break switch (p->speed) { CASE(2400); CASE(9600); CASE(19200); CASE(38400); CASE(57600); CASE(115200); CASE(576000); CASE(1152000); CASE(4000000); CASE(16000000); default: return (EINVAL); } #undef CONC #undef CASE error = sc->sc_methods->im_set_params(sc->sc_handle, p); if (!error) { sc->sc_params = *p; DPRINTF(("irf_set_params: ok\n")); #ifdef DIAGNOSTIC if (p->speed != sc->sc_speed) { sc->sc_speed = p->speed; aprint_verbose_dev(sc->sc_dev, "set speed %u\n", sc->sc_speed); } #endif } else { #ifdef IRFRAME_DEBUG printf("irf_set_params: error=%d\n", error); #endif } return (error); } int irf_reset_params(struct irframe_softc *sc) { struct irda_params params; params.speed = IRDA_DEFAULT_SPEED; params.ebofs = IRDA_DEFAULT_EBOFS; params.maxsize = IRDA_DEFAULT_SIZE; return (irf_set_params(sc, ¶ms)); } int irframeioctl(dev_t dev, u_long cmd, void *addr, int flag, struct lwp *l) { struct irframe_softc *sc; void *vaddr = addr; int error; sc = device_lookup_private(&irframe_cd, IRFRAMEUNIT(dev)); if (sc == NULL) return (ENXIO); if (!device_is_active(sc->sc_dev) || !sc->sc_open) return (EIO); switch (cmd) { case FIONBIO: /* All handled in the upper FS layer. */ error = 0; break; case IRDA_SET_PARAMS: error = irf_set_params(sc, vaddr); break; case IRDA_RESET_PARAMS: error = irf_reset_params(sc); break; case IRDA_GET_SPEEDMASK: error = sc->sc_methods->im_get_speeds(sc->sc_handle, vaddr); break; case IRDA_GET_TURNAROUNDMASK: error = sc->sc_methods->im_get_turnarounds(sc->sc_handle,vaddr); break; default: error = EINVAL; break; } return (error); } int irframepoll(dev_t dev, int events, struct lwp *l) { struct irframe_softc *sc; sc = device_lookup_private(&irframe_cd, IRFRAMEUNIT(dev)); if (sc == NULL) return (POLLHUP); if (!device_is_active(sc->sc_dev) || !sc->sc_open) return (POLLHUP); return (sc->sc_methods->im_poll(sc->sc_handle, events, l)); } int irframekqfilter(dev_t dev, struct knote *kn) { struct irframe_softc *sc; sc = device_lookup_private(&irframe_cd, IRFRAMEUNIT(dev)); if (!device_is_active(sc->sc_dev) || !sc->sc_open) return (1); return (sc->sc_methods->im_kqfilter(sc->sc_handle, kn)); } |
| 8 8 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 | /* $NetBSD: uvm_user.c,v 1.14 2011/02/02 15:13:34 chuck Exp $ */ /* * Copyright (c) 1997 Charles D. Cranor and Washington University. * All rights reserved. * * Redistribution and use in source and binary forms, with or without * modification, are permitted provided that the following conditions * are met: * 1. Redistributions of source code must retain the above copyright * notice, this list of conditions and the following disclaimer. * 2. Redistributions in binary form must reproduce the above copyright * notice, this list of conditions and the following disclaimer in the * documentation and/or other materials provided with the distribution. * * THIS SOFTWARE IS PROVIDED BY THE AUTHOR ``AS IS'' AND ANY EXPRESS OR * IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES * OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. * IN NO EVENT SHALL THE AUTHOR BE LIABLE FOR ANY DIRECT, INDIRECT, * INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT * NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, * DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY * THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT * (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF * THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. * * from: Id: uvm_user.c,v 1.1.2.1 1997/08/14 19:10:41 chuck Exp */ /* * uvm_user.c: high level uvm_allocate/uvm_deallocate interface into vm. */ #include <sys/cdefs.h> __KERNEL_RCSID(0, "$NetBSD: uvm_user.c,v 1.14 2011/02/02 15:13:34 chuck Exp $"); #include <sys/param.h> #include <sys/systm.h> #include <sys/proc.h> #include <uvm/uvm.h> /* * uvm_deallocate: deallocate memory (unmap) */ void uvm_deallocate(struct vm_map *map, vaddr_t start, vsize_t size) { if (size == 0) return; uvm_unmap(map, trunc_page(start), round_page(start + size)); } |
| 6 26 1 25 4 4 3 3 5 2 3 3 2 2 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 197 198 199 200 201 202 203 204 205 206 207 208 209 210 211 212 213 214 215 216 217 218 219 220 221 222 223 224 225 226 227 228 229 230 231 232 233 234 235 236 237 238 239 240 241 242 243 244 245 246 247 248 249 250 251 252 253 254 255 256 257 258 259 260 261 262 263 264 265 266 267 268 269 270 271 272 273 274 275 276 277 278 279 280 281 282 283 284 285 286 287 288 289 290 291 292 293 294 295 296 297 298 299 300 301 302 303 304 305 | /* $NetBSD: random.c,v 1.10 2021/12/28 13:22:43 riastradh Exp $ */ /*- * Copyright (c) 2019 The NetBSD Foundation, Inc. * All rights reserved. * * This code is derived from software contributed to The NetBSD Foundation * by Taylor R. Campbell. * * Redistribution and use in source and binary forms, with or without * modification, are permitted provided that the following conditions * are met: * 1. Redistributions of source code must retain the above copyright * notice, this list of conditions and the following disclaimer. * 2. Redistributions in binary form must reproduce the above copyright * notice, this list of conditions and the following disclaimer in the * documentation and/or other materials provided with the distribution. * * THIS SOFTWARE IS PROVIDED BY THE NETBSD FOUNDATION, INC. AND CONTRIBUTORS * ``AS IS'' AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED * TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR * PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE FOUNDATION OR CONTRIBUTORS * BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR * CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF * SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS * INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN * CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) * ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE * POSSIBILITY OF SUCH DAMAGE. */ /* * /dev/random, /dev/urandom -- stateless version * * For short reads from /dev/urandom, up to 256 bytes, read from a * per-CPU NIST Hash_DRBG instance that is reseeded as soon as the * system has enough entropy. * * For all other reads, instantiate a fresh NIST Hash_DRBG from * the global entropy pool, and draw from it. * * Each read is independent; there is no per-open state. * Concurrent reads from the same open run in parallel. * * Reading from /dev/random may block until entropy is available. * Either device may return short reads if interrupted. */ #include <sys/cdefs.h> __KERNEL_RCSID(0, "$NetBSD: random.c,v 1.10 2021/12/28 13:22:43 riastradh Exp $"); #include <sys/param.h> #include <sys/types.h> #include <sys/atomic.h> #include <sys/conf.h> #include <sys/cprng.h> #include <sys/entropy.h> #include <sys/errno.h> #include <sys/event.h> #include <sys/fcntl.h> #include <sys/kauth.h> #include <sys/kmem.h> #include <sys/lwp.h> #include <sys/poll.h> #include <sys/random.h> #include <sys/rnd.h> #include <sys/rndsource.h> #include <sys/signalvar.h> #include <sys/systm.h> #include <sys/vnode.h> /* IO_NDELAY */ #include "ioconf.h" static dev_type_open(random_open); static dev_type_close(random_close); static dev_type_ioctl(random_ioctl); static dev_type_poll(random_poll); static dev_type_kqfilter(random_kqfilter); static dev_type_read(random_read); static dev_type_write(random_write); const struct cdevsw rnd_cdevsw = { .d_open = random_open, .d_close = random_close, .d_read = random_read, .d_write = random_write, .d_ioctl = random_ioctl, .d_stop = nostop, .d_tty = notty, .d_poll = random_poll, .d_mmap = nommap, .d_kqfilter = random_kqfilter, .d_discard = nodiscard, .d_flag = D_OTHER|D_MPSAFE, }; #define RANDOM_BUFSIZE 512 /* XXX pulled from arse */ /* Entropy source for writes to /dev/random and /dev/urandom */ static krndsource_t user_rndsource; void rndattach(int num) { rnd_attach_source(&user_rndsource, "/dev/random", RND_TYPE_UNKNOWN, RND_FLAG_COLLECT_VALUE); } static int random_open(dev_t dev, int flags, int fmt, struct lwp *l) { /* Validate minor. */ switch (minor(dev)) { case RND_DEV_RANDOM: case RND_DEV_URANDOM: break; default: return ENXIO; } return 0; } static int random_close(dev_t dev, int flags, int fmt, struct lwp *l) { /* Success! */ return 0; } static int random_ioctl(dev_t dev, unsigned long cmd, void *data, int flag, struct lwp *l) { /* * No non-blocking/async options; otherwise defer to * entropy_ioctl. */ switch (cmd) { case FIONBIO: case FIOASYNC: return 0; default: return entropy_ioctl(cmd, data); } } static int random_poll(dev_t dev, int events, struct lwp *l) { /* /dev/random may block; /dev/urandom is always ready. */ switch (minor(dev)) { case RND_DEV_RANDOM: return entropy_poll(events); case RND_DEV_URANDOM: return events & (POLLIN|POLLRDNORM | POLLOUT|POLLWRNORM); default: return 0; } } static int random_kqfilter(dev_t dev, struct knote *kn) { /* Validate the event filter. */ switch (kn->kn_filter) { case EVFILT_READ: case EVFILT_WRITE: break; default: return EINVAL; } /* /dev/random may block; /dev/urandom never does. */ switch (minor(dev)) { case RND_DEV_RANDOM: if (kn->kn_filter == EVFILT_READ) return entropy_kqfilter(kn); /* FALLTHROUGH */ case RND_DEV_URANDOM: kn->kn_fop = &seltrue_filtops; return 0; default: return ENXIO; } } /* * random_read(dev, uio, flags) * * Generate data from a PRNG seeded from the entropy pool. * * - If /dev/random, block until we have full entropy, or fail * with EWOULDBLOCK, and if `depleting' entropy, return at most * the entropy pool's capacity at once. * * - If /dev/urandom, generate data from whatever is in the * entropy pool now. * * On interrupt, return a short read, but not shorter than 256 * bytes (actually, no shorter than RANDOM_BUFSIZE bytes, which is * 512 for hysterical raisins). */ static int random_read(dev_t dev, struct uio *uio, int flags) { int gflags; /* Set the appropriate GRND_* mode. */ switch (minor(dev)) { case RND_DEV_RANDOM: gflags = GRND_RANDOM; break; case RND_DEV_URANDOM: gflags = GRND_INSECURE; break; default: return ENXIO; } /* * Set GRND_NONBLOCK if the user requested IO_NDELAY (i.e., the * file was opened with O_NONBLOCK). */ if (flags & IO_NDELAY) gflags |= GRND_NONBLOCK; /* Defer to getrandom. */ return dogetrandom(uio, gflags); } /* * random_write(dev, uio, flags) * * Enter data from uio into the entropy pool. * * Assume privileged users provide full entropy, and unprivileged * users provide no entropy. If you have a nonuniform source of * data with n bytes of min-entropy, hash it with an XOF like * SHAKE128 into exactly n bytes first. */ static int random_write(dev_t dev, struct uio *uio, int flags) { kauth_cred_t cred = kauth_cred_get(); uint8_t *buf; bool privileged = false, any = false; int error = 0; /* Verify user's authorization to affect the entropy pool. */ error = kauth_authorize_device(cred, KAUTH_DEVICE_RND_ADDDATA, NULL, NULL, NULL, NULL); if (error) return error; /* * Check whether user is privileged. If so, assume user * furnishes full-entropy data; if not, accept user's data but * assume it has zero entropy when we do accounting. If you * want to specify less entropy, use ioctl(RNDADDDATA). */ if (kauth_authorize_device(cred, KAUTH_DEVICE_RND_ADDDATA_ESTIMATE, NULL, NULL, NULL, NULL) == 0) privileged = true; /* Get a buffer for transfers. */ buf = kmem_alloc(RANDOM_BUFSIZE, KM_SLEEP); /* Consume data. */ while (uio->uio_resid) { size_t n = MIN(uio->uio_resid, RANDOM_BUFSIZE); /* Transfer n bytes in and enter them into the pool. */ error = uiomove(buf, n, uio); if (error) break; rnd_add_data(&user_rndsource, buf, n, privileged ? n*NBBY : 0); any = true; /* Now's a good time to yield if needed. */ preempt_point(); /* Check for interruption. */ if (__predict_false(curlwp->l_flag & LW_PENDSIG) && sigispending(curlwp, 0)) { error = EINTR; break; } } /* Zero the buffer and free it. */ explicit_memset(buf, 0, RANDOM_BUFSIZE); kmem_free(buf, RANDOM_BUFSIZE); /* If we added anything, consolidate entropy now. */ if (any) entropy_consolidate(); return error; } |
| 4 1 1 2 1 2 2 1 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 197 198 199 200 201 202 203 204 205 206 207 208 209 210 211 212 213 214 215 216 217 218 219 220 221 222 223 224 225 226 227 228 229 230 231 232 233 234 235 236 237 238 239 240 241 242 243 244 245 246 247 248 249 250 251 252 253 254 255 256 257 258 259 260 261 262 263 264 265 266 267 268 269 270 271 272 273 274 275 276 277 278 279 280 281 282 283 284 285 286 287 288 289 290 291 292 293 294 295 296 297 298 299 300 301 302 303 304 305 306 307 308 309 310 311 312 313 314 315 316 317 318 319 320 321 322 323 324 325 326 327 328 329 330 331 332 333 334 335 336 337 338 339 340 341 342 343 344 345 346 347 348 349 350 351 352 353 354 355 356 357 358 359 360 361 362 363 364 365 366 367 368 369 370 371 372 373 374 375 376 377 378 379 380 381 382 383 384 385 386 387 388 389 390 391 392 393 394 395 396 397 398 399 400 401 402 403 404 405 406 407 408 409 410 411 412 413 414 415 416 417 418 419 420 421 422 423 424 425 426 427 428 429 430 431 432 433 434 435 436 437 438 439 440 441 442 443 444 445 446 447 448 449 450 451 452 453 454 455 456 457 458 459 460 461 462 463 464 465 466 467 468 469 470 471 472 473 474 475 476 477 478 479 480 481 482 483 484 485 486 487 488 489 490 491 492 493 494 495 496 497 498 499 500 501 502 503 504 505 506 507 508 509 510 511 512 513 514 515 516 517 518 519 520 521 522 523 524 525 526 527 528 529 530 531 532 533 534 535 536 537 538 539 540 541 542 543 544 545 546 547 548 549 550 551 552 553 554 555 556 557 558 559 560 561 562 563 564 565 566 567 568 569 570 571 572 573 574 575 576 577 578 579 580 581 582 583 584 585 586 587 588 589 590 591 592 593 594 595 596 597 598 599 600 601 602 603 604 605 606 607 608 609 610 611 612 613 614 615 616 617 618 619 620 621 622 623 624 625 626 627 628 629 630 631 632 633 634 635 636 637 638 639 640 641 642 643 644 645 646 647 648 649 650 651 652 653 654 655 656 657 658 659 660 661 662 663 664 665 666 667 668 669 670 671 672 673 674 675 676 677 678 679 680 681 682 683 684 685 686 687 688 689 690 691 692 693 694 695 696 697 698 699 700 701 702 703 704 705 706 707 708 709 710 711 712 713 714 715 716 717 718 719 720 721 722 723 724 725 726 727 728 729 730 731 732 733 734 735 736 737 738 739 740 741 742 743 744 745 746 747 748 749 750 751 752 753 754 755 756 757 758 759 760 761 762 763 764 765 766 767 768 769 770 771 772 773 774 775 776 777 778 779 780 781 782 783 784 785 786 787 788 789 790 791 792 793 794 795 796 797 798 799 800 801 802 803 804 805 806 807 808 809 810 811 812 813 814 815 816 817 818 819 820 821 822 823 824 825 826 827 828 829 830 831 832 833 834 835 836 837 838 839 840 841 842 843 844 845 846 847 848 849 850 851 852 853 854 855 856 857 858 859 860 861 862 863 864 865 866 867 868 869 870 871 872 873 874 875 876 877 878 879 880 881 882 883 884 885 886 887 888 889 890 891 892 893 894 895 896 897 898 899 900 901 902 903 904 905 906 907 908 909 910 911 912 913 914 915 916 917 918 919 920 921 922 923 924 925 926 927 928 929 930 931 932 933 934 935 936 937 938 939 940 941 942 943 944 945 946 947 948 949 950 951 952 953 954 955 956 957 958 959 960 961 962 963 964 965 966 967 968 969 970 971 972 973 974 975 976 977 978 979 980 981 982 983 984 985 986 987 988 989 990 991 992 993 994 995 996 997 998 999 1000 1001 1002 1003 1004 1005 1006 1007 1008 1009 1010 1011 1012 1013 1014 1015 1016 1017 1018 1019 1020 1021 1022 1023 1024 1025 1026 1027 1028 1029 1030 1031 1032 1033 1034 1035 1036 1037 1038 1039 1040 1041 1042 1043 1044 1045 1046 1047 1048 1049 1050 1051 1052 1053 1054 1055 1056 1057 1058 1059 1060 1061 1062 1063 1064 1065 1066 1067 1068 1069 1070 1071 1072 1073 1074 1075 1076 1077 1078 1079 1080 1081 1082 1083 1084 1085 1086 1087 1088 1089 1090 1091 1092 1093 1094 1095 1096 1097 1098 1099 1100 1101 1102 1103 1104 1105 1106 1107 1108 1109 1110 1111 1112 1113 1114 1115 1116 1117 1118 1119 1120 1121 1122 1123 1124 1125 1126 1127 1128 1129 1130 1131 1132 1133 1134 1135 1136 1137 1138 1139 1140 1141 1142 1143 1144 1145 1146 1147 1148 1149 1150 1151 1152 1153 1154 1155 1156 1157 1158 1159 1160 1161 1162 1163 1164 1165 1166 1167 1168 1169 1170 1171 1172 1173 1174 1175 1176 1177 1178 1179 1180 1181 1182 1183 1184 1185 1186 1187 1188 1189 1190 1191 1192 1193 1194 1195 1196 1197 1198 1199 1200 1201 1202 1203 1204 1205 1206 1207 1208 1209 1210 1211 1212 1213 1214 1215 1216 | /* $NetBSD: scsiconf.c,v 1.302 2022/04/14 16:50:26 pgoyette Exp $ */ /*- * Copyright (c) 1998, 1999, 2004 The NetBSD Foundation, Inc. * All rights reserved. * * This code is derived from software contributed to The NetBSD Foundation * by Charles M. Hannum; Jason R. Thorpe of the Numerical Aerospace * Simulation Facility, NASA Ames Research Center. * * Redistribution and use in source and binary forms, with or without * modification, are permitted provided that the following conditions * are met: * 1. Redistributions of source code must retain the above copyright * notice, this list of conditions and the following disclaimer. * 2. Redistributions in binary form must reproduce the above copyright * notice, this list of conditions and the following disclaimer in the * documentation and/or other materials provided with the distribution. * * THIS SOFTWARE IS PROVIDED BY THE NETBSD FOUNDATION, INC. AND CONTRIBUTORS * ``AS IS'' AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED * TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR * PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE FOUNDATION OR CONTRIBUTORS * BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR * CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF * SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS * INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN * CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) * ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE * POSSIBILITY OF SUCH DAMAGE. */ /* * Originally written by Julian Elischer (julian@tfs.com) * for TRW Financial Systems for use under the MACH(2.5) operating system. * * TRW Financial Systems, in accordance with their agreement with Carnegie * Mellon University, makes this software available to CMU to distribute * or use in any manner that they see fit as long as this message is kept with * the software. For this reason TFS also grants any other persons or * organisations permission to use or modify this software. * * TFS supplies this software to be publicly redistributed * on the understanding that TFS is not responsible for the correct * functioning of this software in any circumstances. * * Ported to run under 386BSD by Julian Elischer (julian@tfs.com) Sept 1992 */ #include <sys/cdefs.h> __KERNEL_RCSID(0, "$NetBSD: scsiconf.c,v 1.302 2022/04/14 16:50:26 pgoyette Exp $"); #include <sys/param.h> #include <sys/systm.h> #include <sys/kernel.h> #include <sys/proc.h> #include <sys/kthread.h> #include <sys/malloc.h> #include <sys/mutex.h> #include <sys/once.h> #include <sys/device.h> #include <sys/conf.h> #include <sys/fcntl.h> #include <sys/scsiio.h> #include <sys/queue.h> #include <sys/atomic.h> #include <sys/kmem.h> #include <dev/scsipi/scsi_all.h> #include <dev/scsipi/scsipi_all.h> #include <dev/scsipi/scsiconf.h> #include "locators.h" static const struct scsipi_periphsw scsi_probe_dev = { NULL, NULL, NULL, NULL, }; struct scsi_initq { struct scsipi_channel *sc_channel; TAILQ_ENTRY(scsi_initq) scsi_initq; }; static ONCE_DECL(scsi_conf_ctrl); static TAILQ_HEAD(, scsi_initq) scsi_initq_head; static kmutex_t scsibus_qlock; static kcondvar_t scsibus_qcv; static int scsi_probe_device(struct scsibus_softc *, int, int); static int scsibusmatch(device_t, cfdata_t, void *); static void scsibusattach(device_t, device_t, void *); static int scsibusdetach(device_t, int flags); static int scsibusrescan(device_t, const char *, const int *); static void scsidevdetached(device_t, device_t); CFATTACH_DECL3_NEW(scsibus, sizeof(struct scsibus_softc), scsibusmatch, scsibusattach, scsibusdetach, NULL, scsibusrescan, scsidevdetached, DVF_DETACH_SHUTDOWN); extern struct cfdriver scsibus_cd; static dev_type_open(scsibusopen); static dev_type_close(scsibusclose); static dev_type_ioctl(scsibusioctl); const struct cdevsw scsibus_cdevsw = { .d_open = scsibusopen, .d_close = scsibusclose, .d_read = noread, .d_write = nowrite, .d_ioctl = scsibusioctl, .d_stop = nostop, .d_tty = notty, .d_poll = nopoll, .d_mmap = nommap, .d_kqfilter = nokqfilter, .d_discard = nodiscard, .d_flag = D_OTHER | D_MPSAFE }; static int scsibusprint(void *, const char *); static void scsibus_discover_thread(void *); static void scsibus_config(struct scsibus_softc *); static int scsibus_init(void) { TAILQ_INIT(&scsi_initq_head); mutex_init(&scsibus_qlock, MUTEX_DEFAULT, IPL_NONE); cv_init(&scsibus_qcv, "scsinitq"); return 0; } static int scsibusmatch(device_t parent, cfdata_t cf, void *aux) { struct scsipi_channel *chan = aux; if (SCSIPI_BUSTYPE_TYPE(chan->chan_bustype->bustype_type) != SCSIPI_BUSTYPE_SCSI) return 0; if (cf->cf_loc[SCSICF_CHANNEL] != chan->chan_channel && cf->cf_loc[SCSICF_CHANNEL] != SCSICF_CHANNEL_DEFAULT) return (0); return (1); } static void scsibusattach(device_t parent, device_t self, void *aux) { struct scsibus_softc *sc = device_private(self); struct scsipi_channel *chan = aux; struct scsi_initq *scsi_initq; if (!pmf_device_register(self, NULL, NULL)) aprint_error_dev(self, "couldn't establish power handler\n"); sc->sc_dev = self; sc->sc_channel = chan; chan->chan_name = device_xname(sc->sc_dev); aprint_naive(": SCSI bus\n"); aprint_normal(": %d target%s, %d lun%s per target\n", chan->chan_ntargets, chan->chan_ntargets == 1 ? "" : "s", chan->chan_nluns, chan->chan_nluns == 1 ? "" : "s"); /* * XXX * newer adapters support more than 256 outstanding commands * per periph and don't use the tag (they eventually allocate one * internally). Right now scsipi always allocate a tag and * is limited to 256 tags, per scsi specs. * this should be revisited */ if (chan->chan_flags & SCSIPI_CHAN_OPENINGS) { if (chan->chan_max_periph > 256) chan->chan_max_periph = 256; } else { if (chan->chan_adapter->adapt_max_periph > 256) chan->chan_adapter->adapt_max_periph = 256; } if (atomic_inc_uint_nv(&chan_running(chan)) == 1) mutex_init(chan_mtx(chan), MUTEX_DEFAULT, IPL_BIO); cv_init(&chan->chan_cv_thr, "scshut"); cv_init(&chan->chan_cv_comp, "sccomp"); cv_init(&chan->chan_cv_xs, "xscmd"); if (scsipi_adapter_addref(chan->chan_adapter)) return; RUN_ONCE(&scsi_conf_ctrl, scsibus_init); /* Initialize the channel structure first */ chan->chan_init_cb = NULL; chan->chan_init_cb_arg = NULL; scsi_initq = malloc(sizeof(struct scsi_initq), M_DEVBUF, M_WAITOK); scsi_initq->sc_channel = chan; TAILQ_INSERT_TAIL(&scsi_initq_head, scsi_initq, scsi_initq); config_pending_incr(sc->sc_dev); if (scsipi_channel_init(chan)) { aprint_error_dev(sc->sc_dev, "failed to init channel\n"); return; } /* * Create the discover thread */ if (kthread_create(PRI_NONE, 0, NULL, scsibus_discover_thread, sc, &chan->chan_dthread, "%s-d", chan->chan_name)) { aprint_error_dev(sc->sc_dev, "unable to create discovery " "thread for channel %d\n", chan->chan_channel); return; } } static void scsibus_discover_thread(void *arg) { struct scsibus_softc *sc = arg; scsibus_config(sc); sc->sc_channel->chan_dthread = NULL; kthread_exit(0); } static void scsibus_config(struct scsibus_softc *sc) { struct scsipi_channel *chan = sc->sc_channel; struct scsi_initq *scsi_initq; #ifndef SCSI_DELAY #define SCSI_DELAY 2 #endif if ((chan->chan_flags & SCSIPI_CHAN_NOSETTLE) == 0 && SCSI_DELAY > 0) { aprint_normal_dev(sc->sc_dev, "waiting %d seconds for devices to settle...\n", SCSI_DELAY); /* ...an identifier we know no one will use... */ kpause("scsidly", false, SCSI_DELAY * hz, NULL); } /* Make sure the devices probe in scsibus order to avoid jitter. */ mutex_enter(&scsibus_qlock); for (;;) { scsi_initq = TAILQ_FIRST(&scsi_initq_head); if (scsi_initq->sc_channel == chan) break; cv_wait(&scsibus_qcv, &scsibus_qlock); } mutex_exit(&scsibus_qlock); scsi_probe_bus(sc, -1, -1); mutex_enter(&scsibus_qlock); TAILQ_REMOVE(&scsi_initq_head, scsi_initq, scsi_initq); cv_broadcast(&scsibus_qcv); mutex_exit(&scsibus_qlock); free(scsi_initq, M_DEVBUF); scsipi_adapter_delref(chan->chan_adapter); config_pending_decr(sc->sc_dev); } static int scsibusdetach(device_t self, int flags) { struct scsibus_softc *sc = device_private(self); struct scsipi_channel *chan = sc->sc_channel; int error; /* * Defer while discovery thread is running */ while (chan->chan_dthread != NULL) kpause("scsibusdet", false, hz, NULL); /* * Detach all of the periphs. */ error = scsipi_target_detach(chan, -1, -1, flags); if (error) return error; pmf_device_deregister(self); /* * Shut down the channel. */ scsipi_channel_shutdown(chan); cv_destroy(&chan->chan_cv_xs); cv_destroy(&chan->chan_cv_comp); cv_destroy(&chan->chan_cv_thr); membar_release(); if (atomic_dec_uint_nv(&chan_running(chan)) == 0) { membar_acquire(); mutex_destroy(chan_mtx(chan)); } return 0; } static int lun_compar(const void *a, const void *b) { const uint16_t * const la = a, * const lb = b; if (*la < *lb) return -1; if (*la > *lb) return 1; return 0; } static int scsi_report_luns(struct scsibus_softc *sc, int target, uint16_t ** const luns, size_t *nluns) { struct scsi_report_luns replun; struct scsi_report_luns_header *rlr; struct scsi_report_luns_lun *lunp; struct scsipi_channel *chan = sc->sc_channel; struct scsipi_inquiry_data inqbuf; struct scsipi_periph *periph; uint16_t tmp; int error; size_t i, rlrlen, rlrlenmin; memset(&replun, 0, sizeof(replun)); periph = scsipi_alloc_periph(M_WAITOK); periph->periph_channel = chan; periph->periph_switch = &scsi_probe_dev; periph->periph_target = target; periph->periph_lun = 0; periph->periph_quirks = chan->chan_defquirks; if ((error = scsipi_inquire(periph, &inqbuf, XS_CTL_DISCOVERY | XS_CTL_SILENT))) goto end2; periph->periph_version = inqbuf.version & SID_ANSII; if (periph->periph_version < 3) { error = ENOTSUP; goto end2; } rlrlen = rlrlenmin = sizeof(*rlr) + sizeof(*lunp) * 1; again: rlr = kmem_zalloc(rlrlen, KM_SLEEP); replun.opcode = SCSI_REPORT_LUNS; replun.selectreport = SELECTREPORT_NORMAL; _lto4b(rlrlen, replun.alloclen); error = scsipi_command(periph, (void *)&replun, sizeof(replun), (void *)rlr, rlrlen, SCSIPIRETRIES, 10000, NULL, XS_CTL_DATA_IN | XS_CTL_DISCOVERY | XS_CTL_SILENT); if (error) goto end; if (sizeof(*rlr) + _4btol(rlr->length) > rlrlen && sizeof(*rlr) + _4btol(rlr->length) <= 32) { const size_t old_rlrlen = rlrlen; rlrlen = sizeof(*rlr) + uimin(_4btol(rlr->length), 16383 * sizeof(*lunp)); kmem_free(rlr, old_rlrlen); rlr = NULL; if (rlrlen < rlrlenmin) { error = EIO; goto end; } goto again; } KASSERT(nluns != NULL); *nluns = (rlrlen - sizeof(*rlr)) / sizeof(*lunp); KASSERT(luns != NULL); *luns = kmem_alloc(*nluns * sizeof(**luns), KM_SLEEP); for (i = 0; i < *nluns; i++) { lunp = &((struct scsi_report_luns_lun *)&rlr[1])[i]; switch (lunp->lun[0] & 0xC0) { default: scsi_print_addr(periph); printf("LUN %016"PRIx64" ignored\n", _8btol(lunp->lun)); (*luns)[i] = 0; break; case 0x40: (*luns)[i] = _2btol(&lunp->lun[0]) & 0x3FFF; break; case 0x00: (*luns)[i] = _2btol(&lunp->lun[0]) & 0x00FF; break; } } kheapsort(*luns, *nluns, sizeof(**luns), lun_compar, &tmp); end: if (rlr) kmem_free(rlr, rlrlen); end2: scsipi_free_periph(periph); return error; } static void scsi_discover_luns(struct scsibus_softc *sc, int target, int minlun, int maxlun) { uint16_t *luns = NULL; /* XXX gcc */ size_t nluns = 0; /* XXX gcc */ if (scsi_report_luns(sc, target, &luns, &nluns) == 0) { for (size_t i = 0; i < nluns; i++) if (luns[i] >= minlun && luns[i] <= maxlun) scsi_probe_device(sc, target, luns[i]); kmem_free(luns, sizeof(*luns) * nluns); return; } for (int lun = minlun; lun <= maxlun; lun++) { /* * See if there's a device present, and configure it. */ if (scsi_probe_device(sc, target, lun) == 0) break; /* otherwise something says we should look further */ } } /* * Probe the requested scsi bus. It must be already set up. * target and lun optionally narrow the search if not -1 */ int scsi_probe_bus(struct scsibus_softc *sc, int target, int lun) { struct scsipi_channel *chan = sc->sc_channel; int maxtarget, mintarget, maxlun, minlun; int error; if (target == -1) { maxtarget = chan->chan_ntargets - 1; mintarget = 0; } else { if (target < 0 || target >= chan->chan_ntargets) return (EINVAL); maxtarget = mintarget = target; } if (lun == -1) { maxlun = chan->chan_nluns - 1; minlun = 0; } else { if (lun < 0 || lun >= chan->chan_nluns) return (EINVAL); maxlun = minlun = lun; } /* * Some HBAs provide an abstracted view of the bus; give them an * opportunity to re-scan it before we do. */ scsipi_adapter_ioctl(chan, SCBUSIOLLSCAN, NULL, 0, curproc); if ((error = scsipi_adapter_addref(chan->chan_adapter)) != 0) goto ret; for (target = mintarget; target <= maxtarget; target++) { if (target == chan->chan_id) continue; scsi_discover_luns(sc, target, minlun, maxlun); /* * Now that we've discovered all of the LUNs on this * I_T Nexus, update the xfer mode for all of them * that we know about. */ scsipi_set_xfer_mode(chan, target, 1); } scsipi_adapter_delref(chan->chan_adapter); ret: return (error); } static int scsibusrescan(device_t sc, const char *ifattr, const int *locators) { KASSERT(ifattr && !strcmp(ifattr, "scsibus")); KASSERT(locators); return (scsi_probe_bus(device_private(sc), locators[SCSIBUSCF_TARGET], locators[SCSIBUSCF_LUN])); } static void scsidevdetached(device_t self, device_t child) { struct scsibus_softc *sc = device_private(self); struct scsipi_channel *chan = sc->sc_channel; struct scsipi_periph *periph; int target, lun; target = device_locator(child, SCSIBUSCF_TARGET); lun = device_locator(child, SCSIBUSCF_LUN); mutex_enter(chan_mtx(chan)); periph = scsipi_lookup_periph_locked(chan, target, lun); KASSERT(periph != NULL && periph->periph_dev == child); scsipi_remove_periph(chan, periph); scsipi_free_periph(periph); mutex_exit(chan_mtx(chan)); } /* * Print out autoconfiguration information for a subdevice. * * This is a slight abuse of 'standard' autoconfiguration semantics, * because 'print' functions don't normally print the colon and * device information. However, in this case that's better than * either printing redundant information before the attach message, * or having the device driver call a special function to print out * the standard device information. */ static int scsibusprint(void *aux, const char *pnp) { struct scsipibus_attach_args *sa = aux; struct scsipi_inquiry_pattern *inqbuf; u_int8_t type; const char *dtype; char vendor[33], product[65], revision[17]; int target, lun; if (pnp != NULL) aprint_normal("%s", pnp); inqbuf = &sa->sa_inqbuf; target = sa->sa_periph->periph_target; lun = sa->sa_periph->periph_lun; type = inqbuf->type & SID_TYPE; dtype = scsipi_dtype(type); strnvisx(vendor, sizeof(vendor), inqbuf->vendor, 8, VIS_TRIM|VIS_SAFE|VIS_OCTAL); strnvisx(product, sizeof(product), inqbuf->product, 16, VIS_TRIM|VIS_SAFE|VIS_OCTAL); strnvisx(revision, sizeof(revision), inqbuf->revision, 4, VIS_TRIM|VIS_SAFE|VIS_OCTAL); aprint_normal(" target %d lun %d: <%s, %s, %s> %s %s%s", target, lun, vendor, product, revision, dtype, inqbuf->removable ? "removable" : "fixed", (sa->sa_periph->periph_opcs != NULL) ? " timeout-info" : ""); return (UNCONF); } static const struct scsi_quirk_inquiry_pattern scsi_quirk_patterns[] = { {{T_DIRECT, T_REMOV, "Apple ", "iPod ", ""}, PQUIRK_START}, {{T_CDROM, T_REMOV, "CHINON ", "CD-ROM CDS-431 ", ""}, PQUIRK_NOLUNS}, {{T_CDROM, T_REMOV, "CHINON ", "CD-ROM CDS-435 ", ""}, PQUIRK_NOLUNS}, {{T_CDROM, T_REMOV, "Chinon ", "CD-ROM CDS-525 ", ""}, PQUIRK_NOLUNS}, {{T_CDROM, T_REMOV, "CHINON ", "CD-ROM CDS-535 ", ""}, PQUIRK_NOLUNS}, {{T_CDROM, T_REMOV, "DEC ", "RRD42 (C) DEC ", ""}, PQUIRK_NOLUNS}, {{T_CDROM, T_REMOV, "DENON ", "DRD-25X ", "V"}, PQUIRK_NOLUNS}, {{T_CDROM, T_REMOV, "GENERIC ", "CRD-BP2 ", ""}, PQUIRK_NOLUNS}, {{T_CDROM, T_REMOV, "HP ", "C4324/C4325 ", ""}, PQUIRK_NOLUNS}, {{T_CDROM, T_REMOV, "IMS ", "CDD521/10 ", "2.06"}, PQUIRK_NOLUNS}, {{T_CDROM, T_REMOV, "MATSHITA", "CD-ROM CR-5XX ", "1.0b"}, PQUIRK_NOLUNS}, {{T_CDROM, T_REMOV, "MEDAVIS ", "RENO CD-ROMX2A ", ""}, PQUIRK_NOLUNS}, {{T_CDROM, T_REMOV, "MEDIAVIS", "CDR-H93MV ", "1.3"}, PQUIRK_NOLUNS}, {{T_CDROM, T_REMOV, "NEC ", "CD-ROM DRIVE:502", ""}, PQUIRK_NOLUNS}, {{T_CDROM, T_REMOV, "NEC ", "CD-ROM DRIVE:55 ", ""}, PQUIRK_NOLUNS}, {{T_CDROM, T_REMOV, "NEC ", "CD-ROM DRIVE:83 ", ""}, PQUIRK_NOLUNS}, {{T_CDROM, T_REMOV, "NEC ", "CD-ROM DRIVE:84 ", ""}, PQUIRK_NOLUNS}, {{T_CDROM, T_REMOV, "NEC ", "CD-ROM DRIVE:841", ""}, PQUIRK_NOLUNS}, {{T_CDROM, T_REMOV, "OLYMPUS ", "CDS620E ", "1.1d"}, PQUIRK_NOLUNS|PQUIRK_NOSYNC|PQUIRK_NOCAPACITY}, {{T_CDROM, T_REMOV, "PIONEER ", "CD-ROM DR-124X ", "1.01"}, PQUIRK_NOLUNS}, {{T_CDROM, T_REMOV, "PLEXTOR ", "CD-ROM PX-4XCS ", "1.01"}, PQUIRK_NOLUNS|PQUIRK_NOSYNC}, {{T_CDROM, T_REMOV, "SONY ", "CD-ROM CDU-541 ", ""}, PQUIRK_NOLUNS}, {{T_CDROM, T_REMOV, "SONY ", "CD-ROM CDU-55S ", ""}, PQUIRK_NOLUNS}, {{T_CDROM, T_REMOV, "SONY ", "CD-ROM CDU-561 ", ""}, PQUIRK_NOLUNS}, {{T_CDROM, T_REMOV, "SONY ", "CD-ROM CDU-76S", ""}, PQUIRK_NOLUNS|PQUIRK_NOSYNC|PQUIRK_NOWIDE}, {{T_CDROM, T_REMOV, "SONY ", "CD-ROM CDU-8003A", ""}, PQUIRK_NOLUNS}, {{T_CDROM, T_REMOV, "SONY ", "CD-ROM CDU-8012 ", ""}, PQUIRK_NOLUNS}, {{T_CDROM, T_REMOV, "TEAC ", "CD-ROM ", "1.06"}, PQUIRK_NOLUNS}, {{T_CDROM, T_REMOV, "TEAC ", "CD-ROM CD-56S ", "1.0B"}, PQUIRK_NOLUNS}, {{T_CDROM, T_REMOV, "TEXEL ", "CD-ROM ", "1.06"}, PQUIRK_NOLUNS}, {{T_CDROM, T_REMOV, "TEXEL ", "CD-ROM DM-XX24 K", "1.09"}, PQUIRK_NOLUNS}, {{T_CDROM, T_REMOV, "TEXEL ", "CD-ROM DM-XX24 K", "1.10"}, PQUIRK_NOLUNS}, {{T_CDROM, T_REMOV, "TOSHIBA ", "XM-4101TASUNSLCD", ""}, PQUIRK_NOLUNS|PQUIRK_NOSYNC}, /* "IBM CDRM00201 !F" 0724 is an IBM OEM Toshiba XM-4101BME */ {{T_CDROM, T_REMOV, "IBM ", "CDRM00201 !F", "0724"}, PQUIRK_NOLUNS|PQUIRK_NOSYNC}, {{T_CDROM, T_REMOV, "ShinaKen", "CD-ROM DM-3x1S", "1.04"}, PQUIRK_NOLUNS}, {{T_CDROM, T_REMOV, "JVC ", "R2626", ""}, PQUIRK_NOLUNS}, {{T_CDROM, T_REMOV, "YAMAHA", "CRW8424S", ""}, PQUIRK_NOLUNS}, {{T_CDROM, T_REMOV, "NEC ", "CD-ROM DRIVE:222", ""}, PQUIRK_NOLUNS|PQUIRK_NOSYNC}, {{T_DIRECT, T_FIXED, "MICROP ", "1588-15MBSUN0669", ""}, PQUIRK_AUTOSAVE}, {{T_DIRECT, T_FIXED, "MICROP ", "2217-15MQ1091501", ""}, PQUIRK_NOSYNCCACHE}, {{T_OPTICAL, T_REMOV, "EPSON ", "OMD-5010 ", "3.08"}, PQUIRK_NOLUNS}, {{T_DIRECT, T_FIXED, "ADAPTEC ", "AEC-4412BD", "1.2A"}, PQUIRK_NOMODESENSE}, {{T_DIRECT, T_FIXED, "ADAPTEC ", "ACB-4000", ""}, PQUIRK_FORCELUNS|PQUIRK_AUTOSAVE|PQUIRK_NOMODESENSE}, {{T_DIRECT, T_FIXED, "DEC ", "RZ55 (C) DEC", ""}, PQUIRK_AUTOSAVE}, {{T_DIRECT, T_FIXED, "EMULEX ", "MD21/S2 ESDI", "A00"}, PQUIRK_FORCELUNS|PQUIRK_AUTOSAVE}, {{T_DIRECT, T_FIXED, "MICROP", "1548-15MZ1077801", "HZ2P"}, PQUIRK_NOTAG}, {{T_DIRECT, T_FIXED, "HP ", "C372", ""}, PQUIRK_NOTAG}, {{T_DIRECT, T_FIXED, "IBMRAID ", "0662S", ""}, PQUIRK_AUTOSAVE}, {{T_DIRECT, T_FIXED, "IBM ", "0663H", ""}, PQUIRK_AUTOSAVE}, {{T_DIRECT, T_FIXED, "IBM", "0664", ""}, PQUIRK_AUTOSAVE}, {{T_DIRECT, T_FIXED, /* improperly report DT-only sync mode */ "IBM ", "DXHS36D", ""}, PQUIRK_CAP_SYNC|PQUIRK_CAP_WIDE16}, {{T_DIRECT, T_FIXED, "IBM ", "DXHS18Y", ""}, PQUIRK_CAP_SYNC|PQUIRK_CAP_WIDE16}, {{T_DIRECT, T_FIXED, "IBM ", "H3171-S2", ""}, PQUIRK_NOLUNS|PQUIRK_AUTOSAVE}, {{T_DIRECT, T_FIXED, "IBM ", "KZ-C", ""}, PQUIRK_AUTOSAVE}, /* Broken IBM disk */ {{T_DIRECT, T_FIXED, "" , "DFRSS2F", ""}, PQUIRK_AUTOSAVE}, {{T_DIRECT, T_FIXED, "Initio ", "", ""}, PQUIRK_NOBIGMODESENSE}, {{T_DIRECT, T_FIXED, "JMicron ", "Generic ", ""}, PQUIRK_NOFUA}, {{T_DIRECT, T_REMOV, "MPL ", "MC-DISK- ", ""}, PQUIRK_NOLUNS}, {{T_DIRECT, T_FIXED, "MAXTOR ", "XT-3280 ", ""}, PQUIRK_NOLUNS}, {{T_DIRECT, T_FIXED, "MAXTOR ", "XT-4380S ", ""}, PQUIRK_NOLUNS}, {{T_DIRECT, T_FIXED, "MAXTOR ", "MXT-1240S ", ""}, PQUIRK_NOLUNS}, {{T_DIRECT, T_FIXED, "MAXTOR ", "XT-4170S ", ""}, PQUIRK_NOLUNS}, {{T_DIRECT, T_FIXED, "MAXTOR ", "XT-8760S", ""}, PQUIRK_NOLUNS}, {{T_DIRECT, T_FIXED, "MAXTOR ", "LXT-213S ", ""}, PQUIRK_NOLUNS}, {{T_DIRECT, T_FIXED, "MAXTOR ", "LXT-213S SUN0207", ""}, PQUIRK_NOLUNS}, {{T_DIRECT, T_FIXED, "MAXTOR ", "LXT-200S ", ""}, PQUIRK_NOLUNS}, {{T_DIRECT, T_FIXED, "MEGADRV ", "EV1000", ""}, PQUIRK_NOMODESENSE}, {{T_DIRECT, T_FIXED, "MICROP", "1991-27MZ", ""}, PQUIRK_NOTAG}, {{T_DIRECT, T_FIXED, "MST ", "SnapLink ", ""}, PQUIRK_NOLUNS}, {{T_DIRECT, T_FIXED, "NEC ", "D3847 ", "0307"}, PQUIRK_NOLUNS}, {{T_DIRECT, T_FIXED, "QUANTUM ", "ELS85S ", ""}, PQUIRK_AUTOSAVE}, {{T_DIRECT, T_FIXED, "QUANTUM ", "LPS525S ", ""}, PQUIRK_NOLUNS}, {{T_DIRECT, T_FIXED, "QUANTUM ", "P105S 910-10-94x", ""}, PQUIRK_NOLUNS}, {{T_DIRECT, T_FIXED, "QUANTUM ", "PD1225S ", ""}, PQUIRK_NOLUNS}, {{T_DIRECT, T_FIXED, "QUANTUM ", "PD210S SUN0207", ""}, PQUIRK_NOLUNS}, {{T_DIRECT, T_FIXED, "QUANTUM ", "ATLAS IV 9 WLS", "0A0A"}, PQUIRK_CAP_NODT}, {{T_DIRECT, T_FIXED, "RODIME ", "RO3000S ", ""}, PQUIRK_NOLUNS}, {{T_DIRECT, T_FIXED, "SEAGATE ", "ST125N ", ""}, PQUIRK_NOLUNS}, {{T_DIRECT, T_FIXED, "SEAGATE ", "ST157N ", ""}, PQUIRK_NOLUNS}, {{T_DIRECT, T_FIXED, "SEAGATE ", "ST296 ", ""}, PQUIRK_NOLUNS}, {{T_DIRECT, T_FIXED, "SEAGATE ", "ST296N ", ""}, PQUIRK_NOLUNS}, {{T_DIRECT, T_FIXED, "SEAGATE ", "ST318404LC ", ""}, PQUIRK_NOLUNS}, {{T_DIRECT, T_FIXED, "SEAGATE ", "ST336753LC ", ""}, PQUIRK_NOLUNS}, {{T_DIRECT, T_FIXED, "SEAGATE ", "ST336753LW ", ""}, PQUIRK_NOLUNS}, {{T_DIRECT, T_FIXED, "SEAGATE ", "ST336754LC ", ""}, PQUIRK_NOLUNS}, {{T_DIRECT, T_FIXED, "SEAGATE ", "ST39236LC ", ""}, PQUIRK_NOLUNS}, {{T_DIRECT, T_FIXED, "SEAGATE ", "ST15150N ", ""}, PQUIRK_NOTAG}, {{T_DIRECT, T_FIXED, "SEAGATE ", "ST19171", ""}, PQUIRK_NOMODESENSE}, {{T_DIRECT, T_FIXED, "SEAGATE ", "ST32430N", ""}, PQUIRK_CAP_SYNC}, {{T_DIRECT, T_FIXED, "SEAGATE ", "ST34501FC ", ""}, PQUIRK_NOMODESENSE}, {{T_DIRECT, T_FIXED, "SEAGATE ", "SX910800N", ""}, PQUIRK_NOTAG}, {{T_DIRECT, T_FIXED, "TOSHIBA ", "MK538FB ", "6027"}, PQUIRK_NOLUNS}, {{T_DIRECT, T_FIXED, "MICROP ", "1924", ""}, PQUIRK_CAP_SYNC}, {{T_DIRECT, T_FIXED, "FUJITSU ", "M2266", ""}, PQUIRK_CAP_SYNC}, {{T_DIRECT, T_FIXED, "FUJITSU ", "M2624S-512 ", ""}, PQUIRK_CAP_SYNC}, {{T_DIRECT, T_FIXED, "SEAGATE ", "SX336704LC" , ""}, PQUIRK_CAP_SYNC | PQUIRK_CAP_WIDE16}, {{T_DIRECT, T_FIXED, "SEAGATE ", "SX173404LC", ""}, PQUIRK_CAP_SYNC | PQUIRK_CAP_WIDE16}, {{T_DIRECT, T_REMOV, "IOMEGA", "ZIP 100", "J.03"}, PQUIRK_NOLUNS|PQUIRK_NOSYNC}, {{T_DIRECT, T_REMOV, "INSITE", "I325VM", ""}, PQUIRK_NOLUNS}, /* XXX: QIC-36 tape behind Emulex adapter. Very broken. */ {{T_SEQUENTIAL, T_REMOV, " ", " ", " "}, PQUIRK_NOLUNS}, {{T_SEQUENTIAL, T_REMOV, "EMULEX ", "MT-02 QIC ", ""}, PQUIRK_NOLUNS}, {{T_SEQUENTIAL, T_REMOV, "CALIPER ", "CP150 ", ""}, PQUIRK_NOLUNS}, {{T_SEQUENTIAL, T_REMOV, "EXABYTE ", "EXB-8200 ", ""}, PQUIRK_NOLUNS}, {{T_SEQUENTIAL, T_REMOV, "SONY ", "GY-10C ", ""}, PQUIRK_NOLUNS}, {{T_SEQUENTIAL, T_REMOV, "SONY ", "SDT-2000 ", "2.09"}, PQUIRK_NOLUNS}, {{T_SEQUENTIAL, T_REMOV, "SONY ", "SDT-5000 ", "3."}, PQUIRK_NOSYNC|PQUIRK_NOWIDE}, {{T_SEQUENTIAL, T_REMOV, "SONY ", "SDT-5200 ", "3."}, PQUIRK_NOLUNS}, {{T_SEQUENTIAL, T_REMOV, "TANDBERG", " TDC 3600 ", ""}, PQUIRK_NOLUNS}, /* Following entry reported as a Tandberg 3600; ref. PR1933 */ {{T_SEQUENTIAL, T_REMOV, "ARCHIVE ", "VIPER 150 21247", ""}, PQUIRK_NOLUNS}, /* Following entry for a Cipher ST150S; ref. PR4171 */ {{T_SEQUENTIAL, T_REMOV, "ARCHIVE ", "VIPER 1500 21247", "2.2G"}, PQUIRK_NOLUNS}, {{T_SEQUENTIAL, T_REMOV, "ARCHIVE ", "Python 28454-XXX", ""}, PQUIRK_NOLUNS}, {{T_SEQUENTIAL, T_REMOV, "WANGTEK ", "5099ES SCSI", ""}, PQUIRK_NOLUNS}, {{T_SEQUENTIAL, T_REMOV, "WANGTEK ", "5150ES SCSI", ""}, PQUIRK_NOLUNS}, {{T_SEQUENTIAL, T_REMOV, "WANGTEK ", "SCSI-36", ""}, PQUIRK_NOLUNS}, {{T_SEQUENTIAL, T_REMOV, "WangDAT ", "Model 1300 ", "02.4"}, PQUIRK_NOSYNC|PQUIRK_NOWIDE}, {{T_SEQUENTIAL, T_REMOV, "WangDAT ", "Model 2600 ", "01.7"}, PQUIRK_NOSYNC|PQUIRK_NOWIDE}, {{T_SEQUENTIAL, T_REMOV, "WangDAT ", "Model 3200 ", "02.2"}, PQUIRK_NOSYNC|PQUIRK_NOWIDE}, {{T_SEQUENTIAL, T_REMOV, "TEAC ", "MT-2ST/N50 ", ""}, PQUIRK_NOLUNS}, {{T_SCANNER, T_FIXED, "RICOH ", "IS60 ", "1R08"}, PQUIRK_NOLUNS}, {{T_SCANNER, T_FIXED, "UMAX ", "Astra 1200S ", "V2.9"}, PQUIRK_NOLUNS}, {{T_SCANNER, T_FIXED, "UMAX ", "Astra 1220S ", ""}, PQUIRK_NOLUNS}, {{T_SCANNER, T_FIXED, "UMAX ", "UMAX S-6E ", "V2.0"}, PQUIRK_NOLUNS}, {{T_SCANNER, T_FIXED, "UMAX ", "UMAX S-12 ", "V2.1"}, PQUIRK_NOLUNS}, {{T_SCANNER, T_FIXED, "ULTIMA ", "A6000C ", ""}, PQUIRK_NOLUNS}, {{T_PROCESSOR, T_FIXED, "ESG-SHV", "SCA HSBP M15", ""}, PQUIRK_NOLUNS}, {{T_PROCESSOR, T_FIXED, "SYMBIOS", "", ""}, PQUIRK_NOLUNS}, {{T_PROCESSOR, T_FIXED, "LITRONIC", "PCMCIA ", ""}, PQUIRK_NOLUNS}, {{T_CHANGER, T_REMOV, "SONY ", "CDL1100 ", ""}, PQUIRK_NOLUNS}, {{T_ENCLOSURE, T_FIXED, "SUN ", "SENA ", ""}, PQUIRK_NOLUNS}, }; /* * given a target and lun, ask the device what * it is, and find the correct driver table * entry. */ static int scsi_probe_device(struct scsibus_softc *sc, int target, int lun) { struct scsipi_channel *chan = sc->sc_channel; struct scsipi_periph *periph; struct scsipi_inquiry_data inqbuf; const struct scsi_quirk_inquiry_pattern *finger; int checkdtype, priority, docontinue, quirks; struct scsipibus_attach_args sa; cfdata_t cf; int locs[SCSIBUSCF_NLOCS]; /* * Assume no more LUNs to search after this one. * If we successfully get Inquiry data and after * merging quirks we find we can probe for more * LUNs, we will. */ docontinue = 0; /* Skip this slot if it is already attached. */ if (scsipi_lookup_periph(chan, target, lun) != NULL) return (docontinue); periph = scsipi_alloc_periph(M_WAITOK); periph->periph_channel = chan; periph->periph_switch = &scsi_probe_dev; periph->periph_target = target; periph->periph_lun = lun; periph->periph_quirks = chan->chan_defquirks; #ifdef SCSIPI_DEBUG if (SCSIPI_DEBUG_TYPE == SCSIPI_BUSTYPE_SCSI && SCSIPI_DEBUG_TARGET == target && SCSIPI_DEBUG_LUN == lun) periph->periph_dbflags |= SCSIPI_DEBUG_FLAGS; #endif /* * Ask the device what it is */ #ifdef SCSI_2_DEF /* some devices need to be told to go to SCSI2 */ /* However some just explode if you tell them this.. leave it out */ scsi_change_def(periph, XS_CTL_DISCOVERY | XS_CTL_SILENT); #endif /* SCSI_2_DEF */ /* Now go ask the device all about itself. */ memset(&inqbuf, 0, sizeof(inqbuf)); { u_int8_t *extension = &inqbuf.flags1; int len = 0; while (len < 3) extension[len++] = '\0'; while (len < 3 + 28) extension[len++] = ' '; while (len < 3 + 28 + 20) extension[len++] = '\0'; while (len < 3 + 28 + 20 + 1) extension[len++] = '\0'; while (len < 3 + 28 + 20 + 1 + 1) extension[len++] = '\0'; while (len < 3 + 28 + 20 + 1 + 1 + (8*2)) extension[len++] = ' '; } if (scsipi_inquire(periph, &inqbuf, XS_CTL_DISCOVERY | XS_CTL_SILENT)) goto bad; periph->periph_type = inqbuf.device & SID_TYPE; if (inqbuf.dev_qual2 & SID_REMOVABLE) periph->periph_flags |= PERIPH_REMOVABLE; periph->periph_version = inqbuf.version & SID_ANSII; /* * Any device qualifier that has the top bit set (qualifier&4 != 0) * is vendor specific and won't match in this switch. * All we do here is throw out bad/negative responses. */ checkdtype = 0; switch (inqbuf.device & SID_QUAL) { case SID_QUAL_LU_PRESENT: checkdtype = 1; break; case SID_QUAL_LU_NOTPRESENT: case SID_QUAL_reserved: case SID_QUAL_LU_NOT_SUPP: goto bad; default: break; } /* Let the adapter driver handle the device separately if it wants. */ if (chan->chan_adapter->adapt_accesschk != NULL && (*chan->chan_adapter->adapt_accesschk)(periph, &sa.sa_inqbuf)) goto bad; if (checkdtype) { switch (periph->periph_type) { case T_DIRECT: case T_SEQUENTIAL: case T_PRINTER: case T_PROCESSOR: case T_WORM: case T_CDROM: case T_SCANNER: case T_OPTICAL: case T_CHANGER: case T_COMM: case T_IT8_1: case T_IT8_2: case T_STORARRAY: case T_ENCLOSURE: case T_SIMPLE_DIRECT: case T_OPTIC_CARD_RW: case T_OBJECT_STORED: default: break; case T_NODEVICE: goto bad; } } sa.sa_periph = periph; sa.sa_inqbuf.type = inqbuf.device; sa.sa_inqbuf.removable = inqbuf.dev_qual2 & SID_REMOVABLE ? T_REMOV : T_FIXED; sa.sa_inqbuf.vendor = inqbuf.vendor; sa.sa_inqbuf.product = inqbuf.product; sa.sa_inqbuf.revision = inqbuf.revision; sa.scsipi_info.scsi_version = inqbuf.version; sa.sa_inqptr = &inqbuf; finger = scsipi_inqmatch( &sa.sa_inqbuf, scsi_quirk_patterns, sizeof(scsi_quirk_patterns)/sizeof(scsi_quirk_patterns[0]), sizeof(scsi_quirk_patterns[0]), &priority); if (finger != NULL) quirks = finger->quirks; else quirks = 0; /* * Determine the operating mode capabilities of the device. */ if (periph->periph_version >= 2) { if ((inqbuf.flags3 & SID_CmdQue) != 0 && (quirks & PQUIRK_NOTAG) == 0) periph->periph_cap |= PERIPH_CAP_TQING; if ((inqbuf.flags3 & SID_Linked) != 0) periph->periph_cap |= PERIPH_CAP_LINKCMDS; if ((inqbuf.flags3 & SID_Sync) != 0 && (quirks & PQUIRK_NOSYNC) == 0) periph->periph_cap |= PERIPH_CAP_SYNC; if ((inqbuf.flags3 & SID_WBus16) != 0 && (quirks & PQUIRK_NOWIDE) == 0) periph->periph_cap |= PERIPH_CAP_WIDE16; if ((inqbuf.flags3 & SID_WBus32) != 0 && (quirks & PQUIRK_NOWIDE) == 0) periph->periph_cap |= PERIPH_CAP_WIDE32; if ((inqbuf.flags3 & SID_SftRe) != 0) periph->periph_cap |= PERIPH_CAP_SFTRESET; if ((inqbuf.flags3 & SID_RelAdr) != 0) periph->periph_cap |= PERIPH_CAP_RELADR; /* SPC-2 */ if (periph->periph_version >= 3 && !(quirks & PQUIRK_CAP_NODT)){ /* * Report ST clocking though CAP_WIDExx/CAP_SYNC. * If the device only supports DT, clear these * flags (DT implies SYNC and WIDE) */ switch (inqbuf.flags4 & SID_Clocking) { case SID_CLOCKING_DT_ONLY: periph->periph_cap &= ~(PERIPH_CAP_SYNC | PERIPH_CAP_WIDE16 | PERIPH_CAP_WIDE32); /* FALLTHROUGH */ case SID_CLOCKING_SD_DT: periph->periph_cap |= PERIPH_CAP_DT; break; default: /* ST only or invalid */ /* nothing to do */ break; } } if (periph->periph_version >= 3) { if (inqbuf.flags4 & SID_IUS) periph->periph_cap |= PERIPH_CAP_IUS; if (inqbuf.flags4 & SID_QAS) periph->periph_cap |= PERIPH_CAP_QAS; } } if (quirks & PQUIRK_CAP_SYNC) periph->periph_cap |= PERIPH_CAP_SYNC; if (quirks & PQUIRK_CAP_WIDE16) periph->periph_cap |= PERIPH_CAP_WIDE16; /* * Now apply any quirks from the table. */ periph->periph_quirks |= quirks; if (periph->periph_version == 0 && (periph->periph_quirks & PQUIRK_FORCELUNS) == 0) periph->periph_quirks |= PQUIRK_NOLUNS; if ((periph->periph_quirks & PQUIRK_NOLUNS) == 0) docontinue = 1; locs[SCSIBUSCF_TARGET] = target; locs[SCSIBUSCF_LUN] = lun; KERNEL_LOCK(1, NULL); if ((cf = config_search(sc->sc_dev, &sa, CFARGS(.submatch = config_stdsubmatch, .locators = locs))) != NULL) { scsipi_insert_periph(chan, periph); /* * Determine supported opcodes and timeouts if available. * Only do this on peripherals reporting SCSI version 3 * or greater - this command isn't in the SCSI-2 spec. and * it causes either timeouts or peripherals disappearing * when sent to some SCSI-1 or SCSI-2 peripherals. */ if (periph->periph_version >= 3) scsipi_get_opcodeinfo(periph); /* * XXX Can't assign periph_dev here, because we'll * XXX need it before config_attach() returns. Must * XXX assign it in periph driver. */ config_attach(sc->sc_dev, cf, &sa, scsibusprint, CFARGS(.locators = locs)); KERNEL_UNLOCK_ONE(NULL); } else { scsibusprint(&sa, device_xname(sc->sc_dev)); aprint_normal(" not configured\n"); KERNEL_UNLOCK_ONE(NULL); goto bad; } return (docontinue); bad: scsipi_free_periph(periph); return (docontinue); } /****** Entry points for user control of the SCSI bus. ******/ static int scsibusopen(dev_t dev, int flag, int fmt, struct lwp *l) { struct scsibus_softc *sc; int error, unit = minor(dev); sc = device_lookup_private(&scsibus_cd, unit); if (sc == NULL) return (ENXIO); if (sc->sc_flags & SCSIBUSF_OPEN) return (EBUSY); if ((error = scsipi_adapter_addref(sc->sc_channel->chan_adapter)) != 0) return (error); sc->sc_flags |= SCSIBUSF_OPEN; return (0); } static int scsibusclose(dev_t dev, int flag, int fmt, struct lwp *l) { struct scsibus_softc *sc; sc = device_lookup_private(&scsibus_cd, minor(dev)); scsipi_adapter_delref(sc->sc_channel->chan_adapter); sc->sc_flags &= ~SCSIBUSF_OPEN; return (0); } static int scsibusioctl(dev_t dev, u_long cmd, void *addr, int flag, struct lwp *l) { struct scsibus_softc *sc; struct scsipi_channel *chan; int error; sc = device_lookup_private(&scsibus_cd, minor(dev)); chan = sc->sc_channel; /* * Enforce write permission for ioctls that change the * state of the bus. Host adapter specific ioctls must * be checked by the adapter driver. */ switch (cmd) { case SCBUSIOSCAN: case SCBUSIODETACH: case SCBUSIORESET: if ((flag & FWRITE) == 0) return (EBADF); } switch (cmd) { case SCBUSIOSCAN: { struct scbusioscan_args *a = (struct scbusioscan_args *)addr; error = scsi_probe_bus(sc, a->sa_target, a->sa_lun); break; } case SCBUSIODETACH: { struct scbusiodetach_args *a = (struct scbusiodetach_args *)addr; error = scsipi_target_detach(chan, a->sa_target, a->sa_lun, 0); break; } case SCBUSIORESET: /* FALLTHROUGH */ default: error = scsipi_adapter_ioctl(chan, cmd, addr, flag, l->l_proc); break; } return (error); } |
| 76 76 76 76 76 76 76 72 72 12 76 6 76 76 105 64 194 194 194 98 98 98 5 193 106 103 61 61 61 60 8 61 61 61 171 128 128 105 61 61 61 61 61 61 61 61 3 61 60 105 127 186 186 186 186 119 16 15 203 202 201 3 3 3 3 3 3 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 197 198 199 200 201 202 203 204 205 206 207 208 209 210 211 212 213 214 215 216 217 218 219 220 221 222 223 224 225 226 227 228 229 230 231 232 233 234 235 236 237 238 239 240 241 242 243 244 245 246 247 248 249 250 251 252 253 254 255 256 257 258 259 260 261 262 263 264 265 266 267 268 269 270 271 272 273 274 275 276 277 278 279 280 281 282 283 284 285 286 287 288 289 290 291 292 293 294 295 296 297 298 299 300 301 302 303 304 305 306 307 308 309 310 311 312 313 314 315 316 317 318 319 320 321 322 323 324 325 326 327 328 329 330 331 332 333 334 335 336 337 338 339 340 341 342 343 344 345 346 347 348 349 350 351 352 353 354 355 356 357 358 359 360 361 362 363 364 365 366 367 368 369 370 371 372 373 374 375 376 377 378 379 380 381 382 383 384 385 386 387 388 389 390 391 392 393 394 395 396 397 398 399 400 401 402 403 404 405 406 407 408 409 410 411 412 413 414 415 416 417 418 419 420 421 422 423 424 425 426 427 428 429 430 431 432 433 434 435 436 437 438 439 440 441 442 443 444 445 446 447 448 449 450 451 452 453 454 455 456 457 458 459 460 461 462 463 464 465 466 467 468 469 470 471 472 473 474 475 476 477 478 479 480 481 482 483 484 485 486 487 488 489 490 491 492 493 494 495 496 497 498 499 500 501 502 503 504 505 506 507 508 509 510 511 512 513 514 515 516 517 518 519 520 521 522 523 524 525 526 527 528 529 530 531 532 533 534 535 536 537 538 539 540 541 542 543 544 545 546 547 548 549 550 551 552 553 554 555 556 557 558 559 560 561 562 563 564 565 566 567 568 569 570 571 572 573 574 575 576 577 578 579 580 581 582 583 584 585 586 587 588 589 590 591 592 593 594 595 596 597 598 599 600 601 602 603 604 605 606 607 608 609 610 611 612 613 614 615 616 617 618 619 620 621 622 623 624 625 626 627 628 629 630 631 632 633 634 635 636 637 638 639 640 641 642 643 644 645 646 647 648 649 650 651 652 653 654 655 656 657 658 659 660 661 662 663 664 665 666 667 668 669 670 671 672 673 674 675 676 677 678 679 680 681 682 683 684 685 686 687 688 689 690 691 692 693 694 695 696 697 698 699 700 701 702 703 704 705 706 707 708 709 710 711 712 713 714 715 716 717 718 719 720 721 722 723 724 725 726 727 728 729 730 731 732 733 734 735 736 737 738 739 740 741 742 743 744 745 746 747 748 749 750 751 752 753 754 755 756 757 758 759 760 761 762 763 764 765 766 767 768 769 770 771 772 773 774 775 776 777 778 779 780 781 782 783 784 785 786 787 788 789 790 791 792 793 794 795 796 797 798 799 800 801 802 803 804 805 806 807 808 809 810 811 812 813 814 815 816 817 818 819 820 821 822 823 824 825 826 827 828 829 830 831 832 833 834 835 836 837 838 839 840 841 842 843 844 845 846 847 848 849 850 851 852 853 854 855 856 857 858 859 860 861 862 863 864 865 866 867 868 869 870 871 872 873 874 875 876 877 878 879 880 881 882 883 884 885 886 887 888 889 890 891 892 893 894 895 896 897 898 899 900 901 902 903 904 905 906 907 908 909 910 911 912 913 914 915 916 917 918 919 920 921 922 923 924 925 926 927 928 929 930 931 932 933 934 935 936 937 938 939 940 941 942 943 944 945 946 947 948 949 950 951 952 953 954 955 956 957 958 959 960 961 962 963 964 965 966 967 968 969 970 971 972 973 974 975 976 977 978 979 980 981 982 983 984 985 986 987 988 989 990 991 992 993 994 995 996 997 998 999 1000 1001 1002 1003 1004 1005 1006 1007 1008 1009 1010 1011 1012 1013 1014 1015 1016 1017 1018 1019 1020 1021 1022 1023 1024 1025 1026 1027 1028 1029 1030 1031 1032 1033 1034 1035 1036 1037 1038 1039 1040 1041 1042 1043 1044 1045 1046 1047 1048 1049 1050 1051 1052 1053 1054 1055 1056 1057 1058 1059 1060 1061 1062 1063 1064 1065 1066 1067 1068 1069 1070 1071 1072 1073 1074 1075 1076 1077 1078 1079 1080 1081 1082 1083 1084 1085 1086 1087 1088 1089 1090 1091 1092 1093 1094 1095 1096 1097 1098 1099 1100 1101 1102 1103 1104 1105 1106 1107 1108 1109 1110 1111 1112 1113 1114 1115 1116 1117 1118 1119 1120 1121 1122 1123 1124 1125 1126 1127 1128 1129 1130 1131 1132 1133 1134 1135 1136 1137 1138 1139 1140 1141 1142 | /* $NetBSD: uvm_bio.c,v 1.126 2021/04/01 06:26:26 simonb Exp $ */ /* * Copyright (c) 1998 Chuck Silvers. * All rights reserved. * * Redistribution and use in source and binary forms, with or without * modification, are permitted provided that the following conditions * are met: * 1. Redistributions of source code must retain the above copyright * notice, this list of conditions and the following disclaimer. * 2. Redistributions in binary form must reproduce the above copyright * notice, this list of conditions and the following disclaimer in the * documentation and/or other materials provided with the distribution. * 3. The name of the author may not be used to endorse or promote products * derived from this software without specific prior written permission. * * THIS SOFTWARE IS PROVIDED BY THE AUTHOR ``AS IS'' AND ANY EXPRESS OR * IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES * OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. * IN NO EVENT SHALL THE AUTHOR BE LIABLE FOR ANY DIRECT, INDIRECT, * INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, * BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; * LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED * AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, * OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY * OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF * SUCH DAMAGE. * */ /* * uvm_bio.c: buffered i/o object mapping cache */ #include <sys/cdefs.h> __KERNEL_RCSID(0, "$NetBSD: uvm_bio.c,v 1.126 2021/04/01 06:26:26 simonb Exp $"); #include "opt_uvmhist.h" #include "opt_ubc.h" #include <sys/param.h> #include <sys/systm.h> #include <sys/kmem.h> #include <sys/kernel.h> #include <sys/proc.h> #include <sys/sysctl.h> #include <sys/vnode.h> #include <sys/bitops.h> /* for ilog2() */ #include <uvm/uvm.h> #include <uvm/uvm_pdpolicy.h> #ifdef PMAP_DIRECT # define UBC_USE_PMAP_DIRECT #endif /* * local functions */ static int ubc_fault(struct uvm_faultinfo *, vaddr_t, struct vm_page **, int, int, vm_prot_t, int); static struct ubc_map *ubc_find_mapping(struct uvm_object *, voff_t); static int ubchash_stats(struct hashstat_sysctl *hs, bool fill); #ifdef UBC_USE_PMAP_DIRECT static int __noinline ubc_uiomove_direct(struct uvm_object *, struct uio *, vsize_t, int, int); static void __noinline ubc_zerorange_direct(struct uvm_object *, off_t, size_t, int); /* XXX disabled by default until the kinks are worked out. */ bool ubc_direct = false; #endif /* * local data structues */ #define UBC_HASH(uobj, offset) \ (((((u_long)(uobj)) >> 8) + (((u_long)(offset)) >> PAGE_SHIFT)) & \ ubc_object.hashmask) #define UBC_QUEUE(offset) \ (&ubc_object.inactive[(((u_long)(offset)) >> ubc_winshift) & \ (UBC_NQUEUES - 1)]) #define UBC_UMAP_ADDR(u) \ (vaddr_t)(ubc_object.kva + (((u) - ubc_object.umap) << ubc_winshift)) #define UMAP_PAGES_LOCKED 0x0001 #define UMAP_MAPPING_CACHED 0x0002 struct ubc_map { struct uvm_object * uobj; /* mapped object */ voff_t offset; /* offset into uobj */ voff_t writeoff; /* write offset */ vsize_t writelen; /* write len */ int refcount; /* refcount on mapping */ int flags; /* extra state */ int advice; LIST_ENTRY(ubc_map) hash; /* hash table */ TAILQ_ENTRY(ubc_map) inactive; /* inactive queue */ LIST_ENTRY(ubc_map) list; /* per-object list */ }; TAILQ_HEAD(ubc_inactive_head, ubc_map); static struct ubc_object { struct uvm_object uobj; /* glue for uvm_map() */ char *kva; /* where ubc_object is mapped */ struct ubc_map *umap; /* array of ubc_map's */ LIST_HEAD(, ubc_map) *hash; /* hashtable for cached ubc_map's */ u_long hashmask; /* mask for hashtable */ struct ubc_inactive_head *inactive; /* inactive queues for ubc_map's */ } ubc_object; const struct uvm_pagerops ubc_pager = { .pgo_fault = ubc_fault, /* ... rest are NULL */ }; /* Use value at least as big as maximum page size supported by architecture */ #define UBC_MAX_WINSHIFT \ ((1 << UBC_WINSHIFT) > MAX_PAGE_SIZE ? UBC_WINSHIFT : ilog2(MAX_PAGE_SIZE)) int ubc_nwins = UBC_NWINS; const int ubc_winshift = UBC_MAX_WINSHIFT; const int ubc_winsize = 1 << UBC_MAX_WINSHIFT; #if defined(PMAP_PREFER) int ubc_nqueues; #define UBC_NQUEUES ubc_nqueues #else #define UBC_NQUEUES 1 #endif #if defined(UBC_STATS) #define UBC_EVCNT_DEFINE(name) \ struct evcnt ubc_evcnt_##name = \ EVCNT_INITIALIZER(EVCNT_TYPE_MISC, NULL, "ubc", #name); \ EVCNT_ATTACH_STATIC(ubc_evcnt_##name); #define UBC_EVCNT_INCR(name) ubc_evcnt_##name.ev_count++ #else /* defined(UBC_STATS) */ #define UBC_EVCNT_DEFINE(name) /* nothing */ #define UBC_EVCNT_INCR(name) /* nothing */ #endif /* defined(UBC_STATS) */ UBC_EVCNT_DEFINE(wincachehit) UBC_EVCNT_DEFINE(wincachemiss) UBC_EVCNT_DEFINE(faultbusy) /* * ubc_init * * init pager private data structures. */ void ubc_init(void) { /* * Make sure ubc_winshift is sane. */ KASSERT(ubc_winshift >= PAGE_SHIFT); /* * init ubc_object. * alloc and init ubc_map's. * init inactive queues. * alloc and init hashtable. * map in ubc_object. */ uvm_obj_init(&ubc_object.uobj, &ubc_pager, true, UVM_OBJ_KERN); ubc_object.umap = kmem_zalloc(ubc_nwins * sizeof(struct ubc_map), KM_SLEEP); if (ubc_object.umap == NULL) panic("ubc_init: failed to allocate ubc_map"); vaddr_t va = (vaddr_t)1L; #ifdef PMAP_PREFER PMAP_PREFER(0, &va, 0, 0); /* kernel is never topdown */ ubc_nqueues = va >> ubc_winshift; if (ubc_nqueues == 0) { ubc_nqueues = 1; } #endif ubc_object.inactive = kmem_alloc(UBC_NQUEUES * sizeof(struct ubc_inactive_head), KM_SLEEP); for (int i = 0; i < UBC_NQUEUES; i++) { TAILQ_INIT(&ubc_object.inactive[i]); } for (int i = 0; i < ubc_nwins; i++) { struct ubc_map *umap; umap = &ubc_object.umap[i]; TAILQ_INSERT_TAIL(&ubc_object.inactive[i & (UBC_NQUEUES - 1)], umap, inactive); } ubc_object.hash = hashinit(ubc_nwins, HASH_LIST, true, &ubc_object.hashmask); for (int i = 0; i <= ubc_object.hashmask; i++) { LIST_INIT(&ubc_object.hash[i]); } if (uvm_map(kernel_map, (vaddr_t *)&ubc_object.kva, ubc_nwins << ubc_winshift, &ubc_object.uobj, 0, (vsize_t)va, UVM_MAPFLAG(UVM_PROT_RW, UVM_PROT_RW, UVM_INH_NONE, UVM_ADV_RANDOM, UVM_FLAG_NOMERGE)) != 0) { panic("ubc_init: failed to map ubc_object"); } hashstat_register("ubchash", ubchash_stats); } void ubchist_init(void) { UVMHIST_INIT(ubchist, 300); } /* * ubc_fault_page: helper of ubc_fault to handle a single page. * * => Caller has UVM object locked. * => Caller will perform pmap_update(). */ static inline int ubc_fault_page(const struct uvm_faultinfo *ufi, const struct ubc_map *umap, struct vm_page *pg, vm_prot_t prot, vm_prot_t access_type, vaddr_t va) { vm_prot_t mask; int error; bool rdonly; KASSERT(rw_write_held(pg->uobject->vmobjlock)); KASSERT((pg->flags & PG_FAKE) == 0); if (pg->flags & PG_RELEASED) { uvm_pagefree(pg); return 0; } if (pg->loan_count != 0) { /* * Avoid unneeded loan break, if possible. */ if ((access_type & VM_PROT_WRITE) == 0) { prot &= ~VM_PROT_WRITE; } if (prot & VM_PROT_WRITE) { struct vm_page *newpg; newpg = uvm_loanbreak(pg); if (newpg == NULL) { uvm_page_unbusy(&pg, 1); return ENOMEM; } pg = newpg; } } /* * Note that a page whose backing store is partially allocated * is marked as PG_RDONLY. * * it's a responsibility of ubc_alloc's caller to allocate backing * blocks before writing to the window. */ KASSERT((pg->flags & PG_RDONLY) == 0 || (access_type & VM_PROT_WRITE) == 0 || pg->offset < umap->writeoff || pg->offset + PAGE_SIZE > umap->writeoff + umap->writelen); rdonly = uvm_pagereadonly_p(pg); mask = rdonly ? ~VM_PROT_WRITE : VM_PROT_ALL; error = pmap_enter(ufi->orig_map->pmap, va, VM_PAGE_TO_PHYS(pg), prot & mask, PMAP_CANFAIL | (access_type & mask)); uvm_pagelock(pg); uvm_pageactivate(pg); uvm_pagewakeup(pg); uvm_pageunlock(pg); pg->flags &= ~PG_BUSY; UVM_PAGE_OWN(pg, NULL); return error; } /* * ubc_fault: fault routine for ubc mapping */ static int ubc_fault(struct uvm_faultinfo *ufi, vaddr_t ign1, struct vm_page **ign2, int ign3, int ign4, vm_prot_t access_type, int flags) { struct uvm_object *uobj; struct ubc_map *umap; vaddr_t va, eva, ubc_offset, slot_offset; struct vm_page *pgs[howmany(ubc_winsize, MIN_PAGE_SIZE)]; int i, error, npages; vm_prot_t prot; UVMHIST_FUNC(__func__); UVMHIST_CALLED(ubchist); /* * no need to try with PGO_LOCKED... * we don't need to have the map locked since we know that * no one will mess with it until our reference is released. */ if (flags & PGO_LOCKED) { uvmfault_unlockall(ufi, NULL, &ubc_object.uobj); flags &= ~PGO_LOCKED; } va = ufi->orig_rvaddr; ubc_offset = va - (vaddr_t)ubc_object.kva; umap = &ubc_object.umap[ubc_offset >> ubc_winshift]; KASSERT(umap->refcount != 0); KASSERT((umap->flags & UMAP_PAGES_LOCKED) == 0); slot_offset = ubc_offset & (ubc_winsize - 1); /* * some platforms cannot write to individual bytes atomically, so * software has to do read/modify/write of larger quantities instead. * this means that the access_type for "write" operations * can be VM_PROT_READ, which confuses us mightily. * * deal with this by resetting access_type based on the info * that ubc_alloc() stores for us. */ access_type = umap->writelen ? VM_PROT_WRITE : VM_PROT_READ; UVMHIST_LOG(ubchist, "va %#jx ubc_offset %#jx access_type %jd", va, ubc_offset, access_type, 0); if ((access_type & VM_PROT_WRITE) != 0) { #ifndef PRIxOFF /* XXX */ #define PRIxOFF "jx" /* XXX */ #endif /* XXX */ KASSERTMSG((trunc_page(umap->writeoff) <= slot_offset), "out of range write: slot=%#"PRIxVSIZE" off=%#"PRIxOFF, slot_offset, (intmax_t)umap->writeoff); KASSERTMSG((slot_offset < umap->writeoff + umap->writelen), "out of range write: slot=%#"PRIxVADDR " off=%#"PRIxOFF" len=%#"PRIxVSIZE, slot_offset, (intmax_t)umap->writeoff, umap->writelen); } /* no umap locking needed since we have a ref on the umap */ uobj = umap->uobj; if ((access_type & VM_PROT_WRITE) == 0) { npages = (ubc_winsize - slot_offset) >> PAGE_SHIFT; } else { npages = (round_page(umap->offset + umap->writeoff + umap->writelen) - (umap->offset + slot_offset)) >> PAGE_SHIFT; flags |= PGO_PASTEOF; } again: memset(pgs, 0, sizeof (pgs)); rw_enter(uobj->vmobjlock, RW_WRITER); UVMHIST_LOG(ubchist, "slot_offset %#jx writeoff %#jx writelen %#jx ", slot_offset, umap->writeoff, umap->writelen, 0); UVMHIST_LOG(ubchist, "getpages uobj %#jx offset %#jx npages %jd", (uintptr_t)uobj, umap->offset + slot_offset, npages, 0); error = (*uobj->pgops->pgo_get)(uobj, umap->offset + slot_offset, pgs, &npages, 0, access_type, umap->advice, flags | PGO_NOBLOCKALLOC | PGO_NOTIMESTAMP); UVMHIST_LOG(ubchist, "getpages error %jd npages %jd", error, npages, 0, 0); if (error == EAGAIN) { kpause("ubc_fault", false, hz >> 2, NULL); goto again; } if (error) { return error; } /* * For virtually-indexed, virtually-tagged caches we should avoid * creating writable mappings when we do not absolutely need them, * since the "compatible alias" trick does not work on such caches. * Otherwise, we can always map the pages writable. */ #ifdef PMAP_CACHE_VIVT prot = VM_PROT_READ | access_type; #else prot = VM_PROT_READ | VM_PROT_WRITE; #endif va = ufi->orig_rvaddr; eva = ufi->orig_rvaddr + (npages << PAGE_SHIFT); UVMHIST_LOG(ubchist, "va %#jx eva %#jx", va, eva, 0, 0); /* * Note: normally all returned pages would have the same UVM object. * However, layered file-systems and e.g. tmpfs, may return pages * which belong to underlying UVM object. In such case, lock is * shared amongst the objects. */ rw_enter(uobj->vmobjlock, RW_WRITER); for (i = 0; va < eva; i++, va += PAGE_SIZE) { struct vm_page *pg; UVMHIST_LOG(ubchist, "pgs[%jd] = %#jx", i, (uintptr_t)pgs[i], 0, 0); pg = pgs[i]; if (pg == NULL || pg == PGO_DONTCARE) { continue; } KASSERT(uobj->vmobjlock == pg->uobject->vmobjlock); error = ubc_fault_page(ufi, umap, pg, prot, access_type, va); if (error) { /* * Flush (there might be pages entered), drop the lock, * and perform uvm_wait(). Note: page will re-fault. */ pmap_update(ufi->orig_map->pmap); rw_exit(uobj->vmobjlock); uvm_wait("ubc_fault"); rw_enter(uobj->vmobjlock, RW_WRITER); } } /* Must make VA visible before the unlock. */ pmap_update(ufi->orig_map->pmap); rw_exit(uobj->vmobjlock); return 0; } /* * local functions */ static struct ubc_map * ubc_find_mapping(struct uvm_object *uobj, voff_t offset) { struct ubc_map *umap; LIST_FOREACH(umap, &ubc_object.hash[UBC_HASH(uobj, offset)], hash) { if (umap->uobj == uobj && umap->offset == offset) { return umap; } } return NULL; } /* * ubc interface functions */ /* * ubc_alloc: allocate a file mapping window */ static void * __noinline ubc_alloc(struct uvm_object *uobj, voff_t offset, vsize_t *lenp, int advice, int flags, struct vm_page **pgs, int *npagesp) { vaddr_t slot_offset, va; struct ubc_map *umap; voff_t umap_offset; int error; UVMHIST_FUNC(__func__); UVMHIST_CALLARGS(ubchist, "uobj %#jx offset %#jx len %#jx", (uintptr_t)uobj, offset, *lenp, 0); KASSERT(*lenp > 0); umap_offset = (offset & ~((voff_t)ubc_winsize - 1)); slot_offset = (vaddr_t)(offset & ((voff_t)ubc_winsize - 1)); *lenp = MIN(*lenp, ubc_winsize - slot_offset); KASSERT(*lenp > 0); rw_enter(ubc_object.uobj.vmobjlock, RW_WRITER); again: /* * The UVM object is already referenced. * Lock order: UBC object -> ubc_map::uobj. */ umap = ubc_find_mapping(uobj, umap_offset); if (umap == NULL) { struct uvm_object *oobj; UBC_EVCNT_INCR(wincachemiss); umap = TAILQ_FIRST(UBC_QUEUE(offset)); if (umap == NULL) { rw_exit(ubc_object.uobj.vmobjlock); kpause("ubc_alloc", false, hz >> 2, NULL); rw_enter(ubc_object.uobj.vmobjlock, RW_WRITER); goto again; } va = UBC_UMAP_ADDR(umap); oobj = umap->uobj; /* * Remove from old hash (if any), add to new hash. */ if (oobj != NULL) { /* * Mapping must be removed before the list entry, * since there is a race with ubc_purge(). */ if (umap->flags & UMAP_MAPPING_CACHED) { umap->flags &= ~UMAP_MAPPING_CACHED; rw_enter(oobj->vmobjlock, RW_WRITER); pmap_remove(pmap_kernel(), va, va + ubc_winsize); pmap_update(pmap_kernel()); rw_exit(oobj->vmobjlock); } LIST_REMOVE(umap, hash); LIST_REMOVE(umap, list); } else { KASSERT((umap->flags & UMAP_MAPPING_CACHED) == 0); } umap->uobj = uobj; umap->offset = umap_offset; LIST_INSERT_HEAD(&ubc_object.hash[UBC_HASH(uobj, umap_offset)], umap, hash); LIST_INSERT_HEAD(&uobj->uo_ubc, umap, list); } else { UBC_EVCNT_INCR(wincachehit); va = UBC_UMAP_ADDR(umap); } if (umap->refcount == 0) { TAILQ_REMOVE(UBC_QUEUE(offset), umap, inactive); } if (flags & UBC_WRITE) { KASSERTMSG(umap->writeoff == 0 && umap->writelen == 0, "ubc_alloc: concurrent writes to uobj %p", uobj); umap->writeoff = slot_offset; umap->writelen = *lenp; } umap->refcount++; umap->advice = advice; rw_exit(ubc_object.uobj.vmobjlock); UVMHIST_LOG(ubchist, "umap %#jx refs %jd va %#jx flags %#jx", (uintptr_t)umap, umap->refcount, (uintptr_t)va, flags); if (flags & UBC_FAULTBUSY) { int npages = (*lenp + (offset & (PAGE_SIZE - 1)) + PAGE_SIZE - 1) >> PAGE_SHIFT; int gpflags = PGO_SYNCIO|PGO_OVERWRITE|PGO_PASTEOF|PGO_NOBLOCKALLOC| PGO_NOTIMESTAMP; int i; KDASSERT(flags & UBC_WRITE); KASSERT(npages <= *npagesp); KASSERT(umap->refcount == 1); UBC_EVCNT_INCR(faultbusy); again_faultbusy: rw_enter(uobj->vmobjlock, RW_WRITER); if (umap->flags & UMAP_MAPPING_CACHED) { umap->flags &= ~UMAP_MAPPING_CACHED; pmap_remove(pmap_kernel(), va, va + ubc_winsize); } memset(pgs, 0, *npagesp * sizeof(pgs[0])); error = (*uobj->pgops->pgo_get)(uobj, trunc_page(offset), pgs, &npages, 0, VM_PROT_READ | VM_PROT_WRITE, advice, gpflags); UVMHIST_LOG(ubchist, "faultbusy getpages %jd", error, 0, 0, 0); if (error) { /* * Flush: the mapping above might have been removed. */ pmap_update(pmap_kernel()); goto out; } for (i = 0; i < npages; i++) { struct vm_page *pg = pgs[i]; KASSERT(pg->uobject == uobj); if (pg->loan_count != 0) { rw_enter(uobj->vmobjlock, RW_WRITER); if (pg->loan_count != 0) { pg = uvm_loanbreak(pg); } if (pg == NULL) { pmap_kremove(va, ubc_winsize); pmap_update(pmap_kernel()); uvm_page_unbusy(pgs, npages); rw_exit(uobj->vmobjlock); uvm_wait("ubc_alloc"); goto again_faultbusy; } rw_exit(uobj->vmobjlock); pgs[i] = pg; } pmap_kenter_pa( va + trunc_page(slot_offset) + (i << PAGE_SHIFT), VM_PAGE_TO_PHYS(pg), VM_PROT_READ | VM_PROT_WRITE, 0); } pmap_update(pmap_kernel()); umap->flags |= UMAP_PAGES_LOCKED; *npagesp = npages; } else { KASSERT((umap->flags & UMAP_PAGES_LOCKED) == 0); } out: return (void *)(va + slot_offset); } /* * ubc_release: free a file mapping window. */ static void __noinline ubc_release(void *va, int flags, struct vm_page **pgs, int npages) { struct ubc_map *umap; struct uvm_object *uobj; vaddr_t umapva; bool unmapped; UVMHIST_FUNC(__func__); UVMHIST_CALLARGS(ubchist, "va %#jx", (uintptr_t)va, 0, 0, 0); umap = &ubc_object.umap[((char *)va - ubc_object.kva) >> ubc_winshift]; umapva = UBC_UMAP_ADDR(umap); uobj = umap->uobj; KASSERT(uobj != NULL); if (umap->flags & UMAP_PAGES_LOCKED) { const voff_t endoff = umap->writeoff + umap->writelen; const voff_t zerolen = round_page(endoff) - endoff; KASSERT(npages == (round_page(endoff) - trunc_page(umap->writeoff)) >> PAGE_SHIFT); KASSERT((umap->flags & UMAP_MAPPING_CACHED) == 0); if (zerolen) { memset((char *)umapva + endoff, 0, zerolen); } umap->flags &= ~UMAP_PAGES_LOCKED; rw_enter(uobj->vmobjlock, RW_WRITER); for (u_int i = 0; i < npages; i++) { struct vm_page *pg = pgs[i]; #ifdef DIAGNOSTIC paddr_t pa; bool rv; rv = pmap_extract(pmap_kernel(), umapva + umap->writeoff + (i << PAGE_SHIFT), &pa); KASSERT(rv); KASSERT(PHYS_TO_VM_PAGE(pa) == pg); #endif pg->flags &= ~PG_FAKE; KASSERTMSG(uvm_pagegetdirty(pg) == UVM_PAGE_STATUS_DIRTY, "page %p not dirty", pg); KASSERT(pg->loan_count == 0); if (uvmpdpol_pageactivate_p(pg)) { uvm_pagelock(pg); uvm_pageactivate(pg); uvm_pageunlock(pg); } } pmap_kremove(umapva, ubc_winsize); pmap_update(pmap_kernel()); uvm_page_unbusy(pgs, npages); rw_exit(uobj->vmobjlock); unmapped = true; } else { unmapped = false; } rw_enter(ubc_object.uobj.vmobjlock, RW_WRITER); umap->writeoff = 0; umap->writelen = 0; umap->refcount--; if (umap->refcount == 0) { if (flags & UBC_UNMAP) { /* * Invalidate any cached mappings if requested. * This is typically used to avoid leaving * incompatible cache aliases around indefinitely. */ rw_enter(uobj->vmobjlock, RW_WRITER); pmap_remove(pmap_kernel(), umapva, umapva + ubc_winsize); pmap_update(pmap_kernel()); rw_exit(uobj->vmobjlock); umap->flags &= ~UMAP_MAPPING_CACHED; LIST_REMOVE(umap, hash); LIST_REMOVE(umap, list); umap->uobj = NULL; TAILQ_INSERT_HEAD(UBC_QUEUE(umap->offset), umap, inactive); } else { if (!unmapped) { umap->flags |= UMAP_MAPPING_CACHED; } TAILQ_INSERT_TAIL(UBC_QUEUE(umap->offset), umap, inactive); } } UVMHIST_LOG(ubchist, "umap %#jx refs %jd", (uintptr_t)umap, umap->refcount, 0, 0); rw_exit(ubc_object.uobj.vmobjlock); } /* * ubc_uiomove: move data to/from an object. */ int ubc_uiomove(struct uvm_object *uobj, struct uio *uio, vsize_t todo, int advice, int flags) { const bool overwrite = (flags & UBC_FAULTBUSY) != 0; struct vm_page *pgs[howmany(ubc_winsize, MIN_PAGE_SIZE)]; voff_t off; int error, npages; KASSERT(todo <= uio->uio_resid); KASSERT(((flags & UBC_WRITE) != 0 && uio->uio_rw == UIO_WRITE) || ((flags & UBC_READ) != 0 && uio->uio_rw == UIO_READ)); #ifdef UBC_USE_PMAP_DIRECT /* * during direct access pages need to be held busy to prevent them * changing identity, and therefore if we read or write an object * into a mapped view of same we could deadlock while faulting. * * avoid the problem by disallowing direct access if the object * might be visible somewhere via mmap(). * * XXX concurrent reads cause thundering herd issues with PG_BUSY. * In the future enable by default for writes or if ncpu<=2, and * make the toggle override that. */ if ((ubc_direct && (flags & UBC_ISMAPPED) == 0) || (flags & UBC_FAULTBUSY) != 0) { return ubc_uiomove_direct(uobj, uio, todo, advice, flags); } #endif off = uio->uio_offset; error = 0; while (todo > 0) { vsize_t bytelen = todo; void *win; npages = __arraycount(pgs); win = ubc_alloc(uobj, off, &bytelen, advice, flags, pgs, &npages); if (error == 0) { error = uiomove(win, bytelen, uio); } if (error != 0 && overwrite) { /* * if we haven't initialized the pages yet, * do it now. it's safe to use memset here * because we just mapped the pages above. */ memset(win, 0, bytelen); } ubc_release(win, flags, pgs, npages); off += bytelen; todo -= bytelen; if (error != 0 && (flags & UBC_PARTIALOK) != 0) { break; } } return error; } /* * ubc_zerorange: set a range of bytes in an object to zero. */ void ubc_zerorange(struct uvm_object *uobj, off_t off, size_t len, int flags) { struct vm_page *pgs[howmany(ubc_winsize, MIN_PAGE_SIZE)]; int npages; #ifdef UBC_USE_PMAP_DIRECT if (ubc_direct || (flags & UBC_FAULTBUSY) != 0) { ubc_zerorange_direct(uobj, off, len, flags); return; } #endif /* * XXXUBC invent kzero() and use it */ while (len) { void *win; vsize_t bytelen = len; npages = __arraycount(pgs); win = ubc_alloc(uobj, off, &bytelen, UVM_ADV_NORMAL, UBC_WRITE, pgs, &npages); memset(win, 0, bytelen); ubc_release(win, flags, pgs, npages); off += bytelen; len -= bytelen; } } #ifdef UBC_USE_PMAP_DIRECT /* Copy data using direct map */ /* * ubc_alloc_direct: allocate a file mapping window using direct map */ static int __noinline ubc_alloc_direct(struct uvm_object *uobj, voff_t offset, vsize_t *lenp, int advice, int flags, struct vm_page **pgs, int *npages) { voff_t pgoff; int error; int gpflags = flags | PGO_NOTIMESTAMP | PGO_SYNCIO; int access_type = VM_PROT_READ; UVMHIST_FUNC(__func__); UVMHIST_CALLED(ubchist); if (flags & UBC_WRITE) { if (flags & UBC_FAULTBUSY) gpflags |= PGO_OVERWRITE | PGO_NOBLOCKALLOC; #if 0 KASSERT(!UVM_OBJ_NEEDS_WRITEFAULT(uobj)); #endif /* * Tell genfs_getpages() we already have the journal lock, * allow allocation past current EOF. */ gpflags |= PGO_JOURNALLOCKED | PGO_PASTEOF; access_type |= VM_PROT_WRITE; } else { /* Don't need the empty blocks allocated, PG_RDONLY is okay */ gpflags |= PGO_NOBLOCKALLOC; } pgoff = (offset & PAGE_MASK); *lenp = MIN(*lenp, ubc_winsize - pgoff); again: *npages = (*lenp + pgoff + PAGE_SIZE - 1) >> PAGE_SHIFT; KASSERT((*npages * PAGE_SIZE) <= ubc_winsize); KASSERT(*lenp + pgoff <= ubc_winsize); memset(pgs, 0, *npages * sizeof(pgs[0])); rw_enter(uobj->vmobjlock, RW_WRITER); error = (*uobj->pgops->pgo_get)(uobj, trunc_page(offset), pgs, npages, 0, access_type, advice, gpflags); UVMHIST_LOG(ubchist, "alloc_direct getpages %jd", error, 0, 0, 0); if (error) { if (error == EAGAIN) { kpause("ubc_alloc_directg", false, hz >> 2, NULL); goto again; } return error; } rw_enter(uobj->vmobjlock, RW_WRITER); for (int i = 0; i < *npages; i++) { struct vm_page *pg = pgs[i]; KASSERT(pg != NULL); KASSERT(pg != PGO_DONTCARE); KASSERT((pg->flags & PG_FAKE) == 0 || (gpflags & PGO_OVERWRITE)); KASSERT(pg->uobject->vmobjlock == uobj->vmobjlock); /* Avoid breaking loan if possible, only do it on write */ if ((flags & UBC_WRITE) && pg->loan_count != 0) { pg = uvm_loanbreak(pg); if (pg == NULL) { uvm_page_unbusy(pgs, *npages); rw_exit(uobj->vmobjlock); uvm_wait("ubc_alloc_directl"); goto again; } pgs[i] = pg; } /* Page must be writable by now */ KASSERT((pg->flags & PG_RDONLY) == 0 || (flags & UBC_WRITE) == 0); /* * XXX For aobj pages. No managed mapping - mark the page * dirty. */ if ((flags & UBC_WRITE) != 0) { uvm_pagemarkdirty(pg, UVM_PAGE_STATUS_DIRTY); } } rw_exit(uobj->vmobjlock); return 0; } static void __noinline ubc_direct_release(struct uvm_object *uobj, int flags, struct vm_page **pgs, int npages) { rw_enter(uobj->vmobjlock, RW_WRITER); for (int i = 0; i < npages; i++) { struct vm_page *pg = pgs[i]; pg->flags &= ~PG_BUSY; UVM_PAGE_OWN(pg, NULL); if (pg->flags & PG_RELEASED) { pg->flags &= ~PG_RELEASED; uvm_pagefree(pg); continue; } if (uvm_pagewanted_p(pg) || uvmpdpol_pageactivate_p(pg)) { uvm_pagelock(pg); uvm_pageactivate(pg); uvm_pagewakeup(pg); uvm_pageunlock(pg); } /* Page was changed, no longer fake and neither clean. */ if (flags & UBC_WRITE) { KASSERTMSG(uvm_pagegetdirty(pg) == UVM_PAGE_STATUS_DIRTY, "page %p not dirty", pg); pg->flags &= ~PG_FAKE; } } rw_exit(uobj->vmobjlock); } static int ubc_uiomove_process(void *win, size_t len, void *arg) { struct uio *uio = (struct uio *)arg; return uiomove(win, len, uio); } static int ubc_zerorange_process(void *win, size_t len, void *arg) { memset(win, 0, len); return 0; } static int __noinline ubc_uiomove_direct(struct uvm_object *uobj, struct uio *uio, vsize_t todo, int advice, int flags) { const bool overwrite = (flags & UBC_FAULTBUSY) != 0; voff_t off; int error, npages; struct vm_page *pgs[howmany(ubc_winsize, MIN_PAGE_SIZE)]; KASSERT(todo <= uio->uio_resid); KASSERT(((flags & UBC_WRITE) != 0 && uio->uio_rw == UIO_WRITE) || ((flags & UBC_READ) != 0 && uio->uio_rw == UIO_READ)); off = uio->uio_offset; error = 0; while (todo > 0) { vsize_t bytelen = todo; error = ubc_alloc_direct(uobj, off, &bytelen, advice, flags, pgs, &npages); if (error != 0) { /* can't do anything, failed to get the pages */ break; } if (error == 0) { error = uvm_direct_process(pgs, npages, off, bytelen, ubc_uiomove_process, uio); } if (overwrite) { voff_t endoff; /* * if we haven't initialized the pages yet due to an * error above, do it now. */ if (error != 0) { (void) uvm_direct_process(pgs, npages, off, bytelen, ubc_zerorange_process, NULL); } off += bytelen; todo -= bytelen; endoff = off & (PAGE_SIZE - 1); /* * zero out the remaining portion of the final page * (if any). */ if (todo == 0 && endoff != 0) { vsize_t zlen = PAGE_SIZE - endoff; (void) uvm_direct_process(pgs + npages - 1, 1, off, zlen, ubc_zerorange_process, NULL); } } else { off += bytelen; todo -= bytelen; } ubc_direct_release(uobj, flags, pgs, npages); if (error != 0 && ISSET(flags, UBC_PARTIALOK)) { break; } } return error; } static void __noinline ubc_zerorange_direct(struct uvm_object *uobj, off_t off, size_t todo, int flags) { int error, npages; struct vm_page *pgs[howmany(ubc_winsize, MIN_PAGE_SIZE)]; flags |= UBC_WRITE; error = 0; while (todo > 0) { vsize_t bytelen = todo; error = ubc_alloc_direct(uobj, off, &bytelen, UVM_ADV_NORMAL, flags, pgs, &npages); if (error != 0) { /* can't do anything, failed to get the pages */ break; } error = uvm_direct_process(pgs, npages, off, bytelen, ubc_zerorange_process, NULL); ubc_direct_release(uobj, flags, pgs, npages); off += bytelen; todo -= bytelen; } } #endif /* UBC_USE_PMAP_DIRECT */ /* * ubc_purge: disassociate ubc_map structures from an empty uvm_object. */ void ubc_purge(struct uvm_object *uobj) { struct ubc_map *umap; vaddr_t va; KASSERT(uobj->uo_npages == 0); /* * Safe to check without lock held, as ubc_alloc() removes * the mapping and list entry in the correct order. */ if (__predict_true(LIST_EMPTY(&uobj->uo_ubc))) { return; } rw_enter(ubc_object.uobj.vmobjlock, RW_WRITER); while ((umap = LIST_FIRST(&uobj->uo_ubc)) != NULL) { KASSERT(umap->refcount == 0); for (va = 0; va < ubc_winsize; va += PAGE_SIZE) { KASSERT(!pmap_extract(pmap_kernel(), va + UBC_UMAP_ADDR(umap), NULL)); } LIST_REMOVE(umap, list); LIST_REMOVE(umap, hash); umap->flags &= ~UMAP_MAPPING_CACHED; umap->uobj = NULL; } rw_exit(ubc_object.uobj.vmobjlock); } static int ubchash_stats(struct hashstat_sysctl *hs, bool fill) { struct ubc_map *umap; uint64_t chain; strlcpy(hs->hash_name, "ubchash", sizeof(hs->hash_name)); strlcpy(hs->hash_desc, "ubc object hash", sizeof(hs->hash_desc)); if (!fill) return 0; hs->hash_size = ubc_object.hashmask + 1; for (size_t i = 0; i < hs->hash_size; i++) { chain = 0; rw_enter(ubc_object.uobj.vmobjlock, RW_READER); LIST_FOREACH(umap, &ubc_object.hash[i], hash) { chain++; } rw_exit(ubc_object.uobj.vmobjlock); if (chain > 0) { hs->hash_used++; hs->hash_items += chain; if (chain > hs->hash_maxchain) hs->hash_maxchain = chain; } preempt_point(); } return 0; } |
| 5 1 4 13 2 11 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 | /* $NetBSD: kern_resource_43.c,v 1.23 2021/09/07 11:43:02 riastradh Exp $ */ /*- * Copyright (c) 1982, 1986, 1991, 1993 * The Regents of the University of California. All rights reserved. * (c) UNIX System Laboratories, Inc. * All or some portions of this file are derived from material licensed * to the University of California by American Telephone and Telegraph * Co. or Unix System Laboratories, Inc. and are reproduced herein with * the permission of UNIX System Laboratories, Inc. * * Redistribution and use in source and binary forms, with or without * modification, are permitted provided that the following conditions * are met: * 1. Redistributions of source code must retain the above copyright * notice, this list of conditions and the following disclaimer. * 2. Redistributions in binary form must reproduce the above copyright * notice, this list of conditions and the following disclaimer in the * documentation and/or other materials provided with the distribution. * 3. Neither the name of the University nor the names of its contributors * may be used to endorse or promote products derived from this software * without specific prior written permission. * * THIS SOFTWARE IS PROVIDED BY THE REGENTS AND CONTRIBUTORS ``AS IS'' AND * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE * ARE DISCLAIMED. IN NO EVENT SHALL THE REGENTS OR CONTRIBUTORS BE LIABLE * FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL * DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS * OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) * HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT * LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY * OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF * SUCH DAMAGE. * * @(#)kern_resource.c 8.5 (Berkeley) 1/21/94 */ #include <sys/cdefs.h> __KERNEL_RCSID(0, "$NetBSD: kern_resource_43.c,v 1.23 2021/09/07 11:43:02 riastradh Exp $"); #if defined(_KERNEL_OPT) #include "opt_compat_netbsd.h" #endif #include <sys/param.h> #include <sys/systm.h> #include <sys/kernel.h> #include <sys/file.h> #include <sys/resourcevar.h> #include <sys/proc.h> #include <sys/mount.h> #include <sys/syscall.h> #include <sys/syscallvar.h> #include <sys/syscallargs.h> #include <compat/common/compat_mod.h> static struct syscall_package kern_resource_43_syscalls[] = { { SYS_compat_43_ogetrlimit, 0, (sy_call_t *)compat_43_sys_getrlimit }, { SYS_compat_43_osetrlimit, 0, (sy_call_t *)compat_43_sys_setrlimit }, { 0, 0, NULL } }; /* ARGSUSED */ int compat_43_sys_getrlimit(struct lwp *l, const struct compat_43_sys_getrlimit_args *uap, register_t *retval) { /* { syscallarg(int) which; syscallarg(struct orlimit *) rlp; } */ struct proc *p = l->l_proc; int which = SCARG(uap, which); struct orlimit olim; if ((u_int)which >= RLIM_NLIMITS) return (EINVAL); memset(&olim, 0, sizeof(olim)); olim.rlim_cur = p->p_rlimit[which].rlim_cur; if (olim.rlim_cur == -1) olim.rlim_cur = 0x7fffffff; olim.rlim_max = p->p_rlimit[which].rlim_max; if (olim.rlim_max == -1) olim.rlim_max = 0x7fffffff; return copyout(&olim, SCARG(uap, rlp), sizeof(olim)); } /* ARGSUSED */ int compat_43_sys_setrlimit(struct lwp *l, const struct compat_43_sys_setrlimit_args *uap, register_t *retval) { /* { syscallarg(int) which; syscallarg(const struct orlimit *) rlp; } */ int which = SCARG(uap, which); struct orlimit olim; struct rlimit lim; int error; error = copyin(SCARG(uap, rlp), &olim, sizeof(struct orlimit)); if (error) return (error); lim.rlim_cur = olim.rlim_cur; lim.rlim_max = olim.rlim_max; return (dosetrlimit(l, l->l_proc, which, &lim)); } int kern_resource_43_init(void) { return syscall_establish(NULL, kern_resource_43_syscalls); } int kern_resource_43_fini(void) { return syscall_disestablish(NULL, kern_resource_43_syscalls); } |
| 18 1 2 2 1 1 1 1 2 1 3 2 77 77 77 75 2 2 1 1 2 2 1 1 3 3 3 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 197 198 199 200 201 202 203 204 205 206 207 208 209 210 211 212 213 214 215 216 217 218 219 220 221 222 223 224 225 226 227 228 229 230 231 232 233 234 235 236 237 238 239 240 241 242 243 244 245 246 247 248 249 250 251 252 253 254 255 256 257 258 259 260 261 262 263 264 265 266 267 268 269 270 271 272 273 274 275 276 277 278 279 280 281 282 283 284 285 286 287 288 289 290 291 292 293 294 295 296 297 298 299 300 301 302 303 304 305 306 307 308 309 310 311 312 313 314 315 316 317 318 319 320 321 322 323 324 325 326 327 328 329 330 331 332 333 334 335 336 337 338 339 340 341 342 343 344 345 346 347 348 349 350 351 352 353 354 355 356 357 358 359 360 361 362 363 364 365 366 367 368 369 370 371 372 373 374 375 376 377 378 379 380 381 382 383 384 385 386 387 388 389 390 391 392 393 394 395 396 397 398 399 400 401 402 403 404 405 406 407 408 409 410 411 412 413 414 415 416 417 418 419 420 421 422 423 424 425 426 427 428 429 430 431 432 433 434 435 436 437 438 439 440 441 442 443 444 445 446 447 448 449 450 451 452 453 454 455 456 457 458 459 460 461 462 463 464 465 466 467 468 469 470 471 472 473 474 475 476 477 478 479 480 481 482 483 484 485 486 487 488 489 490 491 492 493 494 495 496 497 498 499 500 501 502 503 504 505 506 507 508 509 510 511 512 513 514 515 516 517 518 519 520 521 522 523 524 525 526 527 528 529 530 531 532 533 534 535 536 537 538 539 540 541 542 543 544 545 546 547 548 549 550 551 552 553 554 555 556 557 558 559 560 561 562 563 564 565 566 567 568 569 570 571 572 573 574 575 576 577 578 579 580 581 582 583 584 585 586 587 588 589 590 591 592 | /* $NetBSD: kern_cpu.c,v 1.93 2020/10/08 09:16:13 rin Exp $ */ /*- * Copyright (c) 2007, 2008, 2009, 2010, 2012, 2019 The NetBSD Foundation, Inc. * All rights reserved. * * This code is derived from software contributed to The NetBSD Foundation * by Andrew Doran. * * Redistribution and use in source and binary forms, with or without * modification, are permitted provided that the following conditions * are met: * 1. Redistributions of source code must retain the above copyright * notice, this list of conditions and the following disclaimer. * 2. Redistributions in binary form must reproduce the above copyright * notice, this list of conditions and the following disclaimer in the * documentation and/or other materials provided with the distribution. * * THIS SOFTWARE IS PROVIDED BY THE NETBSD FOUNDATION, INC. AND CONTRIBUTORS * ``AS IS'' AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED * TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR * PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE FOUNDATION OR CONTRIBUTORS * BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR * CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF * SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS * INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN * CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) * ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE * POSSIBILITY OF SUCH DAMAGE. */ /*- * Copyright (c)2007 YAMAMOTO Takashi, * All rights reserved. * * Redistribution and use in source and binary forms, with or without * modification, are permitted provided that the following conditions * are met: * 1. Redistributions of source code must retain the above copyright * notice, this list of conditions and the following disclaimer. * 2. Redistributions in binary form must reproduce the above copyright * notice, this list of conditions and the following disclaimer in the * documentation and/or other materials provided with the distribution. * * THIS SOFTWARE IS PROVIDED BY THE AUTHOR AND CONTRIBUTORS ``AS IS'' AND * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE * ARE DISCLAIMED. IN NO EVENT SHALL THE AUTHOR OR CONTRIBUTORS BE LIABLE * FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL * DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS * OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) * HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT * LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY * OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF * SUCH DAMAGE. */ /* * CPU related routines not shared with rump. */ #include <sys/cdefs.h> __KERNEL_RCSID(0, "$NetBSD: kern_cpu.c,v 1.93 2020/10/08 09:16:13 rin Exp $"); #ifdef _KERNEL_OPT #include "opt_cpu_ucode.h" #endif #include <sys/param.h> #include <sys/systm.h> #include <sys/idle.h> #include <sys/sched.h> #include <sys/intr.h> #include <sys/conf.h> #include <sys/cpu.h> #include <sys/cpuio.h> #include <sys/proc.h> #include <sys/percpu.h> #include <sys/kernel.h> #include <sys/kauth.h> #include <sys/xcall.h> #include <sys/pool.h> #include <sys/kmem.h> #include <sys/select.h> #include <sys/namei.h> #include <sys/callout.h> #include <sys/pcu.h> #include <uvm/uvm_extern.h> #include "ioconf.h" /* * If the port has stated that cpu_data is the first thing in cpu_info, * verify that the claim is true. This will prevent them from getting out * of sync. */ #ifdef __HAVE_CPU_DATA_FIRST CTASSERT(offsetof(struct cpu_info, ci_data) == 0); #else CTASSERT(offsetof(struct cpu_info, ci_data) != 0); #endif int (*compat_cpuctl_ioctl)(struct lwp *, u_long, void *) = (void *)enosys; static void cpu_xc_online(struct cpu_info *, void *); static void cpu_xc_offline(struct cpu_info *, void *); dev_type_ioctl(cpuctl_ioctl); const struct cdevsw cpuctl_cdevsw = { .d_open = nullopen, .d_close = nullclose, .d_read = nullread, .d_write = nullwrite, .d_ioctl = cpuctl_ioctl, .d_stop = nullstop, .d_tty = notty, .d_poll = nopoll, .d_mmap = nommap, .d_kqfilter = nokqfilter, .d_discard = nodiscard, .d_flag = D_OTHER | D_MPSAFE }; int mi_cpu_attach(struct cpu_info *ci) { int error; KASSERT(maxcpus > 0); if ((ci->ci_index = ncpu) >= maxcpus) panic("Too many CPUs. Increase MAXCPUS?"); kcpuset_set(kcpuset_attached, cpu_index(ci)); /* * Create a convenience cpuset of just ourselves. */ kcpuset_create(&ci->ci_data.cpu_kcpuset, true); kcpuset_set(ci->ci_data.cpu_kcpuset, cpu_index(ci)); TAILQ_INIT(&ci->ci_data.cpu_ld_locks); __cpu_simple_lock_init(&ci->ci_data.cpu_ld_lock); /* This is useful for eg, per-cpu evcnt */ snprintf(ci->ci_data.cpu_name, sizeof(ci->ci_data.cpu_name), "cpu%d", cpu_index(ci)); if (__predict_false(cpu_infos == NULL)) { size_t ci_bufsize = (maxcpus + 1) * sizeof(struct cpu_info *); cpu_infos = kmem_zalloc(ci_bufsize, KM_SLEEP); } cpu_infos[cpu_index(ci)] = ci; sched_cpuattach(ci); error = create_idle_lwp(ci); if (error != 0) { /* XXX revert sched_cpuattach */ return error; } if (ci == curcpu()) ci->ci_onproc = curlwp; else ci->ci_onproc = ci->ci_data.cpu_idlelwp; percpu_init_cpu(ci); softint_init(ci); callout_init_cpu(ci); xc_init_cpu(ci); pool_cache_cpu_init(ci); selsysinit(ci); cache_cpu_init(ci); TAILQ_INIT(&ci->ci_data.cpu_biodone); ncpu++; ncpuonline++; return 0; } void cpuctlattach(int dummy __unused) { KASSERT(cpu_infos != NULL); } int cpuctl_ioctl(dev_t dev, u_long cmd, void *data, int flag, lwp_t *l) { CPU_INFO_ITERATOR cii; cpustate_t *cs; struct cpu_info *ci; int error, i; u_int id; error = 0; mutex_enter(&cpu_lock); switch (cmd) { case IOC_CPU_SETSTATE: cs = data; error = kauth_authorize_system(l->l_cred, KAUTH_SYSTEM_CPU, KAUTH_REQ_SYSTEM_CPU_SETSTATE, cs, NULL, NULL); if (error != 0) break; if (cs->cs_id >= maxcpus || (ci = cpu_lookup(cs->cs_id)) == NULL) { error = ESRCH; break; } cpu_setintr(ci, cs->cs_intr); /* XXX neglect errors */ error = cpu_setstate(ci, cs->cs_online); break; case IOC_CPU_GETSTATE: cs = data; id = cs->cs_id; memset(cs, 0, sizeof(*cs)); cs->cs_id = id; if (cs->cs_id >= maxcpus || (ci = cpu_lookup(id)) == NULL) { error = ESRCH; break; } if ((ci->ci_schedstate.spc_flags & SPCF_OFFLINE) != 0) cs->cs_online = false; else cs->cs_online = true; if ((ci->ci_schedstate.spc_flags & SPCF_NOINTR) != 0) cs->cs_intr = false; else cs->cs_intr = true; cs->cs_lastmod = (int32_t)ci->ci_schedstate.spc_lastmod; cs->cs_lastmodhi = (int32_t) (ci->ci_schedstate.spc_lastmod >> 32); cs->cs_intrcnt = cpu_intr_count(ci) + 1; cs->cs_hwid = ci->ci_cpuid; break; case IOC_CPU_MAPID: i = 0; for (CPU_INFO_FOREACH(cii, ci)) { if (i++ == *(int *)data) break; } if (ci == NULL) error = ESRCH; else *(int *)data = cpu_index(ci); break; case IOC_CPU_GETCOUNT: *(int *)data = ncpu; break; #ifdef CPU_UCODE case IOC_CPU_UCODE_GET_VERSION: error = cpu_ucode_get_version((struct cpu_ucode_version *)data); break; case IOC_CPU_UCODE_APPLY: error = kauth_authorize_machdep(l->l_cred, KAUTH_MACHDEP_CPU_UCODE_APPLY, NULL, NULL, NULL, NULL); if (error != 0) break; error = cpu_ucode_apply((const struct cpu_ucode *)data); break; #endif default: error = (*compat_cpuctl_ioctl)(l, cmd, data); break; } mutex_exit(&cpu_lock); return error; } struct cpu_info * cpu_lookup(u_int idx) { struct cpu_info *ci; /* * cpu_infos is a NULL terminated array of MAXCPUS + 1 entries, * so an index of MAXCPUS here is ok. See mi_cpu_attach. */ KASSERT(idx <= maxcpus); if (__predict_false(cpu_infos == NULL)) { KASSERT(idx == 0); return curcpu(); } ci = cpu_infos[idx]; KASSERT(ci == NULL || cpu_index(ci) == idx); KASSERTMSG(idx < maxcpus || ci == NULL, "idx %d ci %p", idx, ci); return ci; } static void cpu_xc_offline(struct cpu_info *ci, void *unused) { struct schedstate_percpu *spc, *mspc = NULL; struct cpu_info *target_ci; struct lwp *l; CPU_INFO_ITERATOR cii; int s; /* * Thread that made the cross call (separate context) holds * cpu_lock on our behalf. */ spc = &ci->ci_schedstate; s = splsched(); spc->spc_flags |= SPCF_OFFLINE; splx(s); /* Take the first available CPU for the migration. */ for (CPU_INFO_FOREACH(cii, target_ci)) { mspc = &target_ci->ci_schedstate; if ((mspc->spc_flags & SPCF_OFFLINE) == 0) break; } KASSERT(target_ci != NULL); /* * Migrate all non-bound threads to the other CPU. Note that this * runs from the xcall thread, thus handling of LSONPROC is not needed. */ mutex_enter(&proc_lock); LIST_FOREACH(l, &alllwp, l_list) { struct cpu_info *mci; lwp_lock(l); if (l->l_cpu != ci || (l->l_pflag & (LP_BOUND | LP_INTR))) { lwp_unlock(l); continue; } /* Regular case - no affinity. */ if (l->l_affinity == NULL) { lwp_migrate(l, target_ci); continue; } /* Affinity is set, find an online CPU in the set. */ for (CPU_INFO_FOREACH(cii, mci)) { mspc = &mci->ci_schedstate; if ((mspc->spc_flags & SPCF_OFFLINE) == 0 && kcpuset_isset(l->l_affinity, cpu_index(mci))) break; } if (mci == NULL) { lwp_unlock(l); mutex_exit(&proc_lock); goto fail; } lwp_migrate(l, mci); } mutex_exit(&proc_lock); #if PCU_UNIT_COUNT > 0 pcu_save_all_on_cpu(); #endif #ifdef __HAVE_MD_CPU_OFFLINE cpu_offline_md(); #endif return; fail: /* Just unset the SPCF_OFFLINE flag, caller will check */ s = splsched(); spc->spc_flags &= ~SPCF_OFFLINE; splx(s); } static void cpu_xc_online(struct cpu_info *ci, void *unused) { struct schedstate_percpu *spc; int s; spc = &ci->ci_schedstate; s = splsched(); spc->spc_flags &= ~SPCF_OFFLINE; splx(s); } int cpu_setstate(struct cpu_info *ci, bool online) { struct schedstate_percpu *spc; CPU_INFO_ITERATOR cii; struct cpu_info *ci2; uint64_t where; xcfunc_t func; int nonline; spc = &ci->ci_schedstate; KASSERT(mutex_owned(&cpu_lock)); if (online) { if ((spc->spc_flags & SPCF_OFFLINE) == 0) return 0; func = (xcfunc_t)cpu_xc_online; } else { if ((spc->spc_flags & SPCF_OFFLINE) != 0) return 0; nonline = 0; /* * Ensure that at least one CPU within the processor set * stays online. Revisit this later. */ for (CPU_INFO_FOREACH(cii, ci2)) { if ((ci2->ci_schedstate.spc_flags & SPCF_OFFLINE) != 0) continue; if (ci2->ci_schedstate.spc_psid != spc->spc_psid) continue; nonline++; } if (nonline == 1) return EBUSY; func = (xcfunc_t)cpu_xc_offline; } where = xc_unicast(0, func, ci, NULL, ci); xc_wait(where); if (online) { KASSERT((spc->spc_flags & SPCF_OFFLINE) == 0); ncpuonline++; } else { if ((spc->spc_flags & SPCF_OFFLINE) == 0) { /* If was not set offline, then it is busy */ return EBUSY; } ncpuonline--; } spc->spc_lastmod = time_second; return 0; } #if defined(__HAVE_INTR_CONTROL) static void cpu_xc_intr(struct cpu_info *ci, void *unused) { struct schedstate_percpu *spc; int s; spc = &ci->ci_schedstate; s = splsched(); spc->spc_flags &= ~SPCF_NOINTR; splx(s); } static void cpu_xc_nointr(struct cpu_info *ci, void *unused) { struct schedstate_percpu *spc; int s; spc = &ci->ci_schedstate; s = splsched(); spc->spc_flags |= SPCF_NOINTR; splx(s); } int cpu_setintr(struct cpu_info *ci, bool intr) { struct schedstate_percpu *spc; CPU_INFO_ITERATOR cii; struct cpu_info *ci2; uint64_t where; xcfunc_t func; int nintr; spc = &ci->ci_schedstate; KASSERT(mutex_owned(&cpu_lock)); if (intr) { if ((spc->spc_flags & SPCF_NOINTR) == 0) return 0; func = (xcfunc_t)cpu_xc_intr; } else { if (CPU_IS_PRIMARY(ci)) /* XXX kern/45117 */ return EINVAL; if ((spc->spc_flags & SPCF_NOINTR) != 0) return 0; /* * Ensure that at least one CPU within the system * is handing device interrupts. */ nintr = 0; for (CPU_INFO_FOREACH(cii, ci2)) { if ((ci2->ci_schedstate.spc_flags & SPCF_NOINTR) != 0) continue; if (ci2 == ci) continue; nintr++; } if (nintr == 0) return EBUSY; func = (xcfunc_t)cpu_xc_nointr; } where = xc_unicast(0, func, ci, NULL, ci); xc_wait(where); if (intr) { KASSERT((spc->spc_flags & SPCF_NOINTR) == 0); } else if ((spc->spc_flags & SPCF_NOINTR) == 0) { /* If was not set offline, then it is busy */ return EBUSY; } /* Direct interrupts away from the CPU and record the change. */ cpu_intr_redistribute(); spc->spc_lastmod = time_second; return 0; } #else /* __HAVE_INTR_CONTROL */ int cpu_setintr(struct cpu_info *ci, bool intr) { return EOPNOTSUPP; } u_int cpu_intr_count(struct cpu_info *ci) { return 0; /* 0 == "don't know" */ } #endif /* __HAVE_INTR_CONTROL */ #ifdef CPU_UCODE int cpu_ucode_load(struct cpu_ucode_softc *sc, const char *fwname) { firmware_handle_t fwh; int error; if (sc->sc_blob != NULL) { firmware_free(sc->sc_blob, sc->sc_blobsize); sc->sc_blob = NULL; sc->sc_blobsize = 0; } error = cpu_ucode_md_open(&fwh, sc->loader_version, fwname); if (error != 0) { #ifdef DEBUG printf("ucode: firmware_open(%s) failed: %i\n", fwname, error); #endif goto err0; } sc->sc_blobsize = firmware_get_size(fwh); if (sc->sc_blobsize == 0) { error = EFTYPE; firmware_close(fwh); goto err0; } sc->sc_blob = firmware_malloc(sc->sc_blobsize); if (sc->sc_blob == NULL) { error = ENOMEM; firmware_close(fwh); goto err0; } error = firmware_read(fwh, 0, sc->sc_blob, sc->sc_blobsize); firmware_close(fwh); if (error != 0) goto err1; return 0; err1: firmware_free(sc->sc_blob, sc->sc_blobsize); sc->sc_blob = NULL; sc->sc_blobsize = 0; err0: return error; } #endif |
| 25 14 58 18 71 717 768 1154 1157 58 56 56 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 197 198 199 200 201 202 203 204 205 206 207 208 209 210 211 212 213 214 215 216 217 218 219 220 221 222 223 224 225 226 227 228 229 230 231 232 233 234 235 236 237 238 239 240 241 242 243 244 245 246 247 248 249 250 251 252 253 254 255 256 257 258 259 260 261 262 263 264 265 266 267 268 269 270 271 272 273 274 275 276 277 278 279 280 281 282 283 284 285 286 287 288 289 290 291 292 293 294 295 296 297 298 299 300 301 302 303 304 305 306 307 308 309 310 311 312 313 314 315 316 317 318 319 320 321 322 323 324 325 326 327 328 329 330 331 332 333 334 335 336 337 338 339 340 341 342 343 344 345 346 347 348 349 350 351 352 353 354 355 356 357 358 359 360 361 362 363 364 365 366 367 368 369 370 371 372 | /* $NetBSD: subr_device.c,v 1.13 2022/03/28 12:38:59 riastradh Exp $ */ /* * Copyright (c) 2006, 2021 The NetBSD Foundation, Inc. * All rights reserved. * * Redistribution and use in source and binary forms, with or without * modification, are permitted provided that the following conditions * are met: * 1. Redistributions of source code must retain the above copyright * notice, this list of conditions and the following disclaimer. * 2. Redistributions in binary form must reproduce the above copyright * notice, this list of conditions and the following disclaimer in the * documentation and/or other materials provided with the distribution. * * THIS SOFTWARE IS PROVIDED BY THE NETBSD FOUNDATION, INC. AND CONTRIBUTORS * ``AS IS'' AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED * TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR * PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE FOUNDATION OR CONTRIBUTORS * BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR * CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF * SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS * INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN * CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) * ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE * POSSIBILITY OF SUCH DAMAGE. */ #include <sys/cdefs.h> __KERNEL_RCSID(0, "$NetBSD: subr_device.c,v 1.13 2022/03/28 12:38:59 riastradh Exp $"); #include <sys/param.h> #include <sys/device.h> #include <sys/device_impl.h> #include <sys/systm.h> #include <sys/device_calls.h> /* Root device. */ device_t root_device; /* * devhandle_t accessors / mutators. */ static bool devhandle_is_valid_internal(const devhandle_t * const handlep) { if (handlep->impl == NULL) { return false; } return handlep->impl->type != DEVHANDLE_TYPE_INVALID; } bool devhandle_is_valid(devhandle_t handle) { return devhandle_is_valid_internal(&handle); } devhandle_t devhandle_invalid(void) { static const devhandle_t invalid_devhandle = { .impl = NULL, .uintptr = 0, }; return invalid_devhandle; } devhandle_type_t devhandle_type(devhandle_t handle) { if (!devhandle_is_valid_internal(&handle)) { return DEVHANDLE_TYPE_INVALID; } return handle.impl->type; } int devhandle_compare(devhandle_t handle1, devhandle_t handle2) { devhandle_type_t type1 = devhandle_type(handle1); devhandle_type_t type2 = devhandle_type(handle2); if (type1 == DEVHANDLE_TYPE_INVALID) { return -1; } if (type2 == DEVHANDLE_TYPE_INVALID) { return 1; } if (type1 < type2) { return -1; } if (type1 > type2) { return 1; } /* For private handles, we also compare the impl pointers. */ if (type1 == DEVHANDLE_TYPE_PRIVATE) { intptr_t impl1 = (intptr_t)handle1.impl; intptr_t impl2 = (intptr_t)handle2.impl; if (impl1 < impl2) { return -1; } if (impl1 > impl2) { return 1; } } if (handle1.integer < handle2.integer) { return -1; } if (handle1.integer > handle2.integer) { return 1; } return 0; } device_call_t devhandle_lookup_device_call(devhandle_t handle, const char *name, devhandle_t *call_handlep) { const struct devhandle_impl *impl; device_call_t call; /* * The back-end can override the handle to use for the call, * if needed. */ *call_handlep = handle; for (impl = handle.impl; impl != NULL; impl = impl->super) { if (impl->lookup_device_call != NULL) { call = impl->lookup_device_call(handle, name, call_handlep); if (call != NULL) { return call; } } } return NULL; } void devhandle_impl_inherit(struct devhandle_impl *impl, const struct devhandle_impl *super) { memcpy(impl, super, sizeof(*impl)); impl->super = super; } /* * Accessor functions for the device_t type. */ devclass_t device_class(device_t dev) { return dev->dv_class; } cfdata_t device_cfdata(device_t dev) { return dev->dv_cfdata; } cfdriver_t device_cfdriver(device_t dev) { return dev->dv_cfdriver; } cfattach_t device_cfattach(device_t dev) { return dev->dv_cfattach; } int device_unit(device_t dev) { return dev->dv_unit; } const char * device_xname(device_t dev) { return dev->dv_xname; } device_t device_parent(device_t dev) { return dev->dv_parent; } bool device_activation(device_t dev, devact_level_t level) { int active_flags; active_flags = DVF_ACTIVE; switch (level) { case DEVACT_LEVEL_FULL: active_flags |= DVF_CLASS_SUSPENDED; /*FALLTHROUGH*/ case DEVACT_LEVEL_DRIVER: active_flags |= DVF_DRIVER_SUSPENDED; /*FALLTHROUGH*/ case DEVACT_LEVEL_BUS: active_flags |= DVF_BUS_SUSPENDED; break; } return (dev->dv_flags & active_flags) == DVF_ACTIVE; } bool device_is_active(device_t dev) { int active_flags; active_flags = DVF_ACTIVE; active_flags |= DVF_CLASS_SUSPENDED; active_flags |= DVF_DRIVER_SUSPENDED; active_flags |= DVF_BUS_SUSPENDED; return (dev->dv_flags & active_flags) == DVF_ACTIVE; } bool device_is_enabled(device_t dev) { return (dev->dv_flags & DVF_ACTIVE) == DVF_ACTIVE; } bool device_has_power(device_t dev) { int active_flags; active_flags = DVF_ACTIVE | DVF_BUS_SUSPENDED; return (dev->dv_flags & active_flags) == DVF_ACTIVE; } int device_locator(device_t dev, u_int locnum) { KASSERT(dev->dv_locators != NULL); return dev->dv_locators[locnum]; } void * device_private(device_t dev) { /* * The reason why device_private(NULL) is allowed is to simplify the * work of a lot of userspace request handlers (i.e., c/bdev * handlers) which grab cfdriver_t->cd_units[n]. * It avoids having them test for it to be NULL and only then calling * device_private. */ return dev == NULL ? NULL : dev->dv_private; } void device_set_private(device_t dev, void *private) { KASSERTMSG(dev->dv_private == NULL, "device_set_private(%p, %p):" " device %s already has private set to %p", dev, private, device_xname(dev), device_private(dev)); KASSERT(private != NULL); dev->dv_private = private; } prop_dictionary_t device_properties(device_t dev) { return dev->dv_properties; } /* * device_is_a: * * Returns true if the device is an instance of the specified * driver. */ bool device_is_a(device_t dev, const char *dname) { if (dev == NULL || dev->dv_cfdriver == NULL) { return false; } return strcmp(dev->dv_cfdriver->cd_name, dname) == 0; } /* * device_attached_to_iattr: * * Returns true if the device attached to the specified interface * attribute. */ bool device_attached_to_iattr(device_t dev, const char *iattr) { cfdata_t cfdata = device_cfdata(dev); const struct cfparent *pspec; if (cfdata == NULL || (pspec = cfdata->cf_pspec) == NULL) { return false; } return strcmp(pspec->cfp_iattr, iattr) == 0; } void device_set_handle(device_t dev, devhandle_t handle) { dev->dv_handle = handle; } devhandle_t device_handle(device_t dev) { return dev->dv_handle; } int device_call_generic(device_t dev, const struct device_call_generic *gen) { devhandle_t handle = device_handle(dev); device_call_t call; devhandle_t call_handle; call = devhandle_lookup_device_call(handle, gen->name, &call_handle); if (call == NULL) { return ENOTSUP; } return call(dev, call_handle, gen->args); } int device_enumerate_children(device_t dev, bool (*callback)(device_t, devhandle_t, void *), void *callback_arg) { struct device_enumerate_children_args args = { .callback = callback, .callback_arg = callback_arg, }; return device_call(dev, DEVICE_ENUMERATE_CHILDREN(&args)); } |
| 8 8 8 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 197 198 199 200 201 202 203 204 205 206 207 208 209 210 211 212 213 214 215 216 217 218 219 220 221 222 223 224 225 226 227 228 229 230 231 232 233 234 235 236 237 238 239 240 241 242 243 244 245 246 247 248 249 250 251 252 253 254 255 256 257 258 259 260 261 262 263 264 265 266 267 268 269 270 271 272 273 274 275 276 277 278 279 280 281 282 283 284 285 286 287 288 289 290 291 292 293 294 295 296 297 298 299 300 301 302 303 304 305 306 307 308 309 310 311 312 313 314 315 316 317 318 319 320 321 322 323 324 325 326 327 328 329 330 331 332 333 334 335 336 337 338 339 340 341 342 343 344 345 346 347 348 349 350 351 352 353 354 355 356 357 358 359 360 361 362 363 364 365 366 367 368 369 370 371 372 373 374 375 376 377 378 379 380 381 382 383 384 385 386 387 388 389 390 391 392 393 394 395 396 397 398 399 400 401 402 403 404 405 406 407 408 409 410 411 412 413 414 415 416 417 418 419 420 421 422 423 424 425 426 427 428 429 430 431 432 433 434 435 436 437 438 439 440 441 442 443 444 445 446 447 448 449 450 451 452 453 454 455 456 457 458 459 460 461 462 463 464 465 466 467 468 469 470 471 472 473 474 475 476 477 478 479 480 481 482 483 484 485 486 487 488 489 490 491 492 493 494 495 496 497 498 499 500 501 502 503 504 505 506 507 508 509 510 511 512 513 514 515 516 517 518 519 520 521 522 523 524 525 526 527 528 529 530 531 532 533 534 535 536 537 538 539 540 541 542 543 544 545 546 547 548 549 550 551 552 553 554 555 556 557 558 559 560 561 562 563 564 565 566 567 568 569 570 571 572 573 574 575 576 577 578 579 580 581 582 583 584 585 586 587 588 589 590 591 592 593 594 595 596 597 598 599 600 601 602 603 604 605 606 607 608 609 610 611 612 613 614 615 616 617 618 619 620 621 622 623 624 625 626 627 628 629 630 631 632 633 634 635 636 637 638 639 640 641 642 643 644 645 646 647 648 649 650 651 652 653 654 655 656 657 658 659 660 661 662 663 664 665 666 667 668 669 670 671 672 673 674 675 676 677 678 679 680 681 682 683 684 685 686 687 688 689 690 691 692 693 694 695 696 697 698 699 700 701 702 703 704 705 706 707 708 709 710 711 712 713 714 715 716 717 718 719 720 721 722 723 724 725 726 727 728 729 730 731 732 733 734 735 736 737 738 739 740 741 742 743 744 745 746 747 748 749 750 751 752 753 754 755 756 757 758 759 760 761 762 763 764 765 766 767 768 769 770 771 772 773 774 775 776 777 778 779 780 781 782 783 784 785 786 787 788 789 790 791 792 793 794 795 796 797 798 799 800 801 802 803 804 805 806 807 808 809 810 811 812 813 814 815 816 817 818 819 820 821 822 823 824 825 826 827 828 829 830 831 832 833 834 835 836 837 838 839 840 841 842 843 844 845 846 847 848 849 850 851 852 853 854 855 856 857 858 859 860 861 862 863 864 865 866 867 868 869 870 871 872 873 874 875 876 877 878 879 880 881 882 883 884 885 886 887 888 889 890 891 892 893 894 895 896 897 898 899 900 901 902 903 904 905 906 907 908 909 910 911 912 913 914 915 916 917 918 919 920 921 922 923 924 925 926 927 928 929 930 931 932 933 934 935 936 937 938 939 940 941 942 943 944 945 946 947 948 949 950 951 952 953 954 955 956 957 958 959 960 961 962 963 964 965 966 967 968 969 970 971 972 973 974 975 976 977 978 979 980 981 982 983 984 985 986 987 988 989 990 991 992 993 994 995 996 997 998 999 1000 1001 1002 1003 1004 1005 1006 1007 1008 1009 1010 1011 1012 1013 1014 1015 1016 1017 1018 1019 1020 1021 1022 1023 1024 1025 1026 1027 1028 1029 1030 1031 1032 1033 1034 1035 1036 1037 1038 1039 1040 1041 1042 1043 1044 1045 1046 1047 1048 1049 1050 1051 1052 1053 1054 1055 1056 1057 1058 1059 1060 1061 1062 1063 1064 1065 1066 1067 1068 1069 1070 1071 1072 1073 1074 1075 1076 1077 1078 1079 1080 1081 1082 1083 1084 1085 1086 1087 1088 1089 1090 1091 1092 1093 1094 1095 1096 1097 1098 1099 1100 1101 1102 1103 1104 1105 1106 1107 1108 1109 1110 1111 1112 1113 1114 1115 1116 1117 1118 1119 1120 1121 1122 1123 1124 1125 1126 1127 1128 1129 1130 1131 1132 1133 1134 1135 1136 1137 1138 1139 1140 1141 1142 1143 1144 1145 1146 1147 1148 1149 1150 1151 1152 1153 1154 1155 1156 1157 1158 1159 1160 1161 1162 1163 1164 1165 1166 1167 1168 1169 1170 1171 1172 1173 1174 1175 1176 1177 1178 1179 1180 1181 1182 1183 1184 1185 1186 1187 1188 1189 1190 1191 1192 1193 1194 1195 1196 1197 1198 1199 1200 1201 1202 1203 1204 1205 1206 1207 1208 1209 1210 1211 1212 1213 1214 1215 1216 1217 1218 1219 1220 1221 1222 1223 1224 1225 1226 1227 1228 1229 1230 1231 1232 1233 1234 1235 1236 1237 1238 1239 1240 1241 1242 1243 1244 1245 1246 1247 1248 1249 1250 1251 1252 1253 1254 1255 1256 1257 1258 1259 1260 1261 1262 1263 1264 1265 1266 1267 1268 1269 1270 1271 1272 1273 1274 1275 1276 1277 1278 1279 1280 1281 1282 1283 1284 1285 1286 1287 1288 1289 1290 1291 1292 1293 1294 1295 1296 1297 1298 1299 1300 1301 1302 1303 1304 1305 1306 1307 1308 1309 1310 1311 1312 1313 1314 1315 1316 1317 1318 1319 1320 1321 1322 1323 1324 1325 1326 1327 1328 1329 1330 1331 1332 1333 1334 1335 1336 1337 1338 1339 1340 1341 1342 1343 1344 1345 1346 1347 1348 1349 1350 1351 1352 1353 1354 1355 1356 1357 1358 1359 1360 1361 1362 1363 1364 1365 1366 1367 1368 1369 1370 1371 1372 1373 1374 1375 1376 1377 1378 1379 1380 1381 1382 1383 1384 1385 1386 1387 1388 1389 1390 1391 1392 1393 1394 1395 1396 1397 1398 1399 1400 1401 1402 1403 1404 1405 1406 1407 1408 1409 1410 1411 1412 1413 1414 1415 1416 1417 1418 1419 1420 1421 1422 1423 1424 1425 1426 1427 1428 1429 1430 1431 1432 1433 1434 1435 1436 1437 1438 1439 1440 1441 1442 1443 1444 1445 1446 1447 1448 1449 1450 1451 1452 1453 1454 1455 1456 1457 1458 1459 1460 1461 1462 1463 1464 1465 1466 1467 1468 1469 1470 1471 1472 1473 1474 1475 1476 1477 1478 1479 1480 1481 1482 1483 1484 1485 1486 1487 1488 1489 1490 1491 1492 1493 1494 1495 1496 1497 1498 1499 1500 1501 1502 1503 1504 1505 1506 1507 1508 1509 1510 1511 1512 1513 1514 1515 1516 1517 1518 1519 1520 1521 1522 1523 1524 1525 1526 1527 1528 1529 1530 1531 1532 1533 1534 1535 1536 1537 1538 1539 1540 1541 1542 1543 1544 1545 1546 1547 1548 1549 1550 1551 1552 1553 1554 1555 1556 1557 1558 1559 1560 1561 1562 1563 1564 1565 1566 1567 1568 1569 1570 1571 1572 1573 1574 1575 1576 1577 1578 1579 1580 1581 1582 1583 1584 1585 1586 1587 1588 1589 1590 1591 1592 1593 1594 1595 1596 1597 1598 1599 1600 1601 1602 1603 1604 1605 1606 1607 1608 1609 1610 1611 1612 1613 1614 1615 1616 1617 1618 1619 1620 1621 1622 1623 1624 1625 1626 1627 1628 1629 1630 1631 1632 1633 1634 1635 1636 1637 1638 1639 1640 1641 1642 1643 1644 1645 1646 1647 1648 1649 1650 1651 1652 1653 1654 1655 1656 1657 1658 1659 1660 1661 1662 1663 1664 1665 1666 1667 1668 1669 1670 1671 1672 1673 1674 1675 1676 1677 1678 1679 1680 1681 1682 1683 1684 1685 1686 1687 1688 1689 1690 1691 1692 1693 1694 1695 1696 1697 1698 1699 1700 1701 1702 1703 1704 1705 1706 1707 1708 1709 1710 1711 1712 1713 1714 1715 1716 1717 1718 1719 1720 1721 1722 1723 1724 1725 1726 1727 1728 1729 1730 1731 1732 1733 1734 1735 1736 1737 1738 1739 1740 1741 1742 1743 1744 1745 1746 1747 1748 1749 1750 1751 1752 1753 1754 1755 1756 1757 1758 1759 1760 1761 1762 1763 1764 1765 1766 1767 1768 1769 1770 1771 1772 1773 1774 1775 1776 1777 1778 1779 1780 1781 1782 1783 1784 1785 1786 1787 1788 1789 1790 1791 1792 1793 1794 1795 1796 1797 1798 1799 1800 1801 1802 1803 1804 1805 1806 1807 1808 1809 1810 1811 1812 1813 1814 1815 1816 1817 1818 1819 1820 1821 1822 1823 1824 1825 1826 1827 1828 1829 1830 1831 1832 1833 1834 1835 1836 1837 1838 1839 1840 1841 1842 1843 1844 1845 1846 1847 1848 1849 1850 1851 1852 1853 1854 1855 1856 1857 1858 1859 1860 1861 1862 1863 1864 1865 1866 1867 1868 1869 1870 1871 1872 1873 1874 1875 1876 1877 1878 1879 1880 1881 1882 1883 1884 1885 1886 1887 1888 1889 1890 1891 1892 1893 1894 1895 1896 1897 1898 1899 1900 1901 1902 1903 1904 1905 1906 1907 1908 1909 1910 1911 1912 1913 1914 1915 1916 1917 1918 1919 1920 1921 1922 1923 1924 1925 1926 1927 1928 1929 1930 1931 1932 1933 1934 1935 1936 1937 1938 1939 1940 1941 1942 1943 1944 1945 1946 1947 1948 1949 1950 1951 1952 1953 1954 1955 1956 1957 1958 1959 1960 1961 1962 1963 1964 1965 1966 1967 1968 1969 1970 1971 1972 1973 1974 1975 1976 1977 1978 1979 1980 1981 1982 1983 1984 1985 1986 1987 1988 1989 1990 1991 1992 1993 1994 1995 1996 1997 1998 1999 2000 2001 2002 2003 2004 2005 2006 2007 2008 2009 2010 2011 2012 2013 2014 2015 2016 2017 2018 2019 2020 2021 2022 2023 2024 2025 2026 2027 2028 2029 2030 2031 2032 2033 2034 2035 2036 2037 2038 2039 2040 2041 2042 2043 2044 2045 2046 2047 2048 2049 2050 2051 2052 2053 2054 2055 2056 2057 2058 2059 2060 2061 2062 2063 2064 2065 2066 2067 2068 2069 2070 2071 2072 2073 2074 2075 2076 2077 2078 2079 2080 2081 2082 2083 2084 2085 2086 2087 2088 2089 2090 2091 2092 2093 2094 2095 2096 2097 2098 2099 2100 2101 2102 2103 2104 2105 2106 2107 2108 2109 2110 2111 2112 2113 2114 2115 2116 2117 2118 2119 2120 2121 2122 2123 2124 2125 2126 2127 2128 2129 2130 2131 2132 2133 2134 2135 2136 2137 2138 2139 2140 2141 2142 2143 2144 2145 2146 2147 2148 2149 2150 2151 2152 2153 2154 2155 2156 2157 2158 2159 2160 2161 2162 2163 2164 2165 2166 2167 2168 2169 2170 2171 2172 2173 2174 2175 2176 2177 2178 2179 2180 2181 2182 2183 2184 2185 2186 2187 2188 2189 2190 2191 2192 2193 2194 2195 2196 2197 2198 2199 2200 2201 2202 2203 2204 2205 2206 2207 2208 2209 2210 2211 2212 2213 2214 2215 2216 2217 2218 2219 2220 2221 2222 2223 2224 2225 2226 2227 2228 2229 2230 2231 2232 2233 2234 2235 2236 2237 2238 2239 2240 2241 2242 2243 2244 2245 2246 2247 2248 2249 2250 2251 2252 2253 2254 2255 2256 2257 2258 2259 2260 2261 2262 2263 2264 | /* $NetBSD: if_atu.c,v 1.75 2022/03/03 06:06:52 riastradh Exp $ */ /* $OpenBSD: if_atu.c,v 1.48 2004/12/30 01:53:21 dlg Exp $ */ /* * Copyright (c) 2003, 2004 * Daan Vreeken <Danovitsch@Vitsch.net>. All rights reserved. * * Redistribution and use in source and binary forms, with or without * modification, are permitted provided that the following conditions * are met: * 1. Redistributions of source code must retain the above copyright * notice, this list of conditions and the following disclaimer. * 2. Redistributions in binary form must reproduce the above copyright * notice, this list of conditions and the following disclaimer in the * documentation and/or other materials provided with the distribution. * 3. All advertising materials mentioning features or use of this software * must display the following acknowledgement: * This product includes software developed by Daan Vreeken. * 4. Neither the name of the author nor the names of any co-contributors * may be used to endorse or promote products derived from this software * without specific prior written permission. * * THIS SOFTWARE IS PROVIDED BY Daan Vreeken AND CONTRIBUTORS ``AS IS'' AND * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE * ARE DISCLAIMED. IN NO EVENT SHALL Daan Vreeken OR THE VOICES IN HIS HEAD * BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR * CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF * SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS * INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN * CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) * ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF * THE POSSIBILITY OF SUCH DAMAGE. */ /* * Atmel AT76c503 / AT76c503a / AT76c505 / AT76c505a USB WLAN driver * version 0.5 - 2004-08-03 * * Originally written by Daan Vreeken <Danovitsch @ Vitsch . net> * http://vitsch.net/bsd/atuwi * * Contributed to by : * Chris Whitehouse, Alistair Phillips, Peter Pilka, Martijn van Buul, * Suihong Liang, Arjan van Leeuwen, Stuart Walsh * * Ported to OpenBSD by Theo de Raadt and David Gwynne. * Ported to NetBSD by Jesse Off */ #include <sys/cdefs.h> __KERNEL_RCSID(0, "$NetBSD: if_atu.c,v 1.75 2022/03/03 06:06:52 riastradh Exp $"); #ifdef _KERNEL_OPT #include "opt_usb.h" #endif #include <sys/param.h> #include <sys/sockio.h> #include <sys/mbuf.h> #include <sys/kernel.h> #include <sys/socket.h> #include <sys/systm.h> #include <sys/kthread.h> #include <sys/queue.h> #include <sys/device.h> #include <sys/bus.h> #include <dev/usb/usb.h> #include <dev/usb/usbdi.h> #include <dev/usb/usbdi_util.h> #include <dev/usb/usbdivar.h> #include <dev/usb/usbdevs.h> #include <dev/microcode/atmel/atmel_intersil_fw.h> #include <dev/microcode/atmel/atmel_rfmd2958-smc_fw.h> #include <dev/microcode/atmel/atmel_rfmd2958_fw.h> #include <dev/microcode/atmel/atmel_rfmd_fw.h> #include <net/bpf.h> #include <net/if.h> #include <net/if_dl.h> #include <net/if_media.h> #include <net/if_ether.h> #ifdef INET #include <netinet/in.h> #include <netinet/if_ether.h> #endif #include <net80211/ieee80211_var.h> #include <net80211/ieee80211_radiotap.h> #include <dev/usb/if_atureg.h> #ifdef ATU_DEBUG #define DPRINTF(x) do { if (atudebug) printf x; } while (0) #define DPRINTFN(n,x) do { if (atudebug>(n)) printf x; } while (0) int atudebug = 1; #else #define DPRINTF(x) #define DPRINTFN(n,x) #endif /* * Various supported device vendors/products/radio type. */ static const struct atu_type atu_devs[] = { { USB_VENDOR_3COM, USB_PRODUCT_3COM_3CRSHEW696, RadioRFMD, ATU_NO_QUIRK }, { USB_VENDOR_ABOCOM, USB_PRODUCT_ABOCOM_BWU613, RadioRFMD, ATU_NO_QUIRK }, { USB_VENDOR_ACCTON, USB_PRODUCT_ACCTON_2664W, AT76C503_rfmd_acc, ATU_NO_QUIRK }, { USB_VENDOR_ACERP, USB_PRODUCT_ACERP_AWL300, RadioIntersil, ATU_NO_QUIRK }, { USB_VENDOR_ACERP, USB_PRODUCT_ACERP_AWL400, RadioRFMD, ATU_NO_QUIRK }, { USB_VENDOR_ACTIONTEC, USB_PRODUCT_ACTIONTEC_UAT1, RadioRFMD, ATU_NO_QUIRK }, { USB_VENDOR_ADDTRON, USB_PRODUCT_ADDTRON_AWU120, RadioIntersil, ATU_NO_QUIRK }, { USB_VENDOR_AINCOMM, USB_PRODUCT_AINCOMM_AWU2000B, RadioRFMD2958, ATU_NO_QUIRK }, { USB_VENDOR_ASKEY, USB_PRODUCT_ASKEY_VOYAGER1010, RadioIntersil, ATU_NO_QUIRK }, { USB_VENDOR_ASKEY, USB_PRODUCT_ASKEY_WLL013I, RadioIntersil, ATU_NO_QUIRK }, { USB_VENDOR_ASKEY, USB_PRODUCT_ASKEY_WLL013, RadioRFMD, ATU_NO_QUIRK }, { USB_VENDOR_ATMEL, USB_PRODUCT_ATMEL_AT76C503I1, RadioIntersil, ATU_NO_QUIRK }, { USB_VENDOR_ATMEL, USB_PRODUCT_ATMEL_AT76C503I2, AT76C503_i3863, ATU_NO_QUIRK }, { USB_VENDOR_ATMEL, USB_PRODUCT_ATMEL_AT76C503RFMD, RadioRFMD, ATU_NO_QUIRK }, { USB_VENDOR_ATMEL, USB_PRODUCT_ATMEL_AT76C505RFMD, AT76C505_rfmd, ATU_NO_QUIRK }, { USB_VENDOR_ATMEL, USB_PRODUCT_ATMEL_AT76C505RFMD2958, RadioRFMD2958, ATU_NO_QUIRK }, { USB_VENDOR_ATMEL, USB_PRODUCT_ATMEL_AT76C505A, /* SMC2662 V.4 */ RadioRFMD2958_SMC, ATU_QUIRK_NO_REMAP | ATU_QUIRK_FW_DELAY }, { USB_VENDOR_ATMEL, USB_PRODUCT_ATMEL_AT76C505AS, /* quirk? */ RadioRFMD2958_SMC, ATU_QUIRK_NO_REMAP | ATU_QUIRK_FW_DELAY }, { USB_VENDOR_ATMEL, USB_PRODUCT_ATMEL_WN210, RadioRFMD, ATU_NO_QUIRK }, { USB_VENDOR_BELKIN, USB_PRODUCT_BELKIN_F5D6050, RadioRFMD, ATU_NO_QUIRK }, { USB_VENDOR_CONCEPTRONIC, USB_PRODUCT_CONCEPTRONIC_C11U, RadioIntersil, ATU_NO_QUIRK }, { USB_VENDOR_CONCEPTRONIC, USB_PRODUCT_CONCEPTRONIC_WL210, RadioIntersil, ATU_NO_QUIRK }, { USB_VENDOR_COMPAQ, USB_PRODUCT_COMPAQ_IPAQWLAN, RadioRFMD, ATU_NO_QUIRK }, { USB_VENDOR_COREGA, USB_PRODUCT_COREGA_WLUSB_11_STICK, RadioRFMD2958, ATU_NO_QUIRK }, { USB_VENDOR_DICKSMITH, USB_PRODUCT_DICKSMITH_CHUSB611G, RadioRFMD2958, ATU_NO_QUIRK }, { USB_VENDOR_DICKSMITH, USB_PRODUCT_DICKSMITH_WL200U, RadioRFMD, ATU_NO_QUIRK }, { USB_VENDOR_DICKSMITH, USB_PRODUCT_DICKSMITH_WL240U, RadioRFMD2958, ATU_NO_QUIRK }, { USB_VENDOR_DICKSMITH, USB_PRODUCT_DICKSMITH_XH1153, RadioRFMD, ATU_NO_QUIRK }, { USB_VENDOR_DLINK, USB_PRODUCT_DLINK_DWL120E, RadioRFMD, ATU_NO_QUIRK }, { USB_VENDOR_GIGABYTE, USB_PRODUCT_GIGABYTE_GNWLBM101, RadioRFMD, ATU_NO_QUIRK }, { USB_VENDOR_GIGASET, USB_PRODUCT_GIGASET_WLAN, /* quirk? */ RadioRFMD2958_SMC, ATU_QUIRK_NO_REMAP | ATU_QUIRK_FW_DELAY }, { USB_VENDOR_HP, USB_PRODUCT_HP_HN210W, RadioIntersil, ATU_NO_QUIRK }, { USB_VENDOR_INTEL, USB_PRODUCT_INTEL_AP310, RadioIntersil, ATU_NO_QUIRK }, { USB_VENDOR_IODATA, USB_PRODUCT_IODATA_USBWNB11A, RadioIntersil, ATU_NO_QUIRK }, { USB_VENDOR_LEXAR, USB_PRODUCT_LEXAR_2662WAR, RadioRFMD, ATU_NO_QUIRK }, { USB_VENDOR_LINKSYS, USB_PRODUCT_LINKSYS_WUSB11, RadioIntersil, ATU_NO_QUIRK }, { USB_VENDOR_LINKSYS2, USB_PRODUCT_LINKSYS2_WUSB11, RadioRFMD, ATU_NO_QUIRK }, { USB_VENDOR_LINKSYS2, USB_PRODUCT_LINKSYS2_NWU11B, RadioRFMD, ATU_NO_QUIRK }, { USB_VENDOR_LINKSYS3, USB_PRODUCT_LINKSYS3_WUSB11V28, RadioRFMD2958, ATU_NO_QUIRK }, { USB_VENDOR_MSI, USB_PRODUCT_MSI_WLAN, RadioRFMD2958, ATU_NO_QUIRK }, { USB_VENDOR_NETGEAR2, USB_PRODUCT_NETGEAR2_MA101, RadioIntersil, ATU_NO_QUIRK }, { USB_VENDOR_NETGEAR2, USB_PRODUCT_NETGEAR2_MA101B, RadioRFMD, ATU_NO_QUIRK }, { USB_VENDOR_OQO, USB_PRODUCT_OQO_WIFI01, RadioRFMD2958_SMC, ATU_QUIRK_NO_REMAP | ATU_QUIRK_FW_DELAY }, { USB_VENDOR_PLANEX2, USB_PRODUCT_PLANEX2_GW_US11S, RadioRFMD, ATU_NO_QUIRK }, { USB_VENDOR_SAMSUNG, USB_PRODUCT_SAMSUNG_SWL2100W, AT76C503_i3863, ATU_NO_QUIRK }, { USB_VENDOR_SIEMENS2, USB_PRODUCT_SIEMENS2_WLL013, RadioRFMD, ATU_NO_QUIRK }, { USB_VENDOR_SMC3, USB_PRODUCT_SMC3_2662WV1, RadioIntersil, ATU_NO_QUIRK }, { USB_VENDOR_SMC3, USB_PRODUCT_SMC3_2662WV2, AT76C503_rfmd_acc, ATU_NO_QUIRK }, { USB_VENDOR_TEKRAM, USB_PRODUCT_TEKRAM_U300C, RadioIntersil, ATU_NO_QUIRK }, { USB_VENDOR_ZCOM, USB_PRODUCT_ZCOM_M4Y750, RadioIntersil, ATU_NO_QUIRK }, }; static const struct atu_radfirm { enum atu_radio_type atur_type; unsigned char *atur_internal; size_t atur_internal_sz; unsigned char *atur_external; size_t atur_external_sz; } atu_radfirm[] = { { RadioRFMD, atmel_fw_rfmd_int, sizeof(atmel_fw_rfmd_int), atmel_fw_rfmd_ext, sizeof(atmel_fw_rfmd_ext) }, { RadioRFMD2958, atmel_fw_rfmd2958_int, sizeof(atmel_fw_rfmd2958_int), atmel_fw_rfmd2958_ext, sizeof(atmel_fw_rfmd2958_ext) }, { RadioRFMD2958_SMC, atmel_fw_rfmd2958_smc_int, sizeof(atmel_fw_rfmd2958_smc_int), atmel_fw_rfmd2958_smc_ext, sizeof(atmel_fw_rfmd2958_smc_ext) }, { RadioIntersil, atmel_fw_intersil_int, sizeof(atmel_fw_intersil_int), atmel_fw_intersil_ext, sizeof(atmel_fw_intersil_ext) } }; static int atu_newbuf(struct atu_softc *, struct atu_chain *, struct mbuf *); static void atu_rxeof(struct usbd_xfer *, void *, usbd_status); static void atu_txeof(struct usbd_xfer *, void *, usbd_status); static void atu_start(struct ifnet *); static int atu_ioctl(struct ifnet *, u_long, void *); static int atu_init(struct ifnet *); static void atu_stop(struct ifnet *, int); static void atu_watchdog(struct ifnet *); static usbd_status atu_usb_request(struct atu_softc *, uint8_t, uint8_t, uint16_t, uint16_t, uint16_t, uint8_t *); static int atu_send_command(struct atu_softc *, uint8_t *, int); static int atu_get_cmd_status(struct atu_softc *, uint8_t, uint8_t *); static int atu_wait_completion(struct atu_softc *, uint8_t, uint8_t *); static int atu_send_mib(struct atu_softc *, uint8_t, uint8_t, uint8_t, void *); static int atu_get_mib(struct atu_softc *, uint8_t, uint8_t, uint8_t, uint8_t *); #if 0 int atu_start_ibss(struct atu_softc *); #endif static int atu_start_scan(struct atu_softc *); static int atu_switch_radio(struct atu_softc *, int); static int atu_initial_config(struct atu_softc *); static int atu_join(struct atu_softc *, struct ieee80211_node *); static int8_t atu_get_dfu_state(struct atu_softc *); static uint8_t atu_get_opmode(struct atu_softc *, uint8_t *); static void atu_internal_firmware(device_t); static void atu_external_firmware(device_t); static int atu_get_card_config(struct atu_softc *); static int atu_media_change(struct ifnet *); static void atu_media_status(struct ifnet *, struct ifmediareq *); static int atu_tx_list_init(struct atu_softc *); static int atu_rx_list_init(struct atu_softc *); static void atu_xfer_list_free(struct atu_softc *, struct atu_chain *, int); static void atu_task(void *); static int atu_newstate(struct ieee80211com *, enum ieee80211_state, int); static int atu_tx_start(struct atu_softc *, struct ieee80211_node *, struct atu_chain *, struct mbuf *); static void atu_complete_attach(struct atu_softc *); static uint8_t atu_calculate_padding(int); static int atu_match(device_t, cfdata_t, void *); static void atu_attach(device_t, device_t, void *); static int atu_detach(device_t, int); static int atu_activate(device_t, enum devact); CFATTACH_DECL_NEW(atu, sizeof(struct atu_softc), atu_match, atu_attach, atu_detach, atu_activate); static usbd_status atu_usb_request(struct atu_softc *sc, uint8_t type, uint8_t request, uint16_t value, uint16_t index, uint16_t length, uint8_t *data) { usb_device_request_t req; struct usbd_xfer *xfer; usbd_status err; int total_len = 0, s; req.bmRequestType = type; req.bRequest = request; USETW(req.wValue, value); USETW(req.wIndex, index); USETW(req.wLength, length); #ifdef ATU_DEBUG if (atudebug) { DPRINTFN(20, ("%s: req=%02x val=%02x ind=%02x " "len=%02x\n", device_xname(sc->atu_dev), request, value, index, length)); } #endif /* ATU_DEBUG */ s = splnet(); struct usbd_pipe *pipe0 = usbd_get_pipe0(sc->atu_udev); int error = usbd_create_xfer(pipe0, length, 0, 0, &xfer); if (error) { splx(s); return USBD_IOERROR; } usbd_setup_default_xfer(xfer, sc->atu_udev, 0, 500000, &req, data, length, USBD_SHORT_XFER_OK, NULL); err = usbd_sync_transfer(xfer); usbd_get_xfer_status(xfer, NULL, NULL, &total_len, NULL); #ifdef ATU_DEBUG if (atudebug) { if (type & UT_READ) { DPRINTFN(20, ("%s: transferred %#x bytes in\n", device_xname(sc->atu_dev), total_len)); } else { if (total_len != length) DPRINTF(("%s: wrote only %x bytes\n", device_xname(sc->atu_dev), total_len)); } } #endif /* ATU_DEBUG */ usbd_destroy_xfer(xfer); splx(s); return err; } static int atu_send_command(struct atu_softc *sc, uint8_t *command, int size) { return atu_usb_request(sc, UT_WRITE_VENDOR_DEVICE, 0x0e, 0x0000, 0x0000, size, command); } static int atu_get_cmd_status(struct atu_softc *sc, uint8_t cmd, uint8_t *status) { /* * all other drivers (including Windoze) request 40 bytes of status * and get a short-xfer of just 6 bytes. we can save 34 bytes of * buffer if we just request those 6 bytes in the first place :) */ /* return atu_usb_request(sc, UT_READ_VENDOR_INTERFACE, 0x22, cmd, 0x0000, 40, status); */ return atu_usb_request(sc, UT_READ_VENDOR_INTERFACE, 0x22, cmd, 0x0000, 6, status); } static int atu_wait_completion(struct atu_softc *sc, uint8_t cmd, uint8_t *status) { int idle_count = 0, err; uint8_t statusreq[6]; DPRINTFN(15, ("%s: wait-completion: cmd=%02x\n", device_xname(sc->atu_dev), cmd)); while (1) { err = atu_get_cmd_status(sc, cmd, statusreq); if (err) return err; #ifdef ATU_DEBUG if (atudebug) { DPRINTFN(20, ("%s: status=%s cmd=%02x\n", device_xname(sc->atu_dev), ether_sprintf(statusreq), cmd)); } #endif /* ATU_DEBUG */ /* * during normal operations waiting on STATUS_IDLE * will never happen more than once */ if ((statusreq[5] == STATUS_IDLE) && (idle_count++ > 20)) { DPRINTF(("%s: idle_count > 20!\n", device_xname(sc->atu_dev))); return 0; } if ((statusreq[5] != STATUS_IN_PROGRESS) && (statusreq[5] != STATUS_IDLE)) { if (status != NULL) *status = statusreq[5]; return 0; } usbd_delay_ms(sc->atu_udev, 25); } } static int atu_send_mib(struct atu_softc *sc, uint8_t type, uint8_t size, uint8_t index, void *data) { int err; struct atu_cmd_set_mib request; /* * We don't construct a MIB packet first and then memcpy it into an * Atmel-command-packet, we just construct it the right way at once :) */ memset(&request, 0, sizeof(request)); request.AtCmd = CMD_SET_MIB; USETW(request.AtSize, size + 4); request.MIBType = type; request.MIBSize = size; request.MIBIndex = index; request.MIBReserved = 0; /* * For 1 and 2 byte requests we assume a direct value, * everything bigger than 2 bytes we assume a pointer to the data */ switch (size) { case 0: break; case 1: request.data[0]=(long)data & 0x000000ff; break; case 2: request.data[0]=(long)data & 0x000000ff; request.data[1]=(long)data >> 8; break; default: memcpy(request.data, data, size); break; } err = atu_usb_request(sc, UT_WRITE_VENDOR_DEVICE, 0x0e, 0x0000, 0x0000, size+8, (uByte *)&request); if (err) return err; DPRINTFN(15, ("%s: sendmib : waitcompletion...\n", device_xname(sc->atu_dev))); return atu_wait_completion(sc, CMD_SET_MIB, NULL); } static int atu_get_mib(struct atu_softc *sc, uint8_t type, uint8_t size, uint8_t index, uint8_t *buf) { /* linux/at76c503.c - 478 */ return atu_usb_request(sc, UT_READ_VENDOR_INTERFACE, 0x033, type << 8, index, size, buf); } #if 0 int atu_start_ibss(struct atu_softc *sc) { struct ieee80211com *ic = &sc->sc_ic; int err; struct atu_cmd_start_ibss Request; Request.Cmd = CMD_START_IBSS; Request.Reserved = 0; Request.Size = sizeof(Request) - 4; memset(Request.BSSID, 0x00, sizeof(Request.BSSID)); memset(Request.SSID, 0x00, sizeof(Request.SSID)); memcpy(Request.SSID, ic->ic_des_ssid, ic->ic_des_ssidlen); Request.SSIDSize = ic->ic_des_ssidlen; if (sc->atu_desired_channel != IEEE80211_CHAN_ANY) Request.Channel = (uint8_t)sc->atu_desired_channel; else Request.Channel = ATU_DEFAULT_CHANNEL; Request.BSSType = AD_HOC_MODE; memset(Request.Res, 0x00, sizeof(Request.Res)); /* Write config to adapter */ err = atu_send_command(sc, (uint8_t *)&Request, sizeof(Request)); if (err) { DPRINTF(("%s: start ibss failed!\n", device_xname(sc->atu_dev))); return err; } /* Wait for the adapter to do its thing */ err = atu_wait_completion(sc, CMD_START_IBSS, NULL); if (err) { DPRINTF(("%s: error waiting for start_ibss\n", device_xname(sc->atu_dev))); return err; } /* Get the current BSSID */ err = atu_get_mib(sc, MIB_MAC_MGMT__CURRENT_BSSID, sc->atu_bssid); if (err) { DPRINTF(("%s: could not get BSSID!\n", device_xname(sc->atu_dev))); return err; } DPRINTF(("%s: started a new IBSS (BSSID=%s)\n", device_xname(sc->atu_dev), ether_sprintf(sc->atu_bssid))); return 0; } #endif static int atu_start_scan(struct atu_softc *sc) { struct ieee80211com *ic = &sc->sc_ic; struct atu_cmd_do_scan Scan; usbd_status err; int Cnt; memset(&Scan, 0, sizeof(Scan)); Scan.Cmd = CMD_START_SCAN; Scan.Reserved = 0; USETW(Scan.Size, sizeof(Scan) - 4); /* use the broadcast BSSID (in active scan) */ for (Cnt=0; Cnt<6; Cnt++) Scan.BSSID[Cnt] = 0xff; memset(Scan.SSID, 0x00, sizeof(Scan.SSID)); memcpy(Scan.SSID, ic->ic_des_essid, ic->ic_des_esslen); Scan.SSID_Len = ic->ic_des_esslen; /* default values for scan */ Scan.ScanType = ATU_SCAN_ACTIVE; if (sc->atu_desired_channel != IEEE80211_CHAN_ANY) Scan.Channel = (uint8_t)sc->atu_desired_channel; else Scan.Channel = sc->atu_channel; ic->ic_curchan = &ic->ic_channels[Scan.Channel]; /* we like scans to be quick :) */ /* the time we wait before sending probe's */ USETW(Scan.ProbeDelay, 0); /* the time we stay on one channel */ USETW(Scan.MinChannelTime, 100); USETW(Scan.MaxChannelTime, 200); /* whether or not we scan all channels */ Scan.InternationalScan = 0xc1; #ifdef ATU_DEBUG if (atudebug) { DPRINTFN(20, ("%s: scan cmd len=%02zx\n", device_xname(sc->atu_dev), sizeof(Scan))); } #endif /* ATU_DEBUG */ /* Write config to adapter */ err = atu_send_command(sc, (uint8_t *)&Scan, sizeof(Scan)); if (err) return err; /* * We don't wait for the command to finish... the mgmt-thread will do * that for us */ /* err = atu_wait_completion(sc, CMD_START_SCAN, NULL); if (err) return err; */ return 0; } static int atu_switch_radio(struct atu_softc *sc, int state) { usbd_status err; struct atu_cmd CmdRadio; if (sc->atu_radio == RadioIntersil) { /* * Intersil doesn't seem to need/support switching the radio * on/off */ return 0; } memset(&CmdRadio, 0, sizeof(CmdRadio)); CmdRadio.Cmd = CMD_RADIO_ON; if (sc->atu_radio_on != state) { if (state == 0) CmdRadio.Cmd = CMD_RADIO_OFF; err = atu_send_command(sc, (uint8_t *)&CmdRadio, sizeof(CmdRadio)); if (err) return err; err = atu_wait_completion(sc, CmdRadio.Cmd, NULL); if (err) return err; DPRINTFN(10, ("%s: radio turned %s\n", device_xname(sc->atu_dev), state ? "on" : "off")); sc->atu_radio_on = state; } return 0; } static int atu_initial_config(struct atu_softc *sc) { struct ieee80211com *ic = &sc->sc_ic; uint32_t i; usbd_status err; /* uint8_t rates[4] = {0x82, 0x84, 0x8B, 0x96};*/ uint8_t rates[4] = {0x82, 0x04, 0x0B, 0x16}; struct atu_cmd_card_config cmd; uint8_t reg_domain; DPRINTFN(10, ("%s: sending mac-addr\n", device_xname(sc->atu_dev))); err = atu_send_mib(sc, MIB_MAC_ADDR__ADDR, ic->ic_myaddr); if (err) { DPRINTF(("%s: error setting mac-addr\n", device_xname(sc->atu_dev))); return err; } /* DPRINTF(("%s: sending reg-domain\n", device_xname(sc->atu_dev))); err = atu_send_mib(sc, MIB_PHY__REG_DOMAIN, NR(0x30)); if (err) { DPRINTF(("%s: error setting mac-addr\n", device_xname(sc->atu_dev))); return err; } */ memset(&cmd, 0, sizeof(cmd)); cmd.Cmd = CMD_STARTUP; cmd.Reserved = 0; USETW(cmd.Size, sizeof(cmd) - 4); if (sc->atu_desired_channel != IEEE80211_CHAN_ANY) cmd.Channel = (uint8_t)sc->atu_desired_channel; else cmd.Channel = sc->atu_channel; cmd.AutoRateFallback = 1; memcpy(cmd.BasicRateSet, rates, 4); /* ShortRetryLimit should be 7 according to 802.11 spec */ cmd.ShortRetryLimit = 7; USETW(cmd.RTS_Threshold, 2347); USETW(cmd.FragThreshold, 2346); /* Doesn't seem to work, but we'll set it to 1 anyway */ cmd.PromiscuousMode = 1; /* this goes into the beacon we transmit */ if (ic->ic_flags & IEEE80211_F_PRIVACY) cmd.PrivacyInvoked = 1; else cmd.PrivacyInvoked = 0; cmd.ExcludeUnencrypted = 0; if (ic->ic_flags & IEEE80211_F_PRIVACY) { switch (ic->ic_nw_keys[ic->ic_def_txkey].wk_keylen) { case 5: cmd.EncryptionType = ATU_WEP_40BITS; break; case 13: cmd.EncryptionType = ATU_WEP_104BITS; break; default: cmd.EncryptionType = ATU_WEP_OFF; break; } cmd.WEP_DefaultKeyID = ic->ic_def_txkey; for (i = 0; i < IEEE80211_WEP_NKID; i++) { memcpy(cmd.WEP_DefaultKey[i], ic->ic_nw_keys[i].wk_key, ic->ic_nw_keys[i].wk_keylen); } } /* Setting the SSID here doesn't seem to do anything */ memset(cmd.SSID, 0x00, sizeof(cmd.SSID)); memcpy(cmd.SSID, ic->ic_des_essid, ic->ic_des_esslen); cmd.SSID_Len = ic->ic_des_esslen; cmd.ShortPreamble = 0; USETW(cmd.BeaconPeriod, 100); /* cmd.BeaconPeriod = 65535; */ /* * TODO: * read reg domain MIB_PHY @ 0x17 (1 byte), (reply = 0x30) * we should do something useful with this info. right now it's just * ignored */ err = atu_get_mib(sc, MIB_PHY__REG_DOMAIN, ®_domain); if (err) { DPRINTF(("%s: could not get regdomain!\n", device_xname(sc->atu_dev))); } else { DPRINTF(("%s: in reg domain %#x according to the " "adapter\n", device_xname(sc->atu_dev), reg_domain)); } #ifdef ATU_DEBUG if (atudebug) { DPRINTFN(20, ("%s: configlen=%02zx\n", device_xname(sc->atu_dev), sizeof(cmd))); } #endif /* ATU_DEBUG */ /* Windoze : driver says exclude-unencrypted=1 & encr-type=1 */ err = atu_send_command(sc, (uint8_t *)&cmd, sizeof(cmd)); if (err) return err; err = atu_wait_completion(sc, CMD_STARTUP, NULL); if (err) return err; /* Turn on radio now */ err = atu_switch_radio(sc, 1); if (err) return err; /* preamble type = short */ err = atu_send_mib(sc, MIB_LOCAL__PREAMBLE, NR(PREAMBLE_SHORT)); if (err) return err; /* frag = 1536 */ err = atu_send_mib(sc, MIB_MAC__FRAG, NR(2346)); if (err) return err; /* rts = 1536 */ err = atu_send_mib(sc, MIB_MAC__RTS, NR(2347)); if (err) return err; /* auto rate fallback = 1 */ err = atu_send_mib(sc, MIB_LOCAL__AUTO_RATE_FALLBACK, NR(1)); if (err) return err; /* power mode = full on, no power saving */ err = atu_send_mib(sc, MIB_MAC_MGMT__POWER_MODE, NR(POWER_MODE_ACTIVE)); if (err) return err; DPRINTFN(10, ("%s: completed initial config\n", device_xname(sc->atu_dev))); return 0; } static int atu_join(struct atu_softc *sc, struct ieee80211_node *node) { struct atu_cmd_join join; uint8_t status = 0; /* XXX: GCC */ usbd_status err; memset(&join, 0, sizeof(join)); join.Cmd = CMD_JOIN; join.Reserved = 0x00; USETW(join.Size, sizeof(join) - 4); DPRINTFN(15, ("%s: pre-join sc->atu_bssid=%s\n", device_xname(sc->atu_dev), ether_sprintf(sc->atu_bssid))); DPRINTFN(15, ("%s: mode=%d\n", device_xname(sc->atu_dev), sc->atu_mode)); memcpy(join.bssid, node->ni_bssid, IEEE80211_ADDR_LEN); memset(join.essid, 0x00, 32); memcpy(join.essid, node->ni_essid, node->ni_esslen); join.essid_size = node->ni_esslen; if (node->ni_capinfo & IEEE80211_CAPINFO_IBSS) join.bss_type = AD_HOC_MODE; else join.bss_type = INFRASTRUCTURE_MODE; join.channel = ieee80211_chan2ieee(&sc->sc_ic, node->ni_chan); USETW(join.timeout, ATU_JOIN_TIMEOUT); join.reserved = 0x00; DPRINTFN(10, ("%s: trying to join BSSID=%s\n", device_xname(sc->atu_dev), ether_sprintf(join.bssid))); err = atu_send_command(sc, (uint8_t *)&join, sizeof(join)); if (err) { DPRINTF(("%s: ERROR trying to join IBSS\n", device_xname(sc->atu_dev))); return err; } err = atu_wait_completion(sc, CMD_JOIN, &status); if (err) { DPRINTF(("%s: error joining BSS!\n", device_xname(sc->atu_dev))); return err; } if (status != STATUS_COMPLETE) { DPRINTF(("%s: error joining... [status=%02x]\n", device_xname(sc->atu_dev), status)); return status; } else { DPRINTFN(10, ("%s: joined BSS\n", device_xname(sc->atu_dev))); } return err; } /* * Get the state of the DFU unit */ static int8_t atu_get_dfu_state(struct atu_softc *sc) { uint8_t state; if (atu_usb_request(sc, DFU_GETSTATE, 0, 0, 1, &state)) return -1; return state; } /* * Get MAC opmode */ static uint8_t atu_get_opmode(struct atu_softc *sc, uint8_t *mode) { return atu_usb_request(sc, UT_READ_VENDOR_INTERFACE, 0x33, 0x0001, 0x0000, 1, mode); } /* * Upload the internal firmware into the device */ static void atu_internal_firmware(device_t arg) { struct atu_softc *sc = device_private(arg); u_char state, *ptr = NULL, *firm = NULL, status[6]; int block_size, block = 0, err, i; size_t bytes_left = 0; /* * Uploading firmware is done with the DFU (Device Firmware Upgrade) * interface. See "Universal Serial Bus - Device Class Specification * for Device Firmware Upgrade" pdf for details of the protocol. * Maybe this could be moved to a separate 'firmware driver' once more * device drivers need it... For now we'll just do it here. * * Just for your information, the Atmel's DFU descriptor looks like * this: * * 07 size * 21 type * 01 capabilities : only firmware download, need reset * after download * 13 05 detach timeout : max 1299ms between DFU_DETACH and * reset * 00 04 max bytes of firmware per transaction : 1024 */ /* Choose the right firmware for the device */ for (i = 0; i < __arraycount(atu_radfirm); i++) if (sc->atu_radio == atu_radfirm[i].atur_type) { firm = atu_radfirm[i].atur_internal; bytes_left = atu_radfirm[i].atur_internal_sz; } if (firm == NULL) { aprint_error_dev(arg, "no firmware found\n"); return; } ptr = firm; state = atu_get_dfu_state(sc); while (block >= 0 && state > 0) { switch (state) { case DFUState_DnLoadSync: /* get DFU status */ err = atu_usb_request(sc, DFU_GETSTATUS, 0, 0 , 6, status); if (err) { DPRINTF(("%s: dfu_getstatus failed!\n", device_xname(sc->atu_dev))); return; } /* success means state => DnLoadIdle */ state = DFUState_DnLoadIdle; continue; break; case DFUState_DFUIdle: case DFUState_DnLoadIdle: if (bytes_left>=DFU_MaxBlockSize) block_size = DFU_MaxBlockSize; else block_size = bytes_left; DPRINTFN(15, ("%s: firmware block %d\n", device_xname(sc->atu_dev), block)); err = atu_usb_request(sc, DFU_DNLOAD, block++, 0, block_size, ptr); if (err) { DPRINTF(("%s: dfu_dnload failed\n", device_xname(sc->atu_dev))); return; } ptr += block_size; bytes_left -= block_size; if (block_size == 0) block = -1; break; default: usbd_delay_ms(sc->atu_udev, 100); DPRINTFN(20, ("%s: sleeping for a while\n", device_xname(sc->atu_dev))); break; } state = atu_get_dfu_state(sc); } if (state != DFUState_ManifestSync) { DPRINTF(("%s: state != manifestsync... eek!\n", device_xname(sc->atu_dev))); } err = atu_usb_request(sc, DFU_GETSTATUS, 0, 0, 6, status); if (err) { DPRINTF(("%s: dfu_getstatus failed!\n", device_xname(sc->atu_dev))); return; } DPRINTFN(15, ("%s: sending remap\n", device_xname(sc->atu_dev))); err = atu_usb_request(sc, DFU_REMAP, 0, 0, 0, NULL); if ((err) && !(sc->atu_quirk & ATU_QUIRK_NO_REMAP)) { DPRINTF(("%s: remap failed!\n", device_xname(sc->atu_dev))); return; } /* after a lot of trying and measuring I found out the device needs * about 56 miliseconds after sending the remap command before * it's ready to communicate again. So we'll wait just a little bit * longer than that to be sure... */ usbd_delay_ms(sc->atu_udev, 56+100); aprint_error_dev(arg, "reattaching after firmware upload\n"); usb_needs_reattach(sc->atu_udev); } static void atu_external_firmware(device_t arg) { struct atu_softc *sc = device_private(arg); u_char *ptr = NULL, *firm = NULL; int block_size, block = 0, err, i; size_t bytes_left = 0; for (i = 0; i < __arraycount(atu_radfirm); i++) if (sc->atu_radio == atu_radfirm[i].atur_type) { firm = atu_radfirm[i].atur_external; bytes_left = atu_radfirm[i].atur_external_sz; } if (firm == NULL) { aprint_error_dev(arg, "no firmware found\n"); return; } ptr = firm; while (bytes_left) { if (bytes_left > 1024) block_size = 1024; else block_size = bytes_left; DPRINTFN(15, ("%s: block:%d size:%d\n", device_xname(sc->atu_dev), block, block_size)); err = atu_usb_request(sc, UT_WRITE_VENDOR_DEVICE, 0x0e, 0x0802, block, block_size, ptr); if (err) { DPRINTF(("%s: could not load external firmware " "block\n", device_xname(sc->atu_dev))); return; } ptr += block_size; block++; bytes_left -= block_size; } err = atu_usb_request(sc, UT_WRITE_VENDOR_DEVICE, 0x0e, 0x0802, block, 0, NULL); if (err) { DPRINTF(("%s: could not load last zero-length firmware " "block\n", device_xname(sc->atu_dev))); return; } /* * The SMC2662w V.4 seems to require some time to do its thing with * the external firmware... 20 ms isn't enough, but 21 ms works 100 * times out of 100 tries. We'll wait a bit longer just to be sure */ if (sc->atu_quirk & ATU_QUIRK_FW_DELAY) usbd_delay_ms(sc->atu_udev, 21 + 100); DPRINTFN(10, ("%s: external firmware upload done\n", device_xname(sc->atu_dev))); /* complete configuration after the firmwares have been uploaded */ atu_complete_attach(sc); } static int atu_get_card_config(struct atu_softc *sc) { struct ieee80211com *ic = &sc->sc_ic; struct atu_rfmd_conf rfmd_conf; struct atu_intersil_conf intersil_conf; int err; switch (sc->atu_radio) { case RadioRFMD: case RadioRFMD2958: case RadioRFMD2958_SMC: case AT76C503_rfmd_acc: case AT76C505_rfmd: err = atu_usb_request(sc, UT_READ_VENDOR_INTERFACE, 0x33, 0x0a02, 0x0000, sizeof(rfmd_conf), (uint8_t *)&rfmd_conf); if (err) { DPRINTF(("%s: could not get rfmd config!\n", device_xname(sc->atu_dev))); return err; } memcpy(ic->ic_myaddr, rfmd_conf.MACAddr, IEEE80211_ADDR_LEN); break; case RadioIntersil: case AT76C503_i3863: err = atu_usb_request(sc, UT_READ_VENDOR_INTERFACE, 0x33, 0x0902, 0x0000, sizeof(intersil_conf), (uint8_t *)&intersil_conf); if (err) { DPRINTF(("%s: could not get intersil config!\n", device_xname(sc->atu_dev))); return err; } memcpy(ic->ic_myaddr, intersil_conf.MACAddr, IEEE80211_ADDR_LEN); break; } return 0; } /* * Probe for an AT76c503 chip. */ static int atu_match(device_t parent, cfdata_t match, void *aux) { struct usb_attach_arg *uaa = aux; int i; for (i = 0; i < __arraycount(atu_devs); i++) { const struct atu_type *t = &atu_devs[i]; if (uaa->uaa_vendor == t->atu_vid && uaa->uaa_product == t->atu_pid) { return UMATCH_VENDOR_PRODUCT; } } return UMATCH_NONE; } static int atu_media_change(struct ifnet *ifp) { struct atu_softc *sc = ifp->if_softc; struct ieee80211com *ic = &sc->sc_ic; int err, s; DPRINTFN(10, ("%s: atu_media_change\n", device_xname(sc->atu_dev))); err = ieee80211_media_change(ifp); if (err == ENETRESET) { if ((ifp->if_flags & (IFF_RUNNING|IFF_UP)) == (IFF_RUNNING|IFF_UP)) { s = splnet(); ieee80211_new_state(ic, IEEE80211_S_INIT, -1); atu_initial_config(sc); ieee80211_new_state(ic, IEEE80211_S_SCAN, -1); splx(s); } err = 0; } return err; } static void atu_media_status(struct ifnet *ifp, struct ifmediareq *req) { #ifdef ATU_DEBUG struct atu_softc *sc = ifp->if_softc; #endif /* ATU_DEBUG */ DPRINTFN(10, ("%s: atu_media_status\n", device_xname(sc->atu_dev))); ieee80211_media_status(ifp, req); } static void atu_task(void *arg) { struct atu_softc *sc = (struct atu_softc *)arg; struct ieee80211com *ic = &sc->sc_ic; usbd_status err; int s; DPRINTFN(10, ("%s: atu_task\n", device_xname(sc->atu_dev))); if (sc->sc_state != ATU_S_OK) return; switch (sc->sc_cmd) { case ATU_C_SCAN: err = atu_start_scan(sc); if (err) { DPRINTFN(1, ("%s: atu_task: couldn't start scan!\n", device_xname(sc->atu_dev))); return; } err = atu_wait_completion(sc, CMD_START_SCAN, NULL); if (err) { DPRINTF(("%s: atu_task: error waiting for scan\n", device_xname(sc->atu_dev))); return; } DPRINTF(("%s: ==========================> END OF SCAN!\n", device_xname(sc->atu_dev))); s = splnet(); ieee80211_next_scan(ic); splx(s); DPRINTF(("%s: ----------------------======> END OF SCAN2!\n", device_xname(sc->atu_dev))); break; case ATU_C_JOIN: atu_join(sc, ic->ic_bss); } } static int atu_newstate(struct ieee80211com *ic, enum ieee80211_state nstate, int arg) { struct ifnet *ifp = ic->ic_ifp; struct atu_softc *sc = ifp->if_softc; enum ieee80211_state ostate = ic->ic_state; DPRINTFN(10, ("%s: atu_newstate: %s -> %s\n", device_xname(sc->atu_dev), ieee80211_state_name[ostate], ieee80211_state_name[nstate])); switch (nstate) { case IEEE80211_S_SCAN: memcpy(ic->ic_chan_scan, ic->ic_chan_active, sizeof(ic->ic_chan_active)); ieee80211_node_table_reset(&ic->ic_scan); /* tell the event thread that we want a scan */ sc->sc_cmd = ATU_C_SCAN; usb_add_task(sc->atu_udev, &sc->sc_task, USB_TASKQ_DRIVER); /* handle this ourselves */ ic->ic_state = nstate; return 0; case IEEE80211_S_AUTH: case IEEE80211_S_RUN: if (ostate == IEEE80211_S_SCAN) { sc->sc_cmd = ATU_C_JOIN; usb_add_task(sc->atu_udev, &sc->sc_task, USB_TASKQ_DRIVER); } break; default: /* nothing to do */ break; } return (*sc->sc_newstate)(ic, nstate, arg); } /* * Attach the interface. Allocate softc structures, do * setup and ethernet/BPF attach. */ static void atu_attach(device_t parent, device_t self, void *aux) { struct atu_softc *sc = device_private(self); struct usb_attach_arg *uaa = aux; char *devinfop; usbd_status err; struct usbd_device *dev = uaa->uaa_device; uint8_t mode, channel; int i; sc->atu_dev = self; sc->sc_state = ATU_S_UNCONFIG; aprint_naive("\n"); aprint_normal("\n"); devinfop = usbd_devinfo_alloc(dev, 0); aprint_normal_dev(self, "%s\n", devinfop); usbd_devinfo_free(devinfop); err = usbd_set_config_no(dev, ATU_CONFIG_NO, 1); if (err) { aprint_error_dev(self, "failed to set configuration" ", err=%s\n", usbd_errstr(err)); return; } err = usbd_device2interface_handle(dev, ATU_IFACE_IDX, &sc->atu_iface); if (err) { aprint_error_dev(self, "getting interface handle failed\n"); return; } sc->atu_unit = device_unit(self); sc->atu_udev = dev; /* * look up the radio_type for the device * basically does the same as atu_match */ for (i = 0; i < __arraycount(atu_devs); i++) { const struct atu_type *t = &atu_devs[i]; if (uaa->uaa_vendor == t->atu_vid && uaa->uaa_product == t->atu_pid) { sc->atu_radio = t->atu_radio; sc->atu_quirk = t->atu_quirk; } } /* * Check in the interface descriptor if we're in DFU mode * If we're in DFU mode, we upload the external firmware * If we're not, the PC must have rebooted without power-cycling * the device.. I've tried this out, a reboot only requeres the * external firmware to be reloaded :) * * Hmm. The at76c505a doesn't report a DFU descriptor when it's * in DFU mode... Let's just try to get the opmode */ err = atu_get_opmode(sc, &mode); DPRINTFN(20, ("%s: opmode: %d\n", device_xname(sc->atu_dev), mode)); if (err || (mode != MODE_NETCARD && mode != MODE_NOFLASHNETCARD)) { DPRINTF(("%s: starting internal firmware download\n", device_xname(sc->atu_dev))); atu_internal_firmware(sc->atu_dev); /* * atu_internal_firmware will cause a reset of the device * so we don't want to do any more configuration after this * point. */ return; } if (mode != MODE_NETCARD) { DPRINTFN(15, ("%s: device needs external firmware\n", device_xname(sc->atu_dev))); if (mode != MODE_NOFLASHNETCARD) { DPRINTF(("%s: unexpected opmode=%d\n", device_xname(sc->atu_dev), mode)); } /* * There is no difference in opmode before and after external * firmware upload with the SMC2662 V.4 . So instead we'll try * to read the channel number. If we succeed, external * firmwaremust have been already uploaded... */ if (sc->atu_radio != RadioIntersil) { err = atu_get_mib(sc, MIB_PHY__CHANNEL, &channel); if (!err) { DPRINTF(("%s: external firmware has already" " been downloaded\n", device_xname(sc->atu_dev))); atu_complete_attach(sc); return; } } atu_external_firmware(sc->atu_dev); /* * atu_external_firmware will call atu_complete_attach after * it's finished so we can just return. */ } else { /* all the firmwares are in place, so complete the attach */ atu_complete_attach(sc); } return; } static void atu_complete_attach(struct atu_softc *sc) { struct ieee80211com *ic = &sc->sc_ic; struct ifnet *ifp = &sc->sc_if; usb_interface_descriptor_t *id; usb_endpoint_descriptor_t *ed; usbd_status err; int i; #ifdef ATU_DEBUG struct atu_fw fw; #endif id = usbd_get_interface_descriptor(sc->atu_iface); /* Find endpoints. */ for (i = 0; i < id->bNumEndpoints; i++) { ed = usbd_interface2endpoint_descriptor(sc->atu_iface, i); if (!ed) { DPRINTF(("%s: num_endp:%d\n", device_xname(sc->atu_dev), sc->atu_iface->ui_idesc->bNumEndpoints)); DPRINTF(("%s: couldn't get ep %d\n", device_xname(sc->atu_dev), i)); return; } if (UE_GET_DIR(ed->bEndpointAddress) == UE_DIR_IN && UE_GET_XFERTYPE(ed->bmAttributes) == UE_BULK) { sc->atu_ed[ATU_ENDPT_RX] = ed->bEndpointAddress; } else if (UE_GET_DIR(ed->bEndpointAddress) == UE_DIR_OUT && UE_GET_XFERTYPE(ed->bmAttributes) == UE_BULK) { sc->atu_ed[ATU_ENDPT_TX] = ed->bEndpointAddress; } } /* read device config & get MAC address */ err = atu_get_card_config(sc); if (err) { aprint_error("\n%s: could not get card cfg!\n", device_xname(sc->atu_dev)); return; } #ifdef ATU_DEBUG /* DEBUG : try to get firmware version */ err = atu_get_mib(sc, MIB_FW_VERSION, sizeof(fw), 0, (uint8_t *)&fw); if (!err) { DPRINTFN(15, ("%s: firmware: maj:%d min:%d patch:%d " "build:%d\n", device_xname(sc->atu_dev), fw.major, fw.minor, fw.patch, fw.build)); } else { DPRINTF(("%s: get firmware version failed\n", device_xname(sc->atu_dev))); } #endif /* ATU_DEBUG */ /* Show the world our MAC address */ aprint_normal_dev(sc->atu_dev, "MAC address %s\n", ether_sprintf(ic->ic_myaddr)); sc->atu_cdata.atu_tx_inuse = 0; sc->atu_encrypt = ATU_WEP_OFF; sc->atu_wepkeylen = ATU_WEP_104BITS; sc->atu_wepkey = 0; memset(sc->atu_bssid, 0, ETHER_ADDR_LEN); sc->atu_channel = ATU_DEFAULT_CHANNEL; sc->atu_desired_channel = IEEE80211_CHAN_ANY; sc->atu_mode = INFRASTRUCTURE_MODE; ic->ic_ifp = ifp; ic->ic_phytype = IEEE80211_T_DS; ic->ic_opmode = IEEE80211_M_STA; ic->ic_state = IEEE80211_S_INIT; #ifdef FIXME ic->ic_caps = IEEE80211_C_IBSS | IEEE80211_C_WEP | IEEE80211_C_SCANALL; #else ic->ic_caps = IEEE80211_C_IBSS | IEEE80211_C_WEP; #endif i = 0; ic->ic_sup_rates[IEEE80211_MODE_11B] = ieee80211_std_rateset_11b; for (i = 1; i <= 14; i++) { ic->ic_channels[i].ic_flags = IEEE80211_CHAN_B | IEEE80211_CHAN_PASSIVE; ic->ic_channels[i].ic_freq = ieee80211_ieee2mhz(i, ic->ic_channels[i].ic_flags); } ic->ic_ibss_chan = &ic->ic_channels[0]; ifp->if_softc = sc; memcpy(ifp->if_xname, device_xname(sc->atu_dev), IFNAMSIZ); ifp->if_flags = IFF_BROADCAST | IFF_SIMPLEX | IFF_MULTICAST; ifp->if_init = atu_init; ifp->if_stop = atu_stop; ifp->if_start = atu_start; ifp->if_ioctl = atu_ioctl; ifp->if_watchdog = atu_watchdog; ifp->if_mtu = ATU_DEFAULT_MTU; IFQ_SET_READY(&ifp->if_snd); /* Call MI attach routine. */ if_attach(ifp); ieee80211_ifattach(ic); sc->sc_newstate = ic->ic_newstate; ic->ic_newstate = atu_newstate; /* setup ifmedia interface */ /* XXX media locking needs revisiting */ mutex_init(&sc->sc_media_mtx, MUTEX_DEFAULT, IPL_SOFTUSB); ieee80211_media_init_with_lock(ic, atu_media_change, atu_media_status, &sc->sc_media_mtx); usb_init_task(&sc->sc_task, atu_task, sc, 0); sc->sc_state = ATU_S_OK; } static int atu_detach(device_t self, int flags) { struct atu_softc *sc = device_private(self); struct ifnet *ifp = &sc->sc_if; DPRINTFN(10, ("%s: atu_detach state=%d\n", device_xname(sc->atu_dev), sc->sc_state)); if (sc->sc_state != ATU_S_UNCONFIG) { atu_stop(ifp, 1); ieee80211_ifdetach(&sc->sc_ic); if_detach(ifp); } return 0; } static int atu_activate(device_t self, enum devact act) { struct atu_softc *sc = device_private(self); switch (act) { case DVACT_DEACTIVATE: if (sc->sc_state != ATU_S_UNCONFIG) { if_deactivate(&sc->atu_ec.ec_if); sc->sc_state = ATU_S_DEAD; } return 0; default: return EOPNOTSUPP; } } /* * Initialize an RX descriptor and attach an MBUF cluster. */ static int atu_newbuf(struct atu_softc *sc, struct atu_chain *c, struct mbuf *m) { struct mbuf *m_new = NULL; if (m == NULL) { MGETHDR(m_new, M_DONTWAIT, MT_DATA); if (m_new == NULL) { DPRINTF(("%s: no memory for rx list\n", device_xname(sc->atu_dev))); return ENOBUFS; } MCLGET(m_new, M_DONTWAIT); if (!(m_new->m_flags & M_EXT)) { DPRINTF(("%s: no memory for rx list\n", device_xname(sc->atu_dev))); m_freem(m_new); return ENOBUFS; } m_new->m_len = m_new->m_pkthdr.len = MCLBYTES; } else { m_new = m; m_new->m_len = m_new->m_pkthdr.len = MCLBYTES; m_new->m_data = m_new->m_ext.ext_buf; } c->atu_mbuf = m_new; return 0; } static int atu_rx_list_init(struct atu_softc *sc) { struct atu_cdata *cd = &sc->atu_cdata; struct atu_chain *c; int i; DPRINTFN(15, ("%s: atu_rx_list_init: enter\n", device_xname(sc->atu_dev))); for (i = 0; i < ATU_RX_LIST_CNT; i++) { c = &cd->atu_rx_chain[i]; c->atu_sc = sc; c->atu_idx = i; if (c->atu_xfer == NULL) { int err = usbd_create_xfer(sc->atu_ep[ATU_ENDPT_RX], ATU_RX_BUFSZ, 0, 0, &c->atu_xfer); if (err) return err; c->atu_buf = usbd_get_buffer(c->atu_xfer); if (atu_newbuf(sc, c, NULL) == ENOBUFS) /* XXX free? */ return ENOBUFS; } } return 0; } static int atu_tx_list_init(struct atu_softc *sc) { struct atu_cdata *cd = &sc->atu_cdata; struct atu_chain *c; int i; DPRINTFN(15, ("%s: atu_tx_list_init\n", device_xname(sc->atu_dev))); SLIST_INIT(&cd->atu_tx_free); sc->atu_cdata.atu_tx_inuse = 0; for (i = 0; i < ATU_TX_LIST_CNT; i++) { c = &cd->atu_tx_chain[i]; c->atu_sc = sc; c->atu_idx = i; if (c->atu_xfer == NULL) { int err = usbd_create_xfer(sc->atu_ep[ATU_ENDPT_TX], ATU_TX_BUFSZ, 0, 0, &c->atu_xfer); if (err) { return err; } c->atu_buf = usbd_get_buffer(c->atu_xfer); SLIST_INSERT_HEAD(&cd->atu_tx_free, c, atu_list); } } return 0; } static void atu_xfer_list_free(struct atu_softc *sc, struct atu_chain *ch, int listlen) { int i; /* Free resources. */ for (i = 0; i < listlen; i++) { if (ch[i].atu_buf != NULL) ch[i].atu_buf = NULL; if (ch[i].atu_mbuf != NULL) { m_freem(ch[i].atu_mbuf); ch[i].atu_mbuf = NULL; } if (ch[i].atu_xfer != NULL) { usbd_destroy_xfer(ch[i].atu_xfer); ch[i].atu_xfer = NULL; } } } /* * A frame has been uploaded: pass the resulting mbuf chain up to * the higher level protocols. */ static void atu_rxeof(struct usbd_xfer *xfer, void *priv, usbd_status status) { struct atu_chain *c = (struct atu_chain *)priv; struct atu_softc *sc = c->atu_sc; struct ieee80211com *ic = &sc->sc_ic; struct ifnet *ifp = &sc->sc_if; struct atu_rx_hdr *h; struct ieee80211_frame_min *wh; struct ieee80211_node *ni; struct mbuf *m; uint32_t len; int s; DPRINTFN(25, ("%s: atu_rxeof\n", device_xname(sc->atu_dev))); if (sc->sc_state != ATU_S_OK) return; if ((ifp->if_flags & (IFF_RUNNING|IFF_UP)) != (IFF_RUNNING|IFF_UP)) goto done; if (status != USBD_NORMAL_COMPLETION) { DPRINTF(("%s: status != USBD_NORMAL_COMPLETION\n", device_xname(sc->atu_dev))); if (status == USBD_NOT_STARTED || status == USBD_CANCELLED) { return; } #if 0 if (status == USBD_IOERROR) { DPRINTF(("%s: rx: EEK! lost device?\n", device_xname(sc->atu_dev))); /* * My experience with USBD_IOERROR is that trying to * restart the transfer will always fail and we'll * keep on looping restarting transfers untill someone * pulls the plug of the device. * So we don't restart the transfer, but just let it * die... If someone knows of a situation where we can * recover from USBD_IOERROR, let me know. */ splx(s); return; } #endif /* 0 */ if (usbd_ratecheck(&sc->atu_rx_notice)) { DPRINTF(("%s: usb error on rx: %s\n", device_xname(sc->atu_dev), usbd_errstr(status))); } if (status == USBD_STALLED) usbd_clear_endpoint_stall_async( sc->atu_ep[ATU_ENDPT_RX]); goto done; } usbd_get_xfer_status(xfer, NULL, NULL, &len, NULL); if (len <= 1) { DPRINTF(("%s: atu_rxeof: too short\n", device_xname(sc->atu_dev))); goto done; } else if (len > MCLBYTES) { DPRINTF(("%s: atu_rxeof: too long\n", device_xname(sc->atu_dev))); goto done; } h = (struct atu_rx_hdr *)c->atu_buf; len = UGETW(h->length) - 4; /* XXX magic number */ m = c->atu_mbuf; memcpy(mtod(m, char *), c->atu_buf + ATU_RX_HDRLEN, len); m_set_rcvif(m, ifp); m->m_pkthdr.len = m->m_len = len; wh = mtod(m, struct ieee80211_frame_min *); ni = ieee80211_find_rxnode(ic, wh); if_statinc(ifp, if_ipackets); s = splnet(); if (atu_newbuf(sc, c, NULL) == ENOBUFS) { if_statinc(ifp, if_ierrors); goto done1; /* XXX if we can't allocate, why restart it? */ } if (wh->i_fc[1] & IEEE80211_FC1_WEP) { /* * WEP is decrypted by hardware. Clear WEP bit * header for ieee80211_input(). */ wh->i_fc[1] &= ~IEEE80211_FC1_WEP; } ieee80211_input(ic, m, ni, h->rssi, UGETDW(h->rx_time)); ieee80211_free_node(ni); done1: splx(s); done: /* Setup new transfer. */ usbd_setup_xfer(c->atu_xfer, c, c->atu_buf, ATU_RX_BUFSZ, USBD_SHORT_XFER_OK, USBD_NO_TIMEOUT, atu_rxeof); usbd_transfer(c->atu_xfer); } /* * A frame was downloaded to the chip. It's safe for us to clean up * the list buffers. */ static void atu_txeof(struct usbd_xfer *xfer, void *priv, usbd_status status) { struct atu_chain *c = (struct atu_chain *)priv; struct atu_softc *sc = c->atu_sc; struct ifnet *ifp = &sc->sc_if; usbd_status err; int s; DPRINTFN(25, ("%s: atu_txeof status=%d\n", device_xname(sc->atu_dev), status)); if (c->atu_mbuf) { m_freem(c->atu_mbuf); c->atu_mbuf = NULL; } if (status != USBD_NORMAL_COMPLETION) { if (status == USBD_NOT_STARTED || status == USBD_CANCELLED) return; DPRINTF(("%s: usb error on tx: %s\n", device_xname(sc->atu_dev), usbd_errstr(status))); if (status == USBD_STALLED) usbd_clear_endpoint_stall_async( sc->atu_ep[ATU_ENDPT_TX]); return; } usbd_get_xfer_status(c->atu_xfer, NULL, NULL, NULL, &err); if (err) if_statinc(ifp, if_oerrors); else if_statinc(ifp, if_opackets); s = splnet(); SLIST_INSERT_HEAD(&sc->atu_cdata.atu_tx_free, c, atu_list); sc->atu_cdata.atu_tx_inuse--; if (sc->atu_cdata.atu_tx_inuse == 0) ifp->if_timer = 0; ifp->if_flags &= ~IFF_OACTIVE; splx(s); atu_start(ifp); } static uint8_t atu_calculate_padding(int size) { size %= 64; if (size < 50) return 50 - size; if (size >=61) return 64 + 50 - size; return 0; } static int atu_tx_start(struct atu_softc *sc, struct ieee80211_node *ni, struct atu_chain *c, struct mbuf *m) { int len; struct atu_tx_hdr *h; usbd_status err; uint8_t pad; DPRINTFN(25, ("%s: atu_tx_start\n", device_xname(sc->atu_dev))); /* Don't try to send when we're shutting down the driver */ if (sc->sc_state != ATU_S_OK) { m_freem(m); return EIO; } /* * Copy the mbuf data into a contiguous buffer, leaving * enough room for the atmel headers */ len = m->m_pkthdr.len; m_copydata(m, 0, m->m_pkthdr.len, c->atu_buf + ATU_TX_HDRLEN); h = (struct atu_tx_hdr *)c->atu_buf; memset(h, 0, ATU_TX_HDRLEN); USETW(h->length, len); h->tx_rate = 4; /* XXX rate = auto */ len += ATU_TX_HDRLEN; pad = atu_calculate_padding(len); len += pad; h->padding = pad; c->atu_length = len; c->atu_mbuf = m; usbd_setup_xfer(c->atu_xfer, c, c->atu_buf, c->atu_length, 0, ATU_TX_TIMEOUT, atu_txeof); /* Let's get this thing into the air! */ c->atu_in_xfer = 1; err = usbd_transfer(c->atu_xfer); if (err != USBD_IN_PROGRESS) { DPRINTFN(25, ("%s: atu_tx_start, err=%d", device_xname(sc->atu_dev), err)); c->atu_mbuf = NULL; m_freem(m); return EIO; } return 0; } static void atu_start(struct ifnet *ifp) { struct atu_softc *sc = ifp->if_softc; struct ieee80211com *ic = &sc->sc_ic; struct atu_cdata *cd = &sc->atu_cdata; struct ieee80211_node *ni; struct atu_chain *c; struct mbuf *m = NULL; int s; DPRINTFN(25, ("%s: atu_start: enter\n", device_xname(sc->atu_dev))); if ((ifp->if_flags & IFF_RUNNING) == 0) { return; } if (ifp->if_flags & IFF_OACTIVE) { DPRINTFN(30, ("%s: atu_start: IFF_OACTIVE\n", device_xname(sc->atu_dev))); return; } for (;;) { /* grab a TX buffer */ s = splnet(); c = SLIST_FIRST(&cd->atu_tx_free); if (c != NULL) { SLIST_REMOVE_HEAD(&cd->atu_tx_free, atu_list); cd->atu_tx_inuse++; if (cd->atu_tx_inuse == ATU_TX_LIST_CNT) ifp->if_flags |= IFF_OACTIVE; } splx(s); if (c == NULL) { DPRINTFN(10, ("%s: out of tx xfers\n", device_xname(sc->atu_dev))); ifp->if_flags |= IFF_OACTIVE; break; } /* * Poll the management queue for frames, it has priority over * normal data frames. */ IF_DEQUEUE(&ic->ic_mgtq, m); if (m == NULL) { DPRINTFN(10, ("%s: atu_start: data packet\n", device_xname(sc->atu_dev))); if (ic->ic_state != IEEE80211_S_RUN) { DPRINTFN(25, ("%s: no data till running\n", device_xname(sc->atu_dev))); /* put the xfer back on the list */ s = splnet(); SLIST_INSERT_HEAD(&cd->atu_tx_free, c, atu_list); cd->atu_tx_inuse--; splx(s); break; } IFQ_DEQUEUE(&ifp->if_snd, m); if (m == NULL) { DPRINTFN(25, ("%s: nothing to send\n", device_xname(sc->atu_dev))); s = splnet(); SLIST_INSERT_HEAD(&cd->atu_tx_free, c, atu_list); cd->atu_tx_inuse--; splx(s); break; } bpf_mtap(ifp, m, BPF_D_OUT); ni = ieee80211_find_txnode(ic, mtod(m, struct ether_header *)->ether_dhost); if (ni == NULL) { m_freem(m); goto bad; } m = ieee80211_encap(ic, m, ni); if (m == NULL) goto bad; } else { DPRINTFN(25, ("%s: atu_start: mgmt packet\n", device_xname(sc->atu_dev))); /* * Hack! The referenced node pointer is in the * rcvif field of the packet header. This is * placed there by ieee80211_mgmt_output because * we need to hold the reference with the frame * and there's no other way (other than packet * tags which we consider too expensive to use) * to pass it along. */ ni = M_GETCTX(m, struct ieee80211_node *); M_CLEARCTX(m); /* sc->sc_stats.ast_tx_mgmt++; */ } bpf_mtap3(ic->ic_rawbpf, m, BPF_D_OUT); if (atu_tx_start(sc, ni, c, m)) { bad: s = splnet(); SLIST_INSERT_HEAD(&cd->atu_tx_free, c, atu_list); cd->atu_tx_inuse--; splx(s); /* if_statinc(ifp, if_oerrors); */ if (ni != NULL) ieee80211_free_node(ni); continue; } ifp->if_timer = 5; } } static int atu_init(struct ifnet *ifp) { struct atu_softc *sc = ifp->if_softc; struct ieee80211com *ic = &sc->sc_ic; struct atu_chain *c; usbd_status err; int i, s; s = splnet(); DPRINTFN(10, ("%s: atu_init\n", device_xname(sc->atu_dev))); if (ifp->if_flags & IFF_RUNNING) { splx(s); return 0; } /* Load the multicast filter. */ /*atu_setmulti(sc); */ /* Open RX and TX pipes. */ err = usbd_open_pipe(sc->atu_iface, sc->atu_ed[ATU_ENDPT_RX], USBD_EXCLUSIVE_USE, &sc->atu_ep[ATU_ENDPT_RX]); if (err) { DPRINTF(("%s: open rx pipe failed: %s\n", device_xname(sc->atu_dev), usbd_errstr(err))); splx(s); return EIO; } err = usbd_open_pipe(sc->atu_iface, sc->atu_ed[ATU_ENDPT_TX], USBD_EXCLUSIVE_USE, &sc->atu_ep[ATU_ENDPT_TX]); if (err) { DPRINTF(("%s: open tx pipe failed: %s\n", device_xname(sc->atu_dev), usbd_errstr(err))); splx(s); return EIO; } /* Init TX ring */ if (atu_tx_list_init(sc)) printf("%s: tx list init failed\n", device_xname(sc->atu_dev)); /* Init RX ring */ if (atu_rx_list_init(sc)) printf("%s: rx list init failed\n", device_xname(sc->atu_dev)); /* Start up the receive pipe. */ for (i = 0; i < ATU_RX_LIST_CNT; i++) { c = &sc->atu_cdata.atu_rx_chain[i]; usbd_setup_xfer(c->atu_xfer, c, c->atu_buf, ATU_RX_BUFSZ, USBD_SHORT_XFER_OK, USBD_NO_TIMEOUT, atu_rxeof); usbd_transfer(c->atu_xfer); } DPRINTFN(10, ("%s: starting up using MAC=%s\n", device_xname(sc->atu_dev), ether_sprintf(ic->ic_myaddr))); /* Do initial setup */ err = atu_initial_config(sc); if (err) { DPRINTF(("%s: initial config failed!\n", device_xname(sc->atu_dev))); splx(s); return EIO; } DPRINTFN(10, ("%s: initialised transceiver\n", device_xname(sc->atu_dev))); /* sc->atu_rxfilt = ATU_RXFILT_UNICAST|ATU_RXFILT_BROADCAST; */ /* If we want promiscuous mode, set the allframes bit. */ /* if (ifp->if_flags & IFF_PROMISC) sc->atu_rxfilt |= ATU_RXFILT_PROMISC; */ ifp->if_flags |= IFF_RUNNING; ifp->if_flags &= ~IFF_OACTIVE; splx(s); /* XXX the following HAS to be replaced */ s = splnet(); err = ieee80211_new_state(ic, IEEE80211_S_SCAN, -1); if (err) { DPRINTFN(1, ("%s: atu_init: error calling " "ieee80211_net_state", device_xname(sc->atu_dev))); } splx(s); return 0; } #if 0 && defined(ATU_DEBUG) /* XXX XXX XXX UNUSED */ static void atu_debug_print(struct atu_softc *); static void atu_debug_print(struct atu_softc *sc) { usbd_status err; uint8_t tmp[32]; /* DEBUG */ if ((err = atu_get_mib(sc, MIB_MAC_MGMT__CURRENT_BSSID, tmp))) return; DPRINTF(("%s: DEBUG: current BSSID=%s\n", device_xname(sc->atu_dev), ether_sprintf(tmp))); if ((err = atu_get_mib(sc, MIB_MAC_MGMT__BEACON_PERIOD, tmp))) return; DPRINTF(("%s: DEBUG: beacon period=%d\n", device_xname(sc->atu_dev), tmp[0])); if ((err = atu_get_mib(sc, MIB_MAC_WEP__PRIVACY_INVOKED, tmp))) return; DPRINTF(("%s: DEBUG: privacy invoked=%d\n", device_xname(sc->atu_dev), tmp[0])); if ((err = atu_get_mib(sc, MIB_MAC_WEP__ENCR_LEVEL, tmp))) return; DPRINTF(("%s: DEBUG: encr_level=%d\n", device_xname(sc->atu_dev), tmp[0])); if ((err = atu_get_mib(sc, MIB_MAC_WEP__ICV_ERROR_COUNT, tmp))) return; DPRINTF(("%s: DEBUG: icv error count=%d\n", device_xname(sc->atu_dev), *(short *)tmp)); if ((err = atu_get_mib(sc, MIB_MAC_WEP__EXCLUDED_COUNT, tmp))) return; DPRINTF(("%s: DEBUG: wep excluded count=%d\n", device_xname(sc->atu_dev), *(short *)tmp)); if ((err = atu_get_mib(sc, MIB_MAC_MGMT__POWER_MODE, tmp))) return; DPRINTF(("%s: DEBUG: power mode=%d\n", device_xname(sc->atu_dev), tmp[0])); if ((err = atu_get_mib(sc, MIB_PHY__CHANNEL, tmp))) return; DPRINTF(("%s: DEBUG: channel=%d\n", device_xname(sc->atu_dev), tmp[0])); if ((err = atu_get_mib(sc, MIB_PHY__REG_DOMAIN, tmp))) return; DPRINTF(("%s: DEBUG: reg domain=%d\n", device_xname(sc->atu_dev), tmp[0])); if ((err = atu_get_mib(sc, MIB_LOCAL__SSID_SIZE, tmp))) return; DPRINTF(("%s: DEBUG: ssid size=%d\n", device_xname(sc->atu_dev), tmp[0])); if ((err = atu_get_mib(sc, MIB_LOCAL__BEACON_ENABLE, tmp))) return; DPRINTF(("%s: DEBUG: beacon enable=%d\n", device_xname(sc->atu_dev), tmp[0])); if ((err = atu_get_mib(sc, MIB_LOCAL__AUTO_RATE_FALLBACK, tmp))) return; DPRINTF(("%s: DEBUG: auto rate fallback=%d\n", device_xname(sc->atu_dev), tmp[0])); if ((err = atu_get_mib(sc, MIB_MAC_ADDR__ADDR, tmp))) return; DPRINTF(("%s: DEBUG: mac addr=%s\n", device_xname(sc->atu_dev), ether_sprintf(tmp))); if ((err = atu_get_mib(sc, MIB_MAC__DESIRED_SSID, tmp))) return; DPRINTF(("%s: DEBUG: desired ssid=%s\n", device_xname(sc->atu_dev), tmp)); if ((err = atu_get_mib(sc, MIB_MAC_MGMT__CURRENT_ESSID, tmp))) return; DPRINTF(("%s: DEBUG: current ESSID=%s\n", device_xname(sc->atu_dev), tmp)); } #endif /* ATU_DEBUG */ static int atu_ioctl(struct ifnet *ifp, u_long command, void *data) { struct atu_softc *sc = ifp->if_softc; struct ieee80211com *ic = &sc->sc_ic; int err = 0, s; s = splnet(); switch (command) { default: DPRINTFN(15, ("%s: ieee80211_ioctl (%lu)\n", device_xname(sc->atu_dev), command)); err = ieee80211_ioctl(ic, command, data); break; } if (err == ENETRESET) { if ((ifp->if_flags & (IFF_RUNNING|IFF_UP)) == (IFF_RUNNING|IFF_UP)) { DPRINTF(("%s: atu_ioctl(): netreset %lu\n", device_xname(sc->atu_dev), command)); ieee80211_new_state(ic, IEEE80211_S_INIT, -1); atu_initial_config(sc); ieee80211_new_state(ic, IEEE80211_S_SCAN, -1); } err = 0; } splx(s); return err; } static void atu_watchdog(struct ifnet *ifp) { struct atu_softc *sc = ifp->if_softc; struct atu_chain *c; usbd_status stat; int cnt, s; DPRINTF(("%s: atu_watchdog\n", device_xname(sc->atu_dev))); ifp->if_timer = 0; if (sc->sc_state != ATU_S_OK || (ifp->if_flags & IFF_RUNNING) == 0) return; sc = ifp->if_softc; s = splnet(); if_statinc(ifp, if_oerrors); DPRINTF(("%s: watchdog timeout\n", device_xname(sc->atu_dev))); /* * TODO: * we should change this since we have multiple TX tranfers... */ for (cnt = 0; cnt < ATU_TX_LIST_CNT; cnt++) { c = &sc->atu_cdata.atu_tx_chain[cnt]; if (c->atu_in_xfer) { usbd_get_xfer_status(c->atu_xfer, NULL, NULL, NULL, &stat); atu_txeof(c->atu_xfer, c, stat); } } if (!IFQ_IS_EMPTY(&ifp->if_snd)) atu_start(ifp); splx(s); ieee80211_watchdog(&sc->sc_ic); } /* * Stop the adapter and free any mbufs allocated to the * RX and TX lists. */ static void atu_stop(struct ifnet *ifp, int disable) { struct atu_softc *sc = ifp->if_softc; struct ieee80211com *ic = &sc->sc_ic; struct atu_cdata *cd; int s; s = splnet(); ifp->if_flags &= ~(IFF_RUNNING | IFF_OACTIVE); ifp->if_timer = 0; usb_rem_task_wait(sc->atu_udev, &sc->sc_task, USB_TASKQ_DRIVER, NULL); ieee80211_new_state(ic, IEEE80211_S_INIT, -1); /* Stop transfers. */ if (sc->atu_ep[ATU_ENDPT_RX] != NULL) { usbd_abort_pipe(sc->atu_ep[ATU_ENDPT_RX]); } if (sc->atu_ep[ATU_ENDPT_TX] != NULL) { usbd_abort_pipe(sc->atu_ep[ATU_ENDPT_TX]); } /* Free RX/TX/MGMT list resources. */ cd = &sc->atu_cdata; atu_xfer_list_free(sc, cd->atu_rx_chain, ATU_RX_LIST_CNT); atu_xfer_list_free(sc, cd->atu_tx_chain, ATU_TX_LIST_CNT); /* Close pipes */ if (sc->atu_ep[ATU_ENDPT_RX] != NULL) { usbd_close_pipe(sc->atu_ep[ATU_ENDPT_RX]); sc->atu_ep[ATU_ENDPT_RX] = NULL; } if (sc->atu_ep[ATU_ENDPT_TX] != NULL) { usbd_close_pipe(sc->atu_ep[ATU_ENDPT_TX]); sc->atu_ep[ATU_ENDPT_TX] = NULL; } /* Let's be nice and turn off the radio before we leave */ atu_switch_radio(sc, 0); splx(s); } |
| 8 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 197 198 199 200 201 202 203 204 205 206 207 208 209 210 211 212 213 214 215 216 217 218 219 220 221 222 223 224 225 226 227 228 229 230 231 232 233 234 235 236 237 238 239 240 241 242 243 244 245 246 247 248 249 250 251 252 253 254 255 256 257 258 259 260 261 262 263 264 265 266 267 268 269 270 271 272 273 274 275 276 277 278 279 280 281 282 283 284 285 286 287 288 289 290 291 292 293 294 295 296 297 298 299 300 301 302 303 304 305 306 307 308 309 310 311 312 313 314 315 316 317 318 319 320 321 322 323 324 325 326 327 328 329 330 331 332 333 334 335 336 337 338 339 340 341 342 343 344 345 346 347 348 349 350 351 352 353 354 355 356 357 358 359 360 361 362 363 364 365 366 367 368 369 370 371 372 373 374 375 376 377 378 379 380 381 382 383 384 385 386 387 388 389 390 391 392 393 394 395 396 397 398 399 400 401 402 403 404 405 406 407 408 409 410 411 412 413 414 415 416 417 418 419 420 421 422 423 424 425 426 427 428 429 430 431 432 433 434 435 436 437 438 439 440 441 442 443 444 445 446 447 448 449 450 451 452 453 454 455 456 457 458 459 460 461 462 463 464 465 466 467 468 469 470 471 472 473 474 475 476 477 478 479 480 481 482 483 484 485 486 487 488 489 490 491 492 493 494 495 496 497 498 499 500 501 502 503 504 505 506 507 508 509 510 511 512 513 514 515 516 517 518 519 520 521 522 523 524 525 526 527 528 529 530 531 532 533 534 535 536 537 538 539 540 541 542 543 544 545 546 547 548 549 550 551 552 553 554 555 556 557 558 559 560 561 562 563 564 565 566 567 568 569 570 571 572 573 574 575 576 577 578 579 580 581 582 583 584 585 586 587 588 589 590 591 592 593 594 595 596 597 598 599 600 601 602 603 604 605 606 607 608 609 610 611 612 613 614 615 616 617 618 619 620 621 622 623 624 625 626 627 628 629 630 631 632 633 634 635 636 637 638 639 640 641 642 643 644 645 646 647 648 649 650 651 652 653 654 655 656 657 658 659 660 661 662 663 664 665 666 667 668 669 670 671 672 673 674 675 676 677 678 679 680 681 682 683 684 685 686 687 688 689 690 691 692 693 694 695 696 697 698 699 700 701 702 703 704 705 706 707 708 709 710 711 712 713 714 715 716 717 718 719 720 721 722 723 724 725 726 727 728 729 730 731 732 733 734 735 736 737 738 739 740 741 742 743 744 745 746 747 748 749 750 751 752 753 754 755 756 757 758 759 760 761 762 763 764 765 766 767 768 769 770 771 772 773 774 775 776 777 778 779 780 781 782 783 784 785 786 787 788 789 790 791 792 793 794 795 796 797 798 799 800 801 802 803 804 805 806 807 808 809 810 811 812 813 814 815 816 817 818 819 820 821 822 823 824 825 826 827 828 829 830 831 832 833 834 835 836 837 838 839 840 841 842 843 844 845 846 847 848 849 850 851 852 853 854 855 856 857 858 859 860 861 862 863 864 865 866 867 868 869 870 871 872 873 874 875 876 877 878 879 880 881 882 883 884 885 886 887 888 889 890 891 892 893 894 895 896 897 898 899 900 901 902 903 904 905 906 907 908 909 910 911 912 913 914 915 916 917 918 919 920 921 922 923 924 925 926 927 928 929 930 931 932 933 934 935 936 937 938 939 940 941 942 943 944 945 946 947 948 949 950 951 952 953 954 955 956 957 958 959 960 961 962 963 964 965 966 967 968 969 970 971 972 973 974 975 976 977 978 979 980 981 982 983 984 985 986 987 988 989 990 991 992 993 994 995 996 997 998 999 1000 1001 1002 1003 1004 1005 1006 1007 1008 1009 1010 1011 1012 1013 1014 1015 1016 1017 1018 1019 1020 1021 1022 1023 1024 1025 1026 1027 1028 1029 1030 1031 1032 1033 1034 1035 1036 1037 1038 1039 1040 1041 1042 1043 1044 1045 1046 1047 1048 1049 1050 1051 1052 1053 1054 1055 1056 1057 1058 1059 1060 1061 1062 1063 1064 1065 1066 1067 1068 1069 1070 1071 1072 1073 1074 1075 1076 1077 1078 1079 1080 1081 1082 1083 1084 1085 1086 1087 1088 1089 1090 1091 1092 1093 1094 1095 1096 1097 1098 1099 1100 1101 1102 1103 1104 1105 1106 1107 1108 1109 1110 1111 1112 1113 1114 1115 1116 1117 1118 1119 1120 1121 1122 1123 1124 1125 1126 1127 1128 1129 1130 1131 1132 1133 1134 1135 1136 1137 1138 1139 1140 1141 1142 1143 1144 1145 1146 1147 1148 1149 1150 1151 1152 1153 1154 1155 1156 1157 1158 1159 1160 1161 1162 1163 1164 1165 1166 1167 1168 1169 1170 1171 1172 1173 1174 1175 1176 1177 1178 1179 1180 1181 1182 1183 1184 1185 1186 1187 1188 1189 1190 1191 1192 1193 1194 1195 1196 1197 1198 1199 1200 1201 1202 1203 1204 1205 1206 1207 1208 1209 1210 1211 1212 1213 1214 1215 1216 1217 1218 1219 1220 1221 1222 1223 1224 1225 1226 1227 1228 1229 1230 1231 1232 1233 1234 1235 1236 1237 1238 1239 1240 1241 1242 1243 1244 1245 1246 1247 1248 1249 1250 1251 1252 1253 1254 1255 1256 1257 1258 1259 1260 1261 1262 1263 1264 1265 1266 1267 1268 1269 1270 1271 1272 1273 1274 1275 1276 1277 1278 1279 1280 1281 1282 1283 1284 1285 1286 1287 1288 1289 1290 1291 1292 1293 1294 1295 1296 1297 1298 1299 1300 1301 1302 1303 1304 1305 1306 1307 1308 1309 1310 1311 1312 1313 1314 1315 1316 1317 1318 1319 1320 1321 1322 1323 1324 1325 1326 1327 1328 1329 1330 1331 1332 1333 1334 1335 1336 1337 1338 1339 1340 1341 1342 1343 1344 1345 1346 1347 1348 1349 1350 1351 1352 1353 1354 1355 1356 1357 1358 1359 1360 1361 1362 1363 1364 1365 1366 1367 1368 1369 1370 1371 1372 1373 1374 1375 1376 1377 1378 1379 1380 1381 1382 1383 1384 1385 1386 1387 1388 1389 1390 1391 1392 1393 1394 1395 1396 1397 1398 1399 1400 1401 1402 1403 1404 1405 1406 1407 1408 1409 1410 1411 1412 1413 1414 1415 1416 1417 1418 1419 1420 1421 1422 1423 1424 1425 1426 1427 1428 1429 1430 1431 1432 1433 1434 1435 1436 1437 1438 1439 1440 1441 1442 1443 1444 1445 1446 1447 1448 1449 1450 1451 1452 1453 1454 1455 1456 1457 1458 1459 1460 1461 1462 1463 1464 1465 1466 1467 1468 1469 1470 1471 1472 1473 1474 1475 1476 1477 1478 1479 1480 1481 1482 1483 1484 1485 1486 1487 1488 1489 1490 1491 1492 1493 1494 1495 1496 1497 1498 1499 1500 1501 1502 1503 1504 1505 1506 1507 1508 1509 1510 1511 1512 1513 1514 1515 1516 1517 1518 1519 1520 1521 1522 1523 1524 1525 1526 1527 1528 1529 1530 1531 1532 1533 1534 1535 1536 1537 1538 1539 1540 1541 1542 1543 1544 1545 1546 1547 1548 1549 1550 1551 1552 1553 1554 1555 1556 1557 1558 1559 1560 1561 1562 1563 1564 1565 1566 1567 1568 1569 1570 1571 1572 1573 1574 1575 1576 1577 1578 1579 1580 1581 1582 1583 1584 1585 1586 1587 1588 1589 1590 1591 1592 1593 1594 1595 1596 1597 1598 1599 1600 1601 1602 1603 1604 1605 1606 1607 1608 1609 1610 1611 1612 1613 1614 1615 1616 1617 1618 1619 1620 1621 1622 1623 1624 1625 1626 1627 1628 1629 1630 1631 1632 1633 1634 1635 1636 1637 1638 1639 1640 1641 1642 1643 1644 1645 1646 1647 1648 1649 1650 1651 1652 1653 1654 1655 1656 1657 1658 1659 1660 1661 1662 1663 1664 1665 1666 1667 1668 1669 1670 1671 1672 1673 1674 1675 1676 1677 1678 1679 1680 1681 1682 1683 1684 1685 1686 1687 1688 1689 1690 1691 1692 1693 1694 1695 1696 1697 1698 1699 1700 1701 1702 1703 1704 1705 1706 1707 1708 1709 1710 1711 1712 1713 1714 1715 1716 1717 1718 1719 1720 1721 1722 1723 1724 1725 1726 1727 1728 1729 1730 1731 1732 1733 1734 1735 1736 1737 1738 1739 1740 1741 1742 1743 1744 1745 1746 1747 1748 1749 1750 1751 1752 1753 1754 1755 1756 1757 1758 1759 1760 1761 1762 1763 1764 1765 1766 1767 1768 1769 1770 1771 1772 1773 1774 1775 1776 1777 1778 1779 1780 1781 1782 1783 1784 1785 1786 1787 1788 1789 1790 1791 1792 1793 1794 1795 1796 1797 1798 1799 1800 1801 1802 1803 1804 1805 1806 1807 1808 1809 1810 1811 1812 1813 1814 1815 1816 1817 1818 1819 1820 1821 1822 1823 1824 1825 1826 1827 1828 1829 1830 1831 1832 1833 1834 1835 1836 1837 1838 1839 1840 1841 1842 1843 1844 1845 1846 1847 1848 1849 1850 1851 1852 1853 1854 1855 1856 1857 1858 1859 1860 1861 1862 1863 1864 1865 1866 1867 1868 1869 1870 1871 1872 1873 1874 1875 1876 1877 1878 1879 1880 1881 1882 1883 1884 1885 1886 1887 1888 1889 1890 1891 1892 1893 1894 1895 1896 1897 1898 1899 1900 1901 1902 1903 1904 1905 1906 1907 1908 1909 1910 1911 1912 1913 1914 1915 1916 1917 1918 1919 1920 1921 1922 1923 1924 1925 1926 1927 1928 1929 1930 1931 1932 1933 1934 1935 1936 1937 1938 1939 1940 1941 1942 1943 1944 1945 1946 1947 1948 1949 1950 1951 1952 1953 1954 1955 1956 1957 1958 1959 1960 1961 1962 1963 1964 1965 1966 1967 1968 1969 1970 1971 1972 1973 1974 1975 1976 1977 1978 1979 1980 1981 1982 1983 1984 1985 1986 1987 1988 1989 1990 1991 1992 1993 1994 1995 1996 1997 1998 1999 2000 2001 2002 2003 2004 2005 2006 2007 2008 2009 2010 2011 2012 2013 2014 2015 2016 2017 2018 2019 2020 2021 2022 2023 2024 2025 2026 2027 2028 2029 2030 2031 2032 2033 2034 2035 2036 2037 2038 2039 2040 2041 2042 2043 2044 2045 2046 2047 2048 2049 2050 2051 2052 2053 2054 2055 2056 2057 2058 2059 2060 2061 2062 2063 2064 2065 2066 2067 2068 2069 2070 2071 2072 2073 2074 2075 2076 2077 2078 2079 2080 2081 2082 2083 2084 2085 2086 2087 2088 2089 2090 2091 2092 2093 2094 2095 2096 2097 2098 2099 2100 2101 2102 2103 2104 2105 2106 2107 2108 2109 2110 2111 2112 2113 2114 2115 2116 2117 2118 2119 2120 2121 2122 2123 2124 2125 2126 2127 2128 2129 2130 2131 2132 2133 2134 2135 2136 2137 2138 2139 2140 2141 2142 2143 2144 2145 2146 2147 2148 2149 2150 2151 2152 2153 2154 2155 2156 2157 2158 2159 2160 2161 2162 2163 2164 2165 2166 2167 2168 2169 2170 2171 2172 2173 2174 2175 2176 2177 2178 2179 2180 2181 2182 2183 2184 2185 2186 2187 2188 2189 2190 2191 2192 2193 2194 2195 2196 2197 2198 2199 2200 2201 2202 2203 2204 2205 2206 2207 2208 2209 2210 2211 2212 2213 2214 2215 2216 2217 2218 2219 2220 2221 2222 2223 2224 2225 2226 2227 2228 2229 2230 2231 2232 2233 2234 2235 2236 2237 2238 2239 2240 2241 2242 2243 2244 2245 2246 2247 2248 2249 2250 2251 2252 2253 2254 2255 2256 2257 2258 2259 2260 2261 2262 2263 2264 2265 2266 2267 2268 2269 2270 2271 2272 2273 2274 2275 2276 2277 2278 2279 2280 2281 2282 2283 2284 2285 2286 2287 2288 2289 2290 2291 2292 2293 2294 2295 2296 2297 2298 2299 2300 2301 2302 2303 2304 2305 2306 2307 2308 2309 2310 2311 2312 2313 2314 2315 2316 2317 2318 2319 2320 2321 2322 2323 2324 2325 2326 2327 2328 2329 2330 2331 2332 2333 2334 2335 2336 2337 2338 2339 2340 2341 2342 2343 2344 2345 2346 2347 2348 2349 2350 2351 2352 2353 2354 2355 2356 2357 2358 2359 2360 2361 2362 2363 2364 2365 2366 2367 2368 2369 2370 2371 2372 2373 2374 2375 2376 2377 2378 2379 2380 2381 2382 2383 2384 2385 2386 2387 2388 2389 2390 2391 2392 2393 2394 2395 2396 2397 2398 2399 2400 2401 2402 2403 2404 2405 2406 2407 2408 2409 2410 2411 2412 2413 2414 2415 2416 2417 2418 2419 2420 2421 2422 2423 2424 2425 2426 2427 2428 2429 2430 2431 2432 2433 2434 2435 2436 2437 2438 2439 2440 2441 2442 2443 2444 2445 2446 2447 2448 2449 2450 2451 2452 2453 2454 2455 2456 2457 2458 2459 2460 2461 2462 2463 2464 2465 2466 2467 2468 2469 2470 2471 2472 2473 2474 2475 2476 2477 2478 2479 2480 2481 2482 2483 2484 2485 2486 2487 2488 2489 2490 2491 2492 2493 2494 2495 2496 2497 2498 2499 2500 2501 2502 2503 2504 2505 2506 2507 2508 2509 2510 2511 2512 2513 2514 2515 2516 2517 2518 2519 2520 2521 2522 2523 2524 2525 2526 2527 2528 2529 2530 2531 2532 2533 2534 2535 2536 2537 2538 2539 2540 2541 2542 2543 2544 2545 2546 2547 2548 2549 2550 2551 2552 2553 2554 2555 2556 2557 2558 2559 2560 2561 2562 2563 2564 2565 2566 2567 2568 2569 2570 2571 2572 2573 2574 2575 2576 2577 2578 2579 2580 2581 2582 2583 2584 2585 2586 2587 2588 2589 2590 2591 2592 2593 2594 2595 2596 2597 2598 2599 2600 2601 2602 2603 2604 2605 2606 2607 2608 2609 2610 2611 2612 2613 2614 2615 2616 2617 2618 2619 2620 2621 2622 2623 2624 2625 2626 2627 2628 2629 2630 2631 2632 2633 2634 2635 2636 2637 2638 2639 2640 2641 2642 2643 2644 2645 2646 2647 2648 2649 2650 2651 2652 2653 2654 2655 2656 2657 2658 2659 2660 2661 2662 2663 2664 2665 2666 2667 2668 2669 2670 2671 2672 2673 2674 2675 2676 2677 2678 2679 2680 2681 2682 2683 2684 2685 2686 2687 2688 2689 2690 2691 2692 2693 2694 2695 2696 2697 2698 2699 2700 2701 2702 2703 2704 2705 2706 2707 2708 2709 2710 2711 2712 2713 2714 2715 2716 2717 2718 2719 2720 2721 2722 2723 2724 2725 2726 2727 2728 2729 2730 2731 2732 2733 2734 2735 2736 2737 2738 2739 2740 2741 2742 2743 2744 2745 2746 2747 2748 2749 2750 2751 2752 2753 2754 2755 2756 2757 2758 2759 2760 2761 2762 2763 2764 2765 2766 2767 2768 2769 2770 2771 2772 2773 2774 2775 2776 2777 2778 2779 2780 2781 2782 2783 2784 2785 2786 2787 2788 2789 2790 2791 2792 2793 2794 2795 2796 2797 2798 2799 2800 2801 2802 2803 2804 2805 2806 2807 2808 2809 2810 2811 2812 2813 2814 2815 2816 2817 2818 2819 2820 2821 2822 2823 2824 2825 2826 2827 2828 2829 2830 2831 2832 2833 2834 2835 2836 2837 2838 2839 2840 2841 2842 2843 2844 2845 2846 2847 2848 2849 2850 2851 2852 2853 2854 2855 2856 2857 2858 2859 2860 2861 2862 2863 2864 2865 2866 2867 2868 2869 2870 2871 2872 2873 2874 2875 2876 2877 2878 2879 2880 2881 2882 2883 2884 2885 2886 2887 2888 2889 2890 2891 2892 2893 2894 2895 2896 2897 2898 2899 2900 2901 2902 2903 2904 2905 2906 2907 2908 2909 2910 2911 2912 2913 2914 2915 2916 2917 2918 2919 2920 2921 2922 2923 2924 2925 2926 2927 2928 2929 2930 2931 2932 2933 2934 2935 2936 2937 2938 2939 2940 2941 2942 2943 2944 2945 2946 2947 2948 2949 2950 2951 2952 2953 2954 2955 2956 2957 2958 2959 2960 2961 2962 2963 2964 2965 2966 2967 2968 2969 2970 2971 2972 2973 2974 2975 2976 2977 2978 2979 2980 2981 2982 2983 2984 2985 2986 2987 2988 2989 2990 2991 2992 2993 2994 2995 2996 2997 2998 2999 3000 3001 3002 3003 3004 3005 3006 3007 3008 3009 3010 3011 3012 3013 3014 3015 3016 3017 3018 3019 3020 3021 3022 3023 3024 3025 3026 3027 3028 3029 3030 3031 3032 3033 3034 3035 3036 3037 3038 3039 3040 3041 3042 3043 3044 3045 3046 3047 3048 3049 3050 3051 3052 3053 3054 3055 3056 3057 3058 3059 3060 3061 3062 3063 3064 3065 3066 3067 3068 3069 3070 3071 3072 3073 3074 3075 3076 3077 3078 3079 3080 3081 3082 3083 3084 3085 3086 3087 3088 3089 3090 3091 3092 3093 3094 3095 3096 3097 3098 3099 3100 3101 3102 3103 3104 3105 3106 3107 3108 3109 3110 3111 3112 3113 3114 3115 3116 3117 3118 3119 3120 3121 3122 3123 3124 3125 3126 3127 3128 3129 3130 3131 3132 3133 3134 3135 3136 3137 3138 3139 3140 3141 3142 3143 3144 3145 3146 3147 3148 3149 3150 3151 3152 3153 3154 3155 3156 3157 3158 3159 3160 3161 3162 3163 3164 3165 3166 3167 3168 3169 3170 3171 3172 3173 3174 3175 3176 3177 3178 3179 3180 3181 3182 3183 3184 3185 3186 3187 3188 3189 3190 3191 3192 3193 3194 3195 3196 3197 3198 3199 3200 3201 3202 3203 3204 3205 3206 3207 3208 3209 3210 3211 3212 3213 3214 3215 3216 3217 3218 3219 3220 3221 3222 3223 3224 3225 3226 3227 3228 3229 3230 3231 3232 3233 3234 3235 3236 3237 3238 3239 3240 3241 3242 3243 3244 3245 3246 3247 3248 3249 3250 3251 3252 3253 3254 3255 3256 3257 3258 3259 3260 3261 3262 3263 3264 3265 3266 3267 3268 3269 3270 3271 3272 3273 3274 3275 3276 3277 3278 3279 3280 3281 3282 3283 3284 3285 3286 3287 3288 3289 3290 3291 3292 3293 3294 3295 3296 3297 3298 3299 3300 3301 3302 3303 3304 3305 3306 3307 3308 3309 3310 3311 3312 3313 3314 3315 3316 3317 3318 3319 3320 3321 3322 3323 3324 3325 3326 3327 3328 3329 3330 3331 3332 3333 3334 3335 3336 3337 3338 3339 3340 3341 3342 3343 3344 3345 3346 3347 3348 3349 3350 3351 3352 3353 3354 3355 3356 3357 3358 3359 3360 3361 3362 3363 3364 3365 3366 3367 3368 3369 3370 3371 3372 3373 3374 3375 3376 3377 3378 3379 3380 3381 3382 3383 3384 3385 3386 3387 3388 3389 3390 3391 3392 3393 3394 3395 3396 3397 3398 3399 3400 3401 3402 3403 3404 3405 3406 3407 3408 3409 3410 3411 3412 3413 3414 3415 3416 3417 3418 3419 3420 3421 3422 3423 3424 3425 3426 3427 3428 3429 3430 3431 3432 3433 3434 3435 3436 3437 3438 3439 3440 3441 3442 3443 3444 3445 3446 3447 3448 3449 3450 3451 3452 3453 3454 3455 3456 3457 3458 3459 3460 3461 3462 3463 3464 3465 3466 3467 3468 3469 3470 3471 3472 3473 3474 3475 3476 3477 3478 3479 3480 3481 3482 3483 3484 3485 3486 3487 3488 3489 3490 3491 3492 3493 3494 3495 3496 3497 3498 3499 3500 3501 3502 3503 3504 3505 3506 3507 3508 3509 3510 3511 3512 3513 3514 3515 3516 3517 3518 3519 3520 3521 3522 3523 3524 3525 3526 3527 3528 3529 3530 3531 3532 3533 3534 3535 3536 3537 3538 3539 3540 3541 3542 3543 3544 3545 3546 3547 3548 3549 3550 3551 3552 3553 3554 3555 3556 3557 3558 3559 3560 3561 3562 3563 3564 3565 3566 3567 3568 3569 3570 3571 3572 3573 3574 3575 3576 3577 3578 3579 3580 3581 3582 3583 3584 3585 3586 3587 3588 3589 3590 3591 3592 3593 3594 3595 3596 3597 3598 3599 3600 3601 3602 3603 3604 3605 3606 3607 3608 3609 3610 3611 3612 3613 3614 3615 3616 3617 3618 3619 3620 3621 3622 3623 3624 3625 3626 3627 3628 3629 3630 3631 3632 3633 3634 3635 3636 3637 3638 3639 3640 3641 3642 3643 3644 3645 3646 3647 3648 3649 3650 3651 3652 3653 3654 3655 3656 3657 3658 3659 3660 3661 3662 3663 3664 3665 3666 3667 3668 3669 3670 3671 3672 3673 3674 3675 3676 3677 3678 3679 3680 3681 3682 3683 3684 3685 3686 3687 3688 3689 3690 3691 3692 3693 3694 3695 3696 3697 3698 3699 3700 3701 3702 3703 3704 3705 3706 3707 3708 3709 3710 3711 3712 3713 3714 3715 3716 3717 3718 3719 3720 3721 3722 3723 3724 3725 3726 3727 3728 3729 3730 3731 3732 3733 3734 3735 3736 3737 3738 3739 3740 3741 3742 3743 3744 3745 3746 3747 3748 3749 3750 3751 3752 3753 3754 3755 3756 3757 3758 3759 3760 3761 3762 3763 3764 3765 3766 3767 3768 3769 3770 3771 3772 3773 3774 3775 3776 3777 3778 3779 3780 3781 3782 3783 3784 3785 3786 3787 3788 3789 3790 3791 3792 3793 3794 3795 3796 3797 3798 3799 3800 3801 3802 3803 3804 3805 3806 3807 3808 3809 3810 3811 3812 3813 3814 3815 3816 3817 3818 3819 3820 3821 3822 3823 3824 3825 3826 3827 3828 3829 3830 3831 3832 3833 3834 3835 3836 3837 3838 3839 3840 3841 3842 3843 3844 3845 3846 3847 3848 3849 3850 3851 3852 3853 3854 3855 3856 3857 3858 3859 3860 3861 3862 3863 3864 3865 3866 3867 3868 3869 3870 3871 3872 3873 3874 3875 3876 3877 3878 3879 3880 3881 3882 3883 3884 3885 3886 3887 3888 3889 3890 3891 3892 3893 3894 3895 3896 3897 3898 3899 3900 3901 3902 3903 3904 3905 3906 3907 3908 3909 3910 3911 3912 3913 3914 3915 3916 3917 3918 3919 3920 3921 3922 3923 3924 3925 3926 3927 3928 3929 3930 3931 3932 3933 3934 3935 3936 3937 3938 3939 3940 3941 3942 3943 3944 3945 3946 3947 3948 3949 3950 3951 3952 3953 3954 3955 3956 3957 3958 3959 3960 3961 3962 3963 3964 3965 3966 3967 3968 3969 3970 3971 3972 3973 3974 3975 3976 3977 3978 3979 3980 3981 3982 3983 3984 3985 3986 3987 3988 3989 3990 3991 3992 3993 3994 3995 3996 3997 3998 3999 4000 4001 4002 4003 4004 4005 4006 4007 4008 4009 4010 4011 4012 4013 4014 4015 4016 4017 4018 4019 4020 4021 4022 4023 4024 4025 4026 4027 4028 4029 4030 4031 4032 4033 4034 4035 4036 4037 4038 4039 4040 4041 4042 4043 4044 4045 4046 4047 4048 4049 4050 4051 4052 4053 4054 4055 4056 4057 4058 4059 4060 4061 4062 4063 4064 4065 4066 4067 4068 4069 4070 4071 4072 4073 4074 4075 4076 4077 4078 4079 4080 4081 4082 4083 4084 4085 4086 4087 4088 4089 4090 4091 4092 4093 4094 4095 4096 4097 4098 4099 4100 4101 4102 4103 4104 4105 4106 4107 4108 4109 4110 4111 4112 4113 4114 4115 4116 4117 4118 4119 4120 4121 4122 4123 4124 4125 4126 4127 4128 4129 4130 4131 4132 4133 4134 4135 4136 4137 4138 4139 4140 4141 4142 4143 4144 4145 4146 4147 4148 4149 4150 4151 4152 4153 4154 4155 4156 4157 4158 4159 4160 4161 4162 4163 4164 4165 4166 4167 4168 4169 4170 4171 4172 4173 4174 4175 4176 4177 4178 4179 4180 4181 4182 4183 4184 4185 4186 4187 4188 4189 4190 4191 4192 4193 4194 4195 4196 4197 4198 4199 4200 4201 4202 4203 4204 4205 4206 4207 4208 4209 4210 4211 4212 4213 4214 4215 4216 4217 4218 4219 4220 4221 4222 4223 4224 4225 4226 4227 4228 4229 4230 4231 4232 4233 4234 4235 4236 4237 4238 4239 4240 4241 4242 4243 4244 4245 4246 4247 4248 4249 4250 4251 4252 4253 4254 4255 4256 4257 4258 4259 4260 4261 4262 4263 4264 4265 4266 4267 4268 4269 4270 4271 4272 4273 4274 4275 4276 4277 4278 4279 4280 4281 4282 4283 4284 4285 4286 4287 4288 4289 4290 4291 4292 4293 4294 4295 4296 4297 4298 4299 4300 4301 4302 4303 4304 4305 4306 4307 4308 4309 4310 4311 4312 4313 4314 4315 4316 4317 4318 4319 4320 4321 4322 4323 4324 4325 4326 4327 4328 4329 4330 4331 4332 4333 4334 4335 4336 4337 4338 4339 4340 4341 4342 4343 4344 4345 4346 4347 4348 4349 4350 4351 4352 4353 4354 4355 4356 4357 4358 4359 4360 4361 4362 4363 4364 4365 4366 4367 4368 4369 4370 4371 4372 4373 4374 4375 4376 4377 4378 4379 4380 4381 4382 4383 4384 4385 4386 4387 4388 4389 4390 4391 4392 4393 4394 4395 4396 4397 4398 4399 4400 4401 4402 4403 4404 4405 4406 4407 4408 4409 4410 4411 4412 4413 4414 4415 4416 4417 4418 4419 4420 4421 4422 4423 4424 4425 4426 4427 4428 4429 4430 4431 4432 4433 4434 4435 4436 4437 4438 4439 4440 4441 4442 4443 4444 4445 4446 4447 4448 4449 4450 4451 4452 4453 4454 4455 4456 4457 4458 4459 4460 4461 4462 4463 4464 4465 4466 4467 4468 4469 4470 4471 4472 4473 4474 4475 4476 4477 4478 4479 4480 4481 4482 4483 4484 4485 4486 4487 4488 4489 4490 4491 4492 4493 4494 4495 4496 4497 4498 4499 4500 4501 4502 4503 4504 4505 4506 4507 4508 4509 4510 4511 4512 4513 4514 4515 4516 4517 4518 4519 4520 4521 4522 4523 4524 4525 4526 4527 4528 4529 4530 4531 4532 4533 4534 4535 4536 4537 4538 4539 4540 4541 4542 4543 4544 4545 4546 4547 4548 4549 4550 4551 4552 4553 4554 4555 4556 4557 4558 4559 4560 4561 4562 4563 4564 4565 4566 4567 4568 4569 4570 4571 4572 4573 4574 4575 4576 4577 4578 4579 4580 4581 4582 4583 4584 4585 4586 4587 4588 4589 4590 4591 4592 4593 4594 4595 4596 4597 4598 4599 4600 4601 4602 4603 4604 4605 4606 4607 4608 4609 4610 4611 4612 4613 4614 4615 4616 4617 4618 4619 4620 4621 4622 4623 4624 4625 4626 4627 4628 4629 4630 4631 4632 4633 4634 4635 4636 4637 4638 4639 4640 4641 4642 4643 4644 4645 4646 4647 4648 4649 4650 4651 4652 4653 4654 4655 4656 4657 4658 4659 4660 4661 4662 4663 4664 4665 4666 4667 4668 4669 4670 4671 4672 4673 4674 4675 4676 4677 4678 4679 4680 4681 4682 4683 4684 4685 4686 4687 4688 4689 4690 4691 4692 4693 4694 4695 4696 4697 4698 4699 4700 4701 4702 4703 4704 4705 4706 4707 4708 4709 4710 4711 4712 4713 4714 4715 4716 4717 4718 4719 4720 4721 4722 4723 4724 4725 4726 4727 4728 4729 4730 4731 4732 4733 4734 4735 4736 4737 4738 4739 4740 4741 4742 4743 4744 4745 4746 4747 4748 4749 4750 4751 4752 4753 4754 4755 4756 4757 4758 4759 4760 4761 4762 4763 4764 4765 4766 4767 4768 4769 4770 4771 4772 4773 4774 4775 4776 4777 4778 4779 4780 4781 4782 4783 4784 4785 4786 4787 4788 4789 4790 4791 4792 4793 4794 4795 4796 4797 4798 4799 4800 4801 4802 4803 4804 4805 4806 4807 4808 4809 4810 4811 4812 4813 4814 4815 4816 4817 4818 4819 4820 4821 4822 4823 4824 4825 4826 4827 4828 4829 4830 4831 4832 4833 4834 4835 4836 4837 4838 4839 4840 4841 4842 4843 4844 4845 4846 4847 4848 4849 4850 4851 4852 4853 4854 4855 4856 4857 4858 4859 4860 4861 4862 4863 4864 4865 4866 4867 4868 4869 4870 4871 4872 4873 4874 4875 4876 4877 4878 4879 4880 4881 4882 4883 4884 4885 4886 4887 4888 4889 4890 4891 4892 4893 4894 4895 4896 4897 4898 4899 4900 4901 4902 4903 4904 4905 4906 4907 4908 4909 4910 4911 4912 4913 4914 4915 4916 4917 4918 4919 4920 4921 4922 4923 4924 4925 4926 4927 4928 4929 4930 4931 4932 4933 4934 4935 4936 4937 4938 4939 4940 4941 4942 4943 4944 4945 4946 4947 4948 4949 4950 4951 4952 4953 4954 4955 4956 4957 4958 4959 4960 4961 4962 4963 4964 4965 4966 4967 4968 4969 4970 4971 4972 4973 4974 4975 4976 4977 4978 4979 4980 4981 4982 4983 4984 4985 4986 4987 4988 4989 4990 4991 4992 4993 4994 4995 4996 4997 4998 4999 5000 5001 5002 5003 5004 5005 5006 5007 5008 5009 5010 5011 5012 5013 5014 5015 5016 5017 5018 5019 5020 5021 5022 5023 5024 5025 5026 5027 5028 5029 5030 5031 5032 5033 5034 5035 5036 5037 5038 5039 5040 5041 5042 5043 5044 5045 5046 5047 5048 5049 5050 5051 5052 5053 5054 5055 5056 5057 5058 5059 5060 5061 5062 5063 5064 5065 5066 5067 5068 5069 5070 5071 5072 5073 5074 5075 5076 5077 5078 5079 | /* $NetBSD: if_run.c,v 1.42 2020/06/24 21:06:39 jdolecek Exp $ */ /* $OpenBSD: if_run.c,v 1.90 2012/03/24 15:11:04 jsg Exp $ */ /*- * Copyright (c) 2008-2010 Damien Bergamini <damien.bergamini@free.fr> * * Permission to use, copy, modify, and distribute this software for any * purpose with or without fee is hereby granted, provided that the above * copyright notice and this permission notice appear in all copies. * * THE SOFTWARE IS PROVIDED "AS IS" AND THE AUTHOR DISCLAIMS ALL WARRANTIES * WITH REGARD TO THIS SOFTWARE INCLUDING ALL IMPLIED WARRANTIES OF * MERCHANTABILITY AND FITNESS. IN NO EVENT SHALL THE AUTHOR BE LIABLE FOR * ANY SPECIAL, DIRECT, INDIRECT, OR CONSEQUENTIAL DAMAGES OR ANY DAMAGES * WHATSOEVER RESULTING FROM LOSS OF USE, DATA OR PROFITS, WHETHER IN AN * ACTION OF CONTRACT, NEGLIGENCE OR OTHER TORTIOUS ACTION, ARISING OUT OF * OR IN CONNECTION WITH THE USE OR PERFORMANCE OF THIS SOFTWARE. */ /*- * Ralink Technology RT2700U/RT2800U/RT3000U/RT3900E chipset driver. * http://www.ralinktech.com/ */ #include <sys/cdefs.h> __KERNEL_RCSID(0, "$NetBSD: if_run.c,v 1.42 2020/06/24 21:06:39 jdolecek Exp $"); #ifdef _KERNEL_OPT #include "opt_usb.h" #endif #include <sys/param.h> #include <sys/sockio.h> #include <sys/sysctl.h> #include <sys/mbuf.h> #include <sys/kernel.h> #include <sys/socket.h> #include <sys/systm.h> #include <sys/malloc.h> #include <sys/callout.h> #include <sys/module.h> #include <sys/conf.h> #include <sys/device.h> #include <sys/atomic.h> #include <sys/bus.h> #include <machine/endian.h> #include <sys/intr.h> #include <net/bpf.h> #include <net/if.h> #include <net/if_arp.h> #include <net/if_dl.h> #include <net/if_ether.h> #include <net/if_media.h> #include <net/if_types.h> #include <net80211/ieee80211_var.h> #include <net80211/ieee80211_amrr.h> #include <net80211/ieee80211_radiotap.h> #include <dev/firmload.h> #include <dev/usb/usb.h> #include <dev/usb/usbdi.h> #include <dev/usb/usbdivar.h> #include <dev/usb/usbdi_util.h> #include <dev/usb/usbdevs.h> #include <dev/ic/rt2860reg.h> /* shared with ral(4) */ #include <dev/usb/if_runvar.h> #ifdef RUN_DEBUG #define DPRINTF(x) do { if (run_debug) printf x; } while (0) #define DPRINTFN(n, x) do { if (run_debug >= (n)) printf x; } while (0) int run_debug = 0; #else #define DPRINTF(x) #define DPRINTFN(n, x) #endif #define IEEE80211_HAS_ADDR4(wh) IEEE80211_IS_DSTODS(wh) #define USB_ID(v, p) { USB_VENDOR_##v, USB_PRODUCT_##v##_##p } static const struct usb_devno run_devs[] = { USB_ID(ABOCOM, RT2770), USB_ID(ABOCOM, RT2870), USB_ID(ABOCOM, RT3070), USB_ID(ABOCOM, RT3071), USB_ID(ABOCOM, RT3072), USB_ID(ABOCOM2, RT2870_1), USB_ID(ACCTON, RT2770), USB_ID(ACCTON, RT2870_1), USB_ID(ACCTON, RT2870_2), USB_ID(ACCTON, RT2870_3), USB_ID(ACCTON, RT2870_4), USB_ID(ACCTON, RT2870_5), USB_ID(ACCTON, RT3070), USB_ID(ACCTON, RT3070_1), USB_ID(ACCTON, RT3070_2), USB_ID(ACCTON, RT3070_3), USB_ID(ACCTON, RT3070_4), USB_ID(ACCTON, RT3070_5), USB_ID(ACCTON, RT3070_6), USB_ID(AIRTIES, RT3070), USB_ID(AIRTIES, RT3070_2), USB_ID(ALLWIN, RT2070), USB_ID(ALLWIN, RT2770), USB_ID(ALLWIN, RT2870), USB_ID(ALLWIN, RT3070), USB_ID(ALLWIN, RT3071), USB_ID(ALLWIN, RT3072), USB_ID(ALLWIN, RT3572), USB_ID(AMIGO, RT2870_1), USB_ID(AMIGO, RT2870_2), USB_ID(AMIT, CGWLUSB2GNR), USB_ID(AMIT, RT2870_1), USB_ID(AMIT2, RT2870), USB_ID(ASUSTEK, RT2870_1), USB_ID(ASUSTEK, RT2870_2), USB_ID(ASUSTEK, RT2870_3), USB_ID(ASUSTEK, RT2870_4), USB_ID(ASUSTEK, RT2870_5), USB_ID(ASUSTEK, RT3070), USB_ID(ASUSTEK, RT3070_1), USB_ID(ASUSTEK, USBN53), USB_ID(ASUSTEK, USBN66), USB_ID(ASUSTEK2, USBN11), USB_ID(AZUREWAVE, RT2870_1), USB_ID(AZUREWAVE, RT2870_2), USB_ID(AZUREWAVE, RT3070), USB_ID(AZUREWAVE, RT3070_2), USB_ID(AZUREWAVE, RT3070_3), USB_ID(AZUREWAVE, RT3070_4), USB_ID(AZUREWAVE, RT3070_5), USB_ID(BELKIN, F5D8053V3), USB_ID(BELKIN, F5D8055), USB_ID(BELKIN, F5D8055V2), USB_ID(BELKIN, F6D4050V1), USB_ID(BELKIN, F6D4050V2), USB_ID(BELKIN, F7D1101V2), USB_ID(BELKIN, RT2870_1), USB_ID(BELKIN, RT2870_2), USB_ID(BELKIN, RTL8192CU_2), USB_ID(BEWAN, RT3070), USB_ID(CISCOLINKSYS, AE1000), USB_ID(CISCOLINKSYS, AM10), USB_ID(CISCOLINKSYS2, RT3070), USB_ID(CISCOLINKSYS3, RT3070), USB_ID(CONCEPTRONIC, RT2870_1), USB_ID(CONCEPTRONIC, RT2870_2), USB_ID(CONCEPTRONIC, RT2870_3), USB_ID(CONCEPTRONIC, RT2870_4), USB_ID(CONCEPTRONIC, RT2870_5), USB_ID(CONCEPTRONIC, RT2870_6), USB_ID(CONCEPTRONIC, RT2870_7), USB_ID(CONCEPTRONIC, RT2870_8), USB_ID(CONCEPTRONIC, RT3070_1), USB_ID(CONCEPTRONIC, RT3070_2), USB_ID(CONCEPTRONIC, RT3070_3), USB_ID(COREGA, CGWLUSB300GNM), USB_ID(COREGA, RT2870_1), USB_ID(COREGA, RT2870_2), USB_ID(COREGA, RT2870_3), USB_ID(COREGA, RT3070), USB_ID(CYBERTAN, RT2870), USB_ID(DLINK, RT2870), USB_ID(DLINK, RT3072), USB_ID(DLINK, DWA127), USB_ID(DLINK, DWA140B3), USB_ID(DLINK, DWA160B2), USB_ID(DLINK, DWA162), USB_ID(DLINK2, DWA130), USB_ID(DLINK2, RT2870_1), USB_ID(DLINK2, RT2870_2), USB_ID(DLINK2, RT3070_1), USB_ID(DLINK2, RT3070_2), USB_ID(DLINK2, RT3070_3), USB_ID(DLINK2, RT3070_4), USB_ID(DLINK2, RT3070_5), USB_ID(DLINK2, RT3072), USB_ID(DLINK2, RT3072_1), USB_ID(DVICO, RT3070), USB_ID(EDIMAX, EW7717), USB_ID(EDIMAX, EW7718), USB_ID(EDIMAX, EW7722UTN), USB_ID(EDIMAX, RT2870_1), USB_ID(ENCORE, RT3070), USB_ID(ENCORE, RT3070_2), USB_ID(ENCORE, RT3070_3), USB_ID(GIGABYTE, GNWB31N), USB_ID(GIGABYTE, GNWB32L), USB_ID(GIGABYTE, RT2870_1), USB_ID(GIGASET, RT3070_1), USB_ID(GIGASET, RT3070_2), USB_ID(GUILLEMOT, HWNU300), USB_ID(HAWKING, HWUN2), USB_ID(HAWKING, RT2870_1), USB_ID(HAWKING, RT2870_2), USB_ID(HAWKING, RT2870_3), USB_ID(HAWKING, RT2870_4), USB_ID(HAWKING, RT2870_5), USB_ID(HAWKING, RT3070), USB_ID(IODATA, RT3072_1), USB_ID(IODATA, RT3072_2), USB_ID(IODATA, RT3072_3), USB_ID(IODATA, RT3072_4), USB_ID(LINKSYS4, RT3070), USB_ID(LINKSYS4, WUSB100), USB_ID(LINKSYS4, WUSB54GC_3), USB_ID(LINKSYS4, WUSB600N), USB_ID(LINKSYS4, WUSB600NV2), USB_ID(LOGITEC, LANW300NU2), USB_ID(LOGITEC, LANW300NU2S), USB_ID(LOGITEC, LAN_W300ANU2), USB_ID(LOGITEC, LAN_W450ANU2E), USB_ID(LOGITEC, RT2870_1), USB_ID(LOGITEC, RT2870_2), USB_ID(LOGITEC, RT2870_3), USB_ID(LOGITEC, RT3020), USB_ID(MELCO, RT2870_1), USB_ID(MELCO, RT2870_2), USB_ID(MELCO, WLIUCAG300N), USB_ID(MELCO, WLIUCG300N), USB_ID(MELCO, WLIUCG301N), USB_ID(MELCO, WLIUCGN), USB_ID(MELCO, WLIUCGNHP), USB_ID(MELCO, WLIUCGNM), USB_ID(MELCO, WLIUCGNM2T), USB_ID(MOTOROLA4, RT2770), USB_ID(MOTOROLA4, RT3070), USB_ID(MSI, RT3070), USB_ID(MSI, RT3070_2), USB_ID(MSI, RT3070_3), USB_ID(MSI, RT3070_4), USB_ID(MSI, RT3070_5), USB_ID(MSI, RT3070_6), USB_ID(MSI, RT3070_7), USB_ID(MSI, RT3070_8), USB_ID(MSI, RT3070_9), USB_ID(MSI, RT3070_10), USB_ID(MSI, RT3070_11), USB_ID(MSI, RT3070_12), USB_ID(MSI, RT3070_13), USB_ID(MSI, RT3070_14), USB_ID(MSI, RT3070_15), USB_ID(OVISLINK, RT3071), USB_ID(OVISLINK, RT3072), USB_ID(PARA, RT3070), USB_ID(PEGATRON, RT2870), USB_ID(PEGATRON, RT3070), USB_ID(PEGATRON, RT3070_2), USB_ID(PEGATRON, RT3070_3), USB_ID(PEGATRON, RT3072), USB_ID(PHILIPS, RT2870), USB_ID(PLANEX2, GWUS300MINIS), USB_ID(PLANEX2, GWUSMICRO300), USB_ID(PLANEX2, GWUSMICRON), USB_ID(PLANEX2, GWUS300MINIX), USB_ID(PLANEX2, RT3070), USB_ID(QCOM, RT2870), USB_ID(QUANTA, RT3070), USB_ID(RALINK, RT2070), USB_ID(RALINK, RT2770), USB_ID(RALINK, RT2870), USB_ID(RALINK, RT3070), USB_ID(RALINK, RT3071), USB_ID(RALINK, RT3072), USB_ID(RALINK, RT3370), USB_ID(RALINK, RT3572), USB_ID(RALINK, RT3573), USB_ID(RALINK, RT5370), USB_ID(RALINK, RT5572), USB_ID(RALINK, RT8070), USB_ID(SAMSUNG, RT2870_1), USB_ID(SENAO, RT2870_1), USB_ID(SENAO, RT2870_2), USB_ID(SENAO, RT2870_3), USB_ID(SENAO, RT2870_4), USB_ID(SENAO, RT3070), USB_ID(SENAO, RT3071), USB_ID(SENAO, RT3072), USB_ID(SENAO, RT3072_2), USB_ID(SENAO, RT3072_3), USB_ID(SENAO, RT3072_4), USB_ID(SENAO, RT3072_5), USB_ID(SITECOMEU, RT2870_1), USB_ID(SITECOMEU, RT2870_2), USB_ID(SITECOMEU, RT2870_3), USB_ID(SITECOMEU, RT3070_1), USB_ID(SITECOMEU, RT3070_3), USB_ID(SITECOMEU, RT3072_3), USB_ID(SITECOMEU, RT3072_4), USB_ID(SITECOMEU, RT3072_5), USB_ID(SITECOMEU, RT3072_6), USB_ID(SITECOMEU, WL302), USB_ID(SITECOMEU, WL315), USB_ID(SITECOMEU, WL321), USB_ID(SITECOMEU, WL324), USB_ID(SITECOMEU, WL329), USB_ID(SITECOMEU, WL343), USB_ID(SITECOMEU, WL344), USB_ID(SITECOMEU, WL345), USB_ID(SITECOMEU, WL349V4), USB_ID(SITECOMEU, WL608), USB_ID(SITECOMEU, WLA4000), USB_ID(SITECOMEU, WLA5000), USB_ID(SPARKLAN, RT2870_1), USB_ID(SPARKLAN, RT2870_2), USB_ID(SPARKLAN, RT3070), USB_ID(SWEEX2, LW153), USB_ID(SWEEX2, LW303), USB_ID(SWEEX2, LW313), USB_ID(TOSHIBA, RT3070), USB_ID(UMEDIA, RT2870_1), USB_ID(UMEDIA, TEW645UB), USB_ID(ZCOM, RT2870_1), USB_ID(ZCOM, RT2870_2), USB_ID(ZINWELL, RT2870_1), USB_ID(ZINWELL, RT2870_2), USB_ID(ZINWELL, RT3070), USB_ID(ZINWELL, RT3072), USB_ID(ZINWELL, RT3072_2), USB_ID(ZYXEL, NWD2105), USB_ID(ZYXEL, NWD211AN), USB_ID(ZYXEL, RT2870_1), USB_ID(ZYXEL, RT2870_2), USB_ID(ZYXEL, RT3070), }; static int run_match(device_t, cfdata_t, void *); static void run_attach(device_t, device_t, void *); static int run_detach(device_t, int); static int run_activate(device_t, enum devact); CFATTACH_DECL_NEW(run, sizeof(struct run_softc), run_match, run_attach, run_detach, run_activate); static int run_alloc_rx_ring(struct run_softc *); static void run_free_rx_ring(struct run_softc *); static int run_alloc_tx_ring(struct run_softc *, int); static void run_free_tx_ring(struct run_softc *, int); static int run_load_microcode(struct run_softc *); static int run_reset(struct run_softc *); static int run_read(struct run_softc *, uint16_t, uint32_t *); static int run_read_region_1(struct run_softc *, uint16_t, uint8_t *, int); static int run_write_2(struct run_softc *, uint16_t, uint16_t); static int run_write(struct run_softc *, uint16_t, uint32_t); static int run_write_region_1(struct run_softc *, uint16_t, const uint8_t *, int); static int run_set_region_4(struct run_softc *, uint16_t, uint32_t, int); static int run_efuse_read(struct run_softc *, uint16_t, uint16_t *, int); static int run_efuse_read_2(struct run_softc *, uint16_t, uint16_t *); static int run_eeprom_read_2(struct run_softc *, uint16_t, uint16_t *); static int run_rt2870_rf_write(struct run_softc *, uint8_t, uint32_t); static int run_rt3070_rf_read(struct run_softc *, uint8_t, uint8_t *); static int run_rt3070_rf_write(struct run_softc *, uint8_t, uint8_t); static int run_bbp_read(struct run_softc *, uint8_t, uint8_t *); static int run_bbp_write(struct run_softc *, uint8_t, uint8_t); static int run_mcu_cmd(struct run_softc *, uint8_t, uint16_t); static const char * run_get_rf(uint16_t); static void run_rt3593_get_txpower(struct run_softc *); static void run_get_txpower(struct run_softc *); static int run_read_eeprom(struct run_softc *); static struct ieee80211_node * run_node_alloc(struct ieee80211_node_table *); static int run_media_change(struct ifnet *); static void run_next_scan(void *); static void run_task(void *); static void run_do_async(struct run_softc *, void (*)(struct run_softc *, void *), void *, int); static int run_newstate(struct ieee80211com *, enum ieee80211_state, int); static void run_newstate_cb(struct run_softc *, void *); static int run_updateedca(struct ieee80211com *); static void run_updateedca_cb(struct run_softc *, void *); #ifdef RUN_HWCRYPTO static int run_set_key(struct ieee80211com *, const struct ieee80211_key *, const uint8_t *); static void run_set_key_cb(struct run_softc *, void *); static int run_delete_key(struct ieee80211com *, const struct ieee80211_key *); static void run_delete_key_cb(struct run_softc *, void *); #endif static void run_calibrate_to(void *); static void run_calibrate_cb(struct run_softc *, void *); static void run_newassoc(struct ieee80211_node *, int); static void run_rx_frame(struct run_softc *, uint8_t *, int); static void run_rxeof(struct usbd_xfer *, void *, usbd_status); static void run_txeof(struct usbd_xfer *, void *, usbd_status); static int run_tx(struct run_softc *, struct mbuf *, struct ieee80211_node *); static void run_start(struct ifnet *); static void run_watchdog(struct ifnet *); static int run_ioctl(struct ifnet *, u_long, void *); static void run_select_chan_group(struct run_softc *, int); static void run_iq_calib(struct run_softc *, u_int); static void run_set_agc(struct run_softc *, uint8_t); static void run_set_rx_antenna(struct run_softc *, int); static void run_rt2870_set_chan(struct run_softc *, u_int); static void run_rt3070_set_chan(struct run_softc *, u_int); static void run_rt3572_set_chan(struct run_softc *, u_int); static void run_rt3593_set_chan(struct run_softc *, u_int); static void run_rt5390_set_chan(struct run_softc *, u_int); static void run_rt5592_set_chan(struct run_softc *, u_int); static int run_set_chan(struct run_softc *, struct ieee80211_channel *); static void run_updateprot(struct run_softc *); static void run_enable_tsf_sync(struct run_softc *); static void run_enable_mrr(struct run_softc *); static void run_set_txpreamble(struct run_softc *); static void run_set_basicrates(struct run_softc *); static void run_set_leds(struct run_softc *, uint16_t); static void run_set_bssid(struct run_softc *, const uint8_t *); static void run_set_macaddr(struct run_softc *, const uint8_t *); static void run_updateslot(struct ifnet *); static void run_updateslot_cb(struct run_softc *, void *); static int8_t run_rssi2dbm(struct run_softc *, uint8_t, uint8_t); static void run_rt5390_bbp_init(struct run_softc *); static int run_bbp_init(struct run_softc *); static int run_rt3070_rf_init(struct run_softc *); static int run_rt3593_rf_init(struct run_softc *); static int run_rt5390_rf_init(struct run_softc *); static int run_rt3070_filter_calib(struct run_softc *, uint8_t, uint8_t, uint8_t *); static void run_rt3070_rf_setup(struct run_softc *); static void run_rt3593_rf_setup(struct run_softc *); static void run_rt5390_rf_setup(struct run_softc *); static int run_txrx_enable(struct run_softc *); static int run_adjust_freq_offset(struct run_softc *); static int run_init(struct ifnet *); static void run_stop(struct ifnet *, int); #ifndef IEEE80211_STA_ONLY static int run_setup_beacon(struct run_softc *); #endif static const struct { uint32_t reg; uint32_t val; } rt2870_def_mac[] = { RT2870_DEF_MAC }; static const struct { uint8_t reg; uint8_t val; } rt2860_def_bbp[] = { RT2860_DEF_BBP }, rt5390_def_bbp[] = { RT5390_DEF_BBP }, rt5592_def_bbp[] = { RT5592_DEF_BBP }; /* * Default values for BBP register R196 for RT5592. */ static const uint8_t rt5592_bbp_r196[] = { 0xe0, 0x1f, 0x38, 0x32, 0x08, 0x28, 0x19, 0x0a, 0xff, 0x00, 0x16, 0x10, 0x10, 0x0b, 0x36, 0x2c, 0x26, 0x24, 0x42, 0x36, 0x30, 0x2d, 0x4c, 0x46, 0x3d, 0x40, 0x3e, 0x42, 0x3d, 0x40, 0x3c, 0x34, 0x2c, 0x2f, 0x3c, 0x35, 0x2e, 0x2a, 0x49, 0x41, 0x36, 0x31, 0x30, 0x30, 0x0e, 0x0d, 0x28, 0x21, 0x1c, 0x16, 0x50, 0x4a, 0x43, 0x40, 0x10, 0x10, 0x10, 0x10, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x7d, 0x14, 0x32, 0x2c, 0x36, 0x4c, 0x43, 0x2c, 0x2e, 0x36, 0x30, 0x6e }; static const struct rfprog { uint8_t chan; uint32_t r1, r2, r3, r4; } rt2860_rf2850[] = { RT2860_RF2850 }; static const struct { uint8_t n, r, k; } rt3070_freqs[] = { RT3070_RF3052 }; static const struct rt5592_freqs { uint16_t n; uint8_t k, m, r; } rt5592_freqs_20mhz[] = { RT5592_RF5592_20MHZ },rt5592_freqs_40mhz[] = { RT5592_RF5592_40MHZ }; static const struct { uint8_t reg; uint8_t val; } rt3070_def_rf[] = { RT3070_DEF_RF }, rt3572_def_rf[] = { RT3572_DEF_RF },rt3593_def_rf[] = { RT3593_DEF_RF },rt5390_def_rf[] = { RT5390_DEF_RF },rt5392_def_rf[] = { RT5392_DEF_RF },rt5592_def_rf[] = { RT5592_DEF_RF },rt5592_2ghz_def_rf[] = { RT5592_2GHZ_DEF_RF },rt5592_5ghz_def_rf[] = { RT5592_5GHZ_DEF_RF }; static const struct { u_int firstchan; u_int lastchan; uint8_t reg; uint8_t val; } rt5592_chan_5ghz[] = { RT5592_CHAN_5GHZ }; static int firmware_load(const char *dname, const char *iname, uint8_t **ucodep, size_t *sizep) { firmware_handle_t fh; int error; if ((error = firmware_open(dname, iname, &fh)) != 0) return error; *sizep = firmware_get_size(fh); if ((*ucodep = firmware_malloc(*sizep)) == NULL) { firmware_close(fh); return ENOMEM; } if ((error = firmware_read(fh, 0, *ucodep, *sizep)) != 0) firmware_free(*ucodep, *sizep); firmware_close(fh); return error; } static int run_match(device_t parent, cfdata_t match, void *aux) { struct usb_attach_arg *uaa = aux; return (usb_lookup(run_devs, uaa->uaa_vendor, uaa->uaa_product) != NULL) ? UMATCH_VENDOR_PRODUCT : UMATCH_NONE; } static void run_attach(device_t parent, device_t self, void *aux) { struct run_softc *sc = device_private(self); struct usb_attach_arg *uaa = aux; struct ieee80211com *ic = &sc->sc_ic; struct ifnet *ifp = &sc->sc_if; usb_interface_descriptor_t *id; usb_endpoint_descriptor_t *ed; char *devinfop; int i, nrx, ntx, ntries, error; uint32_t ver; aprint_naive("\n"); aprint_normal("\n"); sc->sc_dev = self; sc->sc_udev = uaa->uaa_device; devinfop = usbd_devinfo_alloc(sc->sc_udev, 0); aprint_normal_dev(sc->sc_dev, "%s\n", devinfop); usbd_devinfo_free(devinfop); error = usbd_set_config_no(sc->sc_udev, 1, 0); if (error != 0) { aprint_error_dev(sc->sc_dev, "failed to set configuration" ", err=%s\n", usbd_errstr(error)); return; } /* get the first interface handle */ error = usbd_device2interface_handle(sc->sc_udev, 0, &sc->sc_iface); if (error != 0) { aprint_error_dev(sc->sc_dev, "could not get interface handle\n"); return; } /* * Find all bulk endpoints. There are 7 bulk endpoints: 1 for RX * and 6 for TX (4 EDCAs + HCCA + Prio). * Update 03-14-2009: some devices like the Planex GW-US300MiniS * seem to have only 4 TX bulk endpoints (Fukaumi Naoki). */ nrx = ntx = 0; id = usbd_get_interface_descriptor(sc->sc_iface); for (i = 0; i < id->bNumEndpoints; i++) { ed = usbd_interface2endpoint_descriptor(sc->sc_iface, i); if (ed == NULL || UE_GET_XFERTYPE(ed->bmAttributes) != UE_BULK) continue; if (UE_GET_DIR(ed->bEndpointAddress) == UE_DIR_IN) { sc->rxq.pipe_no = ed->bEndpointAddress; nrx++; } else if (ntx < RUN_MAXEPOUT) { sc->txq[ntx].pipe_no = ed->bEndpointAddress; ntx++; } } /* make sure we've got them all */ if (nrx < 1 || ntx < RUN_MAXEPOUT) { aprint_error_dev(sc->sc_dev, "missing endpoint\n"); return; } usb_init_task(&sc->sc_task, run_task, sc, 0); callout_init(&sc->scan_to, 0); callout_setfunc(&sc->scan_to, run_next_scan, sc); callout_init(&sc->calib_to, 0); callout_setfunc(&sc->calib_to, run_calibrate_to, sc); sc->amrr.amrr_min_success_threshold = 1; sc->amrr.amrr_max_success_threshold = 10; /* wait for the chip to settle */ for (ntries = 0; ntries < 100; ntries++) { if (run_read(sc, RT2860_ASIC_VER_ID, &ver) != 0) return; if (ver != 0 && ver != 0xffffffff) break; DELAY(10); } if (ntries == 100) { aprint_error_dev(sc->sc_dev, "timeout waiting for NIC to initialize\n"); return; } sc->mac_ver = ver >> 16; sc->mac_rev = ver & 0xffff; /* * Per the comment in run_write_region_1(), "the WRITE_REGION_1 * command is not stable on RT2860", but WRITE_REGION_1 calls * of up to 64 bytes have been tested and found to work with * mac_ver 0x5390, and they reduce the run time of "ifconfig * run0 up" from 30 seconds to a couple of seconds on OHCI. * Enable WRITE_REGION_1 for the tested version only. As other * versions are tested and found to work, they can be added * here. */ if (sc->mac_ver == 0x5390) sc->sc_flags |= RUN_USE_BLOCK_WRITE; /* retrieve RF rev. no and various other things from EEPROM */ run_read_eeprom(sc); aprint_verbose_dev(sc->sc_dev, "MAC/BBP RT%04X (rev 0x%04X), RF %s (MIMO %dT%dR), address %s\n", sc->mac_ver, sc->mac_rev, run_get_rf(sc->rf_rev), sc->ntxchains, sc->nrxchains, ether_sprintf(ic->ic_myaddr)); ic->ic_ifp = ifp; ic->ic_phytype = IEEE80211_T_OFDM; /* not only, but not used */ ic->ic_opmode = IEEE80211_M_STA; /* default to BSS mode */ ic->ic_state = IEEE80211_S_INIT; /* set device capabilities */ ic->ic_caps = IEEE80211_C_MONITOR | /* monitor mode supported */ #ifndef IEEE80211_STA_ONLY IEEE80211_C_IBSS | /* IBSS mode supported */ IEEE80211_C_HOSTAP | /* HostAP mode supported */ #endif IEEE80211_C_SHPREAMBLE | /* short preamble supported */ IEEE80211_C_SHSLOT | /* short slot time supported */ #ifdef RUN_HWCRYPTO IEEE80211_C_WEP | /* WEP */ IEEE80211_C_TKIP | /* TKIP */ IEEE80211_C_AES_CCM | /* AES CCMP */ IEEE80211_C_TKIPMIC | /* TKIPMIC */ #endif IEEE80211_C_WME | /* WME */ IEEE80211_C_WPA; /* WPA/RSN */ if (sc->rf_rev == RT2860_RF_2750 || sc->rf_rev == RT2860_RF_2850 || sc->rf_rev == RT3070_RF_3052 || sc->rf_rev == RT3070_RF_3053 || sc->rf_rev == RT5592_RF_5592) { /* set supported .11a rates */ ic->ic_sup_rates[IEEE80211_MODE_11A] = ieee80211_std_rateset_11a; /* set supported .11a channels */ for (i = 14; i < (int)__arraycount(rt2860_rf2850); i++) { uint8_t chan = rt2860_rf2850[i].chan; ic->ic_channels[chan].ic_freq = ieee80211_ieee2mhz(chan, IEEE80211_CHAN_5GHZ); ic->ic_channels[chan].ic_flags = IEEE80211_CHAN_A; } } /* set supported .11b and .11g rates */ ic->ic_sup_rates[IEEE80211_MODE_11B] = ieee80211_std_rateset_11b; ic->ic_sup_rates[IEEE80211_MODE_11G] = ieee80211_std_rateset_11g; /* set supported .11b and .11g channels (1 through 14) */ for (i = 1; i <= 14; i++) { ic->ic_channels[i].ic_freq = ieee80211_ieee2mhz(i, IEEE80211_CHAN_2GHZ); ic->ic_channels[i].ic_flags = IEEE80211_CHAN_CCK | IEEE80211_CHAN_OFDM | IEEE80211_CHAN_DYN | IEEE80211_CHAN_2GHZ; } ifp->if_softc = sc; ifp->if_flags = IFF_BROADCAST | IFF_SIMPLEX | IFF_MULTICAST; ifp->if_init = run_init; ifp->if_ioctl = run_ioctl; ifp->if_start = run_start; ifp->if_watchdog = run_watchdog; IFQ_SET_READY(&ifp->if_snd); memcpy(ifp->if_xname, device_xname(sc->sc_dev), IFNAMSIZ); if_attach(ifp); ieee80211_ifattach(ic); ic->ic_node_alloc = run_node_alloc; ic->ic_newassoc = run_newassoc; ic->ic_updateslot = run_updateslot; ic->ic_wme.wme_update = run_updateedca; #ifdef RUN_HWCRYPTO ic->ic_crypto.cs_key_set = run_set_key; ic->ic_crypto.cs_key_delete = run_delete_key; #endif /* override state transition machine */ sc->sc_newstate = ic->ic_newstate; ic->ic_newstate = run_newstate; /* XXX media locking needs revisiting */ mutex_init(&sc->sc_media_mtx, MUTEX_DEFAULT, IPL_SOFTUSB); ieee80211_media_init_with_lock(ic, run_media_change, ieee80211_media_status, &sc->sc_media_mtx); bpf_attach2(ifp, DLT_IEEE802_11_RADIO, sizeof(struct ieee80211_frame) + IEEE80211_RADIOTAP_HDRLEN, &sc->sc_drvbpf); sc->sc_rxtap_len = sizeof(sc->sc_rxtapu); sc->sc_rxtap.wr_ihdr.it_len = htole16(sc->sc_rxtap_len); sc->sc_rxtap.wr_ihdr.it_present = htole32(RUN_RX_RADIOTAP_PRESENT); sc->sc_txtap_len = sizeof(sc->sc_txtapu); sc->sc_txtap.wt_ihdr.it_len = htole16(sc->sc_txtap_len); sc->sc_txtap.wt_ihdr.it_present = htole32(RUN_TX_RADIOTAP_PRESENT); ieee80211_announce(ic); usbd_add_drv_event(USB_EVENT_DRIVER_ATTACH, sc->sc_udev, sc->sc_dev); if (!pmf_device_register(self, NULL, NULL)) aprint_error_dev(self, "couldn't establish power handler\n"); } static int run_detach(device_t self, int flags) { struct run_softc *sc = device_private(self); struct ifnet *ifp = &sc->sc_if; struct ieee80211com *ic = &sc->sc_ic; int s; if (ifp->if_softc == NULL) return 0; pmf_device_deregister(self); s = splusb(); sc->sc_flags |= RUN_DETACHING; if (ifp->if_flags & IFF_RUNNING) { run_stop(ifp, 0); callout_halt(&sc->scan_to, NULL); callout_halt(&sc->calib_to, NULL); usb_rem_task_wait(sc->sc_udev, &sc->sc_task, USB_TASKQ_DRIVER, NULL); } ifp->if_flags &= ~(IFF_RUNNING | IFF_OACTIVE); bpf_detach(ifp); ieee80211_ifdetach(ic); if_detach(ifp); splx(s); usbd_add_drv_event(USB_EVENT_DRIVER_DETACH, sc->sc_udev, sc->sc_dev); callout_stop(&sc->scan_to); callout_stop(&sc->calib_to); callout_destroy(&sc->scan_to); callout_destroy(&sc->calib_to); return 0; } static int run_activate(device_t self, enum devact act) { struct run_softc *sc = device_private(self); switch (act) { case DVACT_DEACTIVATE: if_deactivate(sc->sc_ic.ic_ifp); return 0; default: return EOPNOTSUPP; } } static int run_alloc_rx_ring(struct run_softc *sc) { struct run_rx_ring *rxq = &sc->rxq; int i, error; error = usbd_open_pipe(sc->sc_iface, rxq->pipe_no, 0, &rxq->pipeh); if (error != 0) goto fail; for (i = 0; i < RUN_RX_RING_COUNT; i++) { struct run_rx_data *data = &rxq->data[i]; data->sc = sc; /* backpointer for callbacks */ error = usbd_create_xfer(sc->rxq.pipeh, RUN_MAX_RXSZ, 0, 0, &data->xfer); if (error) goto fail; data->buf = usbd_get_buffer(data->xfer); } if (error != 0) fail: run_free_rx_ring(sc); return error; } static void run_free_rx_ring(struct run_softc *sc) { struct run_rx_ring *rxq = &sc->rxq; int i; if (rxq->pipeh != NULL) { usbd_abort_pipe(rxq->pipeh); } for (i = 0; i < RUN_RX_RING_COUNT; i++) { if (rxq->data[i].xfer != NULL) usbd_destroy_xfer(rxq->data[i].xfer); rxq->data[i].xfer = NULL; } if (rxq->pipeh != NULL) { usbd_close_pipe(rxq->pipeh); rxq->pipeh = NULL; } } static int run_alloc_tx_ring(struct run_softc *sc, int qid) { struct run_tx_ring *txq = &sc->txq[qid]; int i, error; uint16_t txwisize; txwisize = sizeof(struct rt2860_txwi); if (sc->mac_ver == 0x5592) txwisize += sizeof(uint32_t); txq->cur = txq->queued = 0; error = usbd_open_pipe(sc->sc_iface, txq->pipe_no, 0, &txq->pipeh); if (error != 0) goto fail; for (i = 0; i < RUN_TX_RING_COUNT; i++) { struct run_tx_data *data = &txq->data[i]; data->sc = sc; /* backpointer for callbacks */ data->qid = qid; error = usbd_create_xfer(txq->pipeh, RUN_MAX_TXSZ, USBD_FORCE_SHORT_XFER, 0, &data->xfer); if (error) goto fail; data->buf = usbd_get_buffer(data->xfer); /* zeroize the TXD + TXWI part */ memset(data->buf, 0, sizeof(struct rt2870_txd) + txwisize); } if (error != 0) fail: run_free_tx_ring(sc, qid); return error; } static void run_free_tx_ring(struct run_softc *sc, int qid) { struct run_tx_ring *txq = &sc->txq[qid]; int i; if (txq->pipeh != NULL) { usbd_abort_pipe(txq->pipeh); usbd_close_pipe(txq->pipeh); txq->pipeh = NULL; } for (i = 0; i < RUN_TX_RING_COUNT; i++) { if (txq->data[i].xfer != NULL) usbd_destroy_xfer(txq->data[i].xfer); txq->data[i].xfer = NULL; } } static int __noinline run_load_microcode(struct run_softc *sc) { usb_device_request_t req; const char *fwname; u_char *ucode = NULL; /* XXX gcc 4.8.3: maybe-uninitialized */ size_t size = 0; /* XXX gcc 4.8.3: maybe-uninitialized */ uint32_t tmp; int ntries, error; /* RT3071/RT3072 use a different firmware */ if (sc->mac_ver != 0x2860 && sc->mac_ver != 0x2872 && sc->mac_ver != 0x3070) fwname = "run-rt3071"; else fwname = "run-rt2870"; if ((error = firmware_load("run", fwname, &ucode, &size)) != 0) { device_printf(sc->sc_dev, "error %d, could not read firmware %s\n", error, fwname); return error; } if (size != 4096) { device_printf(sc->sc_dev, "invalid firmware size (should be 4KB)\n"); firmware_free(ucode, size); return EINVAL; } run_read(sc, RT2860_ASIC_VER_ID, &tmp); /* write microcode image */ run_write_region_1(sc, RT2870_FW_BASE, ucode, size); firmware_free(ucode, size); run_write(sc, RT2860_H2M_MAILBOX_CID, 0xffffffff); run_write(sc, RT2860_H2M_MAILBOX_STATUS, 0xffffffff); req.bmRequestType = UT_WRITE_VENDOR_DEVICE; req.bRequest = RT2870_RESET; USETW(req.wValue, 8); USETW(req.wIndex, 0); USETW(req.wLength, 0); if ((error = usbd_do_request(sc->sc_udev, &req, NULL)) != 0) return error; usbd_delay_ms(sc->sc_udev, 10); run_write(sc, RT2860_H2M_BBPAGENT, 0); run_write(sc, RT2860_H2M_MAILBOX, 0); run_write(sc, RT2860_H2M_INTSRC, 0); if ((error = run_mcu_cmd(sc, RT2860_MCU_CMD_RFRESET, 0)) != 0) return error; /* wait until microcontroller is ready */ for (ntries = 0; ntries < 1000; ntries++) { if ((error = run_read(sc, RT2860_SYS_CTRL, &tmp)) != 0) return error; if (tmp & RT2860_MCU_READY) break; usbd_delay_ms(sc->sc_udev, 10); } if (ntries == 1000) { device_printf(sc->sc_dev, "timeout waiting for MCU to initialize\n"); return ETIMEDOUT; } sc->sc_flags |= RUN_FWLOADED; DPRINTF(("microcode successfully loaded after %d tries\n", ntries)); return 0; } static int __noinline run_reset(struct run_softc *sc) { usb_device_request_t req; req.bmRequestType = UT_WRITE_VENDOR_DEVICE; req.bRequest = RT2870_RESET; USETW(req.wValue, 1); USETW(req.wIndex, 0); USETW(req.wLength, 0); return usbd_do_request(sc->sc_udev, &req, NULL); } static int __noinline run_read(struct run_softc *sc, uint16_t reg, uint32_t *val) { uint32_t tmp; int error; error = run_read_region_1(sc, reg, (uint8_t *)&tmp, sizeof(tmp)); if (error == 0) *val = le32toh(tmp); else *val = 0xffffffff; return error; } static int run_read_region_1(struct run_softc *sc, uint16_t reg, uint8_t *buf, int len) { usb_device_request_t req; req.bmRequestType = UT_READ_VENDOR_DEVICE; req.bRequest = RT2870_READ_REGION_1; USETW(req.wValue, 0); USETW(req.wIndex, reg); USETW(req.wLength, len); return usbd_do_request(sc->sc_udev, &req, buf); } static int run_write_2(struct run_softc *sc, uint16_t reg, uint16_t val) { usb_device_request_t req; req.bmRequestType = UT_WRITE_VENDOR_DEVICE; req.bRequest = RT2870_WRITE_2; USETW(req.wValue, val); USETW(req.wIndex, reg); USETW(req.wLength, 0); return usbd_do_request(sc->sc_udev, &req, NULL); } static int __noinline run_write(struct run_softc *sc, uint16_t reg, uint32_t val) { uint32_t tmp = htole32(val); return run_write_region_1(sc, reg, (uint8_t *)&tmp, sizeof(tmp)); } static int run_write_region_1(struct run_softc *sc, uint16_t reg, const uint8_t *buf, int len) { int error = 0; if (sc->sc_flags & RUN_USE_BLOCK_WRITE) { usb_device_request_t req; /* * NOTE: It appears the WRITE_REGION_1 command cannot be * passed a huge amount of data, which will crash the * firmware. Limit amount of data passed to 64 bytes at a * time. */ while (len > 0) { int delta = MIN(len, 64); req.bmRequestType = UT_WRITE_VENDOR_DEVICE; req.bRequest = RT2870_WRITE_REGION_1; USETW(req.wValue, 0); USETW(req.wIndex, reg); USETW(req.wLength, delta); error = usbd_do_request(sc->sc_udev, &req, __UNCONST(buf)); if (error != 0) break; reg += delta; buf += delta; len -= delta; } } else { /* * NB: the WRITE_REGION_1 command is not stable on RT2860. * We thus issue multiple WRITE_2 commands instead. */ int i; KASSERT((len & 1) == 0); for (i = 0; i < len && error == 0; i += 2) error = run_write_2(sc, reg + i, buf[i] | buf[i + 1] << 8); } return error; } static int run_set_region_4(struct run_softc *sc, uint16_t reg, uint32_t val, int count) { int error = 0; if (sc->sc_flags & RUN_USE_BLOCK_WRITE) { while (count > 0) { int i, delta; uint32_t tmp[16]; delta = MIN(count, __arraycount(tmp)); for (i = 0; i < delta; i++) tmp[i] = htole32(val); error = run_write_region_1(sc, reg, (uint8_t *)tmp, delta * sizeof(uint32_t)); if (error != 0) break; reg += delta * sizeof(uint32_t); count -= delta; } } else { for (; count > 0 && error == 0; count--, reg += 4) error = run_write(sc, reg, val); } return error; } static int run_efuse_read(struct run_softc *sc, uint16_t addr, uint16_t *val, int count) { uint32_t tmp; uint16_t reg; int error, ntries; if ((error = run_read(sc, RT3070_EFUSE_CTRL, &tmp)) != 0) return error; if (count == 2) addr *= 2; /*- * Read one 16-byte block into registers EFUSE_DATA[0-3]: * DATA0: F E D C * DATA1: B A 9 8 * DATA2: 7 6 5 4 * DATA3: 3 2 1 0 */ tmp &= ~(RT3070_EFSROM_MODE_MASK | RT3070_EFSROM_AIN_MASK); tmp |= (addr & ~0xf) << RT3070_EFSROM_AIN_SHIFT | RT3070_EFSROM_KICK; run_write(sc, RT3070_EFUSE_CTRL, tmp); for (ntries = 0; ntries < 100; ntries++) { if ((error = run_read(sc, RT3070_EFUSE_CTRL, &tmp)) != 0) return error; if (!(tmp & RT3070_EFSROM_KICK)) break; usbd_delay_ms(sc->sc_udev, 2); } if (ntries == 100) return ETIMEDOUT; if ((tmp & RT3070_EFUSE_AOUT_MASK) == RT3070_EFUSE_AOUT_MASK) { *val = 0xffff; /* address not found */ return 0; } /* determine to which 32-bit register our 16-bit word belongs */ reg = RT3070_EFUSE_DATA3 - (addr & 0xc); if ((error = run_read(sc, reg, &tmp)) != 0) return error; tmp >>= (8 * (addr & 0x3)); *val = (addr & 1) ? tmp >> 16 : tmp & 0xffff; return 0; } /* Read 16-bit from eFUSE ROM for RT3xxxx. */ static int run_efuse_read_2(struct run_softc *sc, uint16_t addr, uint16_t *val) { return run_efuse_read(sc, addr, val, 2); } static int run_eeprom_read_2(struct run_softc *sc, uint16_t addr, uint16_t *val) { usb_device_request_t req; uint16_t tmp; int error; addr *= 2; req.bmRequestType = UT_READ_VENDOR_DEVICE; req.bRequest = RT2870_EEPROM_READ; USETW(req.wValue, 0); USETW(req.wIndex, addr); USETW(req.wLength, sizeof(tmp)); error = usbd_do_request(sc->sc_udev, &req, &tmp); if (error == 0) *val = le16toh(tmp); else *val = 0xffff; return error; } static __inline int run_srom_read(struct run_softc *sc, uint16_t addr, uint16_t *val) { /* either eFUSE ROM or EEPROM */ return sc->sc_srom_read(sc, addr, val); } static int run_rt2870_rf_write(struct run_softc *sc, uint8_t reg, uint32_t val) { uint32_t tmp; int error, ntries; for (ntries = 0; ntries < 10; ntries++) { if ((error = run_read(sc, RT2860_RF_CSR_CFG0, &tmp)) != 0) return error; if (!(tmp & RT2860_RF_REG_CTRL)) break; } if (ntries == 10) return ETIMEDOUT; /* RF registers are 24-bit on the RT2860 */ tmp = RT2860_RF_REG_CTRL | 24 << RT2860_RF_REG_WIDTH_SHIFT | (val & 0x3fffff) << 2 | (reg & 3); return run_write(sc, RT2860_RF_CSR_CFG0, tmp); } static int run_rt3070_rf_read(struct run_softc *sc, uint8_t reg, uint8_t *val) { uint32_t tmp; int error, ntries; for (ntries = 0; ntries < 100; ntries++) { if ((error = run_read(sc, RT3070_RF_CSR_CFG, &tmp)) != 0) return error; if (!(tmp & RT3070_RF_KICK)) break; } if (ntries == 100) return ETIMEDOUT; tmp = RT3070_RF_KICK | reg << 8; if ((error = run_write(sc, RT3070_RF_CSR_CFG, tmp)) != 0) return error; for (ntries = 0; ntries < 100; ntries++) { if ((error = run_read(sc, RT3070_RF_CSR_CFG, &tmp)) != 0) return error; if (!(tmp & RT3070_RF_KICK)) break; } if (ntries == 100) return ETIMEDOUT; *val = tmp & 0xff; return 0; } static int run_rt3070_rf_write(struct run_softc *sc, uint8_t reg, uint8_t val) { uint32_t tmp; int error, ntries; for (ntries = 0; ntries < 10; ntries++) { if ((error = run_read(sc, RT3070_RF_CSR_CFG, &tmp)) != 0) return error; if (!(tmp & RT3070_RF_KICK)) break; } if (ntries == 10) return ETIMEDOUT; tmp = RT3070_RF_WRITE | RT3070_RF_KICK | reg << 8 | val; return run_write(sc, RT3070_RF_CSR_CFG, tmp); } static int run_bbp_read(struct run_softc *sc, uint8_t reg, uint8_t *val) { uint32_t tmp; int ntries, error; for (ntries = 0; ntries < 10; ntries++) { if ((error = run_read(sc, RT2860_BBP_CSR_CFG, &tmp)) != 0) return error; if (!(tmp & RT2860_BBP_CSR_KICK)) break; } if (ntries == 10) return ETIMEDOUT; tmp = RT2860_BBP_CSR_READ | RT2860_BBP_CSR_KICK | reg << 8; if ((error = run_write(sc, RT2860_BBP_CSR_CFG, tmp)) != 0) return error; for (ntries = 0; ntries < 10; ntries++) { if ((error = run_read(sc, RT2860_BBP_CSR_CFG, &tmp)) != 0) return error; if (!(tmp & RT2860_BBP_CSR_KICK)) break; } if (ntries == 10) return ETIMEDOUT; *val = tmp & 0xff; return 0; } static int run_bbp_write(struct run_softc *sc, uint8_t reg, uint8_t val) { uint32_t tmp; int ntries, error; for (ntries = 0; ntries < 10; ntries++) { if ((error = run_read(sc, RT2860_BBP_CSR_CFG, &tmp)) != 0) return error; if (!(tmp & RT2860_BBP_CSR_KICK)) break; } if (ntries == 10) return ETIMEDOUT; tmp = RT2860_BBP_CSR_KICK | reg << 8 | val; return run_write(sc, RT2860_BBP_CSR_CFG, tmp); } /* * Send a command to the 8051 microcontroller unit. */ static int run_mcu_cmd(struct run_softc *sc, uint8_t cmd, uint16_t arg) { uint32_t tmp; int error, ntries; for (ntries = 0; ntries < 100; ntries++) { if ((error = run_read(sc, RT2860_H2M_MAILBOX, &tmp)) != 0) return error; if (!(tmp & RT2860_H2M_BUSY)) break; } if (ntries == 100) return ETIMEDOUT; tmp = RT2860_H2M_BUSY | RT2860_TOKEN_NO_INTR << 16 | arg; if ((error = run_write(sc, RT2860_H2M_MAILBOX, tmp)) == 0) error = run_write(sc, RT2860_HOST_CMD, cmd); return error; } /* * Add `delta' (signed) to each 4-bit sub-word of a 32-bit word. * Used to adjust per-rate Tx power registers. */ static __inline uint32_t b4inc(uint32_t b32, int8_t delta) { int8_t i, b4; for (i = 0; i < 8; i++) { b4 = b32 & 0xf; b4 += delta; if (b4 < 0) b4 = 0; else if (b4 > 0xf) b4 = 0xf; b32 = b32 >> 4 | b4 << 28; } return b32; } static const char * run_get_rf(uint16_t rev) { switch (rev) { case RT2860_RF_2820: return "RT2820"; case RT2860_RF_2850: return "RT2850"; case RT2860_RF_2720: return "RT2720"; case RT2860_RF_2750: return "RT2750"; case RT3070_RF_3020: return "RT3020"; case RT3070_RF_2020: return "RT2020"; case RT3070_RF_3021: return "RT3021"; case RT3070_RF_3022: return "RT3022"; case RT3070_RF_3052: return "RT3052"; case RT3070_RF_3053: return "RT3053"; case RT5592_RF_5592: return "RT5592"; case RT5390_RF_5370: return "RT5370"; case RT5390_RF_5372: return "RT5372"; } return "unknown"; } static void run_rt3593_get_txpower(struct run_softc *sc) { uint16_t addr, val; int i; /* Read power settings for 2GHz channels. */ for (i = 0; i < 14; i += 2) { addr = (sc->ntxchains == 3) ? RT3593_EEPROM_PWR2GHZ_BASE1 : RT2860_EEPROM_PWR2GHZ_BASE1; run_srom_read(sc, addr + i / 2, &val); sc->txpow1[i + 0] = (int8_t)(val & 0xff); sc->txpow1[i + 1] = (int8_t)(val >> 8); addr = (sc->ntxchains == 3) ? RT3593_EEPROM_PWR2GHZ_BASE2 : RT2860_EEPROM_PWR2GHZ_BASE2; run_srom_read(sc, addr + i / 2, &val); sc->txpow2[i + 0] = (int8_t)(val & 0xff); sc->txpow2[i + 1] = (int8_t)(val >> 8); if (sc->ntxchains == 3) { run_srom_read(sc, RT3593_EEPROM_PWR2GHZ_BASE3 + i / 2, &val); sc->txpow3[i + 0] = (int8_t)(val & 0xff); sc->txpow3[i + 1] = (int8_t)(val >> 8); } } /* Fix broken Tx power entries. */ for (i = 0; i < 14; i++) { if (sc->txpow1[i] > 31) sc->txpow1[i] = 5; if (sc->txpow2[i] > 31) sc->txpow2[i] = 5; if (sc->ntxchains == 3) { if (sc->txpow3[i] > 31) sc->txpow3[i] = 5; } } /* Read power settings for 5GHz channels. */ for (i = 0; i < 40; i += 2) { run_srom_read(sc, RT3593_EEPROM_PWR5GHZ_BASE1 + i / 2, &val); sc->txpow1[i + 14] = (int8_t)(val & 0xff); sc->txpow1[i + 15] = (int8_t)(val >> 8); run_srom_read(sc, RT3593_EEPROM_PWR5GHZ_BASE2 + i / 2, &val); sc->txpow2[i + 14] = (int8_t)(val & 0xff); sc->txpow2[i + 15] = (int8_t)(val >> 8); if (sc->ntxchains == 3) { run_srom_read(sc, RT3593_EEPROM_PWR5GHZ_BASE3 + i / 2, &val); sc->txpow3[i + 14] = (int8_t)(val & 0xff); sc->txpow3[i + 15] = (int8_t)(val >> 8); } } } static void run_get_txpower(struct run_softc *sc) { uint16_t val; int i; /* Read power settings for 2GHz channels. */ for (i = 0; i < 14; i += 2) { run_srom_read(sc, RT2860_EEPROM_PWR2GHZ_BASE1 + i / 2, &val); sc->txpow1[i + 0] = (int8_t)(val & 0xff); sc->txpow1[i + 1] = (int8_t)(val >> 8); if (sc->mac_ver != 0x5390) { run_srom_read(sc, RT2860_EEPROM_PWR2GHZ_BASE2 + i / 2, &val); sc->txpow2[i + 0] = (int8_t)(val & 0xff); sc->txpow2[i + 1] = (int8_t)(val >> 8); } } /* Fix broken Tx power entries. */ for (i = 0; i < 14; i++) { if (sc->mac_ver >= 0x5390) { if (sc->txpow1[i] < 0 || sc->txpow1[i] > 39) sc->txpow1[i] = 5; } else { if (sc->txpow1[i] < 0 || sc->txpow1[i] > 31) sc->txpow1[i] = 5; } if (sc->mac_ver > 0x5390) { if (sc->txpow2[i] < 0 || sc->txpow2[i] > 39) sc->txpow2[i] = 5; } else if (sc->mac_ver < 0x5390) { if (sc->txpow2[i] < 0 || sc->txpow2[i] > 31) sc->txpow2[i] = 5; } DPRINTF(("chan %d: power1=%d, power2=%d\n", rt2860_rf2850[i].chan, sc->txpow1[i], sc->txpow2[i])); } /* Read power settings for 5GHz channels. */ for (i = 0; i < 40; i += 2) { run_srom_read(sc, RT2860_EEPROM_PWR5GHZ_BASE1 + i / 2, &val); sc->txpow1[i + 14] = (int8_t)(val & 0xff); sc->txpow1[i + 15] = (int8_t)(val >> 8); run_srom_read(sc, RT2860_EEPROM_PWR5GHZ_BASE2 + i / 2, &val); sc->txpow2[i + 14] = (int8_t)(val & 0xff); sc->txpow2[i + 15] = (int8_t)(val >> 8); } /* Fix broken Tx power entries. */ for (i = 0; i < 40; i++ ) { if (sc->mac_ver != 0x5592) { if (sc->txpow1[14 + i] < -7 || sc->txpow1[14 + i] > 15) sc->txpow1[14 + i] = 5; if (sc->txpow2[14 + i] < -7 || sc->txpow2[14 + i] > 15) sc->txpow2[14 + i] = 5; } DPRINTF(("chan %d: power1=%d, power2=%d\n", rt2860_rf2850[14 + i].chan, sc->txpow1[14 + i], sc->txpow2[14 + i])); } } static int run_read_eeprom(struct run_softc *sc) { struct ieee80211com *ic = &sc->sc_ic; int8_t delta_2ghz, delta_5ghz; uint32_t tmp; uint16_t val; int ridx, ant, i; /* check whether the ROM is eFUSE ROM or EEPROM */ sc->sc_srom_read = run_eeprom_read_2; if (sc->mac_ver >= 0x3070) { run_read(sc, RT3070_EFUSE_CTRL, &tmp); DPRINTF(("EFUSE_CTRL=0x%08x\n", tmp)); if (tmp & RT3070_SEL_EFUSE) sc->sc_srom_read = run_efuse_read_2; } /* read ROM version */ run_srom_read(sc, RT2860_EEPROM_VERSION, &val); DPRINTF(("EEPROM rev=%d, FAE=%d\n", val & 0xff, val >> 8)); /* read MAC address */ run_srom_read(sc, RT2860_EEPROM_MAC01, &val); ic->ic_myaddr[0] = val & 0xff; ic->ic_myaddr[1] = val >> 8; run_srom_read(sc, RT2860_EEPROM_MAC23, &val); ic->ic_myaddr[2] = val & 0xff; ic->ic_myaddr[3] = val >> 8; run_srom_read(sc, RT2860_EEPROM_MAC45, &val); ic->ic_myaddr[4] = val & 0xff; ic->ic_myaddr[5] = val >> 8; if (sc->mac_ver < 0x3593) { /* read vendor BBP settings */ for (i = 0; i < 10; i++) { run_srom_read(sc, RT2860_EEPROM_BBP_BASE + i, &val); sc->bbp[i].val = val & 0xff; sc->bbp[i].reg = val >> 8; DPRINTF(("BBP%d=0x%02x\n", sc->bbp[i].reg, sc->bbp[i].val)); } if (sc->mac_ver >= 0x3071) { /* read vendor RF settings */ for (i = 0; i < 8; i++) { run_srom_read(sc, RT3071_EEPROM_RF_BASE + i, &val); sc->rf[i].val = val & 0xff; sc->rf[i].reg = val >> 8; DPRINTF(("RF%d=0x%02x\n", sc->rf[i].reg, sc->rf[i].val)); } } } /* read RF frequency offset from EEPROM */ run_srom_read(sc, (sc->mac_ver != 0x3593) ? RT2860_EEPROM_FREQ_LEDS : RT3593_EEPROM_FREQ, &val); sc->freq = ((val & 0xff) != 0xff) ? val & 0xff : 0; DPRINTF(("EEPROM freq offset %d\n", sc->freq & 0xff)); run_srom_read(sc, (sc->mac_ver != 0x3593) ? RT2860_EEPROM_FREQ_LEDS : RT3593_EEPROM_FREQ, &val); if ((val >> 8) != 0xff) { /* read LEDs operating mode */ sc->leds = val >> 8; run_srom_read(sc, (sc->mac_ver != 0x3593) ? RT2860_EEPROM_LED1 : RT3593_EEPROM_LED1, &sc->led[0]); run_srom_read(sc, (sc->mac_ver != 0x3593) ? RT2860_EEPROM_LED2 : RT3593_EEPROM_LED2, &sc->led[1]); run_srom_read(sc, (sc->mac_ver != 0x3593) ? RT2860_EEPROM_LED3 : RT3593_EEPROM_LED3, &sc->led[2]); } else { /* broken EEPROM, use default settings */ sc->leds = 0x01; sc->led[0] = 0x5555; sc->led[1] = 0x2221; sc->led[2] = 0x5627; /* differs from RT2860 */ } DPRINTF(("EEPROM LED mode=0x%02x, LEDs=0x%04x/0x%04x/0x%04x\n", sc->leds, sc->led[0], sc->led[1], sc->led[2])); /* read RF information */ if (sc->mac_ver == 0x5390 || sc->mac_ver == 0x5392) run_srom_read(sc, 0x00, &val); else run_srom_read(sc, RT2860_EEPROM_ANTENNA, &val); if (val == 0xffff) { DPRINTF(("invalid EEPROM antenna info, using default\n")); if (sc->mac_ver == 0x3572) { /* default to RF3052 2T2R */ sc->rf_rev = RT3070_RF_3052; sc->ntxchains = 2; sc->nrxchains = 2; } else if (sc->mac_ver >= 0x3070) { /* default to RF3020 1T1R */ sc->rf_rev = RT3070_RF_3020; sc->ntxchains = 1; sc->nrxchains = 1; } else { /* default to RF2820 1T2R */ sc->rf_rev = RT2860_RF_2820; sc->ntxchains = 1; sc->nrxchains = 2; } } else { if (sc->mac_ver == 0x5390 || sc->mac_ver == 0x5392) { sc->rf_rev = val; run_srom_read(sc, RT2860_EEPROM_ANTENNA, &val); } else sc->rf_rev = (val >> 8) & 0xf; sc->ntxchains = (val >> 4) & 0xf; sc->nrxchains = val & 0xf; } DPRINTF(("EEPROM RF rev=0x%04x chains=%dT%dR\n", sc->rf_rev, sc->ntxchains, sc->nrxchains)); run_srom_read(sc, RT2860_EEPROM_CONFIG, &val); DPRINTF(("EEPROM CFG 0x%04x\n", val)); /* check if driver should patch the DAC issue */ if ((val >> 8) != 0xff) sc->patch_dac = (val >> 15) & 1; if ((val & 0xff) != 0xff) { sc->ext_5ghz_lna = (val >> 3) & 1; sc->ext_2ghz_lna = (val >> 2) & 1; /* check if RF supports automatic Tx access gain control */ sc->calib_2ghz = sc->calib_5ghz = (val >> 1) & 1; /* check if we have a hardware radio switch */ sc->rfswitch = val & 1; } /* Read Tx power settings. */ if (sc->mac_ver == 0x3593) run_rt3593_get_txpower(sc); else run_get_txpower(sc); /* read Tx power compensation for each Tx rate */ run_srom_read(sc, RT2860_EEPROM_DELTAPWR, &val); delta_2ghz = delta_5ghz = 0; if ((val & 0xff) != 0xff && (val & 0x80)) { delta_2ghz = val & 0xf; if (!(val & 0x40)) /* negative number */ delta_2ghz = -delta_2ghz; } val >>= 8; if ((val & 0xff) != 0xff && (val & 0x80)) { delta_5ghz = val & 0xf; if (!(val & 0x40)) /* negative number */ delta_5ghz = -delta_5ghz; } DPRINTF(("power compensation=%d (2GHz), %d (5GHz)\n", delta_2ghz, delta_5ghz)); for (ridx = 0; ridx < 5; ridx++) { uint32_t reg; run_srom_read(sc, RT2860_EEPROM_RPWR + ridx * 2, &val); reg = val; run_srom_read(sc, RT2860_EEPROM_RPWR + ridx * 2 + 1, &val); reg |= (uint32_t)val << 16; sc->txpow20mhz[ridx] = reg; sc->txpow40mhz_2ghz[ridx] = b4inc(reg, delta_2ghz); sc->txpow40mhz_5ghz[ridx] = b4inc(reg, delta_5ghz); DPRINTF(("ridx %d: power 20MHz=0x%08x, 40MHz/2GHz=0x%08x, " "40MHz/5GHz=0x%08x\n", ridx, sc->txpow20mhz[ridx], sc->txpow40mhz_2ghz[ridx], sc->txpow40mhz_5ghz[ridx])); } DPRINTF(("mac_ver %hx\n", sc->mac_ver)); /* read RSSI offsets and LNA gains from EEPROM */ run_srom_read(sc, (sc->mac_ver != 0x3593) ? RT2860_EEPROM_RSSI1_2GHZ : RT3593_EEPROM_RSSI1_2GHZ, &val); sc->rssi_2ghz[0] = val & 0xff; /* Ant A */ sc->rssi_2ghz[1] = val >> 8; /* Ant B */ run_srom_read(sc, (sc->mac_ver != 0x3593) ? RT2860_EEPROM_RSSI2_2GHZ : RT3593_EEPROM_RSSI2_2GHZ, &val); if (sc->mac_ver >= 0x3070) { if (sc->mac_ver == 0x3593) { sc->txmixgain_2ghz = 0; sc->rssi_2ghz[2] = val & 0xff; /* Ant C */ } else { /* * On RT3070 chips (limited to 2 Rx chains), this ROM * field contains the Tx mixer gain for the 2GHz band. */ if ((val & 0xff) != 0xff) sc->txmixgain_2ghz = val & 0x7; } DPRINTF(("tx mixer gain=%u (2GHz)\n", sc->txmixgain_2ghz)); } else { sc->rssi_2ghz[2] = val & 0xff; /* Ant C */ } if (sc->mac_ver == 0x3593) run_srom_read(sc, RT3593_EEPROM_LNA_5GHZ, &val); sc->lna[2] = val >> 8; /* channel group 2 */ run_srom_read(sc, (sc->mac_ver != 0x3593) ? RT2860_EEPROM_RSSI1_5GHZ : RT3593_EEPROM_RSSI1_5GHZ, &val); sc->rssi_5ghz[0] = val & 0xff; /* Ant A */ sc->rssi_5ghz[1] = val >> 8; /* Ant B */ run_srom_read(sc, (sc->mac_ver != 0x3593) ? RT2860_EEPROM_RSSI2_5GHZ : RT3593_EEPROM_RSSI2_5GHZ, &val); if (sc->mac_ver == 0x3572) { /* * On RT3572 chips (limited to 2 Rx chains), this ROM * field contains the Tx mixer gain for the 5GHz band. */ if ((val & 0xff) != 0xff) sc->txmixgain_5ghz = val & 0x7; DPRINTF(("tx mixer gain=%u (5GHz)\n", sc->txmixgain_5ghz)); } else { sc->rssi_5ghz[2] = val & 0xff; /* Ant C */ } if (sc->mac_ver == 0x3593) { sc->txmixgain_5ghz = 0; run_srom_read(sc, RT3593_EEPROM_LNA_5GHZ, &val); } sc->lna[3] = val >> 8; /* channel group 3 */ run_srom_read(sc, (sc->mac_ver != 0x3593) ? RT2860_EEPROM_LNA : RT3593_EEPROM_LNA, &val); sc->lna[0] = val & 0xff; /* channel group 0 */ sc->lna[1] = val >> 8; /* channel group 1 */ /* fix broken 5GHz LNA entries */ if (sc->lna[2] == 0 || sc->lna[2] == 0xff) { DPRINTF(("invalid LNA for channel group %d\n", 2)); sc->lna[2] = sc->lna[1]; } if (sc->lna[3] == 0 || sc->lna[3] == 0xff) { DPRINTF(("invalid LNA for channel group %d\n", 3)); sc->lna[3] = sc->lna[1]; } /* fix broken RSSI offset entries */ for (ant = 0; ant < 3; ant++) { if (sc->rssi_2ghz[ant] < -10 || sc->rssi_2ghz[ant] > 10) { DPRINTF(("invalid RSSI%d offset: %d (2GHz)\n", ant + 1, sc->rssi_2ghz[ant])); sc->rssi_2ghz[ant] = 0; } if (sc->rssi_5ghz[ant] < -10 || sc->rssi_5ghz[ant] > 10) { DPRINTF(("invalid RSSI%d offset: %d (5GHz)\n", ant + 1, sc->rssi_5ghz[ant])); sc->rssi_5ghz[ant] = 0; } } return 0; } static struct ieee80211_node * run_node_alloc(struct ieee80211_node_table *nt) { struct run_node *rn = malloc(sizeof(struct run_node), M_DEVBUF, M_NOWAIT | M_ZERO); return rn ? &rn->ni : NULL; } static int run_media_change(struct ifnet *ifp) { struct run_softc *sc = ifp->if_softc; struct ieee80211com *ic = &sc->sc_ic; uint8_t rate, ridx; int error; error = ieee80211_media_change(ifp); if (error != ENETRESET) return error; if (ic->ic_fixed_rate != IEEE80211_FIXED_RATE_NONE) { rate = ic->ic_sup_rates[ic->ic_curmode]. rs_rates[ic->ic_fixed_rate] & IEEE80211_RATE_VAL; for (ridx = 0; ridx <= RT2860_RIDX_MAX; ridx++) if (rt2860_rates[ridx].rate == rate) break; sc->fixed_ridx = ridx; } if ((ifp->if_flags & (IFF_UP | IFF_RUNNING)) == (IFF_UP | IFF_RUNNING)) run_init(ifp); return 0; } static void run_next_scan(void *arg) { struct run_softc *sc = arg; if (sc->sc_ic.ic_state == IEEE80211_S_SCAN) ieee80211_next_scan(&sc->sc_ic); } static void run_task(void *arg) { struct run_softc *sc = arg; struct run_host_cmd_ring *ring = &sc->cmdq; struct run_host_cmd *cmd; int s; /* process host commands */ s = splusb(); while (ring->next != ring->cur) { cmd = &ring->cmd[ring->next]; splx(s); membar_consumer(); /* callback */ cmd->cb(sc, cmd->data); s = splusb(); atomic_dec_uint(&ring->queued); ring->next = (ring->next + 1) % RUN_HOST_CMD_RING_COUNT; } wakeup(ring); splx(s); } static void run_do_async(struct run_softc *sc, void (*cb)(struct run_softc *, void *), void *arg, int len) { struct run_host_cmd_ring *ring = &sc->cmdq; struct run_host_cmd *cmd; int s; if (sc->sc_flags & RUN_DETACHING) return; s = splusb(); cmd = &ring->cmd[ring->cur]; cmd->cb = cb; KASSERT(len <= sizeof(cmd->data)); memcpy(cmd->data, arg, len); membar_producer(); ring->cur = (ring->cur + 1) % RUN_HOST_CMD_RING_COUNT; /* if there is no pending command already, schedule a task */ if (atomic_inc_uint_nv(&ring->queued) == 1) usb_add_task(sc->sc_udev, &sc->sc_task, USB_TASKQ_DRIVER); splx(s); } static int run_newstate(struct ieee80211com *ic, enum ieee80211_state nstate, int arg) { struct run_softc *sc = ic->ic_ifp->if_softc; struct run_cmd_newstate cmd; callout_stop(&sc->scan_to); callout_stop(&sc->calib_to); /* do it in a process context */ cmd.state = nstate; cmd.arg = arg; run_do_async(sc, run_newstate_cb, &cmd, sizeof(cmd)); return 0; } static void run_newstate_cb(struct run_softc *sc, void *arg) { struct run_cmd_newstate *cmd = arg; struct ifnet *ifp = &sc->sc_if; struct ieee80211com *ic = &sc->sc_ic; enum ieee80211_state ostate; struct ieee80211_node *ni; uint32_t tmp, sta[3]; uint8_t wcid; int s; s = splnet(); ostate = ic->ic_state; if (ostate == IEEE80211_S_RUN) { /* turn link LED off */ run_set_leds(sc, RT2860_LED_RADIO); } switch (cmd->state) { case IEEE80211_S_INIT: if (ostate == IEEE80211_S_RUN) { /* abort TSF synchronization */ run_read(sc, RT2860_BCN_TIME_CFG, &tmp); run_write(sc, RT2860_BCN_TIME_CFG, tmp & ~(RT2860_BCN_TX_EN | RT2860_TSF_TIMER_EN | RT2860_TBTT_TIMER_EN)); } break; case IEEE80211_S_SCAN: run_set_chan(sc, ic->ic_curchan); callout_schedule(&sc->scan_to, hz / 5); break; case IEEE80211_S_AUTH: case IEEE80211_S_ASSOC: run_set_chan(sc, ic->ic_curchan); break; case IEEE80211_S_RUN: run_set_chan(sc, ic->ic_curchan); ni = ic->ic_bss; if (ic->ic_opmode != IEEE80211_M_MONITOR) { run_updateslot(ifp); run_enable_mrr(sc); run_set_txpreamble(sc); run_set_basicrates(sc); run_set_bssid(sc, ni->ni_bssid); } #ifndef IEEE80211_STA_ONLY if (ic->ic_opmode == IEEE80211_M_HOSTAP || ic->ic_opmode == IEEE80211_M_IBSS) (void)run_setup_beacon(sc); #endif if (ic->ic_opmode == IEEE80211_M_STA) { /* add BSS entry to the WCID table */ wcid = RUN_AID2WCID(ni->ni_associd); run_write_region_1(sc, RT2860_WCID_ENTRY(wcid), ni->ni_macaddr, IEEE80211_ADDR_LEN); /* fake a join to init the tx rate */ run_newassoc(ni, 1); } if (ic->ic_opmode != IEEE80211_M_MONITOR) { run_enable_tsf_sync(sc); /* clear statistic registers used by AMRR */ run_read_region_1(sc, RT2860_TX_STA_CNT0, (uint8_t *)sta, sizeof(sta)); /* start calibration timer */ callout_schedule(&sc->calib_to, hz); } /* turn link LED on */ run_set_leds(sc, RT2860_LED_RADIO | (IEEE80211_IS_CHAN_2GHZ(ic->ic_curchan) ? RT2860_LED_LINK_2GHZ : RT2860_LED_LINK_5GHZ)); break; } (void)sc->sc_newstate(ic, cmd->state, cmd->arg); splx(s); } static int run_updateedca(struct ieee80211com *ic) { /* do it in a process context */ run_do_async(ic->ic_ifp->if_softc, run_updateedca_cb, NULL, 0); return 0; } /* ARGSUSED */ static void run_updateedca_cb(struct run_softc *sc, void *arg) { struct ieee80211com *ic = &sc->sc_ic; int s, aci; s = splnet(); /* update MAC TX configuration registers */ for (aci = 0; aci < WME_NUM_AC; aci++) { run_write(sc, RT2860_EDCA_AC_CFG(aci), ic->ic_wme.wme_params[aci].wmep_logcwmax << 16 | ic->ic_wme.wme_params[aci].wmep_logcwmin << 12 | ic->ic_wme.wme_params[aci].wmep_aifsn << 8 | ic->ic_wme.wme_params[aci].wmep_txopLimit); } /* update SCH/DMA registers too */ run_write(sc, RT2860_WMM_AIFSN_CFG, ic->ic_wme.wme_params[WME_AC_VO].wmep_aifsn << 12 | ic->ic_wme.wme_params[WME_AC_VI].wmep_aifsn << 8 | ic->ic_wme.wme_params[WME_AC_BK].wmep_aifsn << 4 | ic->ic_wme.wme_params[WME_AC_BE].wmep_aifsn); run_write(sc, RT2860_WMM_CWMIN_CFG, ic->ic_wme.wme_params[WME_AC_VO].wmep_logcwmin << 12 | ic->ic_wme.wme_params[WME_AC_VI].wmep_logcwmin << 8 | ic->ic_wme.wme_params[WME_AC_BK].wmep_logcwmin << 4 | ic->ic_wme.wme_params[WME_AC_BE].wmep_logcwmin); run_write(sc, RT2860_WMM_CWMAX_CFG, ic->ic_wme.wme_params[WME_AC_VO].wmep_logcwmax << 12 | ic->ic_wme.wme_params[WME_AC_VI].wmep_logcwmax << 8 | ic->ic_wme.wme_params[WME_AC_BK].wmep_logcwmax << 4 | ic->ic_wme.wme_params[WME_AC_BE].wmep_logcwmax); run_write(sc, RT2860_WMM_TXOP0_CFG, ic->ic_wme.wme_params[WME_AC_BK].wmep_txopLimit << 16 | ic->ic_wme.wme_params[WME_AC_BE].wmep_txopLimit); run_write(sc, RT2860_WMM_TXOP1_CFG, ic->ic_wme.wme_params[WME_AC_VO].wmep_txopLimit << 16 | ic->ic_wme.wme_params[WME_AC_VI].wmep_txopLimit); splx(s); } #ifdef RUN_HWCRYPTO static int run_set_key(struct ieee80211com *ic, const struct ieee80211_key *k, const uint8_t *mac) { struct run_softc *sc = ic->ic_ifp->if_softc; struct ieee80211_node *ni = ic->ic_bss; struct run_cmd_key cmd; /* do it in a process context */ cmd.key = *k; cmd.associd = (ni != NULL) ? ni->ni_associd : 0; run_do_async(sc, run_set_key_cb, &cmd, sizeof(cmd)); return 1; } static void run_set_key_cb(struct run_softc *sc, void *arg) { #ifndef IEEE80211_STA_ONLY struct ieee80211com *ic = &sc->sc_ic; #endif struct run_cmd_key *cmd = arg; struct ieee80211_key *k = &cmd->key; uint32_t attr; uint16_t base; uint8_t mode, wcid, iv[8]; /* map net80211 cipher to RT2860 security mode */ switch (k->wk_cipher->ic_cipher) { case IEEE80211_CIPHER_WEP: k->wk_flags |= IEEE80211_KEY_GROUP; /* XXX */ if (k->wk_keylen == 5) mode = RT2860_MODE_WEP40; else mode = RT2860_MODE_WEP104; break; case IEEE80211_CIPHER_TKIP: mode = RT2860_MODE_TKIP; break; case IEEE80211_CIPHER_AES_CCM: mode = RT2860_MODE_AES_CCMP; break; default: return; } if (k->wk_flags & IEEE80211_KEY_GROUP) { wcid = 0; /* NB: update WCID0 for group keys */ base = RT2860_SKEY(0, k->wk_keyix); } else { wcid = RUN_AID2WCID(cmd->associd); base = RT2860_PKEY(wcid); } if (k->wk_cipher->ic_cipher == IEEE80211_CIPHER_TKIP) { run_write_region_1(sc, base, k->wk_key, 16); #ifndef IEEE80211_STA_ONLY if (ic->ic_opmode == IEEE80211_M_HOSTAP) { run_write_region_1(sc, base + 16, &k->wk_key[16], 8); run_write_region_1(sc, base + 24, &k->wk_key[24], 8); } else #endif { run_write_region_1(sc, base + 16, &k->wk_key[24], 8); run_write_region_1(sc, base + 24, &k->wk_key[16], 8); } } else { /* roundup len to 16-bit: XXX fix write_region_1() instead */ run_write_region_1(sc, base, k->wk_key, (k->wk_keylen + 1) & ~1); } if (!(k->wk_flags & IEEE80211_KEY_GROUP) || (k->wk_flags & IEEE80211_KEY_XMIT)) { /* set initial packet number in IV+EIV */ if (k->wk_cipher->ic_cipher == IEEE80211_CIPHER_WEP) { memset(iv, 0, sizeof(iv)); iv[3] = sc->sc_ic.ic_crypto.cs_def_txkey << 6; } else { if (k->wk_cipher->ic_cipher == IEEE80211_CIPHER_TKIP) { iv[0] = k->wk_keytsc >> 8; iv[1] = (iv[0] | 0x20) & 0x7f; iv[2] = k->wk_keytsc; } else /* CCMP */ { iv[0] = k->wk_keytsc; iv[1] = k->wk_keytsc >> 8; iv[2] = 0; } iv[3] = k->wk_keyix << 6 | IEEE80211_WEP_EXTIV; iv[4] = k->wk_keytsc >> 16; iv[5] = k->wk_keytsc >> 24; iv[6] = k->wk_keytsc >> 32; iv[7] = k->wk_keytsc >> 40; } run_write_region_1(sc, RT2860_IVEIV(wcid), iv, 8); } if (k->wk_flags & IEEE80211_KEY_GROUP) { /* install group key */ run_read(sc, RT2860_SKEY_MODE_0_7, &attr); attr &= ~(0xf << (k->wk_keyix * 4)); attr |= mode << (k->wk_keyix * 4); run_write(sc, RT2860_SKEY_MODE_0_7, attr); } else { /* install pairwise key */ run_read(sc, RT2860_WCID_ATTR(wcid), &attr); attr = (attr & ~0xf) | (mode << 1) | RT2860_RX_PKEY_EN; run_write(sc, RT2860_WCID_ATTR(wcid), attr); } } static int run_delete_key(struct ieee80211com *ic, const struct ieee80211_key *k) { struct run_softc *sc = ic->ic_ifp->if_softc; struct ieee80211_node *ni = ic->ic_bss; struct run_cmd_key cmd; /* do it in a process context */ cmd.key = *k; cmd.associd = (ni != NULL) ? ni->ni_associd : 0; run_do_async(sc, run_delete_key_cb, &cmd, sizeof(cmd)); return 1; } static void run_delete_key_cb(struct run_softc *sc, void *arg) { struct run_cmd_key *cmd = arg; struct ieee80211_key *k = &cmd->key; uint32_t attr; uint8_t wcid; if (k->wk_cipher->ic_cipher == IEEE80211_CIPHER_WEP) k->wk_flags |= IEEE80211_KEY_GROUP; /* XXX */ if (k->wk_flags & IEEE80211_KEY_GROUP) { /* remove group key */ run_read(sc, RT2860_SKEY_MODE_0_7, &attr); attr &= ~(0xf << (k->wk_keyix * 4)); run_write(sc, RT2860_SKEY_MODE_0_7, attr); } else { /* remove pairwise key */ wcid = RUN_AID2WCID(cmd->associd); run_read(sc, RT2860_WCID_ATTR(wcid), &attr); attr &= ~0xf; run_write(sc, RT2860_WCID_ATTR(wcid), attr); } } #endif static void run_calibrate_to(void *arg) { /* do it in a process context */ run_do_async(arg, run_calibrate_cb, NULL, 0); /* next timeout will be rescheduled in the calibration task */ } /* ARGSUSED */ static void run_calibrate_cb(struct run_softc *sc, void *arg) { struct ifnet *ifp = &sc->sc_if; uint32_t sta[3]; int s, error; /* read statistic counters (clear on read) and update AMRR state */ error = run_read_region_1(sc, RT2860_TX_STA_CNT0, (uint8_t *)sta, sizeof(sta)); if (error != 0) goto skip; DPRINTF(("retrycnt=%d txcnt=%d failcnt=%d\n", le32toh(sta[1]) >> 16, le32toh(sta[1]) & 0xffff, le32toh(sta[0]) & 0xffff)); s = splnet(); /* count failed TX as errors */ if_statadd(ifp, if_oerrors, le32toh(sta[0]) & 0xffff); sc->amn.amn_retrycnt = (le32toh(sta[0]) & 0xffff) + /* failed TX count */ (le32toh(sta[1]) >> 16); /* TX retransmission count */ sc->amn.amn_txcnt = sc->amn.amn_retrycnt + (le32toh(sta[1]) & 0xffff); /* successful TX count */ ieee80211_amrr_choose(&sc->amrr, sc->sc_ic.ic_bss, &sc->amn); splx(s); skip: callout_schedule(&sc->calib_to, hz); } static void run_newassoc(struct ieee80211_node *ni, int isnew) { struct run_softc *sc = ni->ni_ic->ic_ifp->if_softc; struct run_node *rn = (void *)ni; struct ieee80211_rateset *rs = &ni->ni_rates; uint8_t rate; int ridx, i, j; DPRINTF(("new assoc isnew=%d addr=%s\n", isnew, ether_sprintf(ni->ni_macaddr))); ieee80211_amrr_node_init(&sc->amrr, &sc->amn); /* start at lowest available bit-rate, AMRR will raise */ ni->ni_txrate = 0; for (i = 0; i < rs->rs_nrates; i++) { rate = rs->rs_rates[i] & IEEE80211_RATE_VAL; /* convert 802.11 rate to hardware rate index */ for (ridx = 0; ridx < RT2860_RIDX_MAX; ridx++) if (rt2860_rates[ridx].rate == rate) break; rn->ridx[i] = ridx; /* determine rate of control response frames */ for (j = i; j >= 0; j--) { if ((rs->rs_rates[j] & IEEE80211_RATE_BASIC) && rt2860_rates[rn->ridx[i]].phy == rt2860_rates[rn->ridx[j]].phy) break; } if (j >= 0) { rn->ctl_ridx[i] = rn->ridx[j]; } else { /* no basic rate found, use mandatory one */ rn->ctl_ridx[i] = rt2860_rates[ridx].ctl_ridx; } DPRINTF(("rate=0x%02x ridx=%d ctl_ridx=%d\n", rs->rs_rates[i], rn->ridx[i], rn->ctl_ridx[i])); } } /* * Return the Rx chain with the highest RSSI for a given frame. */ static __inline uint8_t run_maxrssi_chain(struct run_softc *sc, const struct rt2860_rxwi *rxwi) { uint8_t rxchain = 0; if (sc->nrxchains > 1) { if (rxwi->rssi[1] > rxwi->rssi[rxchain]) rxchain = 1; if (sc->nrxchains > 2) if (rxwi->rssi[2] > rxwi->rssi[rxchain]) rxchain = 2; } return rxchain; } static void run_rx_frame(struct run_softc *sc, uint8_t *buf, int dmalen) { struct ieee80211com *ic = &sc->sc_ic; struct ifnet *ifp = &sc->sc_if; struct ieee80211_frame *wh; struct ieee80211_node *ni; struct rt2870_rxd *rxd; struct rt2860_rxwi *rxwi; struct mbuf *m; uint32_t flags; uint16_t len, rxwisize, phy; uint8_t ant, rssi; int s; #ifdef RUN_HWCRYPTO int decrypted = 0; #endif rxwi = (struct rt2860_rxwi *)buf; rxwisize = sizeof(struct rt2860_rxwi); if (sc->mac_ver == 0x5592) rxwisize += sizeof(uint64_t); else if (sc->mac_ver == 0x3593) rxwisize += sizeof(uint32_t); len = le16toh(rxwi->len) & 0xfff; if (__predict_false(len > dmalen)) { DPRINTF(("bad RXWI length %u > %u\n", len, dmalen)); return; } /* Rx descriptor is located at the end */ rxd = (struct rt2870_rxd *)(buf + dmalen); flags = le32toh(rxd->flags); if (__predict_false(flags & (RT2860_RX_CRCERR | RT2860_RX_ICVERR))) { if_statinc(ifp, if_ierrors); return; } wh = (struct ieee80211_frame *)(buf + rxwisize); if (__predict_false((flags & RT2860_RX_MICERR))) { /* report MIC failures to net80211 for TKIP */ ieee80211_notify_michael_failure(ic, wh, 0/* XXX */); if_statinc(ifp, if_ierrors); return; } if (flags & RT2860_RX_L2PAD) { u_int hdrlen = ieee80211_hdrspace(ic, wh); memmove((uint8_t *)wh + 2, wh, hdrlen); wh = (struct ieee80211_frame *)((uint8_t *)wh + 2); } #ifdef RUN_HWCRYPTO if (wh->i_fc[1] & IEEE80211_FC1_WEP) { wh->i_fc[1] &= ~IEEE80211_FC1_WEP; decrypted = 1; } #endif /* could use m_devget but net80211 wants contig mgmt frames */ MGETHDR(m, M_DONTWAIT, MT_DATA); if (__predict_false(m == NULL)) { if_statinc(ifp, if_ierrors); return; } if (len > MHLEN) { if (__predict_true(len <= MCLBYTES)) MCLGET(m, M_DONTWAIT); if (__predict_false(!(m->m_flags & M_EXT))) { if_statinc(ifp, if_ierrors); m_freem(m); return; } } /* finalize mbuf */ m_set_rcvif(m, ifp); memcpy(mtod(m, void *), wh, len); m->m_pkthdr.len = m->m_len = len; ant = run_maxrssi_chain(sc, rxwi); rssi = rxwi->rssi[ant]; if (__predict_false(sc->sc_drvbpf != NULL)) { struct run_rx_radiotap_header *tap = &sc->sc_rxtap; tap->wr_flags = 0; tap->wr_chan_freq = htole16(ic->ic_curchan->ic_freq); tap->wr_chan_flags = htole16(ic->ic_curchan->ic_flags); tap->wr_antsignal = rssi; tap->wr_antenna = ant; tap->wr_dbm_antsignal = run_rssi2dbm(sc, rssi, ant); tap->wr_rate = 2; /* in case it can't be found below */ phy = le16toh(rxwi->phy); switch (phy & RT2860_PHY_MODE) { case RT2860_PHY_CCK: switch ((phy & RT2860_PHY_MCS) & ~RT2860_PHY_SHPRE) { case 0: tap->wr_rate = 2; break; case 1: tap->wr_rate = 4; break; case 2: tap->wr_rate = 11; break; case 3: tap->wr_rate = 22; break; } if (phy & RT2860_PHY_SHPRE) tap->wr_flags |= IEEE80211_RADIOTAP_F_SHORTPRE; break; case RT2860_PHY_OFDM: switch (phy & RT2860_PHY_MCS) { case 0: tap->wr_rate = 12; break; case 1: tap->wr_rate = 18; break; case 2: tap->wr_rate = 24; break; case 3: tap->wr_rate = 36; break; case 4: tap->wr_rate = 48; break; case 5: tap->wr_rate = 72; break; case 6: tap->wr_rate = 96; break; case 7: tap->wr_rate = 108; break; } break; } bpf_mtap2(sc->sc_drvbpf, tap, sc->sc_rxtap_len, m, BPF_D_IN); } s = splnet(); ni = ieee80211_find_rxnode(ic, (struct ieee80211_frame_min *)wh); #ifdef RUN_HWCRYPTO if (decrypted) { uint32_t icflags = ic->ic_flags; ic->ic_flags &= ~IEEE80211_F_DROPUNENC; /* XXX */ ieee80211_input(ic, m, ni, rssi, 0); ic->ic_flags = icflags; } else #endif ieee80211_input(ic, m, ni, rssi, 0); /* node is no longer needed */ ieee80211_free_node(ni); /* * In HostAP mode, ieee80211_input() will enqueue packets in if_snd * without calling if_start(). */ if (!IFQ_IS_EMPTY(&ifp->if_snd) && !(ifp->if_flags & IFF_OACTIVE)) run_start(ifp); splx(s); } static void run_rxeof(struct usbd_xfer *xfer, void *priv, usbd_status status) { struct run_rx_data *data = priv; struct run_softc *sc = data->sc; uint8_t *buf; uint32_t dmalen; int xferlen; uint16_t rxwisize; if (__predict_false(sc->sc_flags & RUN_DETACHING)) return; rxwisize = sizeof(struct rt2860_rxwi); if (sc->mac_ver == 0x5592) rxwisize += sizeof(uint64_t); else if (sc->mac_ver == 0x3593) rxwisize += sizeof(uint32_t); if (__predict_false(status != USBD_NORMAL_COMPLETION)) { DPRINTF(("RX status=%s\n", usbd_errstr(status))); if (status == USBD_STALLED) usbd_clear_endpoint_stall_async(sc->rxq.pipeh); if (status != USBD_CANCELLED) goto skip; return; } usbd_get_xfer_status(xfer, NULL, NULL, &xferlen, NULL); if (__predict_false(xferlen < (int)(sizeof(uint32_t) + rxwisize + sizeof(struct rt2870_rxd)))) { DPRINTF(("xfer too short %d\n", xferlen)); goto skip; } /* HW can aggregate multiple 802.11 frames in a single USB xfer */ buf = data->buf; while (xferlen > 8) { dmalen = le32toh(*(uint32_t *)buf) & 0xffff; if (__predict_false((dmalen >= (uint32_t)-8) || dmalen == 0 || (dmalen & 3) != 0)) { DPRINTF(("bad DMA length %u (%x)\n", dmalen, dmalen)); break; } if (__predict_false(dmalen + 8 > (uint32_t)xferlen)) { DPRINTF(("bad DMA length %u > %d\n", dmalen + 8, xferlen)); break; } run_rx_frame(sc, buf + sizeof(uint32_t), dmalen); buf += dmalen + 8; xferlen -= dmalen + 8; } skip: /* setup a new transfer */ usbd_setup_xfer(xfer, data, data->buf, RUN_MAX_RXSZ, USBD_SHORT_XFER_OK, USBD_NO_TIMEOUT, run_rxeof); status = usbd_transfer(xfer); if (status != USBD_NORMAL_COMPLETION && status != USBD_IN_PROGRESS) device_printf(sc->sc_dev, "requeuing rx failed: %s\n", usbd_errstr(status)); } static void run_txeof(struct usbd_xfer *xfer, void *priv, usbd_status status) { struct run_tx_data *data = priv; struct run_softc *sc = data->sc; struct run_tx_ring *txq = &sc->txq[data->qid]; struct ifnet *ifp = &sc->sc_if; int s; s = splnet(); txq->queued--; sc->qfullmsk &= ~(1 << data->qid); if (__predict_false(status != USBD_NORMAL_COMPLETION)) { if (status == USBD_NOT_STARTED || status == USBD_CANCELLED) return; DPRINTF(("%s: usb error on tx: %s\n", device_xname(sc->sc_dev), usbd_errstr(status))); if (status == USBD_STALLED) usbd_clear_endpoint_stall_async(txq->pipeh); if_statinc(ifp, if_oerrors); splx(s); return; } sc->sc_tx_timer = 0; if_statinc(ifp, if_opackets); ifp->if_flags &= ~IFF_OACTIVE; run_start(ifp); splx(s); } static int run_tx(struct run_softc *sc, struct mbuf *m, struct ieee80211_node *ni) { struct ieee80211com *ic = &sc->sc_ic; struct run_node *rn = (void *)ni; struct ieee80211_frame *wh; #ifndef RUN_HWCRYPTO struct ieee80211_key *k; #endif struct run_tx_ring *ring; struct run_tx_data *data; struct rt2870_txd *txd; struct rt2860_txwi *txwi; uint16_t qos, dur, mcs; uint16_t txwisize; uint8_t type, tid, qid; int hasqos, ridx, ctl_ridx, xferlen; uint8_t pad; usbd_status status; wh = mtod(m, struct ieee80211_frame *); #ifndef RUN_HWCRYPTO if (wh->i_fc[1] & IEEE80211_FC1_WEP) { k = ieee80211_crypto_encap(ic, ni, m); if (k == NULL) { m_freem(m); return ENOBUFS; } /* packet header may have moved, reset our local pointer */ wh = mtod(m, struct ieee80211_frame *); } #endif type = wh->i_fc[0] & IEEE80211_FC0_TYPE_MASK; if ((hasqos = ieee80211_has_qos(wh))) { qos = ((struct ieee80211_qosframe *)wh)->i_qos[0]; tid = qos & IEEE80211_QOS_TID; qid = TID_TO_WME_AC(tid); } else { qos = 0; tid = 0; qid = WME_AC_BE; } ring = &sc->txq[qid]; data = &ring->data[ring->cur]; /* pickup a rate index */ if (IEEE80211_IS_MULTICAST(wh->i_addr1) || type != IEEE80211_FC0_TYPE_DATA) { ridx = (ic->ic_curmode == IEEE80211_MODE_11A) ? RT2860_RIDX_OFDM6 : RT2860_RIDX_CCK1; ctl_ridx = rt2860_rates[ridx].ctl_ridx; } else if (ic->ic_fixed_rate != IEEE80211_FIXED_RATE_NONE) { ridx = sc->fixed_ridx; ctl_ridx = rt2860_rates[ridx].ctl_ridx; } else { ridx = rn->ridx[ni->ni_txrate]; ctl_ridx = rn->ctl_ridx[ni->ni_txrate]; } /* get MCS code from rate index */ mcs = rt2860_rates[ridx].mcs; txwisize = sizeof(struct rt2860_txwi); if (sc->mac_ver == 0x5592) txwisize += sizeof(uint32_t); xferlen = txwisize + m->m_pkthdr.len; /* roundup to 32-bit alignment */ xferlen = (xferlen + 3) & ~3; txd = (struct rt2870_txd *)data->buf; txd->flags = RT2860_TX_QSEL_EDCA; txd->len = htole16(xferlen); /* * Ether both are true or both are false, the header * are nicely aligned to 32-bit. So, no L2 padding. */ if (IEEE80211_HAS_ADDR4(wh) == IEEE80211_QOS_HAS_SEQ(wh)) pad = 0; else pad = 2; /* setup TX Wireless Information */ txwi = (struct rt2860_txwi *)(txd + 1); txwi->flags = 0; txwi->xflags = hasqos ? 0 : RT2860_TX_NSEQ; txwi->wcid = (type == IEEE80211_FC0_TYPE_DATA) ? RUN_AID2WCID(ni->ni_associd) : 0xff; txwi->len = htole16(m->m_pkthdr.len - pad); if (rt2860_rates[ridx].phy == IEEE80211_T_DS) { txwi->phy = htole16(RT2860_PHY_CCK); if (ridx != RT2860_RIDX_CCK1 && (ic->ic_flags & IEEE80211_F_SHPREAMBLE)) mcs |= RT2860_PHY_SHPRE; } else mcs |= RT2860_PHY_OFDM; txwi->phy = htole16(mcs); txwi->txop = RT2860_TX_TXOP_BACKOFF; if (!IEEE80211_IS_MULTICAST(wh->i_addr1) && (!hasqos || (qos & IEEE80211_QOS_ACKPOLICY) != IEEE80211_QOS_ACKPOLICY_NOACK)) { txwi->xflags |= RT2860_TX_ACK; if (ic->ic_flags & IEEE80211_F_SHPREAMBLE) dur = rt2860_rates[ctl_ridx].sp_ack_dur; else dur = rt2860_rates[ctl_ridx].lp_ack_dur; *(uint16_t *)wh->i_dur = htole16(dur); } #ifndef IEEE80211_STA_ONLY /* ask MAC to insert timestamp into probe responses */ if ((wh->i_fc[0] & (IEEE80211_FC0_TYPE_MASK | IEEE80211_FC0_SUBTYPE_MASK)) == (IEEE80211_FC0_TYPE_MGT | IEEE80211_FC0_SUBTYPE_PROBE_RESP)) /* NOTE: beacons do not pass through tx_data() */ txwi->flags |= RT2860_TX_TS; #endif if (__predict_false(sc->sc_drvbpf != NULL)) { struct run_tx_radiotap_header *tap = &sc->sc_txtap; tap->wt_flags = 0; tap->wt_rate = rt2860_rates[ridx].rate; tap->wt_chan_freq = htole16(ic->ic_curchan->ic_freq); tap->wt_chan_flags = htole16(ic->ic_curchan->ic_flags); tap->wt_hwqueue = qid; if (mcs & RT2860_PHY_SHPRE) tap->wt_flags |= IEEE80211_RADIOTAP_F_SHORTPRE; bpf_mtap2(sc->sc_drvbpf, tap, sc->sc_txtap_len, m, BPF_D_OUT); } m_copydata(m, 0, m->m_pkthdr.len, ((uint8_t *)txwi) + txwisize); m_freem(m); xferlen += sizeof(*txd) + 4; usbd_setup_xfer(data->xfer, data, data->buf, xferlen, USBD_FORCE_SHORT_XFER, RUN_TX_TIMEOUT, run_txeof); status = usbd_transfer(data->xfer); if (__predict_false(status != USBD_IN_PROGRESS && status != USBD_NORMAL_COMPLETION)) { device_printf(sc->sc_dev, "queuing tx failed: %s\n", usbd_errstr(status)); return EIO; } ieee80211_free_node(ni); ring->cur = (ring->cur + 1) % RUN_TX_RING_COUNT; if (++ring->queued >= RUN_TX_RING_COUNT) sc->qfullmsk |= 1 << qid; return 0; } static void run_start(struct ifnet *ifp) { struct run_softc *sc = ifp->if_softc; struct ieee80211com *ic = &sc->sc_ic; struct ether_header *eh; struct ieee80211_node *ni; struct mbuf *m; if ((ifp->if_flags & (IFF_RUNNING | IFF_OACTIVE)) != IFF_RUNNING) return; for (;;) { if (sc->qfullmsk != 0) { ifp->if_flags |= IFF_OACTIVE; break; } /* send pending management frames first */ IF_DEQUEUE(&ic->ic_mgtq, m); if (m != NULL) { ni = M_GETCTX(m, struct ieee80211_node *); M_CLEARCTX(m); goto sendit; } if (ic->ic_state != IEEE80211_S_RUN) break; /* encapsulate and send data frames */ IFQ_DEQUEUE(&ifp->if_snd, m); if (m == NULL) break; if (m->m_len < (int)sizeof(*eh) && (m = m_pullup(m, sizeof(*eh))) == NULL) { if_statinc(ifp, if_oerrors); continue; } eh = mtod(m, struct ether_header *); ni = ieee80211_find_txnode(ic, eh->ether_dhost); if (ni == NULL) { m_freem(m); if_statinc(ifp, if_oerrors); continue; } bpf_mtap(ifp, m, BPF_D_OUT); if ((m = ieee80211_encap(ic, m, ni)) == NULL) { ieee80211_free_node(ni); if_statinc(ifp, if_oerrors); continue; } sendit: bpf_mtap3(ic->ic_rawbpf, m, BPF_D_OUT); if (run_tx(sc, m, ni) != 0) { ieee80211_free_node(ni); if_statinc(ifp, if_oerrors); continue; } sc->sc_tx_timer = 5; ifp->if_timer = 1; } } static void run_watchdog(struct ifnet *ifp) { struct run_softc *sc = ifp->if_softc; struct ieee80211com *ic = &sc->sc_ic; ifp->if_timer = 0; if (sc->sc_tx_timer > 0) { if (--sc->sc_tx_timer == 0) { device_printf(sc->sc_dev, "device timeout\n"); /* run_init(ifp); XXX needs a process context! */ if_statinc(ifp, if_oerrors); return; } ifp->if_timer = 1; } ieee80211_watchdog(ic); } static int run_ioctl(struct ifnet *ifp, u_long cmd, void *data) { struct run_softc *sc = ifp->if_softc; struct ieee80211com *ic = &sc->sc_ic; int s, error = 0; s = splnet(); switch (cmd) { case SIOCSIFFLAGS: if ((error = ifioctl_common(ifp, cmd, data)) != 0) break; switch (ifp->if_flags & (IFF_UP|IFF_RUNNING)) { case IFF_UP|IFF_RUNNING: break; case IFF_UP: run_init(ifp); break; case IFF_RUNNING: run_stop(ifp, 1); break; case 0: break; } break; case SIOCADDMULTI: case SIOCDELMULTI: if ((error = ether_ioctl(ifp, cmd, data)) == ENETRESET) { /* setup multicast filter, etc */ error = 0; } break; default: error = ieee80211_ioctl(ic, cmd, data); break; } if (error == ENETRESET) { if ((ifp->if_flags & (IFF_UP | IFF_RUNNING)) == (IFF_UP | IFF_RUNNING)) { run_init(ifp); } error = 0; } splx(s); return error; } static void run_select_chan_group(struct run_softc *sc, int group) { uint32_t tmp; uint8_t agc; run_bbp_write(sc, 62, 0x37 - sc->lna[group]); run_bbp_write(sc, 63, 0x37 - sc->lna[group]); run_bbp_write(sc, 64, 0x37 - sc->lna[group]); if (sc->mac_ver < 0x3572) run_bbp_write(sc, 86, 0x00); if (sc->mac_ver == 0x3593) { run_bbp_write(sc, 77, 0x98); run_bbp_write(sc, 83, (group == 0) ? 0x8a : 0x9a); } if (group == 0) { if (sc->ext_2ghz_lna) { if (sc->mac_ver >= 0x5390) run_bbp_write(sc, 75, 0x52); else { run_bbp_write(sc, 82, 0x62); run_bbp_write(sc, 75, 0x46); } } else { if (sc->mac_ver == 0x5592) { run_bbp_write(sc, 79, 0x1c); run_bbp_write(sc, 80, 0x0e); run_bbp_write(sc, 81, 0x3a); run_bbp_write(sc, 82, 0x62); run_bbp_write(sc, 195, 0x80); run_bbp_write(sc, 196, 0xe0); run_bbp_write(sc, 195, 0x81); run_bbp_write(sc, 196, 0x1f); run_bbp_write(sc, 195, 0x82); run_bbp_write(sc, 196, 0x38); run_bbp_write(sc, 195, 0x83); run_bbp_write(sc, 196, 0x32); run_bbp_write(sc, 195, 0x85); run_bbp_write(sc, 196, 0x28); run_bbp_write(sc, 195, 0x86); run_bbp_write(sc, 196, 0x19); } else if (sc->mac_ver >= 0x5390) { run_bbp_write(sc, 75, 0x50); } else { run_bbp_write(sc, 82, (sc->mac_ver == 0x3593) ? 0x62 : 0x84); run_bbp_write(sc, 75, 0x50); } } } else { if (sc->mac_ver == 0x5592) { run_bbp_write(sc, 79, 0x18); run_bbp_write(sc, 80, 0x08); run_bbp_write(sc, 81, 0x38); run_bbp_write(sc, 82, 0x92); run_bbp_write(sc, 195, 0x80); run_bbp_write(sc, 196, 0xf0); run_bbp_write(sc, 195, 0x81); run_bbp_write(sc, 196, 0x1e); run_bbp_write(sc, 195, 0x82); run_bbp_write(sc, 196, 0x28); run_bbp_write(sc, 195, 0x83); run_bbp_write(sc, 196, 0x20); run_bbp_write(sc, 195, 0x85); run_bbp_write(sc, 196, 0x7f); run_bbp_write(sc, 195, 0x86); run_bbp_write(sc, 196, 0x7f); } else if (sc->mac_ver == 0x3572) run_bbp_write(sc, 82, 0x94); else run_bbp_write(sc, 82, (sc->mac_ver == 0x3593) ? 0x82 : 0xf2); if (sc->ext_5ghz_lna) run_bbp_write(sc, 75, 0x46); else run_bbp_write(sc, 75, 0x50); } run_read(sc, RT2860_TX_BAND_CFG, &tmp); tmp &= ~(RT2860_5G_BAND_SEL_N | RT2860_5G_BAND_SEL_P); tmp |= (group == 0) ? RT2860_5G_BAND_SEL_N : RT2860_5G_BAND_SEL_P; run_write(sc, RT2860_TX_BAND_CFG, tmp); /* enable appropriate Power Amplifiers and Low Noise Amplifiers */ tmp = RT2860_RFTR_EN | RT2860_TRSW_EN | RT2860_LNA_PE0_EN; if (sc->mac_ver == 0x3593) tmp |= RT3593_LNA_PE_G2_EN | RT3593_LNA_PE_A2_EN; if (sc->nrxchains > 1) tmp |= RT2860_LNA_PE1_EN; if (group == 0) { /* 2GHz */ tmp |= RT2860_PA_PE_G0_EN; if (sc->ntxchains > 1) tmp |= RT2860_PA_PE_G1_EN; } else { /* 5GHz */ tmp |= RT2860_PA_PE_A0_EN; if (sc->ntxchains > 1) tmp |= RT2860_PA_PE_A1_EN; if (sc->mac_ver == 0x3593) { if (sc->ntxchains > 2) tmp |= RT3593_PA_PE_G2_EN; } } if (sc->mac_ver == 0x3572) { run_rt3070_rf_write(sc, 8, 0x00); run_write(sc, RT2860_TX_PIN_CFG, tmp); run_rt3070_rf_write(sc, 8, 0x80); } else run_write(sc, RT2860_TX_PIN_CFG, tmp); if (sc->mac_ver == 0x5592) { run_bbp_write(sc, 195, 0x8d); run_bbp_write(sc, 196, 0x1a); } if (sc->mac_ver == 0x3593) { run_read(sc, RT2860_GPIO_CTRL, &tmp); tmp &= ~0x01010000; if (group == 0) tmp |= 0x00010000; tmp = (tmp & ~0x00009090) | 0x00000090; run_write(sc, RT2860_GPIO_CTRL, tmp); } /* set initial AGC value */ if (group == 0) { /* 2GHz band */ if (sc->mac_ver >= 0x3070) agc = 0x1c + sc->lna[0] * 2; else agc = 0x2e + sc->lna[0]; } else { /* 5GHz band */ if (sc->mac_ver == 0x3572) agc = 0x22 + (sc->lna[group] * 5) / 3; else agc = 0x32 + (sc->lna[group] * 5) / 3; } run_set_agc(sc, agc); } static void run_rt2870_set_chan(struct run_softc *sc, u_int chan) { const struct rfprog *rfprog = rt2860_rf2850; uint32_t r2, r3, r4; int8_t txpow1, txpow2; int i; /* find the settings for this channel (we know it exists) */ for (i = 0; rfprog[i].chan != chan; i++); r2 = rfprog[i].r2; if (sc->ntxchains == 1) r2 |= 1 << 12; /* 1T: disable Tx chain 2 */ if (sc->nrxchains == 1) r2 |= 1 << 15 | 1 << 4; /* 1R: disable Rx chains 2 & 3 */ else if (sc->nrxchains == 2) r2 |= 1 << 4; /* 2R: disable Rx chain 3 */ /* use Tx power values from EEPROM */ txpow1 = sc->txpow1[i]; txpow2 = sc->txpow2[i]; if (chan > 14) { if (txpow1 >= 0) txpow1 = txpow1 << 1 | 1; else txpow1 = (7 + txpow1) << 1; if (txpow2 >= 0) txpow2 = txpow2 << 1 | 1; else txpow2 = (7 + txpow2) << 1; } r3 = rfprog[i].r3 | txpow1 << 7; r4 = rfprog[i].r4 | sc->freq << 13 | txpow2 << 4; run_rt2870_rf_write(sc, RT2860_RF1, rfprog[i].r1); run_rt2870_rf_write(sc, RT2860_RF2, r2); run_rt2870_rf_write(sc, RT2860_RF3, r3); run_rt2870_rf_write(sc, RT2860_RF4, r4); usbd_delay_ms(sc->sc_udev, 10); run_rt2870_rf_write(sc, RT2860_RF1, rfprog[i].r1); run_rt2870_rf_write(sc, RT2860_RF2, r2); run_rt2870_rf_write(sc, RT2860_RF3, r3 | 1); run_rt2870_rf_write(sc, RT2860_RF4, r4); usbd_delay_ms(sc->sc_udev, 10); run_rt2870_rf_write(sc, RT2860_RF1, rfprog[i].r1); run_rt2870_rf_write(sc, RT2860_RF2, r2); run_rt2870_rf_write(sc, RT2860_RF3, r3); run_rt2870_rf_write(sc, RT2860_RF4, r4); } static void run_rt3070_set_chan(struct run_softc *sc, u_int chan) { int8_t txpow1, txpow2; uint8_t rf; int i; KASSERT(chan >= 1 && chan <= 14); /* RT3070 is 2GHz only */ /* find the settings for this channel (we know it exists) */ for (i = 0; rt2860_rf2850[i].chan != chan; i++) continue; /* use Tx power values from EEPROM */ txpow1 = sc->txpow1[i]; txpow2 = sc->txpow2[i]; run_rt3070_rf_write(sc, 2, rt3070_freqs[i].n); run_rt3070_rf_write(sc, 3, rt3070_freqs[i].k); run_rt3070_rf_read(sc, 6, &rf); rf = (rf & ~0x03) | rt3070_freqs[i].r; run_rt3070_rf_write(sc, 6, rf); /* set Tx0 power */ run_rt3070_rf_read(sc, 12, &rf); rf = (rf & ~0x1f) | txpow1; run_rt3070_rf_write(sc, 12, rf); /* set Tx1 power */ run_rt3070_rf_read(sc, 13, &rf); rf = (rf & ~0x1f) | txpow2; run_rt3070_rf_write(sc, 13, rf); run_rt3070_rf_read(sc, 1, &rf); rf &= ~0xfc; if (sc->ntxchains == 1) rf |= 1 << 7 | 1 << 5; /* 1T: disable Tx chains 2 & 3 */ else if (sc->ntxchains == 2) rf |= 1 << 7; /* 2T: disable Tx chain 3 */ if (sc->nrxchains == 1) rf |= 1 << 6 | 1 << 4; /* 1R: disable Rx chains 2 & 3 */ else if (sc->nrxchains == 2) rf |= 1 << 6; /* 2R: disable Rx chain 3 */ run_rt3070_rf_write(sc, 1, rf); /* set RF offset */ run_rt3070_rf_read(sc, 23, &rf); rf = (rf & ~0x7f) | sc->freq; run_rt3070_rf_write(sc, 23, rf); /* program RF filter */ run_rt3070_rf_read(sc, 24, &rf); /* Tx */ rf = (rf & ~0x3f) | sc->rf24_20mhz; run_rt3070_rf_write(sc, 24, rf); run_rt3070_rf_read(sc, 31, &rf); /* Rx */ rf = (rf & ~0x3f) | sc->rf24_20mhz; run_rt3070_rf_write(sc, 31, rf); /* enable RF tuning */ run_rt3070_rf_read(sc, 7, &rf); run_rt3070_rf_write(sc, 7, rf | 0x01); } static void run_rt3572_set_chan(struct run_softc *sc, u_int chan) { int8_t txpow1, txpow2; uint32_t tmp; uint8_t rf; int i; /* find the settings for this channel (we know it exists) */ for (i = 0; rt2860_rf2850[i].chan != chan; i++); /* use Tx power values from EEPROM */ txpow1 = sc->txpow1[i]; txpow2 = sc->txpow2[i]; if (chan <= 14) { run_bbp_write(sc, 25, sc->bbp25); run_bbp_write(sc, 26, sc->bbp26); } else { /* enable IQ phase correction */ run_bbp_write(sc, 25, 0x09); run_bbp_write(sc, 26, 0xff); } run_rt3070_rf_write(sc, 2, rt3070_freqs[i].n); run_rt3070_rf_write(sc, 3, rt3070_freqs[i].k); run_rt3070_rf_read(sc, 6, &rf); rf = (rf & ~0x0f) | rt3070_freqs[i].r; rf |= (chan <= 14) ? 0x08 : 0x04; run_rt3070_rf_write(sc, 6, rf); /* set PLL mode */ run_rt3070_rf_read(sc, 5, &rf); rf &= ~(0x08 | 0x04); rf |= (chan <= 14) ? 0x04 : 0x08; run_rt3070_rf_write(sc, 5, rf); /* set Tx power for chain 0 */ if (chan <= 14) rf = 0x60 | txpow1; else rf = 0xe0 | (txpow1 & 0xc) << 1 | (txpow1 & 0x3); run_rt3070_rf_write(sc, 12, rf); /* set Tx power for chain 1 */ if (chan <= 14) rf = 0x60 | txpow2; else rf = 0xe0 | (txpow2 & 0xc) << 1 | (txpow2 & 0x3); run_rt3070_rf_write(sc, 13, rf); /* set Tx/Rx streams */ run_rt3070_rf_read(sc, 1, &rf); rf &= ~0xfc; if (sc->ntxchains == 1) rf |= 1 << 7 | 1 << 5; /* 1T: disable Tx chains 2 & 3 */ else if (sc->ntxchains == 2) rf |= 1 << 7; /* 2T: disable Tx chain 3 */ if (sc->nrxchains == 1) rf |= 1 << 6 | 1 << 4; /* 1R: disable Rx chains 2 & 3 */ else if (sc->nrxchains == 2) rf |= 1 << 6; /* 2R: disable Rx chain 3 */ run_rt3070_rf_write(sc, 1, rf); /* set RF offset */ run_rt3070_rf_read(sc, 23, &rf); rf = (rf & ~0x7f) | sc->freq; run_rt3070_rf_write(sc, 23, rf); /* program RF filter */ rf = sc->rf24_20mhz; run_rt3070_rf_write(sc, 24, rf); /* Tx */ run_rt3070_rf_write(sc, 31, rf); /* Rx */ /* enable RF tuning */ run_rt3070_rf_read(sc, 7, &rf); rf = (chan <= 14) ? 0xd8 : ((rf & ~0xc8) | 0x14); run_rt3070_rf_write(sc, 7, rf); /* TSSI */ rf = (chan <= 14) ? 0xc3 : 0xc0; run_rt3070_rf_write(sc, 9, rf); /* set loop filter 1 */ run_rt3070_rf_write(sc, 10, 0xf1); /* set loop filter 2 */ run_rt3070_rf_write(sc, 11, (chan <= 14) ? 0xb9 : 0x00); /* set tx_mx2_ic */ run_rt3070_rf_write(sc, 15, (chan <= 14) ? 0x53 : 0x43); /* set tx_mx1_ic */ if (chan <= 14) rf = 0x48 | sc->txmixgain_2ghz; else rf = 0x78 | sc->txmixgain_5ghz; run_rt3070_rf_write(sc, 16, rf); /* set tx_lo1 */ run_rt3070_rf_write(sc, 17, 0x23); /* set tx_lo2 */ if (chan <= 14) rf = 0x93; else if (chan <= 64) rf = 0xb7; else if (chan <= 128) rf = 0x74; else rf = 0x72; run_rt3070_rf_write(sc, 19, rf); /* set rx_lo1 */ if (chan <= 14) rf = 0xb3; else if (chan <= 64) rf = 0xf6; else if (chan <= 128) rf = 0xf4; else rf = 0xf3; run_rt3070_rf_write(sc, 20, rf); /* set pfd_delay */ if (chan <= 14) rf = 0x15; else if (chan <= 64) rf = 0x3d; else rf = 0x01; run_rt3070_rf_write(sc, 25, rf); /* set rx_lo2 */ run_rt3070_rf_write(sc, 26, (chan <= 14) ? 0x85 : 0x87); /* set ldo_rf_vc */ run_rt3070_rf_write(sc, 27, (chan <= 14) ? 0x00 : 0x01); /* set drv_cc */ run_rt3070_rf_write(sc, 29, (chan <= 14) ? 0x9b : 0x9f); run_read(sc, RT2860_GPIO_CTRL, &tmp); tmp &= ~0x8080; if (chan <= 14) tmp |= 0x80; run_write(sc, RT2860_GPIO_CTRL, tmp); /* enable RF tuning */ run_rt3070_rf_read(sc, 7, &rf); run_rt3070_rf_write(sc, 7, rf | 0x01); usbd_delay_ms(sc->sc_udev, 2); } static void run_rt3593_set_chan(struct run_softc *sc, u_int chan) { int8_t txpow1, txpow2, txpow3; uint8_t h20mhz, rf; int i; /* find the settings for this channel (we know it exists) */ for (i = 0; rt2860_rf2850[i].chan != chan; i++); /* use Tx power values from EEPROM */ txpow1 = sc->txpow1[i]; txpow2 = sc->txpow2[i]; txpow3 = (sc->ntxchains == 3) ? sc->txpow3[i] : 0; if (chan <= 14) { run_bbp_write(sc, 25, sc->bbp25); run_bbp_write(sc, 26, sc->bbp26); } else { /* Enable IQ phase correction. */ run_bbp_write(sc, 25, 0x09); run_bbp_write(sc, 26, 0xff); } run_rt3070_rf_write(sc, 8, rt3070_freqs[i].n); run_rt3070_rf_write(sc, 9, rt3070_freqs[i].k & 0x0f); run_rt3070_rf_read(sc, 11, &rf); rf = (rf & ~0x03) | (rt3070_freqs[i].r & 0x03); run_rt3070_rf_write(sc, 11, rf); /* Set pll_idoh. */ run_rt3070_rf_read(sc, 11, &rf); rf &= ~0x4c; rf |= (chan <= 14) ? 0x44 : 0x48; run_rt3070_rf_write(sc, 11, rf); if (chan <= 14) rf = txpow1 & 0x1f; else rf = 0x40 | ((txpow1 & 0x18) << 1) | (txpow1 & 0x07); run_rt3070_rf_write(sc, 53, rf); if (chan <= 14) rf = txpow2 & 0x1f; else rf = 0x40 | ((txpow2 & 0x18) << 1) | (txpow2 & 0x07); run_rt3070_rf_write(sc, 55, rf); if (chan <= 14) rf = txpow3 & 0x1f; else rf = 0x40 | ((txpow3 & 0x18) << 1) | (txpow3 & 0x07); run_rt3070_rf_write(sc, 54, rf); rf = RT3070_RF_BLOCK | RT3070_PLL_PD; if (sc->ntxchains == 3) rf |= RT3070_TX0_PD | RT3070_TX1_PD | RT3070_TX2_PD; else rf |= RT3070_TX0_PD | RT3070_TX1_PD; rf |= RT3070_RX0_PD | RT3070_RX1_PD | RT3070_RX2_PD; run_rt3070_rf_write(sc, 1, rf); run_adjust_freq_offset(sc); run_rt3070_rf_write(sc, 31, (chan <= 14) ? 0xa0 : 0x80); h20mhz = (sc->rf24_20mhz & 0x20) >> 5; run_rt3070_rf_read(sc, 30, &rf); rf = (rf & ~0x06) | (h20mhz << 1) | (h20mhz << 2); run_rt3070_rf_write(sc, 30, rf); run_rt3070_rf_read(sc, 36, &rf); if (chan <= 14) rf |= 0x80; else rf &= ~0x80; run_rt3070_rf_write(sc, 36, rf); /* Set vcolo_bs. */ run_rt3070_rf_write(sc, 34, (chan <= 14) ? 0x3c : 0x20); /* Set pfd_delay. */ run_rt3070_rf_write(sc, 12, (chan <= 14) ? 0x1a : 0x12); /* Set vco bias current control. */ run_rt3070_rf_read(sc, 6, &rf); rf &= ~0xc0; if (chan <= 14) rf |= 0x40; else if (chan <= 128) rf |= 0x80; else rf |= 0x40; run_rt3070_rf_write(sc, 6, rf); run_rt3070_rf_read(sc, 30, &rf); rf = (rf & ~0x18) | 0x10; run_rt3070_rf_write(sc, 30, rf); run_rt3070_rf_write(sc, 10, (chan <= 14) ? 0xd3 : 0xd8); run_rt3070_rf_write(sc, 13, (chan <= 14) ? 0x12 : 0x23); run_rt3070_rf_read(sc, 51, &rf); rf = (rf & ~0x03) | 0x01; run_rt3070_rf_write(sc, 51, rf); /* Set tx_mx1_cc. */ run_rt3070_rf_read(sc, 51, &rf); rf &= ~0x1c; rf |= (chan <= 14) ? 0x14 : 0x10; run_rt3070_rf_write(sc, 51, rf); /* Set tx_mx1_ic. */ run_rt3070_rf_read(sc, 51, &rf); rf &= ~0xe0; rf |= (chan <= 14) ? 0x60 : 0x40; run_rt3070_rf_write(sc, 51, rf); /* Set tx_lo1_ic. */ run_rt3070_rf_read(sc, 49, &rf); rf &= ~0x1c; rf |= (chan <= 14) ? 0x0c : 0x08; run_rt3070_rf_write(sc, 49, rf); /* Set tx_lo1_en. */ run_rt3070_rf_read(sc, 50, &rf); run_rt3070_rf_write(sc, 50, rf & ~0x20); /* Set drv_cc. */ run_rt3070_rf_read(sc, 57, &rf); rf &= ~0xfc; rf |= (chan <= 14) ? 0x6c : 0x3c; run_rt3070_rf_write(sc, 57, rf); /* Set rx_mix1_ic, rxa_lnactr, lna_vc, lna_inbias_en and lna_en. */ run_rt3070_rf_write(sc, 44, (chan <= 14) ? 0x93 : 0x9b); /* Set drv_gnd_a, tx_vga_cc_a and tx_mx2_gain. */ run_rt3070_rf_write(sc, 52, (chan <= 14) ? 0x45 : 0x05); /* Enable VCO calibration. */ run_rt3070_rf_read(sc, 3, &rf); rf &= ~RT5390_VCOCAL; rf |= (chan <= 14) ? RT5390_VCOCAL : 0xbe; run_rt3070_rf_write(sc, 3, rf); if (chan <= 14) rf = 0x23; else if (chan <= 64) rf = 0x36; else if (chan <= 128) rf = 0x32; else rf = 0x30; run_rt3070_rf_write(sc, 39, rf); if (chan <= 14) rf = 0xbb; else if (chan <= 64) rf = 0xeb; else if (chan <= 128) rf = 0xb3; else rf = 0x9b; run_rt3070_rf_write(sc, 45, rf); /* Set FEQ/AEQ control. */ run_bbp_write(sc, 105, 0x34); } static void run_rt5390_set_chan(struct run_softc *sc, u_int chan) { int8_t txpow1, txpow2; uint8_t rf; int i; /* find the settings for this channel (we know it exists) */ for (i = 0; rt2860_rf2850[i].chan != chan; i++); /* use Tx power values from EEPROM */ txpow1 = sc->txpow1[i]; txpow2 = sc->txpow2[i]; run_rt3070_rf_write(sc, 8, rt3070_freqs[i].n); run_rt3070_rf_write(sc, 9, rt3070_freqs[i].k & 0x0f); run_rt3070_rf_read(sc, 11, &rf); rf = (rf & ~0x03) | (rt3070_freqs[i].r & 0x03); run_rt3070_rf_write(sc, 11, rf); run_rt3070_rf_read(sc, 49, &rf); rf = (rf & ~0x3f) | (txpow1 & 0x3f); /* The valid range of the RF R49 is 0x00 to 0x27. */ if ((rf & 0x3f) > 0x27) rf = (rf & ~0x3f) | 0x27; run_rt3070_rf_write(sc, 49, rf); if (sc->mac_ver == 0x5392) { run_rt3070_rf_read(sc, 50, &rf); rf = (rf & ~0x3f) | (txpow2 & 0x3f); /* The valid range of the RF R50 is 0x00 to 0x27. */ if ((rf & 0x3f) > 0x27) rf = (rf & ~0x3f) | 0x27; run_rt3070_rf_write(sc, 50, rf); } run_rt3070_rf_read(sc, 1, &rf); rf |= RT3070_RF_BLOCK | RT3070_PLL_PD | RT3070_RX0_PD | RT3070_TX0_PD; if (sc->mac_ver == 0x5392) rf |= RT3070_RX1_PD | RT3070_TX1_PD; run_rt3070_rf_write(sc, 1, rf); if (sc->mac_ver != 0x5392) { run_rt3070_rf_read(sc, 2, &rf); rf |= 0x80; run_rt3070_rf_write(sc, 2, rf); usbd_delay_ms(sc->sc_udev, 10); rf &= 0x7f; run_rt3070_rf_write(sc, 2, rf); } run_adjust_freq_offset(sc); if (sc->mac_ver == 0x5392) { /* Fix for RT5392C. */ if (sc->mac_rev >= 0x0223) { if (chan <= 4) rf = 0x0f; else if (chan >= 5 && chan <= 7) rf = 0x0e; else rf = 0x0d; run_rt3070_rf_write(sc, 23, rf); if (chan <= 4) rf = 0x0c; else if (chan == 5) rf = 0x0b; else if (chan >= 6 && chan <= 7) rf = 0x0a; else if (chan >= 8 && chan <= 10) rf = 0x09; else rf = 0x08; run_rt3070_rf_write(sc, 59, rf); } else { if (chan <= 11) rf = 0x0f; else rf = 0x0b; run_rt3070_rf_write(sc, 59, rf); } } else { /* Fix for RT5390F. */ if (sc->mac_rev >= 0x0502) { if (chan <= 11) rf = 0x43; else rf = 0x23; run_rt3070_rf_write(sc, 55, rf); if (chan <= 11) rf = 0x0f; else if (chan == 12) rf = 0x0d; else rf = 0x0b; run_rt3070_rf_write(sc, 59, rf); } else { run_rt3070_rf_write(sc, 55, 0x44); run_rt3070_rf_write(sc, 59, 0x8f); } } /* Enable VCO calibration. */ run_rt3070_rf_read(sc, 3, &rf); rf |= RT5390_VCOCAL; run_rt3070_rf_write(sc, 3, rf); } static void run_rt5592_set_chan(struct run_softc *sc, u_int chan) { const struct rt5592_freqs *freqs; uint32_t tmp; uint8_t reg, rf, txpow_bound; int8_t txpow1, txpow2; int i; run_read(sc, RT5592_DEBUG_INDEX, &tmp); freqs = (tmp & RT5592_SEL_XTAL) ? rt5592_freqs_40mhz : rt5592_freqs_20mhz; /* find the settings for this channel (we know it exists) */ for (i = 0; rt2860_rf2850[i].chan != chan; i++, freqs++); /* use Tx power values from EEPROM */ txpow1 = sc->txpow1[i]; txpow2 = sc->txpow2[i]; run_read(sc, RT3070_LDO_CFG0, &tmp); tmp &= ~0x1c000000; if (chan > 14) tmp |= 0x14000000; run_write(sc, RT3070_LDO_CFG0, tmp); /* N setting. */ run_rt3070_rf_write(sc, 8, freqs->n & 0xff); run_rt3070_rf_read(sc, 9, &rf); rf &= ~(1 << 4); rf |= ((freqs->n & 0x0100) >> 8) << 4; run_rt3070_rf_write(sc, 9, rf); /* K setting. */ run_rt3070_rf_read(sc, 9, &rf); rf &= ~0x0f; rf |= (freqs->k & 0x0f); run_rt3070_rf_write(sc, 9, rf); /* Mode setting. */ run_rt3070_rf_read(sc, 11, &rf); rf &= ~0x0c; rf |= ((freqs->m - 0x8) & 0x3) << 2; run_rt3070_rf_write(sc, 11, rf); run_rt3070_rf_read(sc, 9, &rf); rf &= ~(1 << 7); rf |= (((freqs->m - 0x8) & 0x4) >> 2) << 7; run_rt3070_rf_write(sc, 9, rf); /* R setting. */ run_rt3070_rf_read(sc, 11, &rf); rf &= ~0x03; rf |= (freqs->r - 0x1); run_rt3070_rf_write(sc, 11, rf); if (chan <= 14) { /* Initialize RF registers for 2GHZ. */ for (i = 0; i < (int)__arraycount(rt5592_2ghz_def_rf); i++) { run_rt3070_rf_write(sc, rt5592_2ghz_def_rf[i].reg, rt5592_2ghz_def_rf[i].val); } rf = (chan <= 10) ? 0x07 : 0x06; run_rt3070_rf_write(sc, 23, rf); run_rt3070_rf_write(sc, 59, rf); run_rt3070_rf_write(sc, 55, 0x43); /* * RF R49/R50 Tx power ALC code. * G-band bit<7:6>=1:0, bit<5:0> range from 0x0 ~ 0x27. */ reg = 2; txpow_bound = 0x27; } else { /* Initialize RF registers for 5GHZ. */ for (i = 0; i < (int)__arraycount(rt5592_5ghz_def_rf); i++) { run_rt3070_rf_write(sc, rt5592_5ghz_def_rf[i].reg, rt5592_5ghz_def_rf[i].val); } for (i = 0; i < (int)__arraycount(rt5592_chan_5ghz); i++) { if (chan >= rt5592_chan_5ghz[i].firstchan && chan <= rt5592_chan_5ghz[i].lastchan) { run_rt3070_rf_write(sc, rt5592_chan_5ghz[i].reg, rt5592_chan_5ghz[i].val); } } /* * RF R49/R50 Tx power ALC code. * A-band bit<7:6>=1:1, bit<5:0> range from 0x0 ~ 0x2b. */ reg = 3; txpow_bound = 0x2b; } /* RF R49 ch0 Tx power ALC code. */ run_rt3070_rf_read(sc, 49, &rf); rf &= ~0xc0; rf |= (reg << 6); rf = (rf & ~0x3f) | (txpow1 & 0x3f); if ((rf & 0x3f) > txpow_bound) rf = (rf & ~0x3f) | txpow_bound; run_rt3070_rf_write(sc, 49, rf); /* RF R50 ch1 Tx power ALC code. */ run_rt3070_rf_read(sc, 50, &rf); rf &= ~(1 << 7 | 1 << 6); rf |= (reg << 6); rf = (rf & ~0x3f) | (txpow2 & 0x3f); if ((rf & 0x3f) > txpow_bound) rf = (rf & ~0x3f) | txpow_bound; run_rt3070_rf_write(sc, 50, rf); /* Enable RF_BLOCK, PLL_PD, RX0_PD, and TX0_PD. */ run_rt3070_rf_read(sc, 1, &rf); rf |= (RT3070_RF_BLOCK | RT3070_PLL_PD | RT3070_RX0_PD | RT3070_TX0_PD); if (sc->ntxchains > 1) rf |= RT3070_TX1_PD; if (sc->nrxchains > 1) rf |= RT3070_RX1_PD; run_rt3070_rf_write(sc, 1, rf); run_rt3070_rf_write(sc, 6, 0xe4); run_rt3070_rf_write(sc, 30, 0x10); run_rt3070_rf_write(sc, 31, 0x80); run_rt3070_rf_write(sc, 32, 0x80); run_adjust_freq_offset(sc); /* Enable VCO calibration. */ run_rt3070_rf_read(sc, 3, &rf); rf |= RT5390_VCOCAL; run_rt3070_rf_write(sc, 3, rf); } static void run_iq_calib(struct run_softc *sc, u_int chan) { uint16_t val; /* Tx0 IQ gain. */ run_bbp_write(sc, 158, 0x2c); if (chan <= 14) run_efuse_read(sc, RT5390_EEPROM_IQ_GAIN_CAL_TX0_2GHZ, &val, 1); else if (chan <= 64) { run_efuse_read(sc, RT5390_EEPROM_IQ_GAIN_CAL_TX0_CH36_TO_CH64_5GHZ, &val, 1); } else if (chan <= 138) { run_efuse_read(sc, RT5390_EEPROM_IQ_GAIN_CAL_TX0_CH100_TO_CH138_5GHZ, &val, 1); } else if (chan <= 165) { run_efuse_read(sc, RT5390_EEPROM_IQ_GAIN_CAL_TX0_CH140_TO_CH165_5GHZ, &val, 1); } else val = 0; run_bbp_write(sc, 159, val); /* Tx0 IQ phase. */ run_bbp_write(sc, 158, 0x2d); if (chan <= 14) { run_efuse_read(sc, RT5390_EEPROM_IQ_PHASE_CAL_TX0_2GHZ, &val, 1); } else if (chan <= 64) { run_efuse_read(sc, RT5390_EEPROM_IQ_PHASE_CAL_TX0_CH36_TO_CH64_5GHZ, &val, 1); } else if (chan <= 138) { run_efuse_read(sc, RT5390_EEPROM_IQ_PHASE_CAL_TX0_CH100_TO_CH138_5GHZ, &val, 1); } else if (chan <= 165) { run_efuse_read(sc, RT5390_EEPROM_IQ_PHASE_CAL_TX0_CH140_TO_CH165_5GHZ, &val, 1); } else val = 0; run_bbp_write(sc, 159, val); /* Tx1 IQ gain. */ run_bbp_write(sc, 158, 0x4a); if (chan <= 14) { run_efuse_read(sc, RT5390_EEPROM_IQ_GAIN_CAL_TX1_2GHZ, &val, 1); } else if (chan <= 64) { run_efuse_read(sc, RT5390_EEPROM_IQ_GAIN_CAL_TX1_CH36_TO_CH64_5GHZ, &val, 1); } else if (chan <= 138) { run_efuse_read(sc, RT5390_EEPROM_IQ_GAIN_CAL_TX1_CH100_TO_CH138_5GHZ, &val, 1); } else if (chan <= 165) { run_efuse_read(sc, RT5390_EEPROM_IQ_GAIN_CAL_TX1_CH140_TO_CH165_5GHZ, &val, 1); } else val = 0; run_bbp_write(sc, 159, val); /* Tx1 IQ phase. */ run_bbp_write(sc, 158, 0x4b); if (chan <= 14) { run_efuse_read(sc, RT5390_EEPROM_IQ_PHASE_CAL_TX1_2GHZ, &val, 1); } else if (chan <= 64) { run_efuse_read(sc, RT5390_EEPROM_IQ_PHASE_CAL_TX1_CH36_TO_CH64_5GHZ, &val, 1); } else if (chan <= 138) { run_efuse_read(sc, RT5390_EEPROM_IQ_PHASE_CAL_TX1_CH100_TO_CH138_5GHZ, &val, 1); } else if (chan <= 165) { run_efuse_read(sc, RT5390_EEPROM_IQ_PHASE_CAL_TX1_CH140_TO_CH165_5GHZ, &val, 1); } else val = 0; run_bbp_write(sc, 159, val); /* RF IQ compensation control. */ run_bbp_write(sc, 158, 0x04); run_efuse_read(sc, RT5390_EEPROM_RF_IQ_COMPENSATION_CTL, &val, 1); run_bbp_write(sc, 159, val); /* RF IQ imbalance compensation control. */ run_bbp_write(sc, 158, 0x03); run_efuse_read(sc, RT5390_EEPROM_RF_IQ_IMBALANCE_COMPENSATION_CTL, &val, 1); run_bbp_write(sc, 159, val); } static void run_set_agc(struct run_softc *sc, uint8_t agc) { uint8_t bbp; if (sc->mac_ver == 0x3572) { run_bbp_read(sc, 27, &bbp); bbp &= ~(0x3 << 5); run_bbp_write(sc, 27, bbp | 0 << 5); /* select Rx0 */ run_bbp_write(sc, 66, agc); run_bbp_write(sc, 27, bbp | 1 << 5); /* select Rx1 */ run_bbp_write(sc, 66, agc); } else run_bbp_write(sc, 66, agc); } static void run_set_rx_antenna(struct run_softc *sc, int aux) { uint32_t tmp; uint8_t bbp152; if (aux) { if (sc->rf_rev == RT5390_RF_5370) { run_bbp_read(sc, 152, &bbp152); bbp152 &= ~0x80; run_bbp_write(sc, 152, bbp152); } else { run_mcu_cmd(sc, RT2860_MCU_CMD_ANTSEL, 0); run_read(sc, RT2860_GPIO_CTRL, &tmp); tmp &= ~0x0808; tmp |= 0x08; run_write(sc, RT2860_GPIO_CTRL, tmp); } } else { if (sc->rf_rev == RT5390_RF_5370) { run_bbp_read(sc, 152, &bbp152); bbp152 |= 0x80; run_bbp_write(sc, 152, bbp152); } else { run_mcu_cmd(sc, RT2860_MCU_CMD_ANTSEL, !aux); run_read(sc, RT2860_GPIO_CTRL, &tmp); tmp &= ~0x0808; run_write(sc, RT2860_GPIO_CTRL, tmp); } } } static int run_set_chan(struct run_softc *sc, struct ieee80211_channel *c) { struct ieee80211com *ic = &sc->sc_ic; u_int chan, group; chan = ieee80211_chan2ieee(ic, c); if (chan == 0 || chan == IEEE80211_CHAN_ANY) return EINVAL; if (sc->mac_ver == 0x5592) run_rt5592_set_chan(sc, chan); else if (sc->mac_ver >= 0x5390) run_rt5390_set_chan(sc, chan); else if (sc->mac_ver == 0x3593) run_rt3593_set_chan(sc, chan); else if (sc->mac_ver == 0x3572) run_rt3572_set_chan(sc, chan); else if (sc->mac_ver >= 0x3070) run_rt3070_set_chan(sc, chan); else run_rt2870_set_chan(sc, chan); /* determine channel group */ if (chan <= 14) group = 0; else if (chan <= 64) group = 1; else if (chan <= 128) group = 2; else group = 3; /* XXX necessary only when group has changed! */ run_select_chan_group(sc, group); usbd_delay_ms(sc->sc_udev, 10); /* Perform IQ calibration. */ if (sc->mac_ver >= 0x5392) run_iq_calib(sc, chan); return 0; } static void run_updateprot(struct run_softc *sc) { struct ieee80211com *ic = &sc->sc_ic; uint32_t tmp; tmp = RT2860_RTSTH_EN | RT2860_PROT_NAV_SHORT | RT2860_TXOP_ALLOW_ALL; /* setup protection frame rate (MCS code) */ tmp |= (ic->ic_curmode == IEEE80211_MODE_11A) ? rt2860_rates[RT2860_RIDX_OFDM6].mcs | RT2860_PHY_OFDM : rt2860_rates[RT2860_RIDX_CCK11].mcs; /* CCK frames don't require protection */ run_write(sc, RT2860_CCK_PROT_CFG, tmp); if (ic->ic_flags & IEEE80211_F_USEPROT) { if (ic->ic_protmode == IEEE80211_PROT_RTSCTS) tmp |= RT2860_PROT_CTRL_RTS_CTS; else if (ic->ic_protmode == IEEE80211_PROT_CTSONLY) tmp |= RT2860_PROT_CTRL_CTS; } run_write(sc, RT2860_OFDM_PROT_CFG, tmp); } static void run_enable_tsf_sync(struct run_softc *sc) { struct ieee80211com *ic = &sc->sc_ic; uint32_t tmp; run_read(sc, RT2860_BCN_TIME_CFG, &tmp); tmp &= ~0x1fffff; tmp |= ic->ic_bss->ni_intval * 16; tmp |= RT2860_TSF_TIMER_EN | RT2860_TBTT_TIMER_EN; if (ic->ic_opmode == IEEE80211_M_STA) { /* * Local TSF is always updated with remote TSF on beacon * reception. */ tmp |= 1 << RT2860_TSF_SYNC_MODE_SHIFT; } #ifndef IEEE80211_STA_ONLY else if (ic->ic_opmode == IEEE80211_M_IBSS) { tmp |= RT2860_BCN_TX_EN; /* * Local TSF is updated with remote TSF on beacon reception * only if the remote TSF is greater than local TSF. */ tmp |= 2 << RT2860_TSF_SYNC_MODE_SHIFT; } else if (ic->ic_opmode == IEEE80211_M_HOSTAP) { tmp |= RT2860_BCN_TX_EN; /* SYNC with nobody */ tmp |= 3 << RT2860_TSF_SYNC_MODE_SHIFT; } #endif run_write(sc, RT2860_BCN_TIME_CFG, tmp); } static void run_enable_mrr(struct run_softc *sc) { #define CCK(mcs) (mcs) #define OFDM(mcs) (1 << 3 | (mcs)) run_write(sc, RT2860_LG_FBK_CFG0, OFDM(6) << 28 | /* 54->48 */ OFDM(5) << 24 | /* 48->36 */ OFDM(4) << 20 | /* 36->24 */ OFDM(3) << 16 | /* 24->18 */ OFDM(2) << 12 | /* 18->12 */ OFDM(1) << 8 | /* 12-> 9 */ OFDM(0) << 4 | /* 9-> 6 */ OFDM(0)); /* 6-> 6 */ run_write(sc, RT2860_LG_FBK_CFG1, CCK(2) << 12 | /* 11->5.5 */ CCK(1) << 8 | /* 5.5-> 2 */ CCK(0) << 4 | /* 2-> 1 */ CCK(0)); /* 1-> 1 */ #undef OFDM #undef CCK } static void run_set_txpreamble(struct run_softc *sc) { uint32_t tmp; run_read(sc, RT2860_AUTO_RSP_CFG, &tmp); if (sc->sc_ic.ic_flags & IEEE80211_F_SHPREAMBLE) tmp |= RT2860_CCK_SHORT_EN; else tmp &= ~RT2860_CCK_SHORT_EN; run_write(sc, RT2860_AUTO_RSP_CFG, tmp); } static void run_set_basicrates(struct run_softc *sc) { struct ieee80211com *ic = &sc->sc_ic; /* set basic rates mask */ if (ic->ic_curmode == IEEE80211_MODE_11B) run_write(sc, RT2860_LEGACY_BASIC_RATE, 0x003); else if (ic->ic_curmode == IEEE80211_MODE_11A) run_write(sc, RT2860_LEGACY_BASIC_RATE, 0x150); else /* 11g */ run_write(sc, RT2860_LEGACY_BASIC_RATE, 0x15f); } static void run_set_leds(struct run_softc *sc, uint16_t which) { (void)run_mcu_cmd(sc, RT2860_MCU_CMD_LEDS, which | (sc->leds & 0x7f)); } static void run_set_bssid(struct run_softc *sc, const uint8_t *bssid) { run_write(sc, RT2860_MAC_BSSID_DW0, bssid[0] | bssid[1] << 8 | bssid[2] << 16 | bssid[3] << 24); run_write(sc, RT2860_MAC_BSSID_DW1, bssid[4] | bssid[5] << 8); } static void run_set_macaddr(struct run_softc *sc, const uint8_t *addr) { run_write(sc, RT2860_MAC_ADDR_DW0, addr[0] | addr[1] << 8 | addr[2] << 16 | addr[3] << 24); run_write(sc, RT2860_MAC_ADDR_DW1, addr[4] | addr[5] << 8 | 0xff << 16); } static void run_updateslot(struct ifnet *ifp) { /* do it in a process context */ run_do_async(ifp->if_softc, run_updateslot_cb, NULL, 0); } /* ARGSUSED */ static void run_updateslot_cb(struct run_softc *sc, void *arg) { uint32_t tmp; run_read(sc, RT2860_BKOFF_SLOT_CFG, &tmp); tmp &= ~0xff; tmp |= (sc->sc_ic.ic_flags & IEEE80211_F_SHSLOT) ? 9 : 20; run_write(sc, RT2860_BKOFF_SLOT_CFG, tmp); } static int8_t run_rssi2dbm(struct run_softc *sc, uint8_t rssi, uint8_t rxchain) { struct ieee80211com *ic = &sc->sc_ic; struct ieee80211_channel *c = ic->ic_curchan; int delta; if (IEEE80211_IS_CHAN_5GHZ(c)) { u_int chan = ieee80211_chan2ieee(ic, c); delta = sc->rssi_5ghz[rxchain]; /* determine channel group */ if (chan <= 64) delta -= sc->lna[1]; else if (chan <= 128) delta -= sc->lna[2]; else delta -= sc->lna[3]; } else delta = sc->rssi_2ghz[rxchain] - sc->lna[0]; return -12 - delta - rssi; } static void run_rt5390_bbp_init(struct run_softc *sc) { u_int i; uint8_t bbp; /* Apply maximum likelihood detection for 2 stream case. */ run_bbp_read(sc, 105, &bbp); if (sc->nrxchains > 1) run_bbp_write(sc, 105, bbp | RT5390_MLD); /* Avoid data lost and CRC error. */ run_bbp_read(sc, 4, &bbp); run_bbp_write(sc, 4, bbp | RT5390_MAC_IF_CTRL); if (sc->mac_ver == 0x5592) { for (i = 0; i < (int)__arraycount(rt5592_def_bbp); i++) { run_bbp_write(sc, rt5592_def_bbp[i].reg, rt5592_def_bbp[i].val); } for (i = 0; i < (int)__arraycount(rt5592_bbp_r196); i++) { run_bbp_write(sc, 195, i + 0x80); run_bbp_write(sc, 196, rt5592_bbp_r196[i]); } } else { for (i = 0; i < (int)__arraycount(rt5390_def_bbp); i++) { run_bbp_write(sc, rt5390_def_bbp[i].reg, rt5390_def_bbp[i].val); } } if (sc->mac_ver == 0x5392) { run_bbp_write(sc, 88, 0x90); run_bbp_write(sc, 95, 0x9a); run_bbp_write(sc, 98, 0x12); run_bbp_write(sc, 106, 0x12); run_bbp_write(sc, 134, 0xd0); run_bbp_write(sc, 135, 0xf6); run_bbp_write(sc, 148, 0x84); } run_bbp_read(sc, 152, &bbp); run_bbp_write(sc, 152, bbp | 0x80); /* Fix BBP254 for RT5592C. */ if (sc->mac_ver == 0x5592 && sc->mac_rev >= 0x0221) { run_bbp_read(sc, 254, &bbp); run_bbp_write(sc, 254, bbp | 0x80); } /* Disable hardware antenna diversity. */ if (sc->mac_ver == 0x5390) run_bbp_write(sc, 154, 0); /* Initialize Rx CCK/OFDM frequency offset report. */ run_bbp_write(sc, 142, 1); run_bbp_write(sc, 143, 57); } static int run_bbp_init(struct run_softc *sc) { int i, error, ntries; uint8_t bbp0; /* wait for BBP to wake up */ for (ntries = 0; ntries < 20; ntries++) { if ((error = run_bbp_read(sc, 0, &bbp0)) != 0) return error; if (bbp0 != 0 && bbp0 != 0xff) break; } if (ntries == 20) return ETIMEDOUT; /* initialize BBP registers to default values */ if (sc->mac_ver >= 0x5390) run_rt5390_bbp_init(sc); else { for (i = 0; i < (int)__arraycount(rt2860_def_bbp); i++) { run_bbp_write(sc, rt2860_def_bbp[i].reg, rt2860_def_bbp[i].val); } } if (sc->mac_ver == 0x3593) { run_bbp_write(sc, 79, 0x13); run_bbp_write(sc, 80, 0x05); run_bbp_write(sc, 81, 0x33); run_bbp_write(sc, 86, 0x46); run_bbp_write(sc, 137, 0x0f); } /* fix BBP84 for RT2860E */ if (sc->mac_ver == 0x2860 && sc->mac_rev != 0x0101) run_bbp_write(sc, 84, 0x19); if (sc->mac_ver >= 0x3070 && (sc->mac_ver != 0x3593 && sc->mac_ver != 0x5592)) { run_bbp_write(sc, 79, 0x13); run_bbp_write(sc, 80, 0x05); run_bbp_write(sc, 81, 0x33); } else if (sc->mac_ver == 0x2860 && sc->mac_rev == 0x0100) { run_bbp_write(sc, 69, 0x16); run_bbp_write(sc, 73, 0x12); } return 0; } static int run_rt3070_rf_init(struct run_softc *sc) { uint32_t tmp; uint8_t rf, target, bbp4; int i; run_rt3070_rf_read(sc, 30, &rf); /* toggle RF R30 bit 7 */ run_rt3070_rf_write(sc, 30, rf | 0x80); usbd_delay_ms(sc->sc_udev, 10); run_rt3070_rf_write(sc, 30, rf & ~0x80); /* initialize RF registers to default value */ if (sc->mac_ver == 0x3572) { for (i = 0; i < (int)__arraycount(rt3572_def_rf); i++) { run_rt3070_rf_write(sc, rt3572_def_rf[i].reg, rt3572_def_rf[i].val); } } else { for (i = 0; i < (int)__arraycount(rt3070_def_rf); i++) { run_rt3070_rf_write(sc, rt3070_def_rf[i].reg, rt3070_def_rf[i].val); } } if (sc->mac_ver == 0x3572) { run_rt3070_rf_read(sc, 6, &rf); run_rt3070_rf_write(sc, 6, rf | 0x40); run_rt3070_rf_write(sc, 31, 0x14); run_read(sc, RT3070_LDO_CFG0, &tmp); tmp &= ~0x1f000000; if (sc->mac_rev < 0x0211 && sc->patch_dac) tmp |= 0x0d000000; /* 1.3V */ else tmp |= 0x01000000; /* 1.2V */ run_write(sc, RT3070_LDO_CFG0, tmp); } else if (sc->mac_ver == 0x3071) { run_rt3070_rf_read(sc, 6, &rf); run_rt3070_rf_write(sc, 6, rf | 0x40); run_rt3070_rf_write(sc, 31, 0x14); run_read(sc, RT3070_LDO_CFG0, &tmp); tmp &= ~0x1f000000; if (sc->mac_rev < 0x0211) tmp |= 0x0d000000; /* 1.35V */ else tmp |= 0x01000000; /* 1.2V */ run_write(sc, RT3070_LDO_CFG0, tmp); /* patch LNA_PE_G1 */ run_read(sc, RT3070_GPIO_SWITCH, &tmp); run_write(sc, RT3070_GPIO_SWITCH, tmp & ~0x20); } else if (sc->mac_ver == 0x3070 && sc->mac_rev < 0x0201) { /* * Change voltage from 1.2V to 1.35V for RT3070. * The DAC issue (RT3070_LD0_CFG0) has been fixed * in RT3070(F). */ run_read(sc, RT3070_LDO_CFG0, &tmp); tmp = (tmp & ~0x0f000000) | 0x0d000000; run_write(sc, RT3070_LDO_CFG0, tmp); } /* select 20MHz bandwidth */ run_rt3070_rf_read(sc, 31, &rf); run_rt3070_rf_write(sc, 31, rf & ~0x20); /* calibrate filter for 20MHz bandwidth */ sc->rf24_20mhz = 0x1f; /* default value */ target = (sc->mac_ver < 0x3071) ? 0x16 : 0x13; run_rt3070_filter_calib(sc, 0x07, target, &sc->rf24_20mhz); /* select 40MHz bandwidth */ run_bbp_read(sc, 4, &bbp4); run_bbp_write(sc, 4, (bbp4 & ~0x08) | 0x10); run_rt3070_rf_read(sc, 31, &rf); run_rt3070_rf_write(sc, 31, rf | 0x20); /* calibrate filter for 40MHz bandwidth */ sc->rf24_40mhz = 0x2f; /* default value */ target = (sc->mac_ver < 0x3071) ? 0x19 : 0x15; run_rt3070_filter_calib(sc, 0x27, target, &sc->rf24_40mhz); /* go back to 20MHz bandwidth */ run_bbp_read(sc, 4, &bbp4); run_bbp_write(sc, 4, bbp4 & ~0x18); if (sc->mac_ver == 0x3572) { /* save default BBP registers 25 and 26 values */ run_bbp_read(sc, 25, &sc->bbp25); run_bbp_read(sc, 26, &sc->bbp26); } else if (sc->mac_rev < 0x0211) run_rt3070_rf_write(sc, 27, 0x03); run_read(sc, RT3070_OPT_14, &tmp); run_write(sc, RT3070_OPT_14, tmp | 1); if (sc->mac_ver == 0x3070 || sc->mac_ver == 0x3071) { run_rt3070_rf_read(sc, 17, &rf); rf &= ~RT3070_TX_LO1; if ((sc->mac_ver == 0x3070 || (sc->mac_ver == 0x3071 && sc->mac_rev >= 0x0211)) && !sc->ext_2ghz_lna) rf |= 0x20; /* fix for long range Rx issue */ if (sc->txmixgain_2ghz >= 1) rf = (rf & ~0x7) | sc->txmixgain_2ghz; run_rt3070_rf_write(sc, 17, rf); } if (sc->mac_ver == 0x3071) { run_rt3070_rf_read(sc, 1, &rf); rf &= ~(RT3070_RX0_PD | RT3070_TX0_PD); rf |= RT3070_RF_BLOCK | RT3070_RX1_PD | RT3070_TX1_PD; run_rt3070_rf_write(sc, 1, rf); run_rt3070_rf_read(sc, 15, &rf); run_rt3070_rf_write(sc, 15, rf & ~RT3070_TX_LO2); run_rt3070_rf_read(sc, 20, &rf); run_rt3070_rf_write(sc, 20, rf & ~RT3070_RX_LO1); run_rt3070_rf_read(sc, 21, &rf); run_rt3070_rf_write(sc, 21, rf & ~RT3070_RX_LO2); } if (sc->mac_ver == 0x3070 || sc->mac_ver == 0x3071) { /* fix Tx to Rx IQ glitch by raising RF voltage */ run_rt3070_rf_read(sc, 27, &rf); rf &= ~0x77; if (sc->mac_rev < 0x0211) rf |= 0x03; run_rt3070_rf_write(sc, 27, rf); } return 0; } static int run_rt3593_rf_init(struct run_softc *sc) { uint32_t tmp; uint8_t rf; int i; /* Disable the GPIO bits 4 and 7 for LNA PE control. */ run_read(sc, RT3070_GPIO_SWITCH, &tmp); tmp &= ~(1 << 4 | 1 << 7); run_write(sc, RT3070_GPIO_SWITCH, tmp); /* Initialize RF registers to default value. */ for (i = 0; i < __arraycount(rt3593_def_rf); i++) { run_rt3070_rf_write(sc, rt3593_def_rf[i].reg, rt3593_def_rf[i].val); } /* Toggle RF R2 to initiate calibration. */ run_rt3070_rf_write(sc, 2, RT5390_RESCAL); /* Initialize RF frequency offset. */ run_adjust_freq_offset(sc); run_rt3070_rf_read(sc, 18, &rf); run_rt3070_rf_write(sc, 18, rf | RT3593_AUTOTUNE_BYPASS); /* * Increase voltage from 1.2V to 1.35V, wait for 1 msec to * decrease voltage back to 1.2V. */ run_read(sc, RT3070_LDO_CFG0, &tmp); tmp = (tmp & ~0x1f000000) | 0x0d000000; run_write(sc, RT3070_LDO_CFG0, tmp); usbd_delay_ms(sc->sc_udev, 1); tmp = (tmp & ~0x1f000000) | 0x01000000; run_write(sc, RT3070_LDO_CFG0, tmp); sc->rf24_20mhz = 0x1f; sc->rf24_40mhz = 0x2f; /* Save default BBP registers 25 and 26 values. */ run_bbp_read(sc, 25, &sc->bbp25); run_bbp_read(sc, 26, &sc->bbp26); run_read(sc, RT3070_OPT_14, &tmp); run_write(sc, RT3070_OPT_14, tmp | 1); return 0; } static int run_rt5390_rf_init(struct run_softc *sc) { uint32_t tmp; uint8_t rf; int i; /* Toggle RF R2 to initiate calibration. */ if (sc->mac_ver == 0x5390) { run_rt3070_rf_read(sc, 2, &rf); run_rt3070_rf_write(sc, 2, rf | RT5390_RESCAL); usbd_delay_ms(sc->sc_udev, 10); run_rt3070_rf_write(sc, 2, rf & ~RT5390_RESCAL); } else { run_rt3070_rf_write(sc, 2, RT5390_RESCAL); usbd_delay_ms(sc->sc_udev, 10); } /* Initialize RF registers to default value. */ if (sc->mac_ver == 0x5592) { for (i = 0; i < __arraycount(rt5592_def_rf); i++) { run_rt3070_rf_write(sc, rt5592_def_rf[i].reg, rt5592_def_rf[i].val); } /* Initialize RF frequency offset. */ run_adjust_freq_offset(sc); } else if (sc->mac_ver == 0x5392) { for (i = 0; i < __arraycount(rt5392_def_rf); i++) { run_rt3070_rf_write(sc, rt5392_def_rf[i].reg, rt5392_def_rf[i].val); } if (sc->mac_rev >= 0x0223) { run_rt3070_rf_write(sc, 23, 0x0f); run_rt3070_rf_write(sc, 24, 0x3e); run_rt3070_rf_write(sc, 51, 0x32); run_rt3070_rf_write(sc, 53, 0x22); run_rt3070_rf_write(sc, 56, 0xc1); run_rt3070_rf_write(sc, 59, 0x0f); } } else { for (i = 0; i < __arraycount(rt5390_def_rf); i++) { run_rt3070_rf_write(sc, rt5390_def_rf[i].reg, rt5390_def_rf[i].val); } if (sc->mac_rev >= 0x0502) { run_rt3070_rf_write(sc, 6, 0xe0); run_rt3070_rf_write(sc, 25, 0x80); run_rt3070_rf_write(sc, 46, 0x73); run_rt3070_rf_write(sc, 53, 0x00); run_rt3070_rf_write(sc, 56, 0x42); run_rt3070_rf_write(sc, 61, 0xd1); } } sc->rf24_20mhz = 0x1f; /* default value */ sc->rf24_40mhz = (sc->mac_ver == 0x5592) ? 0 : 0x2f; if (sc->mac_rev < 0x0211) run_rt3070_rf_write(sc, 27, 0x3); run_read(sc, RT3070_OPT_14, &tmp); run_write(sc, RT3070_OPT_14, tmp | 1); return 0; } static int run_rt3070_filter_calib(struct run_softc *sc, uint8_t init, uint8_t target, uint8_t *val) { uint8_t rf22, rf24; uint8_t bbp55_pb, bbp55_sb, delta; int ntries; /* program filter */ run_rt3070_rf_read(sc, 24, &rf24); rf24 = (rf24 & 0xc0) | init; /* initial filter value */ run_rt3070_rf_write(sc, 24, rf24); /* enable baseband loopback mode */ run_rt3070_rf_read(sc, 22, &rf22); run_rt3070_rf_write(sc, 22, rf22 | 0x01); /* set power and frequency of passband test tone */ run_bbp_write(sc, 24, 0x00); for (ntries = 0; ntries < 100; ntries++) { /* transmit test tone */ run_bbp_write(sc, 25, 0x90); usbd_delay_ms(sc->sc_udev, 10); /* read received power */ run_bbp_read(sc, 55, &bbp55_pb); if (bbp55_pb != 0) break; } if (ntries == 100) return ETIMEDOUT; /* set power and frequency of stopband test tone */ run_bbp_write(sc, 24, 0x06); for (ntries = 0; ntries < 100; ntries++) { /* transmit test tone */ run_bbp_write(sc, 25, 0x90); usbd_delay_ms(sc->sc_udev, 10); /* read received power */ run_bbp_read(sc, 55, &bbp55_sb); delta = bbp55_pb - bbp55_sb; if (delta > target) break; /* reprogram filter */ rf24++; run_rt3070_rf_write(sc, 24, rf24); } if (ntries < 100) { if (rf24 != init) rf24--; /* backtrack */ *val = rf24; run_rt3070_rf_write(sc, 24, rf24); } /* restore initial state */ run_bbp_write(sc, 24, 0x00); /* disable baseband loopback mode */ run_rt3070_rf_read(sc, 22, &rf22); run_rt3070_rf_write(sc, 22, rf22 & ~0x01); return 0; } static void run_rt3070_rf_setup(struct run_softc *sc) { uint8_t bbp, rf; int i; if (sc->mac_ver == 0x3572) { /* enable DC filter */ if (sc->mac_rev >= 0x0201) run_bbp_write(sc, 103, 0xc0); run_bbp_read(sc, 138, &bbp); if (sc->ntxchains == 1) bbp |= 0x20; /* turn off DAC1 */ if (sc->nrxchains == 1) bbp &= ~0x02; /* turn off ADC1 */ run_bbp_write(sc, 138, bbp); if (sc->mac_rev >= 0x0211) { /* improve power consumption */ run_bbp_read(sc, 31, &bbp); run_bbp_write(sc, 31, bbp & ~0x03); } run_rt3070_rf_read(sc, 16, &rf); rf = (rf & ~0x07) | sc->txmixgain_2ghz; run_rt3070_rf_write(sc, 16, rf); } else if (sc->mac_ver == 0x3071) { /* enable DC filter */ if (sc->mac_rev >= 0x0201) run_bbp_write(sc, 103, 0xc0); run_bbp_read(sc, 138, &bbp); if (sc->ntxchains == 1) bbp |= 0x20; /* turn off DAC1 */ if (sc->nrxchains == 1) bbp &= ~0x02; /* turn off ADC1 */ run_bbp_write(sc, 138, bbp); if (sc->mac_rev >= 0x0211) { /* improve power consumption */ run_bbp_read(sc, 31, &bbp); run_bbp_write(sc, 31, bbp & ~0x03); } run_write(sc, RT2860_TX_SW_CFG1, 0); if (sc->mac_rev < 0x0211) { run_write(sc, RT2860_TX_SW_CFG2, sc->patch_dac ? 0x2c : 0x0f); } else run_write(sc, RT2860_TX_SW_CFG2, 0); } else if (sc->mac_ver == 0x3070) { if (sc->mac_rev >= 0x0201) { /* enable DC filter */ run_bbp_write(sc, 103, 0xc0); /* improve power consumption */ run_bbp_read(sc, 31, &bbp); run_bbp_write(sc, 31, bbp & ~0x03); } if (sc->mac_rev < 0x0211) { run_write(sc, RT2860_TX_SW_CFG1, 0); run_write(sc, RT2860_TX_SW_CFG2, 0x2c); } else run_write(sc, RT2860_TX_SW_CFG2, 0); } /* initialize RF registers from ROM for >=RT3071*/ if (sc->mac_ver >= 0x3071) { for (i = 0; i < 10; i++) { if (sc->rf[i].reg == 0 || sc->rf[i].reg == 0xff) continue; run_rt3070_rf_write(sc, sc->rf[i].reg, sc->rf[i].val); } } } static void run_rt3593_rf_setup(struct run_softc *sc) { uint8_t bbp, rf; if (sc->mac_rev >= 0x0211) { /* Enable DC filter. */ run_bbp_write(sc, 103, 0xc0); } run_write(sc, RT2860_TX_SW_CFG1, 0); if (sc->mac_rev < 0x0211) { run_write(sc, RT2860_TX_SW_CFG2, sc->patch_dac ? 0x2c : 0x0f); } else run_write(sc, RT2860_TX_SW_CFG2, 0); run_rt3070_rf_read(sc, 50, &rf); run_rt3070_rf_write(sc, 50, rf & ~RT3593_TX_LO2); run_rt3070_rf_read(sc, 51, &rf); rf = (rf & ~(RT3593_TX_LO1 | 0x0c)) | ((sc->txmixgain_2ghz & 0x07) << 2); run_rt3070_rf_write(sc, 51, rf); run_rt3070_rf_read(sc, 38, &rf); run_rt3070_rf_write(sc, 38, rf & ~RT5390_RX_LO1); run_rt3070_rf_read(sc, 39, &rf); run_rt3070_rf_write(sc, 39, rf & ~RT5390_RX_LO2); run_rt3070_rf_read(sc, 1, &rf); run_rt3070_rf_write(sc, 1, rf & ~(RT3070_RF_BLOCK | RT3070_PLL_PD)); run_rt3070_rf_read(sc, 30, &rf); rf = (rf & ~0x18) | 0x10; run_rt3070_rf_write(sc, 30, rf); /* Apply maximum likelihood detection for 2 stream case. */ run_bbp_read(sc, 105, &bbp); if (sc->nrxchains > 1) run_bbp_write(sc, 105, bbp | RT5390_MLD); /* Avoid data lost and CRC error. */ run_bbp_read(sc, 4, &bbp); run_bbp_write(sc, 4, bbp | RT5390_MAC_IF_CTRL); run_bbp_write(sc, 92, 0x02); run_bbp_write(sc, 82, 0x82); run_bbp_write(sc, 106, 0x05); run_bbp_write(sc, 104, 0x92); run_bbp_write(sc, 88, 0x90); run_bbp_write(sc, 148, 0xc8); run_bbp_write(sc, 47, 0x48); run_bbp_write(sc, 120, 0x50); run_bbp_write(sc, 163, 0x9d); /* SNR mapping. */ run_bbp_write(sc, 142, 0x06); run_bbp_write(sc, 143, 0xa0); run_bbp_write(sc, 142, 0x07); run_bbp_write(sc, 143, 0xa1); run_bbp_write(sc, 142, 0x08); run_bbp_write(sc, 143, 0xa2); run_bbp_write(sc, 31, 0x08); run_bbp_write(sc, 68, 0x0b); run_bbp_write(sc, 105, 0x04); } static void run_rt5390_rf_setup(struct run_softc *sc) { uint8_t bbp, rf; if (sc->mac_rev >= 0x0211) { /* Enable DC filter. */ run_bbp_write(sc, 103, 0xc0); if (sc->mac_ver != 0x5592) { /* Improve power consumption. */ run_bbp_read(sc, 31, &bbp); run_bbp_write(sc, 31, bbp & ~0x03); } } run_bbp_read(sc, 138, &bbp); if (sc->ntxchains == 1) bbp |= 0x20; /* turn off DAC1 */ if (sc->nrxchains == 1) bbp &= ~0x02; /* turn off ADC1 */ run_bbp_write(sc, 138, bbp); run_rt3070_rf_read(sc, 38, &rf); run_rt3070_rf_write(sc, 38, rf & ~RT5390_RX_LO1); run_rt3070_rf_read(sc, 39, &rf); run_rt3070_rf_write(sc, 39, rf & ~RT5390_RX_LO2); /* Avoid data lost and CRC error. */ run_bbp_read(sc, 4, &bbp); run_bbp_write(sc, 4, bbp | RT5390_MAC_IF_CTRL); run_rt3070_rf_read(sc, 30, &rf); rf = (rf & ~0x18) | 0x10; run_rt3070_rf_write(sc, 30, rf); if (sc->mac_ver != 0x5592) { run_write(sc, RT2860_TX_SW_CFG1, 0); if (sc->mac_rev < 0x0211) { run_write(sc, RT2860_TX_SW_CFG2, sc->patch_dac ? 0x2c : 0x0f); } else run_write(sc, RT2860_TX_SW_CFG2, 0); } } static int run_txrx_enable(struct run_softc *sc) { uint32_t tmp; int error, ntries; run_write(sc, RT2860_MAC_SYS_CTRL, RT2860_MAC_TX_EN); for (ntries = 0; ntries < 200; ntries++) { if ((error = run_read(sc, RT2860_WPDMA_GLO_CFG, &tmp)) != 0) return error; if ((tmp & (RT2860_TX_DMA_BUSY | RT2860_RX_DMA_BUSY)) == 0) break; usbd_delay_ms(sc->sc_udev, 50); } if (ntries == 200) return ETIMEDOUT; usbd_delay_ms(sc->sc_udev, 50); tmp |= RT2860_RX_DMA_EN | RT2860_TX_DMA_EN | RT2860_TX_WB_DDONE; run_write(sc, RT2860_WPDMA_GLO_CFG, tmp); /* enable Rx bulk aggregation (set timeout and limit) */ tmp = RT2860_USB_TX_EN | RT2860_USB_RX_EN | RT2860_USB_RX_AGG_EN | RT2860_USB_RX_AGG_TO(128) | RT2860_USB_RX_AGG_LMT(2); run_write(sc, RT2860_USB_DMA_CFG, tmp); /* set Rx filter */ tmp = RT2860_DROP_CRC_ERR | RT2860_DROP_PHY_ERR; if (sc->sc_ic.ic_opmode != IEEE80211_M_MONITOR) { tmp |= RT2860_DROP_UC_NOME | RT2860_DROP_DUPL | RT2860_DROP_CTS | RT2860_DROP_BA | RT2860_DROP_ACK | RT2860_DROP_VER_ERR | RT2860_DROP_CTRL_RSV | RT2860_DROP_CFACK | RT2860_DROP_CFEND; if (sc->sc_ic.ic_opmode == IEEE80211_M_STA) tmp |= RT2860_DROP_RTS | RT2860_DROP_PSPOLL; } run_write(sc, RT2860_RX_FILTR_CFG, tmp); run_write(sc, RT2860_MAC_SYS_CTRL, RT2860_MAC_RX_EN | RT2860_MAC_TX_EN); return 0; } static int run_adjust_freq_offset(struct run_softc *sc) { uint8_t rf, tmp; run_rt3070_rf_read(sc, 17, &rf); tmp = rf; rf = (rf & ~0x7f) | (sc->freq & 0x7f); rf = MIN(rf, 0x5f); if (tmp != rf) run_mcu_cmd(sc, 0x74, (tmp << 8 ) | rf); return 0; } static int run_init(struct ifnet *ifp) { struct run_softc *sc = ifp->if_softc; struct ieee80211com *ic = &sc->sc_ic; uint32_t tmp; uint8_t bbp1, bbp3; int i, error, qid, ridx, ntries; usbd_status status; for (ntries = 0; ntries < 100; ntries++) { if ((error = run_read(sc, RT2860_ASIC_VER_ID, &tmp)) != 0) goto fail; if (tmp != 0 && tmp != 0xffffffff) break; usbd_delay_ms(sc->sc_udev, 10); } if (ntries == 100) { error = ETIMEDOUT; goto fail; } if ((sc->sc_flags & RUN_FWLOADED) == 0 && (error = run_load_microcode(sc)) != 0) { device_printf(sc->sc_dev, "could not load 8051 microcode\n"); goto fail; } if (ifp->if_flags & IFF_RUNNING) run_stop(ifp, 0); /* init host command ring */ sc->cmdq.cur = sc->cmdq.next = sc->cmdq.queued = 0; /* init Tx rings (4 EDCAs) */ for (qid = 0; qid < 4; qid++) { if ((error = run_alloc_tx_ring(sc, qid)) != 0) goto fail; } /* init Rx ring */ if ((error = run_alloc_rx_ring(sc)) != 0) goto fail; IEEE80211_ADDR_COPY(ic->ic_myaddr, CLLADDR(ifp->if_sadl)); run_set_macaddr(sc, ic->ic_myaddr); for (ntries = 0; ntries < 100; ntries++) { if ((error = run_read(sc, RT2860_WPDMA_GLO_CFG, &tmp)) != 0) goto fail; if ((tmp & (RT2860_TX_DMA_BUSY | RT2860_RX_DMA_BUSY)) == 0) break; usbd_delay_ms(sc->sc_udev, 10); } if (ntries == 100) { device_printf(sc->sc_dev, "timeout waiting for DMA engine\n"); error = ETIMEDOUT; goto fail; } tmp &= 0xff0; tmp |= RT2860_TX_WB_DDONE; run_write(sc, RT2860_WPDMA_GLO_CFG, tmp); /* turn off PME_OEN to solve high-current issue */ run_read(sc, RT2860_SYS_CTRL, &tmp); run_write(sc, RT2860_SYS_CTRL, tmp & ~RT2860_PME_OEN); run_write(sc, RT2860_MAC_SYS_CTRL, RT2860_BBP_HRST | RT2860_MAC_SRST); run_write(sc, RT2860_USB_DMA_CFG, 0); if ((error = run_reset(sc)) != 0) { device_printf(sc->sc_dev, "could not reset chipset\n"); goto fail; } run_write(sc, RT2860_MAC_SYS_CTRL, 0); /* init Tx power for all Tx rates (from EEPROM) */ for (ridx = 0; ridx < 5; ridx++) { if (sc->txpow20mhz[ridx] == 0xffffffff) continue; run_write(sc, RT2860_TX_PWR_CFG(ridx), sc->txpow20mhz[ridx]); } for (i = 0; i < (int)__arraycount(rt2870_def_mac); i++) run_write(sc, rt2870_def_mac[i].reg, rt2870_def_mac[i].val); run_write(sc, RT2860_WMM_AIFSN_CFG, 0x00002273); run_write(sc, RT2860_WMM_CWMIN_CFG, 0x00002344); run_write(sc, RT2860_WMM_CWMAX_CFG, 0x000034aa); if (sc->mac_ver >= 0x5390) { run_write(sc, RT2860_TX_SW_CFG0, 4 << RT2860_DLY_PAPE_EN_SHIFT | 4); if (sc->mac_ver >= 0x5392) { run_write(sc, RT2860_MAX_LEN_CFG, 0x00002fff); if (sc->mac_ver == 0x5592) { run_write(sc, RT2860_HT_FBK_CFG1, 0xedcba980); run_write(sc, RT2860_TXOP_HLDR_ET, 0x00000082); } else { run_write(sc, RT2860_HT_FBK_CFG1, 0xedcb4980); run_write(sc, RT2860_LG_FBK_CFG0, 0xedcba322); } } } else if (sc->mac_ver >= 0x3593) { run_write(sc, RT2860_TX_SW_CFG0, 4 << RT2860_DLY_PAPE_EN_SHIFT | 2); } else if (sc->mac_ver >= 0x3070) { /* set delay of PA_PE assertion to 1us (unit of 0.25us) */ run_write(sc, RT2860_TX_SW_CFG0, 4 << RT2860_DLY_PAPE_EN_SHIFT); } /* wait while MAC is busy */ for (ntries = 0; ntries < 100; ntries++) { if ((error = run_read(sc, RT2860_MAC_STATUS_REG, &tmp)) != 0) goto fail; if (!(tmp & (RT2860_RX_STATUS_BUSY | RT2860_TX_STATUS_BUSY))) break; DELAY(1000); } if (ntries == 100) { error = ETIMEDOUT; goto fail; } /* clear Host to MCU mailbox */ run_write(sc, RT2860_H2M_BBPAGENT, 0); run_write(sc, RT2860_H2M_MAILBOX, 0); usbd_delay_ms(sc->sc_udev, 10); if ((error = run_bbp_init(sc)) != 0) { device_printf(sc->sc_dev, "could not initialize BBP\n"); goto fail; } /* abort TSF synchronization */ run_read(sc, RT2860_BCN_TIME_CFG, &tmp); tmp &= ~(RT2860_BCN_TX_EN | RT2860_TSF_TIMER_EN | RT2860_TBTT_TIMER_EN); run_write(sc, RT2860_BCN_TIME_CFG, tmp); /* clear RX WCID search table */ run_set_region_4(sc, RT2860_WCID_ENTRY(0), 0, 512); /* clear Pair-wise key table */ run_set_region_4(sc, RT2860_PKEY(0), 0, 2048); /* clear IV/EIV table */ run_set_region_4(sc, RT2860_IVEIV(0), 0, 512); /* clear WCID attribute table */ run_set_region_4(sc, RT2860_WCID_ATTR(0), 0, 8 * 32); /* clear shared key table */ run_set_region_4(sc, RT2860_SKEY(0, 0), 0, 8 * 32); /* clear shared key mode */ run_set_region_4(sc, RT2860_SKEY_MODE_0_7, 0, 4); /* clear RX WCID search table */ run_set_region_4(sc, RT2860_WCID_ENTRY(0), 0, 512); /* clear WCID attribute table */ run_set_region_4(sc, RT2860_WCID_ATTR(0), 0, 8 * 32); run_read(sc, RT2860_US_CYC_CNT, &tmp); tmp = (tmp & ~0xff) | 0x1e; run_write(sc, RT2860_US_CYC_CNT, tmp); if (sc->mac_rev != 0x0101) run_write(sc, RT2860_TXOP_CTRL_CFG, 0x0000583f); run_write(sc, RT2860_WMM_TXOP0_CFG, 0); run_write(sc, RT2860_WMM_TXOP1_CFG, 48 << 16 | 96); /* write vendor-specific BBP values (from EEPROM) */ if (sc->mac_ver < 0x3593) { for (i = 0; i < 10; i++) { if (sc->bbp[i].reg == 0 || sc->bbp[i].reg == 0xff) continue; run_bbp_write(sc, sc->bbp[i].reg, sc->bbp[i].val); } } /* select Main antenna for 1T1R devices */ if (sc->rf_rev == RT3070_RF_3020 || sc->rf_rev == RT5390_RF_5370) run_set_rx_antenna(sc, 0); /* send LEDs operating mode to microcontroller */ (void)run_mcu_cmd(sc, RT2860_MCU_CMD_LED1, sc->led[0]); (void)run_mcu_cmd(sc, RT2860_MCU_CMD_LED2, sc->led[1]); (void)run_mcu_cmd(sc, RT2860_MCU_CMD_LED3, sc->led[2]); if (sc->mac_ver >= 0x5390) run_rt5390_rf_init(sc); else if (sc->mac_ver == 0x3593) run_rt3593_rf_init(sc); else if (sc->mac_ver >= 0x3070) run_rt3070_rf_init(sc); /* disable non-existing Rx chains */ run_bbp_read(sc, 3, &bbp3); bbp3 &= ~(1 << 3 | 1 << 4); if (sc->nrxchains == 2) bbp3 |= 1 << 3; else if (sc->nrxchains == 3) bbp3 |= 1 << 4; run_bbp_write(sc, 3, bbp3); /* disable non-existing Tx chains */ run_bbp_read(sc, 1, &bbp1); if (sc->ntxchains == 1) bbp1 &= ~(1 << 3 | 1 << 4); run_bbp_write(sc, 1, bbp1); if (sc->mac_ver >= 0x5390) run_rt5390_rf_setup(sc); else if (sc->mac_ver == 0x3593) run_rt3593_rf_setup(sc); else if (sc->mac_ver >= 0x3070) run_rt3070_rf_setup(sc); /* select default channel */ run_set_chan(sc, ic->ic_curchan); /* setup initial protection mode */ run_updateprot(sc); /* turn radio LED on */ run_set_leds(sc, RT2860_LED_RADIO); #ifdef RUN_HWCRYPTO if (ic->ic_flags & IEEE80211_F_PRIVACY) { /* install WEP keys */ for (i = 0; i < IEEE80211_WEP_NKID; i++) (void)run_set_key(ic, &ic->ic_crypto.cs_nw_keys[i], NULL); } #endif for (i = 0; i < RUN_RX_RING_COUNT; i++) { struct run_rx_data *data = &sc->rxq.data[i]; usbd_setup_xfer(data->xfer, data, data->buf, RUN_MAX_RXSZ, USBD_SHORT_XFER_OK, USBD_NO_TIMEOUT, run_rxeof); status = usbd_transfer(data->xfer); if (status != USBD_NORMAL_COMPLETION && status != USBD_IN_PROGRESS) { device_printf(sc->sc_dev, "queuing rx failed: %s\n", usbd_errstr(status)); error = EIO; goto fail; } } if ((error = run_txrx_enable(sc)) != 0) goto fail; ifp->if_flags &= ~IFF_OACTIVE; ifp->if_flags |= IFF_RUNNING; if (ic->ic_opmode == IEEE80211_M_MONITOR) ieee80211_new_state(ic, IEEE80211_S_RUN, -1); else ieee80211_new_state(ic, IEEE80211_S_SCAN, -1); if (error != 0) fail: run_stop(ifp, 1); return error; } static void run_stop(struct ifnet *ifp, int disable) { struct run_softc *sc = ifp->if_softc; struct ieee80211com *ic = &sc->sc_ic; uint32_t tmp; int ntries, qid; if (ifp->if_flags & IFF_RUNNING) run_set_leds(sc, 0); /* turn all LEDs off */ sc->sc_tx_timer = 0; ifp->if_timer = 0; ifp->if_flags &= ~(IFF_RUNNING | IFF_OACTIVE); callout_stop(&sc->scan_to); callout_stop(&sc->calib_to); ieee80211_new_state(ic, IEEE80211_S_INIT, -1); /* wait for all queued asynchronous commands to complete */ while (sc->cmdq.queued > 0) tsleep(&sc->cmdq, 0, "cmdq", 0); /* disable Tx/Rx */ run_read(sc, RT2860_MAC_SYS_CTRL, &tmp); tmp &= ~(RT2860_MAC_RX_EN | RT2860_MAC_TX_EN); run_write(sc, RT2860_MAC_SYS_CTRL, tmp); /* wait for pending Tx to complete */ for (ntries = 0; ntries < 100; ntries++) { if (run_read(sc, RT2860_TXRXQ_PCNT, &tmp) != 0) break; if ((tmp & RT2860_TX2Q_PCNT_MASK) == 0) break; } DELAY(1000); run_write(sc, RT2860_USB_DMA_CFG, 0); /* reset adapter */ run_write(sc, RT2860_MAC_SYS_CTRL, RT2860_BBP_HRST | RT2860_MAC_SRST); run_write(sc, RT2860_MAC_SYS_CTRL, 0); /* reset Tx and Rx rings */ sc->qfullmsk = 0; for (qid = 0; qid < 4; qid++) run_free_tx_ring(sc, qid); run_free_rx_ring(sc); } #ifndef IEEE80211_STA_ONLY static int run_setup_beacon(struct run_softc *sc) { struct ieee80211com *ic = &sc->sc_ic; struct rt2860_txwi txwi; struct mbuf *m; uint16_t txwisize; int ridx; if ((m = ieee80211_beacon_alloc(ic, ic->ic_bss, &sc->sc_bo)) == NULL) return ENOBUFS; memset(&txwi, 0, sizeof(txwi)); txwi.wcid = 0xff; txwi.len = htole16(m->m_pkthdr.len); /* send beacons at the lowest available rate */ ridx = (ic->ic_curmode == IEEE80211_MODE_11A) ? RT2860_RIDX_OFDM6 : RT2860_RIDX_CCK1; txwi.phy = htole16(rt2860_rates[ridx].mcs); if (rt2860_rates[ridx].phy == IEEE80211_T_OFDM) txwi.phy |= htole16(RT2860_PHY_OFDM); txwi.txop = RT2860_TX_TXOP_HT; txwi.flags = RT2860_TX_TS; txwisize = (sc->mac_ver == 0x5592) ? sizeof(txwi) + sizeof(uint32_t) : sizeof(txwi); run_write_region_1(sc, RT2860_BCN_BASE(0), (uint8_t *)&txwi, txwisize); run_write_region_1(sc, RT2860_BCN_BASE(0) + txwisize, mtod(m, uint8_t *), (m->m_pkthdr.len + 1) & ~1); m_freem(m); return 0; } #endif MODULE(MODULE_CLASS_DRIVER, if_run, NULL); #ifdef _MODULE #include "ioconf.c" #endif static int if_run_modcmd(modcmd_t cmd, void *arg) { int error = 0; switch (cmd) { case MODULE_CMD_INIT: #ifdef _MODULE error = config_init_component(cfdriver_ioconf_run, cfattach_ioconf_run, cfdata_ioconf_run); #endif return error; case MODULE_CMD_FINI: #ifdef _MODULE error = config_fini_component(cfdriver_ioconf_run, cfattach_ioconf_run, cfdata_ioconf_run); #endif return error; default: return ENOTTY; } } |
| 12 4 6 4 10 10 10 10 3 7 2 5 2 1 1 22 1 21 15 15 15 1 5 16 16 4 2 5 5 5 5 2 2 2 2 2 19 19 2 15 16 7 2 3 13 13 13 1 1 11 2 3 10 1 10 11 10 10 5 2 1 1 1 1 1 1 1 1 1 2 22 24 24 24 23 2 20 24 22 22 2 2 23 5 24 31 36 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 197 198 199 200 201 202 203 204 205 206 207 208 209 210 211 212 213 214 215 216 217 218 219 220 221 222 223 224 225 226 227 228 229 230 231 232 233 234 235 236 237 238 239 240 241 242 243 244 245 246 247 248 249 250 251 252 253 254 255 256 257 258 259 260 261 262 263 264 265 266 267 268 269 270 271 272 273 274 275 276 277 278 279 280 281 282 283 284 285 286 287 288 289 290 291 292 293 294 295 296 297 298 299 300 301 302 303 304 305 306 307 308 309 310 311 312 313 314 315 316 317 318 319 320 321 322 323 324 325 326 327 328 329 330 331 332 333 334 335 336 337 338 339 340 341 342 343 344 345 346 347 348 349 350 351 352 353 354 355 356 357 358 359 360 361 362 363 364 365 366 367 368 369 370 371 372 373 374 375 376 377 378 379 380 381 382 383 384 385 386 387 388 389 390 391 392 393 394 395 396 397 398 399 400 401 402 403 404 405 406 407 408 409 410 411 412 413 414 415 416 417 418 419 420 421 422 423 424 425 426 427 428 429 430 431 432 433 434 435 436 437 438 439 440 441 442 443 444 445 446 447 448 449 450 451 452 453 454 455 456 457 458 459 460 461 462 463 464 465 466 467 468 469 470 471 472 473 474 475 476 477 478 479 480 481 482 483 484 485 486 487 488 489 490 491 492 493 494 495 496 497 498 499 500 501 502 503 504 505 506 507 508 509 510 511 512 513 514 515 516 517 518 519 520 521 522 523 524 525 526 527 528 529 530 531 532 533 534 535 536 537 538 539 540 541 542 543 544 545 546 547 548 549 550 551 552 553 554 555 556 557 558 559 560 561 562 563 564 565 566 567 568 569 570 571 572 573 574 575 576 577 578 579 580 581 582 583 584 585 586 587 588 589 590 591 592 593 594 595 596 597 598 599 600 601 602 603 604 605 606 607 608 609 610 611 612 613 614 615 616 617 618 619 620 621 622 623 624 625 626 627 628 629 630 631 632 633 634 635 636 637 638 639 640 641 642 643 644 645 646 647 648 649 650 651 652 653 654 655 656 657 658 659 660 661 662 663 664 665 666 667 668 669 670 671 672 673 674 675 676 677 678 679 680 681 682 683 684 685 686 687 688 689 690 691 692 693 694 695 696 697 698 699 700 701 702 703 704 705 706 707 708 709 710 711 712 713 714 715 716 717 718 719 720 721 722 723 724 725 726 727 728 729 730 731 732 733 734 735 736 737 738 739 740 741 742 743 744 745 746 747 748 749 750 751 752 753 754 755 756 757 758 759 760 761 762 763 764 765 766 767 768 769 770 771 772 773 774 775 776 777 778 779 780 781 782 783 784 785 786 787 788 789 790 791 792 793 794 795 796 797 798 799 800 801 802 803 804 805 806 807 808 809 810 811 812 813 814 815 816 817 818 819 820 821 822 823 824 825 826 827 828 829 830 831 832 833 834 835 836 837 838 839 840 841 842 843 844 845 846 847 848 849 850 851 852 853 854 855 856 857 858 859 860 861 862 863 864 865 866 867 868 869 870 871 872 873 874 875 876 877 878 879 880 881 882 883 884 885 886 887 888 889 890 891 892 893 894 895 896 897 898 899 900 901 902 903 904 905 906 907 908 909 910 911 912 913 914 915 916 917 918 919 920 921 922 923 924 925 926 927 928 929 930 931 932 933 934 935 936 937 938 939 940 941 942 943 944 945 946 947 948 949 950 951 952 953 954 955 956 957 958 959 960 961 962 963 964 965 966 967 968 969 970 971 972 973 974 975 976 977 978 979 980 981 982 983 984 985 986 987 988 989 990 991 992 993 994 995 996 997 998 999 1000 1001 1002 1003 1004 1005 1006 1007 1008 1009 1010 1011 1012 1013 1014 1015 1016 1017 1018 1019 1020 1021 1022 1023 1024 1025 1026 1027 1028 1029 1030 1031 1032 1033 1034 1035 1036 1037 1038 1039 1040 1041 1042 1043 1044 1045 1046 1047 1048 1049 1050 1051 1052 1053 1054 1055 1056 1057 1058 1059 1060 1061 1062 1063 1064 1065 1066 1067 1068 1069 1070 1071 1072 1073 1074 1075 1076 1077 1078 1079 1080 1081 1082 1083 1084 1085 1086 1087 1088 1089 1090 1091 1092 1093 1094 1095 1096 1097 1098 1099 1100 1101 1102 1103 1104 1105 1106 1107 1108 1109 1110 1111 1112 1113 1114 1115 1116 1117 1118 1119 1120 1121 1122 1123 1124 1125 1126 1127 1128 1129 1130 1131 1132 1133 1134 1135 1136 1137 1138 1139 1140 1141 1142 1143 1144 1145 1146 1147 1148 1149 1150 1151 1152 1153 1154 1155 1156 1157 1158 1159 1160 1161 1162 1163 1164 1165 1166 1167 1168 1169 1170 1171 1172 1173 1174 1175 1176 1177 1178 1179 1180 1181 1182 1183 1184 1185 1186 1187 1188 1189 1190 1191 1192 1193 1194 1195 1196 1197 1198 1199 1200 1201 1202 1203 1204 1205 1206 1207 1208 1209 1210 1211 1212 1213 1214 1215 1216 1217 1218 1219 1220 1221 1222 1223 1224 1225 1226 1227 1228 1229 1230 1231 1232 1233 1234 1235 1236 1237 1238 1239 1240 1241 1242 1243 1244 1245 1246 1247 1248 1249 1250 1251 1252 1253 1254 1255 1256 1257 1258 1259 1260 1261 1262 1263 1264 1265 1266 1267 1268 1269 1270 1271 1272 1273 1274 1275 1276 1277 1278 1279 1280 1281 1282 1283 1284 1285 1286 1287 1288 1289 1290 1291 1292 1293 1294 1295 1296 1297 1298 1299 1300 1301 1302 1303 1304 1305 1306 1307 1308 1309 1310 1311 1312 1313 1314 1315 1316 1317 1318 1319 1320 1321 1322 1323 1324 1325 1326 1327 1328 1329 1330 1331 1332 1333 1334 1335 1336 1337 1338 1339 1340 1341 1342 1343 1344 1345 1346 1347 1348 1349 1350 1351 1352 1353 1354 1355 1356 1357 1358 1359 1360 1361 1362 1363 1364 1365 1366 1367 1368 1369 1370 1371 1372 1373 1374 1375 1376 1377 1378 1379 1380 1381 1382 1383 1384 1385 1386 1387 1388 1389 1390 1391 1392 1393 1394 1395 1396 1397 1398 1399 1400 1401 1402 1403 1404 1405 1406 1407 1408 1409 1410 1411 1412 1413 1414 1415 1416 1417 1418 1419 1420 1421 1422 1423 1424 1425 1426 1427 1428 1429 1430 1431 1432 1433 1434 1435 1436 1437 1438 1439 1440 1441 1442 1443 1444 1445 1446 1447 1448 1449 1450 1451 1452 1453 1454 1455 1456 1457 1458 1459 1460 1461 1462 1463 1464 1465 1466 1467 1468 1469 1470 1471 1472 1473 1474 1475 1476 1477 1478 1479 1480 1481 1482 1483 1484 1485 1486 1487 1488 1489 1490 1491 1492 1493 1494 1495 1496 1497 1498 1499 1500 1501 1502 1503 1504 1505 1506 1507 1508 1509 1510 1511 1512 1513 1514 1515 1516 1517 1518 1519 1520 1521 1522 1523 1524 1525 1526 1527 1528 1529 1530 1531 1532 1533 1534 1535 1536 1537 1538 1539 1540 1541 1542 1543 1544 1545 1546 1547 1548 1549 1550 1551 1552 1553 1554 1555 1556 1557 1558 1559 1560 1561 1562 1563 1564 1565 1566 1567 1568 1569 1570 1571 1572 1573 1574 1575 1576 1577 1578 1579 1580 1581 1582 1583 1584 1585 1586 1587 1588 1589 1590 1591 1592 1593 1594 1595 1596 1597 1598 1599 1600 1601 1602 1603 1604 1605 1606 1607 1608 1609 1610 1611 1612 1613 1614 1615 1616 1617 1618 1619 1620 1621 1622 1623 1624 1625 1626 1627 1628 1629 1630 1631 1632 1633 1634 1635 1636 1637 1638 1639 1640 1641 1642 1643 1644 1645 1646 1647 1648 1649 1650 1651 1652 1653 1654 1655 1656 1657 1658 1659 1660 1661 1662 1663 1664 1665 1666 1667 1668 1669 1670 1671 1672 1673 1674 1675 1676 1677 1678 1679 1680 1681 1682 1683 1684 1685 1686 1687 1688 1689 1690 1691 1692 1693 1694 1695 1696 1697 1698 1699 1700 1701 1702 1703 1704 1705 1706 1707 1708 1709 1710 1711 1712 1713 1714 1715 1716 1717 1718 1719 1720 1721 1722 1723 1724 1725 1726 1727 1728 1729 1730 1731 1732 1733 1734 1735 1736 1737 1738 1739 1740 1741 1742 1743 1744 1745 1746 1747 1748 1749 1750 1751 1752 1753 1754 1755 1756 1757 1758 1759 1760 1761 1762 1763 1764 1765 1766 1767 1768 1769 1770 1771 1772 1773 1774 1775 1776 1777 1778 1779 1780 1781 1782 1783 1784 1785 1786 1787 1788 1789 1790 1791 1792 1793 1794 1795 1796 1797 1798 1799 1800 1801 | /* $NetBSD: rtsock_shared.c,v 1.22 2022/07/01 21:22:23 riastradh Exp $ */ /* * Copyright (C) 1995, 1996, 1997, and 1998 WIDE Project. * All rights reserved. * * Redistribution and use in source and binary forms, with or without * modification, are permitted provided that the following conditions * are met: * 1. Redistributions of source code must retain the above copyright * notice, this list of conditions and the following disclaimer. * 2. Redistributions in binary form must reproduce the above copyright * notice, this list of conditions and the following disclaimer in the * documentation and/or other materials provided with the distribution. * 3. Neither the name of the project nor the names of its contributors * may be used to endorse or promote products derived from this software * without specific prior written permission. * * THIS SOFTWARE IS PROVIDED BY THE PROJECT AND CONTRIBUTORS ``AS IS'' AND * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE * ARE DISCLAIMED. IN NO EVENT SHALL THE PROJECT OR CONTRIBUTORS BE LIABLE * FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL * DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS * OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) * HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT * LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY * OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF * SUCH DAMAGE. */ /* * Copyright (c) 1988, 1991, 1993 * The Regents of the University of California. All rights reserved. * * Redistribution and use in source and binary forms, with or without * modification, are permitted provided that the following conditions * are met: * 1. Redistributions of source code must retain the above copyright * notice, this list of conditions and the following disclaimer. * 2. Redistributions in binary form must reproduce the above copyright * notice, this list of conditions and the following disclaimer in the * documentation and/or other materials provided with the distribution. * 3. Neither the name of the University nor the names of its contributors * may be used to endorse or promote products derived from this software * without specific prior written permission. * * THIS SOFTWARE IS PROVIDED BY THE REGENTS AND CONTRIBUTORS ``AS IS'' AND * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE * ARE DISCLAIMED. IN NO EVENT SHALL THE REGENTS OR CONTRIBUTORS BE LIABLE * FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL * DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS * OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) * HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT * LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY * OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF * SUCH DAMAGE. * * @(#)rtsock.c 8.7 (Berkeley) 10/12/95 */ #include <sys/cdefs.h> __KERNEL_RCSID(0, "$NetBSD: rtsock_shared.c,v 1.22 2022/07/01 21:22:23 riastradh Exp $"); #ifdef _KERNEL_OPT #include "opt_inet.h" #include "opt_net_mpsafe.h" #endif #include <sys/param.h> #include <sys/systm.h> #include <sys/proc.h> #include <sys/socket.h> #include <sys/socketvar.h> #include <sys/domain.h> #include <sys/protosw.h> #include <sys/sysctl.h> #include <sys/kauth.h> #include <sys/kmem.h> #include <sys/intr.h> #include <sys/condvar.h> #include <sys/compat_stub.h> #include <net/if.h> #include <net/if_llatbl.h> #include <net/if_types.h> #include <net/route.h> #include <net/raw_cb.h> #include <netinet/in_var.h> #include <netinet/if_inarp.h> #include <netmpls/mpls.h> #include <compat/net/if.h> #include <compat/net/route.h> /* sa_family is after sa_len, rest is data */ #define _SA_MINSIZE (offsetof(struct sockaddr, sa_family) + \ sizeof(((struct sockaddr *)0)->sa_family)) #ifdef COMPAT_RTSOCK /* * These are used when #include-d from compat/common/rtsock_50.c */ #define RTM_XVERSION RTM_OVERSION #define RTM_XNEWADDR RTM_ONEWADDR #define RTM_XDELADDR RTM_ODELADDR #define RTM_XCHGADDR RTM_OCHGADDR #define RT_XADVANCE(a,b) RT_OADVANCE(a,b) #define RT_XROUNDUP(n) RT_OROUNDUP(n) #define PF_XROUTE PF_OROUTE #define rt_xmsghdr rt_msghdr50 #define if_xmsghdr if_msghdr /* if_msghdr50 is for RTM_OIFINFO */ #define ifa_xmsghdr ifa_msghdr50 #define if_xannouncemsghdr if_announcemsghdr50 #define COMPATNAME(x) compat_50_ ## x #define DOMAINNAME "oroute" #define COMPATCALL(name, args) \ MODULE_HOOK_CALL_VOID(rtsock_ ## name ## _50_hook, args, __nothing); #define RTS_CTASSERT(x) __nothing CTASSERT(sizeof(struct ifa_xmsghdr) == 20); DOMAIN_DEFINE(compat_50_routedomain); /* forward declare and add to link set */ #else /* COMPAT_RTSOCK */ /* * These are used when #include-d from compat/common/rtsock_50.c */ #define RTM_XVERSION RTM_VERSION #define RTM_XNEWADDR RTM_NEWADDR #define RTM_XDELADDR RTM_DELADDR #define RTM_XCHGADDR RTM_CHGADDR #define RT_XADVANCE(a,b) RT_ADVANCE(a,b) #define RT_XROUNDUP(n) RT_ROUNDUP(n) #define PF_XROUTE PF_ROUTE #define rt_xmsghdr rt_msghdr #define if_xmsghdr if_msghdr #define ifa_xmsghdr ifa_msghdr #define if_xannouncemsghdr if_announcemsghdr #define COMPATNAME(x) x #define DOMAINNAME "route" #define COMPATCALL(name, args) __nothing; #define RTS_CTASSERT(x) CTASSERT(x) CTASSERT(sizeof(struct ifa_xmsghdr) == 32); DOMAIN_DEFINE(routedomain); /* forward declare and add to link set */ #endif /* COMPAT_RTSOCK */ #ifdef RTSOCK_DEBUG #define RT_IN_PRINT(info, b, a) (in_print((b), sizeof(b), \ &((const struct sockaddr_in *)(info)->rti_info[(a)])->sin_addr), (b)) #endif /* RTSOCK_DEBUG */ struct route_info COMPATNAME(route_info) = { .ri_dst = { .sa_len = 2, .sa_family = PF_XROUTE, }, .ri_src = { .sa_len = 2, .sa_family = PF_XROUTE, }, .ri_maxqlen = IFQ_MAXLEN, }; static void COMPATNAME(route_init)(void); static int COMPATNAME(route_output)(struct mbuf *, struct socket *); static int rt_xaddrs(u_char, const char *, const char *, struct rt_addrinfo *); static struct mbuf *rt_makeifannouncemsg(struct ifnet *, int, int, struct rt_addrinfo *); static int rt_msg2(int, struct rt_addrinfo *, void *, struct rt_walkarg *, int *); static void _rt_setmetrics(int, const struct rt_xmsghdr *, struct rtentry *); static void rtm_setmetrics(const struct rtentry *, struct rt_xmsghdr *); static void rt_adjustcount(int, int); static const struct protosw COMPATNAME(route_protosw)[]; struct routecb { struct rawcb rocb_rcb; unsigned int rocb_msgfilter; #define RTMSGFILTER(m) (1U << (m)) char *rocb_missfilter; size_t rocb_missfilterlen; }; #define sotoroutecb(so) ((struct routecb *)(so)->so_pcb) static struct rawcbhead rt_rawcb; #ifdef NET_MPSAFE static kmutex_t *rt_so_mtx; static bool rt_updating = false; static kcondvar_t rt_update_cv; #endif static void rt_adjustcount(int af, int cnt) { struct route_cb * const cb = &COMPATNAME(route_info).ri_cb; cb->any_count += cnt; switch (af) { case AF_INET: cb->ip_count += cnt; return; #ifdef INET6 case AF_INET6: cb->ip6_count += cnt; return; #endif case AF_MPLS: cb->mpls_count += cnt; return; } } static int COMPATNAME(route_filter)(struct mbuf *m, struct sockproto *proto, struct rawcb *rp) { struct routecb *rop = (struct routecb *)rp; struct rt_xmsghdr rtm; KASSERT(m != NULL); KASSERT(proto != NULL); KASSERT(rp != NULL); /* Wrong family for this socket. */ if (proto->sp_family != PF_ROUTE) return ENOPROTOOPT; /* If no filter set, just return. */ if (rop->rocb_msgfilter == 0 && rop->rocb_missfilterlen == 0) return 0; /* Ensure we can access rtm_type */ if (m->m_len < offsetof(struct rt_xmsghdr, rtm_type) + sizeof(rtm.rtm_type)) return EINVAL; m_copydata(m, offsetof(struct rt_xmsghdr, rtm_type), sizeof(rtm.rtm_type), &rtm.rtm_type); if (rtm.rtm_type >= sizeof(rop->rocb_msgfilter) * CHAR_BIT) return EINVAL; /* If the rtm type is filtered out, return a positive. */ if (rop->rocb_msgfilter != 0 && !(rop->rocb_msgfilter & RTMSGFILTER(rtm.rtm_type))) return EEXIST; if (rop->rocb_missfilterlen != 0 && rtm.rtm_type == RTM_MISS) { __CTASSERT(RTAX_DST == 0); struct sockaddr_storage ss; struct sockaddr *dst = (struct sockaddr *)&ss, *sa; char *cp = rop->rocb_missfilter; char *ep = cp + rop->rocb_missfilterlen; /* Ensure we can access sa_len */ if (m->m_pkthdr.len < sizeof(rtm) + _SA_MINSIZE) return EINVAL; m_copydata(m, sizeof(rtm) + offsetof(struct sockaddr, sa_len), sizeof(ss.ss_len), &ss.ss_len); if (ss.ss_len < _SA_MINSIZE || ss.ss_len > sizeof(ss) || m->m_pkthdr.len < sizeof(rtm) + ss.ss_len) return EINVAL; /* Copy out the destination sockaddr */ m_copydata(m, sizeof(rtm), ss.ss_len, &ss); /* Find a matching sockaddr in the filter */ while (cp < ep) { sa = (struct sockaddr *)cp; if (sa->sa_len == dst->sa_len && memcmp(sa, dst, sa->sa_len) == 0) break; cp += RT_XROUNDUP(sa->sa_len); } if (cp == ep) return EEXIST; } /* Passed the filter. */ return 0; } static void rt_pr_init(void) { LIST_INIT(&rt_rawcb); } static int COMPATNAME(route_attach)(struct socket *so, int proto) { struct rawcb *rp; struct routecb *rop; int s, error; KASSERT(sotorawcb(so) == NULL); rop = kmem_zalloc(sizeof(*rop), KM_SLEEP); rp = &rop->rocb_rcb; rp->rcb_len = sizeof(*rop); so->so_pcb = rp; s = splsoftnet(); #ifdef NET_MPSAFE KASSERT(so->so_lock == NULL); mutex_obj_hold(rt_so_mtx); so->so_lock = rt_so_mtx; solock(so); #endif if ((error = raw_attach(so, proto, &rt_rawcb)) == 0) { rt_adjustcount(rp->rcb_proto.sp_protocol, 1); rp->rcb_laddr = &COMPATNAME(route_info).ri_src; rp->rcb_faddr = &COMPATNAME(route_info).ri_dst; rp->rcb_filter = COMPATNAME(route_filter); } splx(s); if (error) { kmem_free(rop, sizeof(*rop)); so->so_pcb = NULL; return error; } soisconnected(so); so->so_options |= SO_USELOOPBACK; KASSERT(solocked(so)); return error; } static void COMPATNAME(route_detach)(struct socket *so) { struct rawcb *rp = sotorawcb(so); struct routecb *rop = (struct routecb *)rp; int s; KASSERT(rp != NULL); KASSERT(solocked(so)); s = splsoftnet(); if (rop->rocb_missfilterlen != 0) kmem_free(rop->rocb_missfilter, rop->rocb_missfilterlen); rt_adjustcount(rp->rcb_proto.sp_protocol, -1); raw_detach(so); splx(s); } static int COMPATNAME(route_accept)(struct socket *so, struct sockaddr *nam) { KASSERT(solocked(so)); panic("route_accept"); return EOPNOTSUPP; } static int COMPATNAME(route_bind)(struct socket *so, struct sockaddr *nam, struct lwp *l) { KASSERT(solocked(so)); return EOPNOTSUPP; } static int COMPATNAME(route_listen)(struct socket *so, struct lwp *l) { KASSERT(solocked(so)); return EOPNOTSUPP; } static int COMPATNAME(route_connect)(struct socket *so, struct sockaddr *nam, struct lwp *l) { KASSERT(solocked(so)); return EOPNOTSUPP; } static int COMPATNAME(route_connect2)(struct socket *so, struct socket *so2) { KASSERT(solocked(so)); return EOPNOTSUPP; } static int COMPATNAME(route_disconnect)(struct socket *so) { struct rawcb *rp = sotorawcb(so); int s; KASSERT(solocked(so)); KASSERT(rp != NULL); s = splsoftnet(); soisdisconnected(so); raw_disconnect(rp); splx(s); return 0; } static int COMPATNAME(route_shutdown)(struct socket *so) { int s; KASSERT(solocked(so)); /* * Mark the connection as being incapable of further input. */ s = splsoftnet(); socantsendmore(so); splx(s); return 0; } static int COMPATNAME(route_abort)(struct socket *so) { KASSERT(solocked(so)); panic("route_abort"); return EOPNOTSUPP; } static int COMPATNAME(route_ioctl)(struct socket *so, u_long cmd, void *nam, struct ifnet * ifp) { return EOPNOTSUPP; } static int COMPATNAME(route_stat)(struct socket *so, struct stat *ub) { KASSERT(solocked(so)); return 0; } static int COMPATNAME(route_peeraddr)(struct socket *so, struct sockaddr *nam) { struct rawcb *rp = sotorawcb(so); KASSERT(solocked(so)); KASSERT(rp != NULL); KASSERT(nam != NULL); if (rp->rcb_faddr == NULL) return ENOTCONN; raw_setpeeraddr(rp, nam); return 0; } static int COMPATNAME(route_sockaddr)(struct socket *so, struct sockaddr *nam) { struct rawcb *rp = sotorawcb(so); KASSERT(solocked(so)); KASSERT(rp != NULL); KASSERT(nam != NULL); if (rp->rcb_faddr == NULL) return ENOTCONN; raw_setsockaddr(rp, nam); return 0; } static int COMPATNAME(route_rcvd)(struct socket *so, int flags, struct lwp *l) { KASSERT(solocked(so)); return EOPNOTSUPP; } static int COMPATNAME(route_recvoob)(struct socket *so, struct mbuf *m, int flags) { KASSERT(solocked(so)); return EOPNOTSUPP; } static int COMPATNAME(route_send)(struct socket *so, struct mbuf *m, struct sockaddr *nam, struct mbuf *control, struct lwp *l) { int error = 0; int s; KASSERT(solocked(so)); KASSERT(so->so_proto == &COMPATNAME(route_protosw)[0]); s = splsoftnet(); error = raw_send(so, m, nam, control, l, &COMPATNAME(route_output)); splx(s); return error; } static int COMPATNAME(route_sendoob)(struct socket *so, struct mbuf *m, struct mbuf *control) { KASSERT(solocked(so)); m_freem(m); m_freem(control); return EOPNOTSUPP; } static int COMPATNAME(route_purgeif)(struct socket *so, struct ifnet *ifp) { panic("route_purgeif"); return EOPNOTSUPP; } #if defined(INET) || defined(INET6) static int route_get_sdl_index(struct rt_addrinfo *info, int *sdl_index) { struct rtentry *nrt; int error; error = rtrequest1(RTM_GET, info, &nrt); if (error != 0) return error; /* * nrt->rt_ifp->if_index may not be correct * due to changing to ifplo0. */ *sdl_index = satosdl(nrt->rt_gateway)->sdl_index; rt_unref(nrt); return 0; } #endif static void route_get_sdl(const struct ifnet *ifp, const struct sockaddr *dst, struct sockaddr_dl *sdl, int *flags) { struct llentry *la; KASSERT(ifp != NULL); IF_AFDATA_RLOCK(ifp); switch (dst->sa_family) { case AF_INET: la = lla_lookup(LLTABLE(ifp), 0, dst); break; case AF_INET6: la = lla_lookup(LLTABLE6(ifp), 0, dst); break; default: la = NULL; KASSERTMSG(0, "Invalid AF=%d\n", dst->sa_family); break; } IF_AFDATA_RUNLOCK(ifp); void *a = (LLE_IS_VALID(la) && (la->la_flags & LLE_VALID) == LLE_VALID) ? &la->ll_addr : NULL; a = sockaddr_dl_init(sdl, sizeof(*sdl), ifp->if_index, ifp->if_type, NULL, 0, a, ifp->if_addrlen); KASSERT(a != NULL); if (la != NULL) { *flags = la->la_flags; LLE_RUNLOCK(la); } } static int route_output_report(struct rtentry *rt, struct rt_addrinfo *info, struct rt_xmsghdr *rtm, struct rt_xmsghdr **new_rtm) { int len, error; if (rtm->rtm_addrs & (RTA_IFP | RTA_IFA)) { const struct ifaddr *rtifa; const struct ifnet *ifp = rt->rt_ifp; info->rti_info[RTAX_IFP] = ifp->if_dl->ifa_addr; /* rtifa used to be simply rt->rt_ifa. * If rt->rt_ifa != NULL, then * rt_get_ifa() != NULL. So this * ought to still be safe. --dyoung */ rtifa = rt_get_ifa(rt); info->rti_info[RTAX_IFA] = rtifa->ifa_addr; #ifdef RTSOCK_DEBUG if (info->rti_info[RTAX_IFA]->sa_family == AF_INET) { char ibuf[INET_ADDRSTRLEN]; char abuf[INET_ADDRSTRLEN]; printf("%s: copying out RTAX_IFA %s " "for info->rti_info[RTAX_DST] %s " "ifa_getifa %p ifa_seqno %p\n", __func__, RT_IN_PRINT(info, ibuf, RTAX_IFA), RT_IN_PRINT(info, abuf, RTAX_DST), (void *)rtifa->ifa_getifa, rtifa->ifa_seqno); } #endif /* RTSOCK_DEBUG */ if (ifp->if_flags & IFF_POINTOPOINT) info->rti_info[RTAX_BRD] = rtifa->ifa_dstaddr; else info->rti_info[RTAX_BRD] = NULL; rtm->rtm_index = ifp->if_index; } error = rt_msg2(rtm->rtm_type, info, NULL, NULL, &len); if (error) return error; if (len > rtm->rtm_msglen) { struct rt_xmsghdr *old_rtm = rtm; R_Malloc(*new_rtm, struct rt_xmsghdr *, len); if (*new_rtm == NULL) return ENOBUFS; (void)memcpy(*new_rtm, old_rtm, old_rtm->rtm_msglen); rtm = *new_rtm; } (void)rt_msg2(rtm->rtm_type, info, rtm, NULL, 0); rtm->rtm_flags = rt->rt_flags; rtm_setmetrics(rt, rtm); rtm->rtm_addrs = info->rti_addrs; return 0; } /*ARGSUSED*/ int COMPATNAME(route_output)(struct mbuf *m, struct socket *so) { struct sockproto proto = { .sp_family = PF_XROUTE, }; struct rt_xmsghdr hdr; struct rt_xmsghdr *rtm = NULL; struct rt_xmsghdr *old_rtm = NULL, *new_rtm = NULL; struct rtentry *rt = NULL; struct rtentry *saved_nrt = NULL; struct rt_addrinfo info; int len, error = 0; sa_family_t family; struct sockaddr_dl sdl; int bound = curlwp_bind(); bool do_rt_free = false; struct sockaddr_storage netmask; #define senderr(e) do { error = e; goto flush;} while (/*CONSTCOND*/ 0) if (m == NULL || ((m->m_len < sizeof(int32_t)) && (m = m_pullup(m, sizeof(int32_t))) == NULL)) { error = ENOBUFS; goto out; } if ((m->m_flags & M_PKTHDR) == 0) panic("%s", __func__); len = m->m_pkthdr.len; if (len < sizeof(*rtm)) { info.rti_info[RTAX_DST] = NULL; senderr(EINVAL); } m_copydata(m, 0, sizeof(hdr), &hdr); if (len != hdr.rtm_msglen) { info.rti_info[RTAX_DST] = NULL; senderr(EINVAL); } R_Malloc(rtm, struct rt_xmsghdr *, len); if (rtm == NULL) { info.rti_info[RTAX_DST] = NULL; senderr(ENOBUFS); } m_copydata(m, 0, len, rtm); if (rtm->rtm_version != RTM_XVERSION) { info.rti_info[RTAX_DST] = NULL; senderr(EPROTONOSUPPORT); } rtm->rtm_pid = curproc->p_pid; memset(&info, 0, sizeof(info)); info.rti_addrs = rtm->rtm_addrs; if (rt_xaddrs(rtm->rtm_type, (const char *)(rtm + 1), len + (char *)rtm, &info)) { senderr(EINVAL); } info.rti_flags = rtm->rtm_flags; if (info.rti_info[RTAX_DST] == NULL || (info.rti_info[RTAX_DST]->sa_family >= AF_MAX)) { senderr(EINVAL); } #ifdef RTSOCK_DEBUG if (info.rti_info[RTAX_DST]->sa_family == AF_INET) { char abuf[INET_ADDRSTRLEN]; printf("%s: extracted info.rti_info[RTAX_DST] %s\n", __func__, RT_IN_PRINT(&info, abuf, RTAX_DST)); } #endif /* RTSOCK_DEBUG */ if (info.rti_info[RTAX_GATEWAY] != NULL && (info.rti_info[RTAX_GATEWAY]->sa_family >= AF_MAX)) { senderr(EINVAL); } /* * Verify that the socket has the appropriate privilege; RTM_GET * is the only operation the non-superuser is allowed. */ if (kauth_authorize_network(so->so_cred, KAUTH_NETWORK_ROUTE, 0, rtm, NULL, NULL) != 0) senderr(EACCES); /* * route(8) passes a sockaddr truncated with prefixlen. * The kernel doesn't expect such sockaddr and need to * use a buffer that is big enough for the sockaddr expected * (padded with 0's). We keep the original length of the sockaddr. */ if (info.rti_info[RTAX_NETMASK]) { /* * Use the family of RTAX_DST, because RTAX_NETMASK * can have a zero family if it comes from the radix * tree via rt_mask(). */ socklen_t sa_len = sockaddr_getsize_by_family( info.rti_info[RTAX_DST]->sa_family); socklen_t masklen = sockaddr_getlen( info.rti_info[RTAX_NETMASK]); if (sa_len != 0 && sa_len > masklen) { KASSERT(sa_len <= sizeof(netmask)); memcpy(&netmask, info.rti_info[RTAX_NETMASK], masklen); memset((char *)&netmask + masklen, 0, sa_len - masklen); info.rti_info[RTAX_NETMASK] = sstocsa(&netmask); } } switch (rtm->rtm_type) { case RTM_ADD: if (info.rti_info[RTAX_GATEWAY] == NULL) { senderr(EINVAL); } #if defined(INET) || defined(INET6) /* support for new ARP/NDP code with keeping backcompat */ if (info.rti_info[RTAX_GATEWAY]->sa_family == AF_LINK) { const struct sockaddr_dl *sdlp = satocsdl(info.rti_info[RTAX_GATEWAY]); /* Allow routing requests by interface index */ if (sdlp->sdl_nlen == 0 && sdlp->sdl_alen == 0 && sdlp->sdl_slen == 0) goto fallback; /* * Old arp binaries don't set the sdl_index * so we have to complement it. */ int sdl_index = sdlp->sdl_index; if (sdl_index == 0) { error = route_get_sdl_index(&info, &sdl_index); if (error != 0) goto fallback; } else if ( info.rti_info[RTAX_DST]->sa_family == AF_INET) { /* * XXX workaround for SIN_PROXY case; proxy arp * entry should be in an interface that has * a network route including the destination, * not a local (link) route that may not be a * desired place, for example a tap. */ const struct sockaddr_inarp *sina = (const struct sockaddr_inarp *) info.rti_info[RTAX_DST]; if (sina->sin_other & SIN_PROXY) { error = route_get_sdl_index(&info, &sdl_index); if (error != 0) goto fallback; } } error = lla_rt_output(rtm->rtm_type, rtm->rtm_flags, rtm->rtm_rmx.rmx_expire, &info, sdl_index); break; } fallback: #endif /* defined(INET) || defined(INET6) */ error = rtrequest1(rtm->rtm_type, &info, &saved_nrt); if (error == 0) { _rt_setmetrics(rtm->rtm_inits, rtm, saved_nrt); rt_unref(saved_nrt); } break; case RTM_DELETE: #if defined(INET) || defined(INET6) /* support for new ARP/NDP code */ if (info.rti_info[RTAX_GATEWAY] && (info.rti_info[RTAX_GATEWAY]->sa_family == AF_LINK) && (rtm->rtm_flags & RTF_LLDATA) != 0) { const struct sockaddr_dl *sdlp = satocsdl(info.rti_info[RTAX_GATEWAY]); error = lla_rt_output(rtm->rtm_type, rtm->rtm_flags, rtm->rtm_rmx.rmx_expire, &info, sdlp->sdl_index); rtm->rtm_flags &= ~RTF_UP; break; } #endif error = rtrequest1(rtm->rtm_type, &info, &saved_nrt); if (error != 0) break; rt = saved_nrt; do_rt_free = true; info.rti_info[RTAX_DST] = rt_getkey(rt); info.rti_info[RTAX_GATEWAY] = rt->rt_gateway; info.rti_info[RTAX_NETMASK] = rt_mask(rt); info.rti_info[RTAX_TAG] = rt_gettag(rt); error = route_output_report(rt, &info, rtm, &new_rtm); if (error) senderr(error); if (new_rtm != NULL) { old_rtm = rtm; rtm = new_rtm; } break; case RTM_GET: case RTM_CHANGE: case RTM_LOCK: /* XXX This will mask info.rti_info[RTAX_DST] with * info.rti_info[RTAX_NETMASK] before * searching. It did not used to do that. --dyoung */ rt = NULL; error = rtrequest1(RTM_GET, &info, &rt); if (error != 0) senderr(error); if (rtm->rtm_type != RTM_GET) {/* XXX: too grotty */ if (memcmp(info.rti_info[RTAX_DST], rt_getkey(rt), info.rti_info[RTAX_DST]->sa_len) != 0) senderr(ESRCH); if (info.rti_info[RTAX_NETMASK] == NULL && rt_mask(rt) != NULL) senderr(ETOOMANYREFS); } /* * XXX if arp/ndp requests an L2 entry, we have to obtain * it from lltable while for the route command we have to * return a route as it is. How to distinguish them? * For newer arp/ndp, RTF_LLDATA flag set by arp/ndp * indicates an L2 entry is requested. For old arp/ndp * binaries, we check RTF_UP flag is NOT set; it works * by the fact that arp/ndp don't set it while the route * command sets it. */ if (((rtm->rtm_flags & RTF_LLDATA) != 0 || (rtm->rtm_flags & RTF_UP) == 0) && rtm->rtm_type == RTM_GET && sockaddr_cmp(rt_getkey(rt), info.rti_info[RTAX_DST]) != 0) { int ll_flags = 0; route_get_sdl(rt->rt_ifp, info.rti_info[RTAX_DST], &sdl, &ll_flags); info.rti_info[RTAX_GATEWAY] = sstocsa(&sdl); error = route_output_report(rt, &info, rtm, &new_rtm); if (error) senderr(error); if (new_rtm != NULL) { old_rtm = rtm; rtm = new_rtm; } rtm->rtm_flags |= RTF_LLDATA; rtm->rtm_flags &= ~RTF_CONNECTED; rtm->rtm_flags |= (ll_flags & LLE_STATIC) ? RTF_STATIC : 0; break; } switch (rtm->rtm_type) { case RTM_GET: info.rti_info[RTAX_DST] = rt_getkey(rt); info.rti_info[RTAX_GATEWAY] = rt->rt_gateway; info.rti_info[RTAX_NETMASK] = rt_mask(rt); info.rti_info[RTAX_TAG] = rt_gettag(rt); error = route_output_report(rt, &info, rtm, &new_rtm); if (error) senderr(error); if (new_rtm != NULL) { old_rtm = rtm; rtm = new_rtm; } break; case RTM_CHANGE: #ifdef NET_MPSAFE /* * Release rt_so_mtx to avoid a deadlock with route_intr * and also serialize updating routes to avoid another. */ if (rt_updating) { /* Release to allow the updater to proceed */ rt_unref(rt); rt = NULL; } while (rt_updating) { error = cv_wait_sig(&rt_update_cv, rt_so_mtx); if (error != 0) goto flush; } if (rt == NULL) { error = rtrequest1(RTM_GET, &info, &rt); if (error != 0) goto flush; } rt_updating = true; mutex_exit(rt_so_mtx); error = rt_update_prepare(rt); if (error == 0) { error = rt_update(rt, &info, rtm); rt_update_finish(rt); } mutex_enter(rt_so_mtx); rt_updating = false; cv_broadcast(&rt_update_cv); #else error = rt_update(rt, &info, rtm); #endif if (error != 0) goto flush; /*FALLTHROUGH*/ case RTM_LOCK: rt->rt_rmx.rmx_locks &= ~(rtm->rtm_inits); rt->rt_rmx.rmx_locks |= (rtm->rtm_inits & rtm->rtm_rmx.rmx_locks); break; } break; default: senderr(EOPNOTSUPP); } flush: if (rtm) { if (error) rtm->rtm_errno = error; else rtm->rtm_flags |= RTF_DONE; } family = info.rti_info[RTAX_DST] ? info.rti_info[RTAX_DST]->sa_family : 0; /* We cannot free old_rtm until we have stopped using the * pointers in info, some of which may point to sockaddrs * in old_rtm. */ if (old_rtm != NULL) Free(old_rtm); if (rt) { if (do_rt_free) { #ifdef NET_MPSAFE /* * Release rt_so_mtx to avoid a deadlock with * route_intr. */ mutex_exit(rt_so_mtx); rt_free(rt); mutex_enter(rt_so_mtx); #else rt_free(rt); #endif } else rt_unref(rt); } { struct rawcb *rp = NULL; /* * Check to see if we don't want our own messages. */ if ((so->so_options & SO_USELOOPBACK) == 0) { if (COMPATNAME(route_info).ri_cb.any_count <= 1) { if (rtm) Free(rtm); m_freem(m); goto out; } /* There is another listener, so construct message */ rp = sotorawcb(so); } if (rtm) { m_copyback(m, 0, rtm->rtm_msglen, rtm); if (m->m_pkthdr.len < rtm->rtm_msglen) { m_freem(m); m = NULL; } else if (m->m_pkthdr.len > rtm->rtm_msglen) m_adj(m, rtm->rtm_msglen - m->m_pkthdr.len); Free(rtm); } if (rp) rp->rcb_proto.sp_family = 0; /* Avoid us */ if (family) proto.sp_protocol = family; if (m) raw_input(m, &proto, &COMPATNAME(route_info).ri_src, &COMPATNAME(route_info).ri_dst, &rt_rawcb); if (rp) rp->rcb_proto.sp_family = PF_XROUTE; } out: curlwp_bindx(bound); return error; } static int route_ctloutput(int op, struct socket *so, struct sockopt *sopt) { struct routecb *rop = sotoroutecb(so); int error = 0; unsigned char *rtm_type, *cp, *ep; size_t len; unsigned int msgfilter; struct sockaddr *sa; KASSERT(solocked(so)); if (sopt->sopt_level != AF_ROUTE) { error = ENOPROTOOPT; } else switch (op) { case PRCO_SETOPT: switch (sopt->sopt_name) { case RO_MSGFILTER: msgfilter = 0; for (rtm_type = sopt->sopt_data, len = sopt->sopt_size; len != 0; rtm_type++, len -= sizeof(*rtm_type)) { /* Guard against overflowing our storage. */ if (*rtm_type >= sizeof(msgfilter) * CHAR_BIT) { error = EOVERFLOW; break; } msgfilter |= RTMSGFILTER(*rtm_type); } if (error == 0) rop->rocb_msgfilter = msgfilter; break; case RO_MISSFILTER: /* Validate the data */ len = 0; cp = sopt->sopt_data; ep = cp + sopt->sopt_size; while (cp < ep) { if (ep - cp < offsetof(struct sockaddr, sa_len) + sizeof(sa->sa_len)) break; if (++len > RO_FILTSA_MAX) { error = ENOBUFS; break; } sa = (struct sockaddr *)cp; if (sa->sa_len < _SA_MINSIZE || sa->sa_len >sizeof(struct sockaddr_storage)) return EINVAL; cp += RT_XROUNDUP(sa->sa_len); } if (cp != ep) { if (error == 0) error = EINVAL; break; } if (rop->rocb_missfilterlen != 0) kmem_free(rop->rocb_missfilter, rop->rocb_missfilterlen); if (sopt->sopt_size != 0) { rop->rocb_missfilter = kmem_alloc(sopt->sopt_size, KM_SLEEP); if (rop->rocb_missfilter == NULL) { rop->rocb_missfilterlen = 0; error = ENOBUFS; break; } } else rop->rocb_missfilter = NULL; rop->rocb_missfilterlen = sopt->sopt_size; if (rop->rocb_missfilterlen != 0) memcpy(rop->rocb_missfilter, sopt->sopt_data, rop->rocb_missfilterlen); break; default: error = ENOPROTOOPT; break; } break; case PRCO_GETOPT: switch (sopt->sopt_name) { case RO_MSGFILTER: error = ENOTSUP; break; default: error = ENOPROTOOPT; break; } } return error; } static void _rt_setmetrics(int which, const struct rt_xmsghdr *in, struct rtentry *out) { #define metric(f, e) if (which & (f)) out->rt_rmx.e = in->rtm_rmx.e; metric(RTV_RPIPE, rmx_recvpipe); metric(RTV_SPIPE, rmx_sendpipe); metric(RTV_SSTHRESH, rmx_ssthresh); metric(RTV_RTT, rmx_rtt); metric(RTV_RTTVAR, rmx_rttvar); metric(RTV_HOPCOUNT, rmx_hopcount); metric(RTV_MTU, rmx_mtu); #undef metric if (which & RTV_EXPIRE) { out->rt_rmx.rmx_expire = in->rtm_rmx.rmx_expire ? time_wall_to_mono(in->rtm_rmx.rmx_expire) : 0; } } static void rtm_setmetrics(const struct rtentry *in, struct rt_xmsghdr *out) { #define metric(e) out->rtm_rmx.e = in->rt_rmx.e; metric(rmx_recvpipe); metric(rmx_sendpipe); metric(rmx_ssthresh); metric(rmx_rtt); metric(rmx_rttvar); metric(rmx_hopcount); metric(rmx_mtu); metric(rmx_locks); #undef metric out->rtm_rmx.rmx_expire = in->rt_rmx.rmx_expire ? time_mono_to_wall(in->rt_rmx.rmx_expire) : 0; } static int rt_xaddrs(u_char rtmtype, const char *cp, const char *cplim, struct rt_addrinfo *rtinfo) { const struct sockaddr *sa = NULL; /* Quell compiler warning */ int i; for (i = 0; i < RTAX_MAX && cp < cplim; i++) { if ((rtinfo->rti_addrs & (1 << i)) == 0) continue; rtinfo->rti_info[i] = sa = (const struct sockaddr *)cp; RT_XADVANCE(cp, sa); } /* * Check for extra addresses specified, except RTM_GET asking * for interface info. */ if (rtmtype == RTM_GET) { if (((rtinfo->rti_addrs & (~((1 << RTAX_IFP) | (1 << RTAX_IFA)))) & (~0U << i)) != 0) return 1; } else if ((rtinfo->rti_addrs & (~0U << i)) != 0) return 1; /* Check for bad data length. */ if (cp != cplim) { if (i == RTAX_NETMASK + 1 && sa != NULL && cp - RT_XROUNDUP(sa->sa_len) + sa->sa_len == cplim) /* * The last sockaddr was info.rti_info[RTAX_NETMASK]. * We accept this for now for the sake of old * binaries or third party softwares. */ ; else return 1; } return 0; } static int rt_getlen(int type) { RTS_CTASSERT(__alignof(struct ifa_msghdr) >= sizeof(uint64_t)); RTS_CTASSERT(__alignof(struct if_msghdr) >= sizeof(uint64_t)); RTS_CTASSERT(__alignof(struct if_announcemsghdr) >= sizeof(uint64_t)); RTS_CTASSERT(__alignof(struct rt_msghdr) >= sizeof(uint64_t)); switch (type) { case RTM_ODELADDR: case RTM_ONEWADDR: case RTM_OCHGADDR: if (rtsock_iflist_70_hook.hooked) return sizeof(struct ifa_msghdr70); else { #ifdef RTSOCK_DEBUG printf("%s: unsupported RTM type %d\n", __func__, type); #endif return -1; } case RTM_DELADDR: case RTM_NEWADDR: case RTM_CHGADDR: return sizeof(struct ifa_xmsghdr); case RTM_OOIFINFO: if (rtsock_iflist_14_hook.hooked) return sizeof(struct if_msghdr14); else { #ifdef RTSOCK_DEBUG printf("%s: unsupported RTM type RTM_OOIFINFO\n", __func__); #endif return -1; } case RTM_OIFINFO: if (rtsock_iflist_50_hook.hooked) return sizeof(struct if_msghdr50); else { #ifdef RTSOCK_DEBUG printf("%s: unsupported RTM type RTM_OIFINFO\n", __func__); #endif return -1; } case RTM_IFINFO: return sizeof(struct if_xmsghdr); case RTM_IFANNOUNCE: case RTM_IEEE80211: return sizeof(struct if_xannouncemsghdr); default: return sizeof(struct rt_xmsghdr); } } struct mbuf * COMPATNAME(rt_msg1)(int type, struct rt_addrinfo *rtinfo, void *data, int datalen) { struct rt_xmsghdr *rtm; struct mbuf *m; int i; const struct sockaddr *sa; int len, dlen; m = m_gethdr(M_DONTWAIT, MT_DATA); if (m == NULL) return m; MCLAIM(m, &COMPATNAME(routedomain).dom_mowner); if ((len = rt_getlen(type)) == -1) goto out; if (len > MHLEN + MLEN) panic("%s: message too long", __func__); else if (len > MHLEN) { m->m_next = m_get(M_DONTWAIT, MT_DATA); if (m->m_next == NULL) goto out; MCLAIM(m->m_next, m->m_owner); m->m_pkthdr.len = len; m->m_len = MHLEN; m->m_next->m_len = len - MHLEN; } else { m->m_pkthdr.len = m->m_len = len; } m_reset_rcvif(m); m_copyback(m, 0, datalen, data); if (len > datalen) (void)memset(mtod(m, char *) + datalen, 0, len - datalen); rtm = mtod(m, struct rt_xmsghdr *); for (i = 0; i < RTAX_MAX; i++) { if ((sa = rtinfo->rti_info[i]) == NULL) continue; rtinfo->rti_addrs |= (1 << i); dlen = RT_XROUNDUP(sa->sa_len); m_copyback(m, len, sa->sa_len, sa); if (dlen != sa->sa_len) { /* * Up to 7 + 1 nul's since roundup is to * sizeof(uint64_t) (8 bytes) */ m_copyback(m, len + sa->sa_len, dlen - sa->sa_len, "\0\0\0\0\0\0\0"); } len += dlen; } if (m->m_pkthdr.len != len) goto out; rtm->rtm_msglen = len; rtm->rtm_version = RTM_XVERSION; rtm->rtm_type = type; return m; out: m_freem(m); return NULL; } /* * rt_msg2 * * fills 'cp' or 'w'.w_tmem with the routing socket message and * returns the length of the message in 'lenp'. * * if walkarg is 0, cp is expected to be 0 or a buffer large enough to hold * the message * otherwise walkarg's w_needed is updated and if the user buffer is * specified and w_needed indicates space exists the information is copied * into the temp space (w_tmem). w_tmem is [re]allocated if necessary, * if the allocation fails ENOBUFS is returned. */ static int rt_msg2(int type, struct rt_addrinfo *rtinfo, void *cpv, struct rt_walkarg *w, int *lenp) { int i; int len, dlen, second_time = 0; char *cp0, *cp = cpv; rtinfo->rti_addrs = 0; again: if ((len = rt_getlen(type)) == -1) return EINVAL; if ((cp0 = cp) != NULL) cp += len; for (i = 0; i < RTAX_MAX; i++) { const struct sockaddr *sa; if ((sa = rtinfo->rti_info[i]) == NULL) continue; rtinfo->rti_addrs |= (1 << i); dlen = RT_XROUNDUP(sa->sa_len); if (cp) { int diff = dlen - sa->sa_len; (void)memcpy(cp, sa, (size_t)sa->sa_len); cp += sa->sa_len; if (diff > 0) { (void)memset(cp, 0, (size_t)diff); cp += diff; } } len += dlen; } if (cp == NULL && w != NULL && !second_time) { struct rt_walkarg *rw = w; rw->w_needed += len; if (rw->w_needed <= 0 && rw->w_where) { if (rw->w_tmemsize < len) { if (rw->w_tmem) kmem_free(rw->w_tmem, rw->w_tmemsize); rw->w_tmem = kmem_zalloc(len, KM_SLEEP); rw->w_tmemsize = len; } if (rw->w_tmem) { cp = rw->w_tmem; second_time = 1; goto again; } else { rw->w_tmemneeded = len; return ENOBUFS; } } } if (cp) { struct rt_xmsghdr *rtm = (struct rt_xmsghdr *)cp0; rtm->rtm_version = RTM_XVERSION; rtm->rtm_type = type; rtm->rtm_msglen = len; } if (lenp) *lenp = len; return 0; } /* * This routine is called to generate a message from the routing * socket indicating that a redirect has occurred, a routing lookup * has failed, or that a protocol has detected timeouts to a particular * destination. */ void COMPATNAME(rt_missmsg)(int type, const struct rt_addrinfo *rtinfo, int flags, int error) { struct rt_xmsghdr rtm; struct mbuf *m; const struct sockaddr *sa = rtinfo->rti_info[RTAX_DST]; struct rt_addrinfo info = *rtinfo; COMPATCALL(rt_missmsg, (type, rtinfo, flags, error)); if (COMPATNAME(route_info).ri_cb.any_count == 0) return; memset(&rtm, 0, sizeof(rtm)); rtm.rtm_pid = curproc->p_pid; rtm.rtm_flags = RTF_DONE | flags; rtm.rtm_errno = error; m = COMPATNAME(rt_msg1)(type, &info, &rtm, sizeof(rtm)); if (m == NULL) return; mtod(m, struct rt_xmsghdr *)->rtm_addrs = info.rti_addrs; COMPATNAME(route_enqueue)(m, sa ? sa->sa_family : 0); } /* * This routine is called to generate a message from the routing * socket indicating that the status of a network interface has changed. */ void COMPATNAME(rt_ifmsg)(struct ifnet *ifp) { struct if_xmsghdr ifm; struct mbuf *m; struct rt_addrinfo info; COMPATCALL(rt_ifmsg, (ifp)); if (COMPATNAME(route_info).ri_cb.any_count == 0) return; (void)memset(&info, 0, sizeof(info)); (void)memset(&ifm, 0, sizeof(ifm)); ifm.ifm_index = ifp->if_index; ifm.ifm_flags = ifp->if_flags; if_export_if_data(ifp, &ifm.ifm_data, false); ifm.ifm_addrs = 0; m = COMPATNAME(rt_msg1)(RTM_IFINFO, &info, &ifm, sizeof(ifm)); if (m == NULL) return; COMPATNAME(route_enqueue)(m, 0); MODULE_HOOK_CALL_VOID(rtsock_oifmsg_14_hook, (ifp), __nothing); MODULE_HOOK_CALL_VOID(rtsock_oifmsg_50_hook, (ifp), __nothing); } /* * This is called to generate messages from the routing socket * indicating a network interface has had addresses associated with it. * if we ever reverse the logic and replace messages TO the routing * socket indicate a request to configure interfaces, then it will * be unnecessary as the routing socket will automatically generate * copies of it. */ static void COMPATNAME(rt_addrmsg0)(int cmd, struct ifaddr *ifa, int error, struct rtentry *rt, const struct sockaddr *src) { #define cmdpass(__cmd, __pass) (((__cmd) << 2) | (__pass)) struct rt_addrinfo info; const struct sockaddr *sa; int pass; struct mbuf *m; struct ifnet *ifp; struct rt_xmsghdr rtm; struct ifa_xmsghdr ifam; int ncmd; KASSERT(ifa != NULL); KASSERT(ifa->ifa_addr != NULL); ifp = ifa->ifa_ifp; if (cmd == RTM_ADD && vec_sctp_add_ip_address != NULL) { (*vec_sctp_add_ip_address)(ifa); } else if (cmd == RTM_DELETE && vec_sctp_delete_ip_address != NULL) { (*vec_sctp_delete_ip_address)(ifa); } COMPATCALL(rt_addrmsg_rt, (cmd, ifa, error, rt)); if (COMPATNAME(route_info).ri_cb.any_count == 0) return; for (pass = 1; pass < 3; pass++) { memset(&info, 0, sizeof(info)); switch (cmdpass(cmd, pass)) { case cmdpass(RTM_ADD, 1): case cmdpass(RTM_CHANGE, 1): case cmdpass(RTM_DELETE, 2): case cmdpass(RTM_NEWADDR, 1): case cmdpass(RTM_DELADDR, 1): case cmdpass(RTM_CHGADDR, 1): switch (cmd) { case RTM_ADD: ncmd = RTM_XNEWADDR; break; case RTM_DELETE: ncmd = RTM_XDELADDR; break; case RTM_CHANGE: ncmd = RTM_XCHGADDR; break; case RTM_NEWADDR: ncmd = RTM_XNEWADDR; break; case RTM_DELADDR: ncmd = RTM_XDELADDR; break; case RTM_CHGADDR: ncmd = RTM_XCHGADDR; break; default: panic("%s: unknown command %d", __func__, cmd); } MODULE_HOOK_CALL_VOID(rtsock_newaddr_70_hook, (ncmd, ifa), __nothing); info.rti_info[RTAX_IFA] = sa = ifa->ifa_addr; KASSERT(ifp->if_dl != NULL); info.rti_info[RTAX_IFP] = ifp->if_dl->ifa_addr; info.rti_info[RTAX_NETMASK] = ifa->ifa_netmask; info.rti_info[RTAX_BRD] = ifa->ifa_dstaddr; info.rti_info[RTAX_AUTHOR] = src; memset(&ifam, 0, sizeof(ifam)); ifam.ifam_index = ifp->if_index; ifam.ifam_metric = ifa->ifa_metric; ifam.ifam_flags = ifa->ifa_flags; #ifndef COMPAT_RTSOCK ifam.ifam_pid = curproc->p_pid; ifam.ifam_addrflags = if_addrflags(ifa); #endif m = COMPATNAME(rt_msg1)(ncmd, &info, &ifam, sizeof(ifam)); if (m == NULL) continue; mtod(m, struct ifa_xmsghdr *)->ifam_addrs = info.rti_addrs; break; case cmdpass(RTM_ADD, 2): case cmdpass(RTM_CHANGE, 2): case cmdpass(RTM_DELETE, 1): if (rt == NULL) continue; info.rti_info[RTAX_NETMASK] = rt_mask(rt); info.rti_info[RTAX_DST] = sa = rt_getkey(rt); info.rti_info[RTAX_GATEWAY] = rt->rt_gateway; memset(&rtm, 0, sizeof(rtm)); rtm.rtm_pid = curproc->p_pid; rtm.rtm_index = ifp->if_index; rtm.rtm_flags |= rt->rt_flags; rtm.rtm_errno = error; m = COMPATNAME(rt_msg1)(cmd, &info, &rtm, sizeof(rtm)); if (m == NULL) continue; mtod(m, struct rt_xmsghdr *)->rtm_addrs = info.rti_addrs; break; default: continue; } KASSERTMSG(m != NULL, "called with wrong command"); COMPATNAME(route_enqueue)(m, sa ? sa->sa_family : 0); } #undef cmdpass } void COMPATNAME(rt_addrmsg)(int cmd, struct ifaddr *ifa) { COMPATNAME(rt_addrmsg0)(cmd, ifa, 0, NULL, NULL); } void COMPATNAME(rt_addrmsg_rt)(int cmd, struct ifaddr *ifa, int error, struct rtentry *rt) { COMPATNAME(rt_addrmsg0)(cmd, ifa, error, rt, NULL); } void COMPATNAME(rt_addrmsg_src)(int cmd, struct ifaddr *ifa, const struct sockaddr *src) { COMPATNAME(rt_addrmsg0)(cmd, ifa, 0, NULL, src); } static struct mbuf * rt_makeifannouncemsg(struct ifnet *ifp, int type, int what, struct rt_addrinfo *info) { struct if_xannouncemsghdr ifan; memset(info, 0, sizeof(*info)); memset(&ifan, 0, sizeof(ifan)); ifan.ifan_index = ifp->if_index; strlcpy(ifan.ifan_name, ifp->if_xname, sizeof(ifan.ifan_name)); ifan.ifan_what = what; return COMPATNAME(rt_msg1)(type, info, &ifan, sizeof(ifan)); } /* * This is called to generate routing socket messages indicating * network interface arrival and departure. */ void COMPATNAME(rt_ifannouncemsg)(struct ifnet *ifp, int what) { struct mbuf *m; struct rt_addrinfo info; COMPATCALL(rt_ifannouncemsg, (ifp, what)); if (COMPATNAME(route_info).ri_cb.any_count == 0) return; m = rt_makeifannouncemsg(ifp, RTM_IFANNOUNCE, what, &info); if (m == NULL) return; COMPATNAME(route_enqueue)(m, 0); } /* * This is called to generate routing socket messages indicating * IEEE80211 wireless events. * XXX we piggyback on the RTM_IFANNOUNCE msg format in a clumsy way. */ void COMPATNAME(rt_ieee80211msg)(struct ifnet *ifp, int what, void *data, size_t data_len) { struct mbuf *m; struct rt_addrinfo info; COMPATCALL(rt_ieee80211msg, (ifp, what, data, data_len)); if (COMPATNAME(route_info).ri_cb.any_count == 0) return; m = rt_makeifannouncemsg(ifp, RTM_IEEE80211, what, &info); if (m == NULL) return; /* * Append the ieee80211 data. Try to stick it in the * mbuf containing the ifannounce msg; otherwise allocate * a new mbuf and append. * * NB: we assume m is a single mbuf. */ if (data_len > M_TRAILINGSPACE(m)) { struct mbuf *n = m_get(M_NOWAIT, MT_DATA); if (n == NULL) { m_freem(m); return; } (void)memcpy(mtod(n, void *), data, data_len); n->m_len = data_len; m->m_next = n; } else if (data_len > 0) { (void)memcpy(mtod(m, uint8_t *) + m->m_len, data, data_len); m->m_len += data_len; } if (m->m_flags & M_PKTHDR) m->m_pkthdr.len += data_len; mtod(m, struct if_xannouncemsghdr *)->ifan_msglen += data_len; COMPATNAME(route_enqueue)(m, 0); } /* * Routing message software interrupt routine */ static void COMPATNAME(route_intr)(void *cookie) { struct sockproto proto = { .sp_family = PF_XROUTE, }; struct route_info * const ri = &COMPATNAME(route_info); struct mbuf *m; SOFTNET_KERNEL_LOCK_UNLESS_NET_MPSAFE(); for (;;) { IFQ_LOCK(&ri->ri_intrq); IF_DEQUEUE(&ri->ri_intrq, m); IFQ_UNLOCK(&ri->ri_intrq); if (m == NULL) break; proto.sp_protocol = M_GETCTX(m, uintptr_t); #ifdef NET_MPSAFE mutex_enter(rt_so_mtx); #endif raw_input(m, &proto, &ri->ri_src, &ri->ri_dst, &rt_rawcb); #ifdef NET_MPSAFE mutex_exit(rt_so_mtx); #endif } SOFTNET_KERNEL_UNLOCK_UNLESS_NET_MPSAFE(); } /* * Enqueue a message to the software interrupt routine. */ void COMPATNAME(route_enqueue)(struct mbuf *m, int family) { struct route_info * const ri = &COMPATNAME(route_info); int wasempty; IFQ_LOCK(&ri->ri_intrq); if (IF_QFULL(&ri->ri_intrq)) { printf("%s: queue full, dropped message\n", __func__); IF_DROP(&ri->ri_intrq); IFQ_UNLOCK(&ri->ri_intrq); m_freem(m); } else { wasempty = IF_IS_EMPTY(&ri->ri_intrq); M_SETCTX(m, (uintptr_t)family); IF_ENQUEUE(&ri->ri_intrq, m); IFQ_UNLOCK(&ri->ri_intrq); if (wasempty) { kpreempt_disable(); softint_schedule(ri->ri_sih); kpreempt_enable(); } } } static void COMPATNAME(route_init)(void) { struct route_info * const ri = &COMPATNAME(route_info); #ifndef COMPAT_RTSOCK rt_init(); #ifdef NET_MPSAFE rt_so_mtx = mutex_obj_alloc(MUTEX_DEFAULT, IPL_NONE); cv_init(&rt_update_cv, "rtsock_cv"); #endif sysctl_net_route_setup(NULL, PF_ROUTE, "rtable"); #endif ri->ri_intrq.ifq_maxlen = ri->ri_maxqlen; ri->ri_sih = softint_establish(SOFTINT_NET | SOFTINT_MPSAFE, COMPATNAME(route_intr), NULL); IFQ_LOCK_INIT(&ri->ri_intrq); #ifdef MBUFTRACE MOWNER_ATTACH(&COMPATNAME(routedomain).dom_mowner); #endif } /* * Definitions of protocols supported in the ROUTE domain. */ #ifndef COMPAT_RTSOCK PR_WRAP_USRREQS(route); #else PR_WRAP_USRREQS(compat_50_route); #endif static const struct pr_usrreqs route_usrreqs = { .pr_attach = COMPATNAME(route_attach_wrapper), .pr_detach = COMPATNAME(route_detach_wrapper), .pr_accept = COMPATNAME(route_accept_wrapper), .pr_bind = COMPATNAME(route_bind_wrapper), .pr_listen = COMPATNAME(route_listen_wrapper), .pr_connect = COMPATNAME(route_connect_wrapper), .pr_connect2 = COMPATNAME(route_connect2_wrapper), .pr_disconnect = COMPATNAME(route_disconnect_wrapper), .pr_shutdown = COMPATNAME(route_shutdown_wrapper), .pr_abort = COMPATNAME(route_abort_wrapper), .pr_ioctl = COMPATNAME(route_ioctl_wrapper), .pr_stat = COMPATNAME(route_stat_wrapper), .pr_peeraddr = COMPATNAME(route_peeraddr_wrapper), .pr_sockaddr = COMPATNAME(route_sockaddr_wrapper), .pr_rcvd = COMPATNAME(route_rcvd_wrapper), .pr_recvoob = COMPATNAME(route_recvoob_wrapper), .pr_send = COMPATNAME(route_send_wrapper), .pr_sendoob = COMPATNAME(route_sendoob_wrapper), .pr_purgeif = COMPATNAME(route_purgeif_wrapper), }; static const struct protosw COMPATNAME(route_protosw)[] = { { .pr_type = SOCK_RAW, .pr_domain = &COMPATNAME(routedomain), .pr_flags = PR_ATOMIC|PR_ADDR, .pr_ctlinput = raw_ctlinput, .pr_ctloutput = route_ctloutput, .pr_usrreqs = &route_usrreqs, .pr_init = rt_pr_init, }, }; struct domain COMPATNAME(routedomain) = { .dom_family = PF_XROUTE, .dom_name = DOMAINNAME, .dom_init = COMPATNAME(route_init), .dom_protosw = COMPATNAME(route_protosw), .dom_protoswNPROTOSW = &COMPATNAME(route_protosw)[__arraycount(COMPATNAME(route_protosw))], #ifdef MBUFTRACE .dom_mowner = MOWNER_INIT("route", "rtm"), #endif }; |
| 19 19 2 12 17 11 26 26 1 1 44 44 16 1 2 1 22 28 12 13 21 24 7 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 197 198 199 200 201 202 203 204 205 206 207 208 209 210 211 212 213 214 215 216 217 218 219 220 221 222 223 224 225 226 227 228 229 230 231 232 233 234 235 236 237 238 239 240 241 242 243 244 245 246 247 248 249 250 251 252 253 254 255 256 257 258 259 260 261 262 263 264 265 266 267 268 269 270 271 272 273 274 275 276 277 278 279 280 281 282 283 284 285 286 287 288 289 290 291 292 293 294 295 296 297 298 299 300 301 302 303 304 305 306 307 308 309 310 311 312 313 314 315 316 317 318 319 320 321 322 323 324 325 326 327 328 329 330 331 332 333 334 335 336 337 338 339 340 341 342 343 344 345 346 347 348 349 350 351 352 353 354 355 356 357 358 359 360 361 362 363 364 365 366 367 | /* $NetBSD: uvm_readahead.c,v 1.13 2020/05/19 21:45:35 ad Exp $ */ /*- * Copyright (c)2003, 2005, 2009 YAMAMOTO Takashi, * All rights reserved. * * Redistribution and use in source and binary forms, with or without * modification, are permitted provided that the following conditions * are met: * 1. Redistributions of source code must retain the above copyright * notice, this list of conditions and the following disclaimer. * 2. Redistributions in binary form must reproduce the above copyright * notice, this list of conditions and the following disclaimer in the * documentation and/or other materials provided with the distribution. * * THIS SOFTWARE IS PROVIDED BY THE AUTHOR AND CONTRIBUTORS ``AS IS'' AND * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE * ARE DISCLAIMED. IN NO EVENT SHALL THE AUTHOR OR CONTRIBUTORS BE LIABLE * FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL * DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS * OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) * HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT * LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY * OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF * SUCH DAMAGE. */ /* * uvm_object read-ahead * * TODO: * - tune. * - handle multiple streams. * - find a better way to deal with PGO_LOCKED pager requests. * (currently just ignored) * - consider the amount of memory in the system. * - consider the speed of the underlying device. * - consider filesystem block size / block layout. */ #include <sys/cdefs.h> __KERNEL_RCSID(0, "$NetBSD: uvm_readahead.c,v 1.13 2020/05/19 21:45:35 ad Exp $"); #include <sys/param.h> #include <sys/pool.h> #include <uvm/uvm.h> #include <uvm/uvm_readahead.h> #if defined(READAHEAD_DEBUG) #define DPRINTF(a) printf a #else /* defined(READAHEAD_DEBUG) */ #define DPRINTF(a) /* nothing */ #endif /* defined(READAHEAD_DEBUG) */ /* * uvm_ractx: read-ahead context. */ struct uvm_ractx { int ra_flags; #define RA_VALID 1 off_t ra_winstart; /* window start offset */ size_t ra_winsize; /* window size */ off_t ra_next; /* next offset to read-ahead */ }; #if defined(sun2) || defined(sun3) /* XXX: on sun2 and sun3 MAXPHYS is 0xe000 */ #undef MAXPHYS #define MAXPHYS 0x8000 /* XXX */ #endif #define RA_WINSIZE_INIT MAXPHYS /* initial window size */ #define RA_WINSIZE_MAX (MAXPHYS * 16) /* max window size */ #define RA_WINSIZE_SEQENTIAL RA_WINSIZE_MAX /* fixed window size used for SEQUENTIAL hint */ #define RA_MINSIZE (MAXPHYS * 2) /* min size to start i/o */ #define RA_IOCHUNK MAXPHYS /* read-ahead i/o chunk size */ static off_t ra_startio(struct uvm_object *, off_t, size_t); static struct uvm_ractx *ra_allocctx(void); static void ra_freectx(struct uvm_ractx *); static struct pool_cache ractx_cache; /* * uvm_ra_init: initialize readahead module. */ void uvm_ra_init(void) { pool_cache_bootstrap(&ractx_cache, sizeof(struct uvm_ractx), 0, 0, 0, "ractx", NULL, IPL_NONE, NULL, NULL, NULL); } static struct uvm_ractx * ra_allocctx(void) { return pool_cache_get(&ractx_cache, PR_NOWAIT); } static void ra_freectx(struct uvm_ractx *ra) { pool_cache_put(&ractx_cache, ra); } /* * ra_startio: start i/o for read-ahead. * * => start i/o for each RA_IOCHUNK sized chunk. * => return offset to which we started i/o. */ static off_t ra_startio(struct uvm_object *uobj, off_t off, size_t sz) { const off_t endoff = off + sz; DPRINTF(("%s: uobj=%p, off=%" PRIu64 ", endoff=%" PRIu64 "\n", __func__, uobj, off, endoff)); KASSERT(rw_write_held(uobj->vmobjlock)); /* * Don't issue read-ahead if the last page of the range is already cached. * The assumption is that since the access is sequential, the intermediate * pages would have similar LRU stats, and hence likely to be still in cache * too. This speeds up I/O using cache, since it avoids lookups and temporary * allocations done by full pgo_get. */ struct vm_page *pg = uvm_pagelookup(uobj, trunc_page(endoff - 1)); if (pg != NULL) { DPRINTF(("%s: off=%" PRIu64 ", sz=%zu already cached\n", __func__, off, sz)); return endoff; } off = trunc_page(off); while (off < endoff) { const size_t chunksize = RA_IOCHUNK; int error; size_t donebytes; int npages; int orignpages; size_t bytelen; KASSERT((chunksize & (chunksize - 1)) == 0); KASSERT((off & PAGE_MASK) == 0); bytelen = ((off + chunksize) & -(off_t)chunksize) - off; KASSERT((bytelen & PAGE_MASK) == 0); npages = orignpages = bytelen >> PAGE_SHIFT; KASSERT(npages != 0); /* * use UVM_ADV_RANDOM to avoid recursion. */ error = (*uobj->pgops->pgo_get)(uobj, off, NULL, &npages, 0, VM_PROT_READ, UVM_ADV_RANDOM, PGO_NOTIMESTAMP); rw_enter(uobj->vmobjlock, RW_WRITER); DPRINTF(("%s: off=%" PRIu64 ", bytelen=%zu -> %d\n", __func__, off, bytelen, error)); if (error != 0 && error != EBUSY) { if (error != EINVAL) { /* maybe past EOF */ DPRINTF(("%s: error=%d\n", __func__, error)); } break; } KASSERT(orignpages == npages); donebytes = orignpages << PAGE_SHIFT; off += donebytes; } return off; } /* ------------------------------------------------------------ */ /* * uvm_ra_allocctx: allocate a context. */ struct uvm_ractx * uvm_ra_allocctx(void) { struct uvm_ractx *ra; ra = ra_allocctx(); if (ra != NULL) { ra->ra_flags = 0; } return ra; } /* * uvm_ra_freectx: free a context. */ void uvm_ra_freectx(struct uvm_ractx *ra) { KASSERT(ra != NULL); ra_freectx(ra); } /* * uvm_ra_request: update a read-ahead context and start i/o if appropriate. * * => called when [reqoff, reqoff+reqsize) is requested. * => object must be locked by caller, will return locked. */ void uvm_ra_request(struct uvm_ractx *ra, int advice, struct uvm_object *uobj, off_t reqoff, size_t reqsize) { KASSERT(rw_write_held(uobj->vmobjlock)); if (ra == NULL || advice == UVM_ADV_RANDOM) { return; } if (advice == UVM_ADV_SEQUENTIAL) { /* * always do read-ahead with a large window. */ if ((ra->ra_flags & RA_VALID) == 0) { ra->ra_winstart = ra->ra_next = 0; ra->ra_flags |= RA_VALID; } if (reqoff < ra->ra_winstart) { ra->ra_next = reqoff; } ra->ra_winsize = RA_WINSIZE_SEQENTIAL; goto do_readahead; } /* * a request with UVM_ADV_NORMAL hint. (ie. no hint) * * we keep a sliding window in order to determine: * - if the previous read-ahead was successful or not. * - how many bytes to read-ahead. */ /* * if it's the first request for this context, * initialize context and return. */ if ((ra->ra_flags & RA_VALID) == 0) { initialize: ra->ra_winstart = ra->ra_next = reqoff + reqsize; ra->ra_winsize = RA_WINSIZE_INIT; ra->ra_flags |= RA_VALID; goto done; } /* * if it isn't in our window, * initialize context and return. * (read-ahead miss) */ if (reqoff < ra->ra_winstart || ra->ra_winstart + ra->ra_winsize < reqoff) { /* * ... unless we seem to be reading the same chunk repeatedly. * * XXX should have some margin? */ if (reqoff + reqsize == ra->ra_winstart) { DPRINTF(("%s: %p: same block: off=%" PRIu64 ", size=%zd, winstart=%" PRIu64 "\n", __func__, ra, reqoff, reqsize, ra->ra_winstart)); goto done; } goto initialize; } /* * it's in our window. (read-ahead hit) * - start read-ahead i/o if appropriate. * - advance and enlarge window. */ do_readahead: /* * don't bother to read-ahead behind current request. */ if (reqoff > ra->ra_next) { ra->ra_next = reqoff; } /* * try to make [reqoff, reqoff+ra_winsize) in-core. * note that [reqoff, ra_next) is considered already done. */ if (reqoff + ra->ra_winsize > ra->ra_next) { off_t raoff = MAX(reqoff, ra->ra_next); size_t rasize = reqoff + ra->ra_winsize - ra->ra_next; #if defined(DIAGNOSTIC) if (rasize > RA_WINSIZE_MAX) { printf("%s: corrupted context", __func__); rasize = RA_WINSIZE_MAX; } #endif /* defined(DIAGNOSTIC) */ /* * issue read-ahead only if we can start big enough i/o. * otherwise we end up with a stream of small i/o. */ if (rasize >= RA_MINSIZE) { off_t next; next = ra_startio(uobj, raoff, rasize); ra->ra_next = next; } } /* * update window. * * enlarge window by reqsize, so that it grows in a predictable manner * regardless of the size of each read(2). */ ra->ra_winstart = reqoff + reqsize; ra->ra_winsize = MIN(RA_WINSIZE_MAX, ra->ra_winsize + reqsize); done:; } int uvm_readahead(struct uvm_object *uobj, off_t off, off_t size) { /* * don't allow too much read-ahead. */ if (size > RA_WINSIZE_MAX) { size = RA_WINSIZE_MAX; } rw_enter(uobj->vmobjlock, RW_WRITER); ra_startio(uobj, off, size); rw_exit(uobj->vmobjlock); return 0; } |
| 42 65 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 197 198 199 200 201 202 203 204 205 206 207 208 209 210 211 212 213 214 215 216 217 218 219 220 221 222 223 224 225 226 227 228 229 230 231 232 233 234 235 236 237 238 239 240 241 242 243 244 245 246 247 248 249 250 251 252 253 254 255 256 257 258 259 260 261 262 263 264 265 266 267 268 269 270 271 272 273 274 275 276 277 278 279 280 281 282 283 284 285 286 287 288 289 290 291 292 293 294 295 296 297 298 299 300 301 302 303 304 305 306 307 308 309 310 311 312 313 314 315 316 317 318 319 320 321 322 323 324 325 326 327 328 329 330 331 332 333 334 335 336 337 338 339 340 341 342 343 344 345 346 347 348 349 350 351 352 353 354 355 356 357 358 359 360 361 362 363 364 365 366 367 368 369 370 371 372 373 374 375 376 377 378 379 380 381 382 383 384 385 386 387 388 389 390 391 392 393 394 395 396 397 398 399 400 401 402 403 404 405 406 407 408 409 410 411 412 413 414 415 416 417 418 419 420 421 422 423 424 425 426 427 428 429 430 431 432 433 434 435 436 437 438 439 440 441 442 443 444 445 446 447 448 449 450 451 452 453 454 455 456 457 458 459 460 461 462 463 464 465 466 467 468 469 470 471 472 473 474 475 476 477 478 479 480 481 482 483 484 485 486 487 488 489 490 491 492 493 494 495 496 497 498 499 500 501 502 503 504 505 506 507 508 509 510 511 512 513 514 515 516 517 518 519 520 521 522 523 524 525 526 527 528 529 530 531 532 533 534 535 536 537 538 539 540 541 542 543 544 545 546 547 548 549 550 551 552 553 554 555 556 557 558 559 560 561 562 563 564 565 566 567 568 569 570 571 572 573 574 575 576 577 578 579 580 581 582 583 584 585 586 587 588 589 590 591 592 593 594 595 596 597 598 599 600 601 602 603 604 605 606 607 608 609 610 | /* $NetBSD: bpf.h,v 1.78 2022/06/20 08:20:09 yamaguchi Exp $ */ /* * Copyright (c) 1990, 1991, 1993 * The Regents of the University of California. All rights reserved. * * This code is derived from the Stanford/CMU enet packet filter, * (net/enet.c) distributed as part of 4.3BSD, and code contributed * to Berkeley by Steven McCanne and Van Jacobson both of Lawrence * Berkeley Laboratory. * * Redistribution and use in source and binary forms, with or without * modification, are permitted provided that the following conditions * are met: * 1. Redistributions of source code must retain the above copyright * notice, this list of conditions and the following disclaimer. * 2. Redistributions in binary form must reproduce the above copyright * notice, this list of conditions and the following disclaimer in the * documentation and/or other materials provided with the distribution. * 3. Neither the name of the University nor the names of its contributors * may be used to endorse or promote products derived from this software * without specific prior written permission. * * THIS SOFTWARE IS PROVIDED BY THE REGENTS AND CONTRIBUTORS ``AS IS'' AND * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE * ARE DISCLAIMED. IN NO EVENT SHALL THE REGENTS OR CONTRIBUTORS BE LIABLE * FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL * DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS * OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) * HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT * LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY * OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF * SUCH DAMAGE. * * @(#)bpf.h 8.2 (Berkeley) 1/9/95 * @(#) Header: bpf.h,v 1.36 97/06/12 14:29:53 leres Exp (LBL) */ #ifndef _NET_BPF_H_ #define _NET_BPF_H_ #include <sys/ioccom.h> #include <sys/time.h> /* BSD style release date */ #define BPF_RELEASE 199606 /* Date when COP instructions and external memory have been released. */ #define BPF_COP_EXTMEM_RELEASE 20140624 __BEGIN_DECLS typedef int bpf_int32; typedef u_int bpf_u_int32; /* * Alignment macros. BPF_WORDALIGN rounds up to the next * even multiple of BPF_ALIGNMENT. */ #define BPF_ALIGNMENT sizeof(long) #define BPF_ALIGNMENT32 sizeof(int) #define BPF_WORDALIGN(x) (((x)+(BPF_ALIGNMENT-1))&~(BPF_ALIGNMENT-1)) #define BPF_WORDALIGN32(x) (((x)+(BPF_ALIGNMENT32-1))&~(BPF_ALIGNMENT32-1)) #define BPF_MAXINSNS 512 #define BPF_DFLTBUFSIZE (1024*1024) /* default static upper limit */ #define BPF_MAXBUFSIZE (1024*1024*16) /* hard limit on sysctl'able value */ #define BPF_MINBUFSIZE 32 /* * Structure for BIOCSETF. */ struct bpf_program { u_int bf_len; struct bpf_insn *bf_insns; }; /* * Struct returned by BIOCGSTATS and net.bpf.stats sysctl. */ struct bpf_stat { uint64_t bs_recv; /* number of packets received */ uint64_t bs_drop; /* number of packets dropped */ uint64_t bs_capt; /* number of packets captured */ uint64_t bs_padding[13]; }; /* * Struct returned by BIOCGSTATSOLD. */ struct bpf_stat_old { u_int bs_recv; /* number of packets received */ u_int bs_drop; /* number of packets dropped */ }; /* * Struct return by BIOCVERSION. This represents the version number of * the filter language described by the instruction encodings below. * bpf understands a program iff kernel_major == filter_major && * kernel_minor >= filter_minor, that is, if the value returned by the * running kernel has the same major number and a minor number equal * equal to or less than the filter being downloaded. Otherwise, the * results are undefined, meaning an error may be returned or packets * may be accepted haphazardly. * It has nothing to do with the source code version. */ struct bpf_version { u_short bv_major; u_short bv_minor; }; /* Current version number of filter architecture. */ #define BPF_MAJOR_VERSION 1 #define BPF_MINOR_VERSION 1 /* * BPF ioctls * * The first set is for compatibility with Sun's pcc style * header files. If your using gcc, we assume that you * have run fixincludes so the latter set should work. */ #define BIOCGBLEN _IOR('B', 102, u_int) #define BIOCSBLEN _IOWR('B', 102, u_int) #define BIOCSETF _IOW('B', 103, struct bpf_program) #define BIOCFLUSH _IO('B', 104) #define BIOCPROMISC _IO('B', 105) #define BIOCGDLT _IOR('B', 106, u_int) #define BIOCGETIF _IOR('B', 107, struct ifreq) #define BIOCSETIF _IOW('B', 108, struct ifreq) #ifdef COMPAT_50 #include <compat/sys/time.h> #define BIOCSORTIMEOUT _IOW('B', 109, struct timeval50) #define BIOCGORTIMEOUT _IOR('B', 110, struct timeval50) #endif #define BIOCGSTATS _IOR('B', 111, struct bpf_stat) #define BIOCGSTATSOLD _IOR('B', 111, struct bpf_stat_old) #define BIOCIMMEDIATE _IOW('B', 112, u_int) #define BIOCVERSION _IOR('B', 113, struct bpf_version) #define BIOCSTCPF _IOW('B', 114, struct bpf_program) #define BIOCSUDPF _IOW('B', 115, struct bpf_program) #define BIOCGHDRCMPLT _IOR('B', 116, u_int) #define BIOCSHDRCMPLT _IOW('B', 117, u_int) #define BIOCSDLT _IOW('B', 118, u_int) #define BIOCGDLTLIST _IOWR('B', 119, struct bpf_dltlist) #define BIOCGDIRECTION _IOR('B', 120, u_int) #define BIOCSDIRECTION _IOW('B', 121, u_int) #define BIOCSRTIMEOUT _IOW('B', 122, struct timeval) #define BIOCGRTIMEOUT _IOR('B', 123, struct timeval) #define BIOCGFEEDBACK _IOR('B', 124, u_int) #define BIOCSFEEDBACK _IOW('B', 125, u_int) #define BIOCFEEDBACK BIOCSFEEDBACK /* FreeBSD name */ #define BIOCLOCK _IO('B', 126) #define BIOCSETWF _IOW('B', 127, struct bpf_program) /* Obsolete */ #define BIOCGSEESENT BIOCGDIRECTION #define BIOCSSEESENT BIOCSDIRECTION /* * Packet directions. * BPF_D_IN = 0, BPF_D_INOUT =1 for backward compatibility of BIOC[GS]SEESENT. */ #define BPF_D_IN 0 /* See incoming packets */ #define BPF_D_INOUT 1 /* See incoming and outgoing packets */ #define BPF_D_OUT 2 /* See outgoing packets */ /* * Structure prepended to each packet. This is "wire" format, so we * cannot change it unfortunately to 64 bit times on 32 bit systems [yet]. */ struct bpf_timeval { long tv_sec; long tv_usec; }; struct bpf_timeval32 { int32_t tv_sec; int32_t tv_usec; }; struct bpf_hdr { struct bpf_timeval bh_tstamp; /* time stamp */ uint32_t bh_caplen; /* length of captured portion */ uint32_t bh_datalen; /* original length of packet */ uint16_t bh_hdrlen; /* length of bpf header (this struct plus alignment padding) */ }; struct bpf_hdr32 { struct bpf_timeval32 bh_tstamp; /* time stamp */ uint32_t bh_caplen; /* length of captured portion */ uint32_t bh_datalen; /* original length of packet */ uint16_t bh_hdrlen; /* length of bpf header (this struct plus alignment padding) */ }; /* * Because the structure above is not a multiple of 4 bytes, some compilers * will insist on inserting padding; hence, sizeof(struct bpf_hdr) won't work. * Only the kernel needs to know about it; applications use bh_hdrlen. * XXX To save a few bytes on 32-bit machines, we avoid end-of-struct * XXX padding by using the size of the header data elements. This is * XXX fail-safe: on new machines, we just use the 'safe' sizeof. */ #ifdef _KERNEL #if defined(__arm32__) || defined(__i386__) || defined(__m68k__) || \ defined(__mips__) || defined(__ns32k__) || defined(__vax__) || \ defined(__sh__) || (defined(__sparc__) && !defined(__sparc64__)) #define SIZEOF_BPF_HDR 18 #define SIZEOF_BPF_HDR32 18 #else #define SIZEOF_BPF_HDR sizeof(struct bpf_hdr) #define SIZEOF_BPF_HDR32 sizeof(struct bpf_hdr32) #endif #endif /* Pull in data-link level type codes. */ #include <net/dlt.h> /* * The instruction encodings. */ /* instruction classes */ #define BPF_CLASS(code) ((code) & 0x07) #define BPF_LD 0x00 #define BPF_LDX 0x01 #define BPF_ST 0x02 #define BPF_STX 0x03 #define BPF_ALU 0x04 #define BPF_JMP 0x05 #define BPF_RET 0x06 #define BPF_MISC 0x07 /* ld/ldx fields */ #define BPF_SIZE(code) ((code) & 0x18) #define BPF_W 0x00 #define BPF_H 0x08 #define BPF_B 0x10 /* 0x18 reserved; used by BSD/OS */ #define BPF_MODE(code) ((code) & 0xe0) #define BPF_IMM 0x00 #define BPF_ABS 0x20 #define BPF_IND 0x40 #define BPF_MEM 0x60 #define BPF_LEN 0x80 #define BPF_MSH 0xa0 /* 0xc0 reserved; used by BSD/OS */ /* 0xe0 reserved; used by BSD/OS */ /* alu/jmp fields */ #define BPF_OP(code) ((code) & 0xf0) #define BPF_ADD 0x00 #define BPF_SUB 0x10 #define BPF_MUL 0x20 #define BPF_DIV 0x30 #define BPF_OR 0x40 #define BPF_AND 0x50 #define BPF_LSH 0x60 #define BPF_RSH 0x70 #define BPF_NEG 0x80 #define BPF_MOD 0x90 #define BPF_XOR 0xa0 /* 0xb0 reserved */ /* 0xc0 reserved */ /* 0xd0 reserved */ /* 0xe0 reserved */ /* 0xf0 reserved */ #define BPF_JA 0x00 #define BPF_JEQ 0x10 #define BPF_JGT 0x20 #define BPF_JGE 0x30 #define BPF_JSET 0x40 /* 0x50 reserved; used by BSD/OS */ /* 0x60 reserved */ /* 0x70 reserved */ /* 0x80 reserved */ /* 0x90 reserved */ /* 0xa0 reserved */ /* 0xb0 reserved */ /* 0xc0 reserved */ /* 0xd0 reserved */ /* 0xe0 reserved */ /* 0xf0 reserved */ #define BPF_SRC(code) ((code) & 0x08) #define BPF_K 0x00 #define BPF_X 0x08 /* ret - BPF_K and BPF_X also apply */ #define BPF_RVAL(code) ((code) & 0x18) #define BPF_A 0x10 /* 0x18 reserved */ /* misc */ #define BPF_MISCOP(code) ((code) & 0xf8) #define BPF_TAX 0x00 /* 0x10 reserved */ /* 0x18 reserved */ #define BPF_COP 0x20 /* 0x28 reserved */ /* 0x30 reserved */ /* 0x38 reserved */ #define BPF_COPX 0x40 /* XXX: also used by BSD/OS */ /* 0x48 reserved */ /* 0x50 reserved */ /* 0x58 reserved */ /* 0x60 reserved */ /* 0x68 reserved */ /* 0x70 reserved */ /* 0x78 reserved */ #define BPF_TXA 0x80 /* 0x88 reserved */ /* 0x90 reserved */ /* 0x98 reserved */ /* 0xa0 reserved */ /* 0xa8 reserved */ /* 0xb0 reserved */ /* 0xb8 reserved */ /* 0xc0 reserved; used by BSD/OS */ /* 0xc8 reserved */ /* 0xd0 reserved */ /* 0xd8 reserved */ /* 0xe0 reserved */ /* 0xe8 reserved */ /* 0xf0 reserved */ /* 0xf8 reserved */ /* * The instruction data structure. */ struct bpf_insn { uint16_t code; u_char jt; u_char jf; uint32_t k; }; /* * Auxiliary data, for use when interpreting a filter intended for the * Linux kernel when the kernel rejects the filter (requiring us to * run it in userland). It contains VLAN tag information. */ struct bpf_aux_data { u_short vlan_tag_present; u_short vlan_tag; }; /* * Macros for insn array initializers. */ #define BPF_STMT(code, k) { (uint16_t)(code), 0, 0, k } #define BPF_JUMP(code, k, jt, jf) { (uint16_t)(code), jt, jf, k } /* * Number of scratch memory words (for BPF_LD|BPF_MEM and BPF_ST). */ #define BPF_MEMWORDS 16 /* * bpf_memword_init_t: bits indicate which words in the external memory * store will be initialised by the caller before BPF program execution. */ typedef uint32_t bpf_memword_init_t; #define BPF_MEMWORD_INIT(k) (UINT32_C(1) << (k)) /* Note: two most significant bits are reserved by bpfjit. */ __CTASSERT(BPF_MEMWORDS + 2 <= sizeof(bpf_memword_init_t) * NBBY); #ifdef _KERNEL /* * Max number of external memory words (for BPF_LD|BPF_MEM and BPF_ST). */ #define BPF_MAX_MEMWORDS 30 __CTASSERT(BPF_MAX_MEMWORDS >= BPF_MEMWORDS); __CTASSERT(BPF_MAX_MEMWORDS + 2 <= sizeof(bpf_memword_init_t) * NBBY); #endif /* * Structure to retrieve available DLTs for the interface. */ struct bpf_dltlist { u_int bfl_len; /* number of bfd_list array */ u_int *bfl_list; /* array of DLTs */ }; struct bpf_ctx; typedef struct bpf_ctx bpf_ctx_t; typedef struct bpf_args { const uint8_t * pkt; size_t wirelen; size_t buflen; /* * The following arguments are used only by some kernel * subsystems. * They aren't required for classical bpf filter programs. * For such programs, bpfjit generated code doesn't read * those arguments at all. Note however that bpf interpreter * always needs a pointer to memstore. */ uint32_t * mem; /* pointer to external memory store */ void * arg; /* auxiliary argument for a copfunc */ } bpf_args_t; #if defined(_KERNEL) || defined(__BPF_PRIVATE) typedef uint32_t (*bpf_copfunc_t)(const bpf_ctx_t *, bpf_args_t *, uint32_t); struct bpf_ctx { /* * BPF coprocessor functions and the number of them. */ const bpf_copfunc_t * copfuncs; size_t nfuncs; /* * The number of memory words in the external memory store. * There may be up to BPF_MAX_MEMWORDS words; if zero is set, * then the internal memory store is used which has a fixed * number of words (BPF_MEMWORDS). */ size_t extwords; /* * The bitmask indicating which words in the external memstore * will be initialised by the caller. */ bpf_memword_init_t preinited; }; #endif #ifdef _KERNEL #include <net/bpfjit.h> #include <net/if.h> struct bpf_if; struct bpf_ops { void (*bpf_attach)(struct ifnet *, u_int, u_int, struct bpf_if **); void (*bpf_detach)(struct ifnet *); void (*bpf_change_type)(struct ifnet *, u_int, u_int); void (*bpf_mtap)(struct bpf_if *, struct mbuf *, u_int); void (*bpf_mtap2)(struct bpf_if *, void *, u_int, struct mbuf *, u_int); void (*bpf_mtap_af)(struct bpf_if *, uint32_t, struct mbuf *, u_int); void (*bpf_mtap_sl_in)(struct bpf_if *, u_char *, struct mbuf **); void (*bpf_mtap_sl_out)(struct bpf_if *, u_char *, struct mbuf *); void (*bpf_mtap_softint_init)(struct ifnet *); void (*bpf_mtap_softint)(struct ifnet *, struct mbuf *); int (*bpf_register_track_event)(struct bpf_if **, void (*)(struct bpf_if *, struct ifnet *, int, int)); int (*bpf_deregister_track_event)(struct bpf_if **, void (*)(struct bpf_if *, struct ifnet *, int, int)); }; extern struct bpf_ops *bpf_ops; static __inline void bpf_attach(struct ifnet *_ifp, u_int _dlt, u_int _hdrlen) { bpf_ops->bpf_attach(_ifp, _dlt, _hdrlen, &_ifp->if_bpf); } static __inline void bpf_attach2(struct ifnet *_ifp, u_int _dlt, u_int _hdrlen, struct bpf_if **_dp) { bpf_ops->bpf_attach(_ifp, _dlt, _hdrlen, _dp); } static __inline void bpf_mtap(struct ifnet *_ifp, struct mbuf *_m, u_int _direction) { if (_ifp->if_bpf) { if (_ifp->if_bpf_mtap) { _ifp->if_bpf_mtap(_ifp->if_bpf, _m, _direction); } else { bpf_ops->bpf_mtap(_ifp->if_bpf, _m, _direction); } } } static __inline void bpf_mtap2(struct bpf_if *_bpf, void *_data, u_int _dlen, struct mbuf *_m, u_int _direction) { bpf_ops->bpf_mtap2(_bpf, _data, _dlen, _m, _direction); } static __inline void bpf_mtap3(struct bpf_if *_bpf, struct mbuf *_m, u_int _direction) { if (_bpf) bpf_ops->bpf_mtap(_bpf, _m, _direction); } static __inline void bpf_mtap_af(struct ifnet *_ifp, uint32_t _af, struct mbuf *_m, u_int _direction) { if (_ifp->if_bpf) bpf_ops->bpf_mtap_af(_ifp->if_bpf, _af, _m, _direction); } static __inline void bpf_change_type(struct ifnet *_ifp, u_int _dlt, u_int _hdrlen) { bpf_ops->bpf_change_type(_ifp, _dlt, _hdrlen); } static __inline bool bpf_peers_present(struct bpf_if *dp) { /* * Our code makes sure the driver visible pointer is NULL * whenever there is no listener on this tap. */ return dp != NULL; } static __inline void bpf_detach(struct ifnet *_ifp) { bpf_ops->bpf_detach(_ifp); } static __inline void bpf_mtap_sl_in(struct ifnet *_ifp, u_char *_hdr, struct mbuf **_m) { bpf_ops->bpf_mtap_sl_in(_ifp->if_bpf, _hdr, _m); } static __inline void bpf_mtap_sl_out(struct ifnet *_ifp, u_char *_hdr, struct mbuf *_m) { if (_ifp->if_bpf) bpf_ops->bpf_mtap_sl_out(_ifp->if_bpf, _hdr, _m); } static __inline void bpf_mtap_softint_init(struct ifnet *_ifp) { bpf_ops->bpf_mtap_softint_init(_ifp); } static __inline void bpf_mtap_softint(struct ifnet *_ifp, struct mbuf *_m) { if (_ifp->if_bpf) bpf_ops->bpf_mtap_softint(_ifp, _m); } static __inline int bpf_register_track_event(struct bpf_if **_dp, void (*_fun)(struct bpf_if *, struct ifnet *, int, int)) { if (bpf_ops->bpf_register_track_event == NULL) return ENXIO; return bpf_ops->bpf_register_track_event(_dp, _fun); } static __inline int bpf_deregister_track_event(struct bpf_if **_dp, void (*_fun)(struct bpf_if *, struct ifnet *, int, int)) { if (bpf_ops->bpf_deregister_track_event == NULL) return ENXIO; return bpf_ops->bpf_deregister_track_event(_dp, _fun); } void bpf_setops(void); void bpf_ops_handover_enter(struct bpf_ops *); void bpf_ops_handover_exit(void); void bpfilterattach(int); bpf_ctx_t *bpf_create(void); void bpf_destroy(bpf_ctx_t *); int bpf_set_cop(bpf_ctx_t *, const bpf_copfunc_t *, size_t); int bpf_set_extmem(bpf_ctx_t *, size_t, bpf_memword_init_t); u_int bpf_filter_ext(const bpf_ctx_t *, const struct bpf_insn *, bpf_args_t *); int bpf_validate_ext(const bpf_ctx_t *, const struct bpf_insn *, int); bpfjit_func_t bpf_jit_generate(bpf_ctx_t *, void *, size_t); void bpf_jit_freecode(bpfjit_func_t); #endif int bpf_validate(const struct bpf_insn *, int); u_int bpf_filter(const struct bpf_insn *, const u_char *, u_int, u_int); u_int bpf_filter_with_aux_data(const struct bpf_insn *, const u_char *, u_int, u_int, const struct bpf_aux_data *); /* * events to be tracked by bpf_register_track_event callbacks */ #define BPF_TRACK_EVENT_ATTACH 1 #define BPF_TRACK_EVENT_DETACH 2 __END_DECLS #endif /* !_NET_BPF_H_ */ |
| 10 5 2 3 5 2 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 | /* $NetBSD: sysv_msg_50.c,v 1.5 2019/12/15 16:48:26 tsutsui Exp $ */ /*- * Copyright (c) 1999 The NetBSD Foundation, Inc. * All rights reserved. * * This code is derived from software contributed to The NetBSD Foundation * by Jason R. Thorpe of the Numerical Aerospace Simulation Facility, * NASA Ames Research Center. * * Redistribution and use in source and binary forms, with or without * modification, are permitted provided that the following conditions * are met: * 1. Redistributions of source code must retain the above copyright * notice, this list of conditions and the following disclaimer. * 2. Redistributions in binary form must reproduce the above copyright * notice, this list of conditions and the following disclaimer in the * documentation and/or other materials provided with the distribution. * * THIS SOFTWARE IS PROVIDED BY THE NETBSD FOUNDATION, INC. AND CONTRIBUTORS * ``AS IS'' AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED * TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR * PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE FOUNDATION OR CONTRIBUTORS * BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR * CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF * SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS * INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN * CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) * ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE * POSSIBILITY OF SUCH DAMAGE. */ #include <sys/cdefs.h> __KERNEL_RCSID(0, "$NetBSD: sysv_msg_50.c,v 1.5 2019/12/15 16:48:26 tsutsui Exp $"); #if defined(_KERNEL_OPT) #include "opt_compat_netbsd.h" #endif #include <sys/param.h> #include <sys/systm.h> #include <sys/signal.h> #include <sys/proc.h> #include <sys/msg.h> #include <compat/sys/msg.h> #ifndef SYSVMSG #define SYSVMSG #endif #include <sys/syscallargs.h> int compat_50_sys___msgctl13(struct lwp *l, const struct compat_50_sys___msgctl13_args *uap, register_t *retval) { /* { syscallarg(int) msqid; syscallarg(int) cmd; syscallarg(struct msqid_ds13 *) buf; } */ struct msqid_ds msqbuf; struct msqid_ds13 omsqbuf; int cmd, error; cmd = SCARG(uap, cmd); if (cmd == IPC_SET) { error = copyin(SCARG(uap, buf), &omsqbuf, sizeof(omsqbuf)); if (error) return (error); __msqid_ds13_to_native(&omsqbuf, &msqbuf); } error = msgctl1(l, SCARG(uap, msqid), cmd, (cmd == IPC_SET || cmd == IPC_STAT) ? &msqbuf : NULL); if (error == 0 && cmd == IPC_STAT) { __native_to_msqid_ds13(&msqbuf, &omsqbuf); error = copyout(&omsqbuf, SCARG(uap, buf), sizeof(omsqbuf)); } return (error); } |
| 13 31 31 21 8 6 6 2 2 1 16 9 9 25 9 1 2 13 18 18 1 18 1 1 2 1 2 18 18 18 18 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 197 198 199 200 201 202 203 204 205 206 207 208 209 210 211 212 213 214 215 216 217 218 219 220 221 222 223 224 225 226 227 228 229 230 231 232 233 234 235 236 237 238 239 240 241 242 243 244 245 246 247 248 249 250 251 252 253 254 255 256 257 258 259 260 261 262 263 264 265 266 267 268 269 270 271 272 273 274 275 276 277 278 279 280 281 282 283 284 285 286 287 288 289 290 291 292 293 294 295 296 297 298 299 300 301 302 303 304 305 306 307 308 309 310 311 312 313 314 315 316 317 318 319 320 321 322 323 324 325 326 327 328 329 330 331 332 333 334 335 336 337 338 339 340 341 342 343 344 345 346 347 348 349 350 351 352 353 354 355 356 357 358 359 360 361 362 363 364 365 366 367 368 369 370 371 372 373 374 375 376 377 378 379 380 381 382 383 384 385 386 387 388 389 390 391 392 393 394 395 396 397 398 399 400 401 402 403 404 405 406 407 408 409 410 411 412 413 414 415 416 417 418 419 420 421 422 423 424 425 426 427 428 429 430 431 432 433 434 435 436 437 438 439 440 441 442 443 444 445 446 447 448 449 450 451 452 453 454 455 456 457 458 459 460 461 462 463 464 465 466 467 468 469 470 471 472 473 474 475 476 477 478 479 480 481 482 483 484 485 486 487 488 489 490 491 492 493 494 495 496 497 498 499 500 501 502 503 504 505 506 507 508 509 510 511 512 513 514 515 516 517 518 519 520 521 522 523 524 525 526 527 528 529 530 531 532 533 534 535 536 537 538 539 540 541 542 543 544 545 546 547 548 549 550 551 552 553 554 555 556 557 558 559 560 561 562 563 564 565 566 567 568 569 570 571 572 573 574 575 576 577 578 579 580 581 582 583 584 585 586 587 588 589 590 591 592 593 594 595 596 597 598 599 600 601 602 603 604 605 606 607 608 609 610 611 612 613 614 615 616 617 618 619 620 621 622 623 624 625 626 627 628 629 630 631 632 633 634 635 636 637 638 639 640 641 642 643 644 645 646 647 648 649 650 651 652 653 654 655 656 657 658 659 660 661 662 663 664 665 666 667 668 669 670 671 672 673 674 675 676 677 678 679 680 681 682 683 684 685 686 687 688 689 690 691 692 693 694 695 696 697 698 699 700 701 702 703 704 705 706 707 708 709 710 711 712 713 714 715 716 717 718 719 720 721 722 723 724 725 726 727 728 729 730 731 732 733 734 735 736 737 738 739 740 741 742 743 744 745 746 747 748 749 750 751 752 753 754 755 756 757 758 759 760 761 762 763 764 765 766 767 768 769 770 771 772 773 774 775 776 777 778 779 780 781 782 783 784 785 786 787 788 789 790 791 792 793 794 795 796 797 798 799 800 801 802 803 804 805 806 807 808 809 810 811 812 813 814 815 816 817 818 819 820 821 822 823 824 825 826 827 828 829 830 831 832 833 834 835 836 837 838 839 840 841 842 843 844 845 846 847 848 849 850 851 852 853 854 855 856 857 858 859 860 861 862 863 864 865 866 867 868 869 870 871 872 873 874 875 876 877 878 879 880 881 882 883 884 885 886 887 888 889 890 891 892 893 894 895 896 897 898 899 900 901 902 903 904 905 906 907 908 909 910 911 912 913 914 915 916 917 918 919 920 921 922 923 924 925 926 927 928 929 930 931 932 933 934 935 936 937 938 939 940 941 942 943 944 945 946 947 948 949 950 951 952 953 954 955 956 957 958 959 960 961 962 963 964 965 966 967 968 969 970 971 972 973 974 975 976 977 978 979 980 981 982 983 984 985 986 987 988 989 990 991 992 993 994 995 996 997 998 999 1000 1001 1002 1003 1004 1005 1006 1007 1008 1009 1010 1011 1012 1013 1014 1015 1016 1017 1018 1019 1020 1021 1022 1023 1024 1025 1026 1027 1028 1029 1030 1031 1032 1033 1034 1035 1036 1037 1038 1039 1040 1041 1042 1043 1044 1045 1046 1047 1048 1049 1050 1051 1052 1053 1054 1055 1056 1057 1058 1059 1060 1061 1062 1063 1064 1065 1066 1067 1068 1069 1070 1071 1072 1073 1074 1075 1076 1077 1078 1079 1080 1081 1082 1083 1084 1085 1086 1087 1088 1089 1090 1091 1092 1093 1094 1095 1096 1097 1098 1099 1100 1101 1102 1103 1104 1105 1106 1107 1108 1109 1110 1111 1112 1113 1114 1115 1116 1117 1118 1119 1120 1121 1122 1123 1124 1125 1126 1127 1128 1129 1130 1131 1132 1133 1134 1135 1136 1137 1138 1139 1140 1141 1142 1143 1144 1145 1146 1147 1148 1149 1150 1151 1152 1153 1154 1155 1156 1157 1158 1159 1160 1161 1162 1163 1164 1165 1166 1167 1168 1169 1170 1171 1172 1173 1174 1175 1176 1177 1178 1179 1180 1181 1182 1183 1184 1185 1186 1187 1188 1189 1190 1191 1192 1193 1194 1195 1196 1197 1198 1199 1200 1201 1202 1203 1204 1205 1206 1207 1208 1209 1210 1211 1212 1213 1214 1215 1216 1217 1218 1219 1220 1221 1222 1223 1224 1225 1226 1227 1228 1229 1230 1231 1232 1233 1234 1235 1236 1237 1238 1239 1240 1241 1242 1243 1244 1245 1246 1247 1248 1249 1250 1251 1252 1253 1254 1255 1256 1257 1258 1259 1260 1261 1262 1263 1264 1265 1266 1267 1268 1269 1270 1271 1272 1273 1274 1275 1276 1277 1278 1279 1280 1281 1282 1283 1284 1285 1286 1287 1288 1289 1290 1291 1292 1293 1294 1295 1296 1297 1298 1299 1300 1301 1302 1303 1304 1305 1306 1307 1308 1309 1310 1311 1312 1313 1314 1315 1316 1317 1318 1319 1320 1321 1322 1323 1324 1325 1326 1327 1328 1329 1330 1331 1332 1333 1334 1335 1336 1337 1338 1339 1340 1341 1342 1343 1344 1345 1346 1347 1348 1349 1350 1351 1352 1353 1354 1355 1356 1357 1358 1359 1360 1361 1362 1363 1364 1365 1366 1367 1368 1369 1370 1371 1372 1373 1374 1375 1376 1377 1378 1379 1380 1381 1382 1383 1384 1385 1386 1387 1388 1389 1390 1391 1392 1393 1394 1395 1396 1397 1398 1399 1400 1401 1402 1403 1404 1405 1406 1407 1408 1409 1410 1411 1412 1413 1414 1415 1416 1417 1418 1419 1420 1421 1422 1423 1424 1425 1426 1427 1428 1429 1430 1431 1432 1433 1434 1435 1436 1437 1438 1439 1440 1441 1442 1443 1444 1445 1446 1447 1448 1449 1450 1451 1452 1453 1454 1455 1456 1457 1458 1459 1460 1461 1462 1463 1464 1465 1466 1467 1468 1469 1470 1471 1472 1473 1474 1475 1476 1477 1478 1479 1480 1481 1482 1483 1484 1485 1486 1487 1488 1489 1490 1491 1492 1493 1494 1495 1496 1497 1498 1499 1500 1501 1502 1503 1504 1505 1506 1507 1508 1509 1510 1511 1512 1513 1514 1515 1516 1517 1518 1519 1520 1521 1522 1523 1524 1525 1526 1527 1528 1529 1530 1531 1532 1533 1534 1535 1536 1537 1538 1539 1540 1541 1542 1543 1544 1545 1546 1547 1548 1549 1550 1551 1552 1553 1554 1555 1556 1557 1558 1559 1560 1561 1562 1563 1564 1565 1566 1567 1568 1569 1570 1571 1572 1573 1574 1575 1576 1577 1578 1579 1580 1581 1582 1583 1584 1585 1586 1587 1588 1589 1590 1591 1592 1593 1594 1595 1596 1597 1598 1599 1600 1601 1602 1603 1604 1605 1606 1607 1608 1609 1610 1611 1612 1613 1614 1615 1616 1617 1618 1619 1620 1621 1622 1623 1624 1625 1626 1627 1628 1629 1630 1631 1632 1633 1634 1635 1636 1637 1638 1639 1640 1641 1642 1643 1644 1645 1646 1647 1648 1649 1650 1651 1652 1653 1654 1655 1656 1657 1658 1659 1660 1661 1662 1663 1664 1665 1666 1667 1668 1669 1670 1671 1672 1673 1674 1675 1676 1677 1678 1679 1680 1681 1682 1683 1684 1685 1686 1687 1688 1689 1690 1691 1692 1693 1694 1695 1696 1697 1698 1699 1700 1701 1702 1703 1704 1705 1706 1707 1708 1709 1710 1711 1712 1713 1714 1715 1716 1717 1718 1719 1720 1721 1722 1723 1724 1725 1726 1727 1728 1729 1730 1731 1732 1733 1734 1735 1736 1737 1738 1739 1740 1741 1742 1743 1744 1745 1746 1747 1748 1749 1750 1751 1752 1753 1754 1755 1756 1757 1758 1759 1760 1761 1762 1763 1764 1765 1766 1767 1768 1769 1770 1771 1772 1773 1774 1775 1776 1777 1778 1779 1780 1781 1782 1783 1784 1785 1786 1787 1788 1789 1790 1791 1792 1793 1794 1795 1796 1797 1798 1799 1800 1801 1802 1803 1804 1805 1806 1807 1808 1809 1810 | /* $NetBSD: ip6_input.c,v 1.224 2021/02/19 14:52:00 christos Exp $ */ /* $KAME: ip6_input.c,v 1.188 2001/03/29 05:34:31 itojun Exp $ */ /* * Copyright (C) 1995, 1996, 1997, and 1998 WIDE Project. * All rights reserved. * * Redistribution and use in source and binary forms, with or without * modification, are permitted provided that the following conditions * are met: * 1. Redistributions of source code must retain the above copyright * notice, this list of conditions and the following disclaimer. * 2. Redistributions in binary form must reproduce the above copyright * notice, this list of conditions and the following disclaimer in the * documentation and/or other materials provided with the distribution. * 3. Neither the name of the project nor the names of its contributors * may be used to endorse or promote products derived from this software * without specific prior written permission. * * THIS SOFTWARE IS PROVIDED BY THE PROJECT AND CONTRIBUTORS ``AS IS'' AND * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE * ARE DISCLAIMED. IN NO EVENT SHALL THE PROJECT OR CONTRIBUTORS BE LIABLE * FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL * DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS * OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) * HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT * LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY * OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF * SUCH DAMAGE. */ /* * Copyright (c) 1982, 1986, 1988, 1993 * The Regents of the University of California. All rights reserved. * * Redistribution and use in source and binary forms, with or without * modification, are permitted provided that the following conditions * are met: * 1. Redistributions of source code must retain the above copyright * notice, this list of conditions and the following disclaimer. * 2. Redistributions in binary form must reproduce the above copyright * notice, this list of conditions and the following disclaimer in the * documentation and/or other materials provided with the distribution. * 3. Neither the name of the University nor the names of its contributors * may be used to endorse or promote products derived from this software * without specific prior written permission. * * THIS SOFTWARE IS PROVIDED BY THE REGENTS AND CONTRIBUTORS ``AS IS'' AND * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE * ARE DISCLAIMED. IN NO EVENT SHALL THE REGENTS OR CONTRIBUTORS BE LIABLE * FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL * DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS * OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) * HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT * LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY * OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF * SUCH DAMAGE. * * @(#)ip_input.c 8.2 (Berkeley) 1/4/94 */ #include <sys/cdefs.h> __KERNEL_RCSID(0, "$NetBSD: ip6_input.c,v 1.224 2021/02/19 14:52:00 christos Exp $"); #ifdef _KERNEL_OPT #include "opt_gateway.h" #include "opt_inet.h" #include "opt_inet6.h" #include "opt_ipsec.h" #include "opt_net_mpsafe.h" #endif #include <sys/param.h> #include <sys/systm.h> #include <sys/mbuf.h> #include <sys/domain.h> #include <sys/protosw.h> #include <sys/socket.h> #include <sys/socketvar.h> #include <sys/errno.h> #include <sys/time.h> #include <sys/kernel.h> #include <sys/syslog.h> #include <sys/proc.h> #include <sys/sysctl.h> #include <sys/cprng.h> #include <sys/percpu.h> #include <net/if.h> #include <net/if_types.h> #include <net/if_dl.h> #include <net/route.h> #include <net/pktqueue.h> #include <net/pfil.h> #include <netinet/in.h> #include <netinet/in_systm.h> #ifdef INET #include <netinet/ip.h> #include <netinet/ip_var.h> #include <netinet/ip_icmp.h> #endif /* INET */ #include <netinet/ip6.h> #include <netinet/portalgo.h> #include <netinet6/in6_var.h> #include <netinet6/ip6_var.h> #include <netinet6/ip6_private.h> #include <netinet6/in6_pcb.h> #include <netinet/icmp6.h> #include <netinet6/scope6_var.h> #include <netinet6/in6_ifattach.h> #include <netinet6/nd6.h> #ifdef IPSEC #include <netipsec/ipsec.h> #include <netipsec/ipsec6.h> #include <netipsec/key.h> #endif /* IPSEC */ #include <netinet6/ip6protosw.h> #include "faith.h" extern struct domain inet6domain; u_char ip6_protox[IPPROTO_MAX]; pktqueue_t *ip6_pktq __read_mostly; pfil_head_t *inet6_pfil_hook; percpu_t *ip6stat_percpu; percpu_t *ip6_forward_rt_percpu __cacheline_aligned; static void ip6intr(void *); static void ip6_input(struct mbuf *, struct ifnet *); static bool ip6_badaddr(struct ip6_hdr *); static struct m_tag *ip6_setdstifaddr(struct mbuf *, const struct in6_ifaddr *); static struct m_tag *ip6_addaux(struct mbuf *); static struct m_tag *ip6_findaux(struct mbuf *); static void ip6_delaux(struct mbuf *); static int ip6_process_hopopts(struct mbuf *, u_int8_t *, int, u_int32_t *, u_int32_t *); static struct mbuf *ip6_pullexthdr(struct mbuf *, size_t, int); static void sysctl_net_inet6_ip6_setup(struct sysctllog **); #ifdef NET_MPSAFE #define SOFTNET_LOCK() mutex_enter(softnet_lock) #define SOFTNET_UNLOCK() mutex_exit(softnet_lock) #else #define SOFTNET_LOCK() KASSERT(mutex_owned(softnet_lock)) #define SOFTNET_UNLOCK() KASSERT(mutex_owned(softnet_lock)) #endif /* Ensure that non packed structures are the desired size. */ __CTASSERT(sizeof(struct ip6_hdr) == 40); __CTASSERT(sizeof(struct ip6_ext) == 2); __CTASSERT(sizeof(struct ip6_hbh) == 2); __CTASSERT(sizeof(struct ip6_dest) == 2); __CTASSERT(sizeof(struct ip6_opt) == 2); __CTASSERT(sizeof(struct ip6_opt_jumbo) == 6); __CTASSERT(sizeof(struct ip6_opt_nsap) == 4); __CTASSERT(sizeof(struct ip6_opt_tunnel) == 3); __CTASSERT(sizeof(struct ip6_opt_router) == 4); __CTASSERT(sizeof(struct ip6_rthdr) == 4); __CTASSERT(sizeof(struct ip6_rthdr0) == 8); __CTASSERT(sizeof(struct ip6_frag) == 8); /* * IP6 initialization: fill in IP6 protocol switch table. * All protocols not implemented in kernel go to raw IP6 protocol handler. */ void ip6_init(void) { const struct ip6protosw *pr; int i; in6_init(); sysctl_net_inet6_ip6_setup(NULL); pr = (const struct ip6protosw *)pffindproto(PF_INET6, IPPROTO_RAW, SOCK_RAW); if (pr == 0) panic("ip6_init"); for (i = 0; i < IPPROTO_MAX; i++) ip6_protox[i] = pr - inet6sw; for (pr = (const struct ip6protosw *)inet6domain.dom_protosw; pr < (const struct ip6protosw *)inet6domain.dom_protoswNPROTOSW; pr++) if (pr->pr_domain->dom_family == PF_INET6 && pr->pr_protocol && pr->pr_protocol != IPPROTO_RAW) ip6_protox[pr->pr_protocol] = pr - inet6sw; ip6_pktq = pktq_create(IFQ_MAXLEN, ip6intr, NULL); KASSERT(ip6_pktq != NULL); scope6_init(); addrsel_policy_init(); nd6_init(); frag6_init(); #ifdef GATEWAY ip6flow_init(ip6_hashsize); #endif /* Register our Packet Filter hook. */ inet6_pfil_hook = pfil_head_create(PFIL_TYPE_AF, (void *)AF_INET6); KASSERT(inet6_pfil_hook != NULL); ip6stat_percpu = percpu_alloc(sizeof(uint64_t) * IP6_NSTATS); ip6_forward_rt_percpu = rtcache_percpu_alloc(); } /* * IP6 input interrupt handling. Just pass the packet to ip6_input. */ static void ip6intr(void *arg __unused) { struct mbuf *m; SOFTNET_KERNEL_LOCK_UNLESS_NET_MPSAFE(); while ((m = pktq_dequeue(ip6_pktq)) != NULL) { struct psref psref; struct ifnet *rcvif = m_get_rcvif_psref(m, &psref); if (rcvif == NULL) { IP6_STATINC(IP6_STAT_IFDROP); m_freem(m); continue; } /* * Drop the packet if IPv6 is disabled on the interface. */ if ((ND_IFINFO(rcvif)->flags & ND6_IFF_IFDISABLED)) { m_put_rcvif_psref(rcvif, &psref); IP6_STATINC(IP6_STAT_IFDROP); m_freem(m); continue; } ip6_input(m, rcvif); m_put_rcvif_psref(rcvif, &psref); } SOFTNET_KERNEL_UNLOCK_UNLESS_NET_MPSAFE(); } static void ip6_input(struct mbuf *m, struct ifnet *rcvif) { struct ip6_hdr *ip6; int hit, off = sizeof(struct ip6_hdr), nest; u_int32_t plen; u_int32_t rtalert = ~0; int nxt, ours = 0, rh_present = 0, frg_present; struct ifnet *deliverifp = NULL; int srcrt = 0; struct rtentry *rt = NULL; union { struct sockaddr dst; struct sockaddr_in6 dst6; } u; struct route *ro; KASSERT(rcvif != NULL); /* * make sure we don't have onion peering information into m_tag. */ ip6_delaux(m); /* * mbuf statistics */ if (m->m_flags & M_EXT) { if (m->m_next) IP6_STATINC(IP6_STAT_MEXT2M); else IP6_STATINC(IP6_STAT_MEXT1); } else { #define M2MMAX 32 if (m->m_next) { if (m->m_flags & M_LOOP) /*XXX*/ IP6_STATINC(IP6_STAT_M2M + lo0ifp->if_index); else if (rcvif->if_index < M2MMAX) IP6_STATINC(IP6_STAT_M2M + rcvif->if_index); else IP6_STATINC(IP6_STAT_M2M); } else IP6_STATINC(IP6_STAT_M1); #undef M2MMAX } in6_ifstat_inc(rcvif, ifs6_in_receive); IP6_STATINC(IP6_STAT_TOTAL); /* * If the IPv6 header is not aligned, slurp it up into a new * mbuf with space for link headers, in the event we forward * it. Otherwise, if it is aligned, make sure the entire base * IPv6 header is in the first mbuf of the chain. */ if (M_GET_ALIGNED_HDR(&m, struct ip6_hdr, true) != 0) { /* XXXJRT new stat, please */ IP6_STATINC(IP6_STAT_TOOSMALL); in6_ifstat_inc(rcvif, ifs6_in_hdrerr); return; } ip6 = mtod(m, struct ip6_hdr *); if ((ip6->ip6_vfc & IPV6_VERSION_MASK) != IPV6_VERSION) { IP6_STATINC(IP6_STAT_BADVERS); in6_ifstat_inc(rcvif, ifs6_in_hdrerr); goto bad; } if (ip6_badaddr(ip6)) { IP6_STATINC(IP6_STAT_BADSCOPE); in6_ifstat_inc(rcvif, ifs6_in_addrerr); goto bad; } /* * Assume that we can create a fast-forward IP flow entry * based on this packet. */ m->m_flags |= M_CANFASTFWD; /* * Run through list of hooks for input packets. If there are any * filters which require that additional packets in the flow are * not fast-forwarded, they must clear the M_CANFASTFWD flag. * Note that filters must _never_ set this flag, as another filter * in the list may have previously cleared it. * * Don't call hooks if the packet has already been processed by * IPsec (encapsulated, tunnel mode). */ #if defined(IPSEC) if (!ipsec_used || !ipsec_skip_pfil(m)) #else if (1) #endif { struct in6_addr odst; int error; odst = ip6->ip6_dst; error = pfil_run_hooks(inet6_pfil_hook, &m, rcvif, PFIL_IN); if (error != 0 || m == NULL) { IP6_STATINC(IP6_STAT_PFILDROP_IN); return; } if (m->m_len < sizeof(struct ip6_hdr)) { if ((m = m_pullup(m, sizeof(struct ip6_hdr))) == NULL) { IP6_STATINC(IP6_STAT_TOOSMALL); in6_ifstat_inc(rcvif, ifs6_in_hdrerr); return; } } ip6 = mtod(m, struct ip6_hdr *); srcrt = !IN6_ARE_ADDR_EQUAL(&odst, &ip6->ip6_dst); } IP6_STATINC(IP6_STAT_NXTHIST + ip6->ip6_nxt); #ifdef ALTQ if (altq_input != NULL) { SOFTNET_LOCK(); if ((*altq_input)(m, AF_INET6) == 0) { SOFTNET_UNLOCK(); /* packet is dropped by traffic conditioner */ return; } SOFTNET_UNLOCK(); } #endif /* * Disambiguate address scope zones (if there is ambiguity). * We first make sure that the original source or destination address * is not in our internal form for scoped addresses. Such addresses * are not necessarily invalid spec-wise, but we cannot accept them due * to the usage conflict. * in6_setscope() then also checks and rejects the cases where src or * dst are the loopback address and the receiving interface * is not loopback. */ if (__predict_false( m_makewritable(&m, 0, sizeof(struct ip6_hdr), M_DONTWAIT))) { IP6_STATINC(IP6_STAT_IDROPPED); goto bad; } ip6 = mtod(m, struct ip6_hdr *); if (in6_clearscope(&ip6->ip6_src) || in6_clearscope(&ip6->ip6_dst)) { IP6_STATINC(IP6_STAT_BADSCOPE); /* XXX */ goto bad; } if (in6_setscope(&ip6->ip6_src, rcvif, NULL) || in6_setscope(&ip6->ip6_dst, rcvif, NULL)) { IP6_STATINC(IP6_STAT_BADSCOPE); goto bad; } ro = rtcache_percpu_getref(ip6_forward_rt_percpu); /* * Multicast check */ if (IN6_IS_ADDR_MULTICAST(&ip6->ip6_dst)) { bool ingroup; in6_ifstat_inc(rcvif, ifs6_in_mcast); /* * See if we belong to the destination multicast group on the * arrival interface. */ ingroup = in6_multi_group(&ip6->ip6_dst, rcvif); if (ingroup) { ours = 1; } else if (!ip6_mrouter) { uint64_t *ip6s = IP6_STAT_GETREF(); ip6s[IP6_STAT_NOTMEMBER]++; ip6s[IP6_STAT_CANTFORWARD]++; IP6_STAT_PUTREF(); in6_ifstat_inc(rcvif, ifs6_in_discard); goto bad_unref; } deliverifp = rcvif; goto hbhcheck; } sockaddr_in6_init(&u.dst6, &ip6->ip6_dst, 0, 0, 0); /* * Unicast check */ rt = rtcache_lookup2(ro, &u.dst, 1, &hit); if (hit) IP6_STATINC(IP6_STAT_FORWARD_CACHEHIT); else IP6_STATINC(IP6_STAT_FORWARD_CACHEMISS); /* * Accept the packet if the forwarding interface to the destination * (according to the routing table) is the loopback interface, * unless the associated route has a gateway. * * We don't explicitly match ip6_dst against an interface here. It * is already done in rtcache_lookup2: rt->rt_ifp->if_type will be * IFT_LOOP if the packet is for us. * * Note that this approach causes to accept a packet if there is a * route to the loopback interface for the destination of the packet. * But we think it's even useful in some situations, e.g. when using * a special daemon which wants to intercept the packet. */ if (rt != NULL && (rt->rt_flags & (RTF_HOST|RTF_GATEWAY)) == RTF_HOST && rt->rt_ifp->if_type == IFT_LOOP) { struct in6_ifaddr *ia6 = (struct in6_ifaddr *)rt->rt_ifa; int addrok; if (ia6->ia6_flags & IN6_IFF_ANYCAST) m->m_flags |= M_ANYCAST6; /* * packets to a tentative, duplicated, or somehow invalid * address must not be accepted. */ if (ia6->ia6_flags & IN6_IFF_NOTREADY) addrok = 0; else if (ia6->ia6_flags & IN6_IFF_DETACHED && !IN6_IS_ADDR_UNSPECIFIED(&ip6->ip6_src)) { /* Allow internal traffic to DETACHED addresses */ struct sockaddr_in6 sin6; int s; memset(&sin6, 0, sizeof(sin6)); sin6.sin6_family = AF_INET6; sin6.sin6_len = sizeof(sin6); sin6.sin6_addr = ip6->ip6_src; s = pserialize_read_enter(); addrok = (ifa_ifwithaddr(sin6tosa(&sin6)) != NULL); pserialize_read_exit(s); } else addrok = 1; if (addrok) { /* this address is ready */ ours = 1; deliverifp = ia6->ia_ifp; /* correct? */ goto hbhcheck; } else { /* address is not ready, so discard the packet. */ char ip6bufs[INET6_ADDRSTRLEN]; char ip6bufd[INET6_ADDRSTRLEN]; nd6log(LOG_INFO, "packet to an unready address %s->%s\n", IN6_PRINT(ip6bufs, &ip6->ip6_src), IN6_PRINT(ip6bufd, &ip6->ip6_dst)); IP6_STATINC(IP6_STAT_IDROPPED); goto bad_unref; } } /* * FAITH (Firewall Aided Internet Translator) */ #if defined(NFAITH) && 0 < NFAITH if (ip6_keepfaith) { if (rt != NULL && rt->rt_ifp != NULL && rt->rt_ifp->if_type == IFT_FAITH) { /* XXX do we need more sanity checks? */ ours = 1; deliverifp = rt->rt_ifp; /* faith */ goto hbhcheck; } } #endif /* * Now there is no reason to process the packet if it's not our own * and we're not a router. */ if (!ip6_forwarding) { IP6_STATINC(IP6_STAT_CANTFORWARD); in6_ifstat_inc(rcvif, ifs6_in_discard); goto bad_unref; } hbhcheck: /* * Record address information into m_tag, if we don't have one yet. * Note that we are unable to record it, if the address is not listed * as our interface address (e.g. multicast addresses, addresses * within FAITH prefixes and such). */ if (deliverifp && ip6_getdstifaddr(m) == NULL) { struct in6_ifaddr *ia6; int s = pserialize_read_enter(); ia6 = in6_ifawithifp(deliverifp, &ip6->ip6_dst); /* Depends on ip6_setdstifaddr never sleep */ if (ia6 != NULL && ip6_setdstifaddr(m, ia6) == NULL) { /* * XXX maybe we should drop the packet here, * as we could not provide enough information * to the upper layers. */ } pserialize_read_exit(s); } /* * Process Hop-by-Hop options header if it's contained. * m may be modified in ip6_hopopts_input(). * If a JumboPayload option is included, plen will also be modified. */ plen = (u_int32_t)ntohs(ip6->ip6_plen); if (ip6->ip6_nxt == IPPROTO_HOPOPTS) { struct ip6_hbh *hbh; if (ip6_hopopts_input(&plen, &rtalert, &m, &off)) { /* m already freed */ in6_ifstat_inc(rcvif, ifs6_in_discard); rtcache_unref(rt, ro); rtcache_percpu_putref(ip6_forward_rt_percpu); return; } /* adjust pointer */ ip6 = mtod(m, struct ip6_hdr *); /* * if the payload length field is 0 and the next header field * indicates Hop-by-Hop Options header, then a Jumbo Payload * option MUST be included. */ if (ip6->ip6_plen == 0 && plen == 0) { /* * Note that if a valid jumbo payload option is * contained, ip6_hopopts_input() must set a valid * (non-zero) payload length to the variable plen. */ IP6_STATINC(IP6_STAT_BADOPTIONS); in6_ifstat_inc(rcvif, ifs6_in_discard); in6_ifstat_inc(rcvif, ifs6_in_hdrerr); icmp6_error(m, ICMP6_PARAM_PROB, ICMP6_PARAMPROB_HEADER, (char *)&ip6->ip6_plen - (char *)ip6); rtcache_unref(rt, ro); rtcache_percpu_putref(ip6_forward_rt_percpu); return; } IP6_EXTHDR_GET(hbh, struct ip6_hbh *, m, sizeof(struct ip6_hdr), sizeof(struct ip6_hbh)); if (hbh == NULL) { IP6_STATINC(IP6_STAT_TOOSHORT); rtcache_unref(rt, ro); rtcache_percpu_putref(ip6_forward_rt_percpu); return; } KASSERT(ACCESSIBLE_POINTER(hbh, struct ip6_hdr)); nxt = hbh->ip6h_nxt; /* * accept the packet if a router alert option is included * and we act as an IPv6 router. */ if (rtalert != ~0 && ip6_forwarding) ours = 1; } else nxt = ip6->ip6_nxt; /* * Check that the amount of data in the buffers is at least much as * the IPv6 header would have us expect. Trim mbufs if longer than we * expect. Drop packet if shorter than we expect. */ if (m->m_pkthdr.len - sizeof(struct ip6_hdr) < plen) { IP6_STATINC(IP6_STAT_TOOSHORT); in6_ifstat_inc(rcvif, ifs6_in_truncated); goto bad_unref; } if (m->m_pkthdr.len > sizeof(struct ip6_hdr) + plen) { if (m->m_len == m->m_pkthdr.len) { m->m_len = sizeof(struct ip6_hdr) + plen; m->m_pkthdr.len = sizeof(struct ip6_hdr) + plen; } else m_adj(m, sizeof(struct ip6_hdr) + plen - m->m_pkthdr.len); } /* * Forward if desirable. */ if (IN6_IS_ADDR_MULTICAST(&ip6->ip6_dst)) { /* * If we are acting as a multicast router, all * incoming multicast packets are passed to the * kernel-level multicast forwarding function. * The packet is returned (relatively) intact; if * ip6_mforward() returns a non-zero value, the packet * must be discarded, else it may be accepted below. */ if (ip6_mrouter != NULL) { int error; SOFTNET_LOCK(); error = ip6_mforward(ip6, rcvif, m); SOFTNET_UNLOCK(); if (error != 0) { rtcache_unref(rt, ro); rtcache_percpu_putref(ip6_forward_rt_percpu); IP6_STATINC(IP6_STAT_CANTFORWARD); goto bad; } } if (!ours) { IP6_STATINC(IP6_STAT_CANTFORWARD); goto bad_unref; } } else if (!ours) { rtcache_unref(rt, ro); rtcache_percpu_putref(ip6_forward_rt_percpu); ip6_forward(m, srcrt, rcvif); return; } ip6 = mtod(m, struct ip6_hdr *); /* * Malicious party may be able to use IPv4 mapped addr to confuse * tcp/udp stack and bypass security checks (act as if it was from * 127.0.0.1 by using IPv6 src ::ffff:127.0.0.1). Be cautious. * * For SIIT end node behavior, you may want to disable the check. * However, you will become vulnerable to attacks using IPv4 mapped * source. */ if (IN6_IS_ADDR_V4MAPPED(&ip6->ip6_src) || IN6_IS_ADDR_V4MAPPED(&ip6->ip6_dst)) { IP6_STATINC(IP6_STAT_BADSCOPE); in6_ifstat_inc(rcvif, ifs6_in_addrerr); goto bad_unref; } #ifdef IFA_STATS if (deliverifp != NULL) { struct in6_ifaddr *ia6; int s = pserialize_read_enter(); ia6 = in6_ifawithifp(deliverifp, &ip6->ip6_dst); if (ia6) ia6->ia_ifa.ifa_data.ifad_inbytes += m->m_pkthdr.len; pserialize_read_exit(s); } #endif IP6_STATINC(IP6_STAT_DELIVERED); in6_ifstat_inc(deliverifp, ifs6_in_deliver); nest = 0; if (rt != NULL) { rtcache_unref(rt, ro); rt = NULL; } rtcache_percpu_putref(ip6_forward_rt_percpu); rh_present = 0; frg_present = 0; while (nxt != IPPROTO_DONE) { if (ip6_hdrnestlimit && (++nest > ip6_hdrnestlimit)) { IP6_STATINC(IP6_STAT_TOOMANYHDR); in6_ifstat_inc(rcvif, ifs6_in_hdrerr); goto bad; } M_VERIFY_PACKET(m); /* * protection against faulty packet - there should be * more sanity checks in header chain processing. */ if (m->m_pkthdr.len < off) { IP6_STATINC(IP6_STAT_TOOSHORT); in6_ifstat_inc(rcvif, ifs6_in_truncated); goto bad; } if (nxt == IPPROTO_ROUTING) { if (rh_present++) { in6_ifstat_inc(rcvif, ifs6_in_hdrerr); IP6_STATINC(IP6_STAT_BADOPTIONS); goto bad; } } else if (nxt == IPPROTO_FRAGMENT) { if (frg_present++) { in6_ifstat_inc(rcvif, ifs6_in_hdrerr); IP6_STATINC(IP6_STAT_BADOPTIONS); goto bad; } } #ifdef IPSEC if (ipsec_used) { /* * Enforce IPsec policy checking if we are seeing last * header. Note that we do not visit this with * protocols with pcb layer code - like udp/tcp/raw ip. */ if ((inet6sw[ip6_protox[nxt]].pr_flags & PR_LASTHDR) != 0) { int error; error = ipsec_ip_input_checkpolicy(m, false); if (error) { IP6_STATINC(IP6_STAT_IPSECDROP_IN); goto bad; } } } #endif nxt = (*inet6sw[ip6_protox[nxt]].pr_input)(&m, &off, nxt); } return; bad_unref: rtcache_unref(rt, ro); rtcache_percpu_putref(ip6_forward_rt_percpu); bad: m_freem(m); return; } static bool ip6_badaddr(struct ip6_hdr *ip6) { /* Check against address spoofing/corruption. */ if (IN6_IS_ADDR_MULTICAST(&ip6->ip6_src) || IN6_IS_ADDR_UNSPECIFIED(&ip6->ip6_dst)) { return true; } /* * The following check is not documented in specs. A malicious * party may be able to use IPv4 mapped addr to confuse tcp/udp stack * and bypass security checks (act as if it was from 127.0.0.1 by using * IPv6 src ::ffff:127.0.0.1). Be cautious. * * This check chokes if we are in an SIIT cloud. As none of BSDs * support IPv4-less kernel compilation, we cannot support SIIT * environment at all. So, it makes more sense for us to reject any * malicious packets for non-SIIT environment, than try to do a * partial support for SIIT environment. */ if (IN6_IS_ADDR_V4MAPPED(&ip6->ip6_src) || IN6_IS_ADDR_V4MAPPED(&ip6->ip6_dst)) { return true; } /* * Reject packets with IPv4-compatible IPv6 addresses (RFC4291). */ if (IN6_IS_ADDR_V4COMPAT(&ip6->ip6_src) || IN6_IS_ADDR_V4COMPAT(&ip6->ip6_dst)) { return true; } return false; } /* * set/grab in6_ifaddr correspond to IPv6 destination address. */ static struct m_tag * ip6_setdstifaddr(struct mbuf *m, const struct in6_ifaddr *ia) { struct m_tag *mtag; struct ip6aux *ip6a; mtag = ip6_addaux(m); if (mtag == NULL) return NULL; ip6a = (struct ip6aux *)(mtag + 1); if (in6_setscope(&ip6a->ip6a_src, ia->ia_ifp, &ip6a->ip6a_scope_id)) { IP6_STATINC(IP6_STAT_BADSCOPE); return NULL; } ip6a->ip6a_src = ia->ia_addr.sin6_addr; ip6a->ip6a_flags = ia->ia6_flags; return mtag; } const struct ip6aux * ip6_getdstifaddr(struct mbuf *m) { struct m_tag *mtag; mtag = ip6_findaux(m); if (mtag != NULL) return (struct ip6aux *)(mtag + 1); else return NULL; } /* * Hop-by-Hop options header processing. If a valid jumbo payload option is * included, the real payload length will be stored in plenp. * * rtalertp - XXX: should be stored more smart way */ int ip6_hopopts_input(u_int32_t *plenp, u_int32_t *rtalertp, struct mbuf **mp, int *offp) { struct mbuf *m = *mp; int off = *offp, hbhlen; struct ip6_hbh *hbh; /* validation of the length of the header */ IP6_EXTHDR_GET(hbh, struct ip6_hbh *, m, sizeof(struct ip6_hdr), sizeof(struct ip6_hbh)); if (hbh == NULL) { IP6_STATINC(IP6_STAT_TOOSHORT); return -1; } hbhlen = (hbh->ip6h_len + 1) << 3; IP6_EXTHDR_GET(hbh, struct ip6_hbh *, m, sizeof(struct ip6_hdr), hbhlen); if (hbh == NULL) { IP6_STATINC(IP6_STAT_TOOSHORT); return -1; } KASSERT(ACCESSIBLE_POINTER(hbh, struct ip6_hdr)); off += hbhlen; hbhlen -= sizeof(struct ip6_hbh); if (ip6_process_hopopts(m, (u_int8_t *)hbh + sizeof(struct ip6_hbh), hbhlen, rtalertp, plenp) < 0) return -1; *offp = off; *mp = m; return 0; } /* * Search header for all Hop-by-hop options and process each option. * This function is separate from ip6_hopopts_input() in order to * handle a case where the sending node itself process its hop-by-hop * options header. In such a case, the function is called from ip6_output(). * * The function assumes that hbh header is located right after the IPv6 header * (RFC2460 p7), opthead is pointer into data content in m, and opthead to * opthead + hbhlen is located in continuous memory region. */ static int ip6_process_hopopts(struct mbuf *m, u_int8_t *opthead, int hbhlen, u_int32_t *rtalertp, u_int32_t *plenp) { struct ip6_hdr *ip6; int optlen = 0; u_int8_t *opt = opthead; u_int16_t rtalert_val; u_int32_t jumboplen; const int erroff = sizeof(struct ip6_hdr) + sizeof(struct ip6_hbh); for (; hbhlen > 0; hbhlen -= optlen, opt += optlen) { switch (*opt) { case IP6OPT_PAD1: optlen = 1; break; case IP6OPT_PADN: if (hbhlen < IP6OPT_MINLEN) { IP6_STATINC(IP6_STAT_TOOSMALL); goto bad; } optlen = *(opt + 1) + 2; break; case IP6OPT_RTALERT: /* XXX may need check for alignment */ if (hbhlen < IP6OPT_RTALERT_LEN) { IP6_STATINC(IP6_STAT_TOOSMALL); goto bad; } if (*(opt + 1) != IP6OPT_RTALERT_LEN - 2) { IP6_STATINC(IP6_STAT_BADOPTIONS); icmp6_error(m, ICMP6_PARAM_PROB, ICMP6_PARAMPROB_HEADER, erroff + opt + 1 - opthead); return (-1); } optlen = IP6OPT_RTALERT_LEN; memcpy((void *)&rtalert_val, (void *)(opt + 2), 2); *rtalertp = ntohs(rtalert_val); break; case IP6OPT_JUMBO: /* XXX may need check for alignment */ if (hbhlen < IP6OPT_JUMBO_LEN) { IP6_STATINC(IP6_STAT_TOOSMALL); goto bad; } if (*(opt + 1) != IP6OPT_JUMBO_LEN - 2) { IP6_STATINC(IP6_STAT_BADOPTIONS); icmp6_error(m, ICMP6_PARAM_PROB, ICMP6_PARAMPROB_HEADER, erroff + opt + 1 - opthead); return (-1); } optlen = IP6OPT_JUMBO_LEN; /* * IPv6 packets that have non 0 payload length * must not contain a jumbo payload option. */ ip6 = mtod(m, struct ip6_hdr *); if (ip6->ip6_plen) { IP6_STATINC(IP6_STAT_BADOPTIONS); icmp6_error(m, ICMP6_PARAM_PROB, ICMP6_PARAMPROB_HEADER, erroff + opt - opthead); return (-1); } /* * We may see jumbolen in unaligned location, so * we'd need to perform memcpy(). */ memcpy(&jumboplen, opt + 2, sizeof(jumboplen)); jumboplen = (u_int32_t)htonl(jumboplen); #if 1 /* * if there are multiple jumbo payload options, * *plenp will be non-zero and the packet will be * rejected. * the behavior may need some debate in ipngwg - * multiple options does not make sense, however, * there's no explicit mention in specification. */ if (*plenp != 0) { IP6_STATINC(IP6_STAT_BADOPTIONS); icmp6_error(m, ICMP6_PARAM_PROB, ICMP6_PARAMPROB_HEADER, erroff + opt + 2 - opthead); return (-1); } #endif /* * jumbo payload length must be larger than 65535. */ if (jumboplen <= IPV6_MAXPACKET) { IP6_STATINC(IP6_STAT_BADOPTIONS); icmp6_error(m, ICMP6_PARAM_PROB, ICMP6_PARAMPROB_HEADER, erroff + opt + 2 - opthead); return (-1); } *plenp = jumboplen; break; default: /* unknown option */ if (hbhlen < IP6OPT_MINLEN) { IP6_STATINC(IP6_STAT_TOOSMALL); goto bad; } optlen = ip6_unknown_opt(opt, m, erroff + opt - opthead); if (optlen == -1) return (-1); optlen += 2; break; } } return (0); bad: m_freem(m); return (-1); } /* * Unknown option processing. * The third argument `off' is the offset from the IPv6 header to the option, * which is necessary if the IPv6 header the and option header and IPv6 header * is not continuous in order to return an ICMPv6 error. */ int ip6_unknown_opt(u_int8_t *optp, struct mbuf *m, int off) { struct ip6_hdr *ip6; switch (IP6OPT_TYPE(*optp)) { case IP6OPT_TYPE_SKIP: /* ignore the option */ return ((int)*(optp + 1)); case IP6OPT_TYPE_DISCARD: /* silently discard */ m_freem(m); return (-1); case IP6OPT_TYPE_FORCEICMP: /* send ICMP even if multicasted */ IP6_STATINC(IP6_STAT_BADOPTIONS); icmp6_error(m, ICMP6_PARAM_PROB, ICMP6_PARAMPROB_OPTION, off); return (-1); case IP6OPT_TYPE_ICMP: /* send ICMP if not multicasted */ IP6_STATINC(IP6_STAT_BADOPTIONS); ip6 = mtod(m, struct ip6_hdr *); if (IN6_IS_ADDR_MULTICAST(&ip6->ip6_dst) || (m->m_flags & (M_BCAST|M_MCAST))) m_freem(m); else icmp6_error(m, ICMP6_PARAM_PROB, ICMP6_PARAMPROB_OPTION, off); return (-1); } m_freem(m); /* XXX: NOTREACHED */ return (-1); } void ip6_savecontrol(struct in6pcb *in6p, struct mbuf **mp, struct ip6_hdr *ip6, struct mbuf *m) { struct socket *so = in6p->in6p_socket; #ifdef RFC2292 #define IS2292(x, y) ((in6p->in6p_flags & IN6P_RFC2292) ? (x) : (y)) #else #define IS2292(x, y) (y) #endif KASSERT(m->m_flags & M_PKTHDR); if (SOOPT_TIMESTAMP(so->so_options)) mp = sbsavetimestamp(so->so_options, mp); /* some OSes call this logic with IPv4 packet, for SO_TIMESTAMP */ if ((ip6->ip6_vfc & IPV6_VERSION_MASK) != IPV6_VERSION) return; /* RFC 2292 sec. 5 */ if ((in6p->in6p_flags & IN6P_PKTINFO) != 0) { struct in6_pktinfo pi6; memcpy(&pi6.ipi6_addr, &ip6->ip6_dst, sizeof(struct in6_addr)); in6_clearscope(&pi6.ipi6_addr); /* XXX */ pi6.ipi6_ifindex = m->m_pkthdr.rcvif_index; *mp = sbcreatecontrol(&pi6, sizeof(pi6), IS2292(IPV6_2292PKTINFO, IPV6_PKTINFO), IPPROTO_IPV6); if (*mp) mp = &(*mp)->m_next; } if (in6p->in6p_flags & IN6P_HOPLIMIT) { int hlim = ip6->ip6_hlim & 0xff; *mp = sbcreatecontrol(&hlim, sizeof(hlim), IS2292(IPV6_2292HOPLIMIT, IPV6_HOPLIMIT), IPPROTO_IPV6); if (*mp) mp = &(*mp)->m_next; } if ((in6p->in6p_flags & IN6P_TCLASS) != 0) { u_int32_t flowinfo; int tclass; flowinfo = (u_int32_t)ntohl(ip6->ip6_flow & IPV6_FLOWINFO_MASK); flowinfo >>= 20; tclass = flowinfo & 0xff; *mp = sbcreatecontrol(&tclass, sizeof(tclass), IPV6_TCLASS, IPPROTO_IPV6); if (*mp) mp = &(*mp)->m_next; } /* * IPV6_HOPOPTS socket option. Recall that we required super-user * privilege for the option (see ip6_ctloutput), but it might be too * strict, since there might be some hop-by-hop options which can be * returned to normal user. * See also RFC3542 section 8 (or RFC2292 section 6). */ if ((in6p->in6p_flags & IN6P_HOPOPTS) != 0) { /* * Check if a hop-by-hop options header is contatined in the * received packet, and if so, store the options as ancillary * data. Note that a hop-by-hop options header must be * just after the IPv6 header, which fact is assured through * the IPv6 input processing. */ struct ip6_hdr *xip6 = mtod(m, struct ip6_hdr *); if (xip6->ip6_nxt == IPPROTO_HOPOPTS) { struct ip6_hbh *hbh; int hbhlen; struct mbuf *ext; ext = ip6_pullexthdr(m, sizeof(struct ip6_hdr), xip6->ip6_nxt); if (ext == NULL) { IP6_STATINC(IP6_STAT_TOOSHORT); return; } hbh = mtod(ext, struct ip6_hbh *); hbhlen = (hbh->ip6h_len + 1) << 3; if (hbhlen != ext->m_len) { m_freem(ext); IP6_STATINC(IP6_STAT_TOOSHORT); return; } /* * XXX: We copy whole the header even if a jumbo * payload option is included, which option is to * be removed before returning in the RFC 2292. * Note: this constraint is removed in RFC3542. */ *mp = sbcreatecontrol(hbh, hbhlen, IS2292(IPV6_2292HOPOPTS, IPV6_HOPOPTS), IPPROTO_IPV6); if (*mp) mp = &(*mp)->m_next; m_freem(ext); } } /* IPV6_DSTOPTS and IPV6_RTHDR socket options */ if (in6p->in6p_flags & (IN6P_DSTOPTS | IN6P_RTHDR)) { struct ip6_hdr *xip6 = mtod(m, struct ip6_hdr *); int nxt = xip6->ip6_nxt, off = sizeof(struct ip6_hdr); /* * Search for destination options headers or routing * header(s) through the header chain, and stores each * header as ancillary data. * Note that the order of the headers remains in * the chain of ancillary data. */ for (;;) { /* is explicit loop prevention necessary? */ struct ip6_ext *ip6e = NULL; int elen; struct mbuf *ext = NULL; /* * if it is not an extension header, don't try to * pull it from the chain. */ switch (nxt) { case IPPROTO_DSTOPTS: case IPPROTO_ROUTING: case IPPROTO_HOPOPTS: case IPPROTO_AH: /* is it possible? */ break; default: goto loopend; } ext = ip6_pullexthdr(m, off, nxt); if (ext == NULL) { IP6_STATINC(IP6_STAT_TOOSHORT); return; } ip6e = mtod(ext, struct ip6_ext *); if (nxt == IPPROTO_AH) elen = (ip6e->ip6e_len + 2) << 2; else elen = (ip6e->ip6e_len + 1) << 3; if (elen != ext->m_len) { m_freem(ext); IP6_STATINC(IP6_STAT_TOOSHORT); return; } KASSERT(ACCESSIBLE_POINTER(ip6e, struct ip6_hdr)); switch (nxt) { case IPPROTO_DSTOPTS: if (!(in6p->in6p_flags & IN6P_DSTOPTS)) break; *mp = sbcreatecontrol(ip6e, elen, IS2292(IPV6_2292DSTOPTS, IPV6_DSTOPTS), IPPROTO_IPV6); if (*mp) mp = &(*mp)->m_next; break; case IPPROTO_ROUTING: if (!(in6p->in6p_flags & IN6P_RTHDR)) break; *mp = sbcreatecontrol(ip6e, elen, IS2292(IPV6_2292RTHDR, IPV6_RTHDR), IPPROTO_IPV6); if (*mp) mp = &(*mp)->m_next; break; case IPPROTO_HOPOPTS: case IPPROTO_AH: /* is it possible? */ break; default: /* * other cases have been filtered in the above. * none will visit this case. here we supply * the code just in case (nxt overwritten or * other cases). */ m_freem(ext); goto loopend; } /* proceed with the next header. */ off += elen; nxt = ip6e->ip6e_nxt; ip6e = NULL; m_freem(ext); ext = NULL; } loopend: ; } } #undef IS2292 void ip6_notify_pmtu(struct in6pcb *in6p, const struct sockaddr_in6 *dst, uint32_t *mtu) { struct socket *so; struct mbuf *m_mtu; struct ip6_mtuinfo mtuctl; so = in6p->in6p_socket; if (mtu == NULL) return; KASSERT(so != NULL); memset(&mtuctl, 0, sizeof(mtuctl)); /* zero-clear for safety */ mtuctl.ip6m_mtu = *mtu; mtuctl.ip6m_addr = *dst; if (sa6_recoverscope(&mtuctl.ip6m_addr)) return; if ((m_mtu = sbcreatecontrol(&mtuctl, sizeof(mtuctl), IPV6_PATHMTU, IPPROTO_IPV6)) == NULL) return; if (sbappendaddr(&so->so_rcv, (const struct sockaddr *)dst, NULL, m_mtu) == 0) { soroverflow(so); m_freem(m_mtu); } else sorwakeup(so); return; } /* * pull single extension header from mbuf chain. returns single mbuf that * contains the result, or NULL on error. */ static struct mbuf * ip6_pullexthdr(struct mbuf *m, size_t off, int nxt) { struct ip6_ext ip6e; size_t elen; struct mbuf *n; if (off + sizeof(ip6e) > m->m_pkthdr.len) return NULL; m_copydata(m, off, sizeof(ip6e), (void *)&ip6e); if (nxt == IPPROTO_AH) elen = (ip6e.ip6e_len + 2) << 2; else elen = (ip6e.ip6e_len + 1) << 3; if (off + elen > m->m_pkthdr.len) return NULL; MGET(n, M_DONTWAIT, MT_DATA); if (n && elen >= MLEN) { MCLGET(n, M_DONTWAIT); if ((n->m_flags & M_EXT) == 0) { m_free(n); n = NULL; } } if (!n) return NULL; n->m_len = 0; if (elen >= M_TRAILINGSPACE(n)) { m_free(n); return NULL; } m_copydata(m, off, elen, mtod(n, void *)); n->m_len = elen; return n; } /* * Get offset to the previous header followed by the header * currently processed. */ int ip6_get_prevhdr(struct mbuf *m, int off) { struct ip6_hdr *ip6 = mtod(m, struct ip6_hdr *); if (off == sizeof(struct ip6_hdr)) { return offsetof(struct ip6_hdr, ip6_nxt); } else if (off < sizeof(struct ip6_hdr)) { panic("%s: off < sizeof(struct ip6_hdr)", __func__); } else { int len, nlen, nxt; struct ip6_ext ip6e; nxt = ip6->ip6_nxt; len = sizeof(struct ip6_hdr); nlen = 0; while (len < off) { m_copydata(m, len, sizeof(ip6e), &ip6e); switch (nxt) { case IPPROTO_FRAGMENT: nlen = sizeof(struct ip6_frag); break; case IPPROTO_AH: nlen = (ip6e.ip6e_len + 2) << 2; break; default: nlen = (ip6e.ip6e_len + 1) << 3; break; } len += nlen; nxt = ip6e.ip6e_nxt; } return (len - nlen); } } /* * get next header offset. m will be retained. */ int ip6_nexthdr(struct mbuf *m, int off, int proto, int *nxtp) { struct ip6_hdr ip6; struct ip6_ext ip6e; struct ip6_frag fh; /* just in case */ if (m == NULL) panic("%s: m == NULL", __func__); if ((m->m_flags & M_PKTHDR) == 0 || m->m_pkthdr.len < off) return -1; switch (proto) { case IPPROTO_IPV6: /* do not chase beyond intermediate IPv6 headers */ if (off != 0) return -1; if (m->m_pkthdr.len < off + sizeof(ip6)) return -1; m_copydata(m, off, sizeof(ip6), (void *)&ip6); if (nxtp) *nxtp = ip6.ip6_nxt; off += sizeof(ip6); return off; case IPPROTO_FRAGMENT: /* * terminate parsing if it is not the first fragment, * it does not make sense to parse through it. */ if (m->m_pkthdr.len < off + sizeof(fh)) return -1; m_copydata(m, off, sizeof(fh), (void *)&fh); if ((fh.ip6f_offlg & IP6F_OFF_MASK) != 0) return -1; if (nxtp) *nxtp = fh.ip6f_nxt; off += sizeof(struct ip6_frag); return off; case IPPROTO_AH: if (m->m_pkthdr.len < off + sizeof(ip6e)) return -1; m_copydata(m, off, sizeof(ip6e), (void *)&ip6e); if (nxtp) *nxtp = ip6e.ip6e_nxt; off += (ip6e.ip6e_len + 2) << 2; if (m->m_pkthdr.len < off) return -1; return off; case IPPROTO_HOPOPTS: case IPPROTO_ROUTING: case IPPROTO_DSTOPTS: if (m->m_pkthdr.len < off + sizeof(ip6e)) return -1; m_copydata(m, off, sizeof(ip6e), (void *)&ip6e); if (nxtp) *nxtp = ip6e.ip6e_nxt; off += (ip6e.ip6e_len + 1) << 3; if (m->m_pkthdr.len < off) return -1; return off; case IPPROTO_NONE: case IPPROTO_ESP: case IPPROTO_IPCOMP: /* give up */ return -1; default: return -1; } } /* * get offset for the last header in the chain. m will be kept untainted. */ int ip6_lasthdr(struct mbuf *m, int off, int proto, int *nxtp) { int newoff; int nxt; if (!nxtp) { nxt = -1; nxtp = &nxt; } for (;;) { newoff = ip6_nexthdr(m, off, proto, nxtp); if (newoff < 0) return off; else if (newoff < off) return -1; /* invalid */ else if (newoff == off) return newoff; off = newoff; proto = *nxtp; } } static struct m_tag * ip6_addaux(struct mbuf *m) { struct m_tag *mtag; mtag = m_tag_find(m, PACKET_TAG_INET6); if (!mtag) { mtag = m_tag_get(PACKET_TAG_INET6, sizeof(struct ip6aux), M_NOWAIT); if (mtag) { m_tag_prepend(m, mtag); memset(mtag + 1, 0, sizeof(struct ip6aux)); } } return mtag; } static struct m_tag * ip6_findaux(struct mbuf *m) { struct m_tag *mtag; mtag = m_tag_find(m, PACKET_TAG_INET6); return mtag; } static void ip6_delaux(struct mbuf *m) { struct m_tag *mtag; mtag = m_tag_find(m, PACKET_TAG_INET6); if (mtag) m_tag_delete(m, mtag); } /* * System control for IP6 */ const u_char inet6ctlerrmap[PRC_NCMDS] = { 0, 0, 0, 0, 0, EMSGSIZE, EHOSTDOWN, EHOSTUNREACH, EHOSTUNREACH, EHOSTUNREACH, ECONNREFUSED, ECONNREFUSED, EMSGSIZE, EHOSTUNREACH, 0, 0, 0, 0, 0, 0, ENOPROTOOPT }; extern int sysctl_net_inet6_addrctlpolicy(SYSCTLFN_ARGS); static int sysctl_net_inet6_ip6_stats(SYSCTLFN_ARGS) { return (NETSTAT_SYSCTL(ip6stat_percpu, IP6_NSTATS)); } static void sysctl_net_inet6_ip6_setup(struct sysctllog **clog) { sysctl_createv(clog, 0, NULL, NULL, CTLFLAG_PERMANENT, CTLTYPE_NODE, "inet6", SYSCTL_DESCR("PF_INET6 related settings"), NULL, 0, NULL, 0, CTL_NET, PF_INET6, CTL_EOL); sysctl_createv(clog, 0, NULL, NULL, CTLFLAG_PERMANENT, CTLTYPE_NODE, "ip6", SYSCTL_DESCR("IPv6 related settings"), NULL, 0, NULL, 0, CTL_NET, PF_INET6, IPPROTO_IPV6, CTL_EOL); sysctl_createv(clog, 0, NULL, NULL, CTLFLAG_PERMANENT|CTLFLAG_READWRITE, CTLTYPE_INT, "forwarding", SYSCTL_DESCR("Enable forwarding of INET6 datagrams"), NULL, 0, &ip6_forwarding, 0, CTL_NET, PF_INET6, IPPROTO_IPV6, IPV6CTL_FORWARDING, CTL_EOL); sysctl_createv(clog, 0, NULL, NULL, CTLFLAG_PERMANENT|CTLFLAG_READWRITE, CTLTYPE_INT, "redirect", SYSCTL_DESCR("Enable sending of ICMPv6 redirect messages"), NULL, 0, &ip6_sendredirects, 0, CTL_NET, PF_INET6, IPPROTO_IPV6, IPV6CTL_SENDREDIRECTS, CTL_EOL); sysctl_createv(clog, 0, NULL, NULL, CTLFLAG_PERMANENT|CTLFLAG_READWRITE, CTLTYPE_INT, "hlim", SYSCTL_DESCR("Hop limit for an INET6 datagram"), NULL, 0, &ip6_defhlim, 0, CTL_NET, PF_INET6, IPPROTO_IPV6, IPV6CTL_DEFHLIM, CTL_EOL); sysctl_createv(clog, 0, NULL, NULL, CTLFLAG_PERMANENT|CTLFLAG_READWRITE, CTLTYPE_INT, "maxfragpackets", SYSCTL_DESCR("Maximum number of fragments to buffer " "for reassembly"), NULL, 0, &ip6_maxfragpackets, 0, CTL_NET, PF_INET6, IPPROTO_IPV6, IPV6CTL_MAXFRAGPACKETS, CTL_EOL); sysctl_createv(clog, 0, NULL, NULL, CTLFLAG_PERMANENT|CTLFLAG_READWRITE, CTLTYPE_INT, "keepfaith", SYSCTL_DESCR("Activate faith interface"), NULL, 0, &ip6_keepfaith, 0, CTL_NET, PF_INET6, IPPROTO_IPV6, IPV6CTL_KEEPFAITH, CTL_EOL); sysctl_createv(clog, 0, NULL, NULL, CTLFLAG_PERMANENT|CTLFLAG_READWRITE, CTLTYPE_INT, "log_interval", SYSCTL_DESCR("Minimum interval between logging " "unroutable packets"), NULL, 0, &ip6_log_interval, 0, CTL_NET, PF_INET6, IPPROTO_IPV6, IPV6CTL_LOG_INTERVAL, CTL_EOL); sysctl_createv(clog, 0, NULL, NULL, CTLFLAG_PERMANENT|CTLFLAG_READWRITE, CTLTYPE_INT, "hdrnestlimit", SYSCTL_DESCR("Maximum number of nested IPv6 headers"), NULL, 0, &ip6_hdrnestlimit, 0, CTL_NET, PF_INET6, IPPROTO_IPV6, IPV6CTL_HDRNESTLIMIT, CTL_EOL); sysctl_createv(clog, 0, NULL, NULL, CTLFLAG_PERMANENT|CTLFLAG_READWRITE, CTLTYPE_INT, "dad_count", SYSCTL_DESCR("Number of Duplicate Address Detection " "probes to send"), NULL, 0, &ip6_dad_count, 0, CTL_NET, PF_INET6, IPPROTO_IPV6, IPV6CTL_DAD_COUNT, CTL_EOL); sysctl_createv(clog, 0, NULL, NULL, CTLFLAG_PERMANENT|CTLFLAG_READWRITE, CTLTYPE_INT, "auto_flowlabel", SYSCTL_DESCR("Assign random IPv6 flow labels"), NULL, 0, &ip6_auto_flowlabel, 0, CTL_NET, PF_INET6, IPPROTO_IPV6, IPV6CTL_AUTO_FLOWLABEL, CTL_EOL); sysctl_createv(clog, 0, NULL, NULL, CTLFLAG_PERMANENT|CTLFLAG_READWRITE, CTLTYPE_INT, "defmcasthlim", SYSCTL_DESCR("Default multicast hop limit"), NULL, 0, &ip6_defmcasthlim, 0, CTL_NET, PF_INET6, IPPROTO_IPV6, IPV6CTL_DEFMCASTHLIM, CTL_EOL); sysctl_createv(clog, 0, NULL, NULL, CTLFLAG_PERMANENT, CTLTYPE_STRING, "kame_version", SYSCTL_DESCR("KAME Version"), NULL, 0, __UNCONST(__KAME_VERSION), 0, CTL_NET, PF_INET6, IPPROTO_IPV6, IPV6CTL_KAME_VERSION, CTL_EOL); sysctl_createv(clog, 0, NULL, NULL, CTLFLAG_PERMANENT|CTLFLAG_READWRITE, CTLTYPE_INT, "use_deprecated", SYSCTL_DESCR("Allow use of deprecated addresses as " "source addresses"), NULL, 0, &ip6_use_deprecated, 0, CTL_NET, PF_INET6, IPPROTO_IPV6, IPV6CTL_USE_DEPRECATED, CTL_EOL); sysctl_createv(clog, 0, NULL, NULL, CTLFLAG_PERMANENT #ifndef INET6_BINDV6ONLY |CTLFLAG_READWRITE, #endif CTLTYPE_INT, "v6only", SYSCTL_DESCR("Disallow PF_INET6 sockets from connecting " "to PF_INET sockets"), NULL, 0, &ip6_v6only, 0, CTL_NET, PF_INET6, IPPROTO_IPV6, IPV6CTL_V6ONLY, CTL_EOL); sysctl_createv(clog, 0, NULL, NULL, CTLFLAG_PERMANENT|CTLFLAG_READWRITE, CTLTYPE_INT, "anonportmin", SYSCTL_DESCR("Lowest ephemeral port number to assign"), sysctl_net_inet_ip_ports, 0, &ip6_anonportmin, 0, CTL_NET, PF_INET6, IPPROTO_IPV6, IPV6CTL_ANONPORTMIN, CTL_EOL); sysctl_createv(clog, 0, NULL, NULL, CTLFLAG_PERMANENT|CTLFLAG_READWRITE, CTLTYPE_INT, "anonportmax", SYSCTL_DESCR("Highest ephemeral port number to assign"), sysctl_net_inet_ip_ports, 0, &ip6_anonportmax, 0, CTL_NET, PF_INET6, IPPROTO_IPV6, IPV6CTL_ANONPORTMAX, CTL_EOL); #ifndef IPNOPRIVPORTS sysctl_createv(clog, 0, NULL, NULL, CTLFLAG_PERMANENT|CTLFLAG_READWRITE, CTLTYPE_INT, "lowportmin", SYSCTL_DESCR("Lowest privileged ephemeral port number " "to assign"), sysctl_net_inet_ip_ports, 0, &ip6_lowportmin, 0, CTL_NET, PF_INET6, IPPROTO_IPV6, IPV6CTL_LOWPORTMIN, CTL_EOL); sysctl_createv(clog, 0, NULL, NULL, CTLFLAG_PERMANENT|CTLFLAG_READWRITE, CTLTYPE_INT, "lowportmax", SYSCTL_DESCR("Highest privileged ephemeral port number " "to assign"), sysctl_net_inet_ip_ports, 0, &ip6_lowportmax, 0, CTL_NET, PF_INET6, IPPROTO_IPV6, IPV6CTL_LOWPORTMAX, CTL_EOL); #endif /* IPNOPRIVPORTS */ sysctl_createv(clog, 0, NULL, NULL, CTLFLAG_PERMANENT|CTLFLAG_READWRITE, CTLTYPE_INT, "auto_linklocal", SYSCTL_DESCR("Default value of per-interface flag for " "adding an IPv6 link-local address to " "interfaces when attached"), NULL, 0, &ip6_auto_linklocal, 0, CTL_NET, PF_INET6, IPPROTO_IPV6, IPV6CTL_AUTO_LINKLOCAL, CTL_EOL); sysctl_createv(clog, 0, NULL, NULL, CTLFLAG_PERMANENT|CTLFLAG_READONLY, CTLTYPE_STRUCT, "addctlpolicy", SYSCTL_DESCR("Return the current address control" " policy"), sysctl_net_inet6_addrctlpolicy, 0, NULL, 0, CTL_NET, PF_INET6, IPPROTO_IPV6, IPV6CTL_ADDRCTLPOLICY, CTL_EOL); sysctl_createv(clog, 0, NULL, NULL, CTLFLAG_PERMANENT|CTLFLAG_READWRITE, CTLTYPE_INT, "prefer_tempaddr", SYSCTL_DESCR("Prefer temporary address as source " "address"), NULL, 0, &ip6_prefer_tempaddr, 0, CTL_NET, PF_INET6, IPPROTO_IPV6, CTL_CREATE, CTL_EOL); sysctl_createv(clog, 0, NULL, NULL, CTLFLAG_PERMANENT|CTLFLAG_READWRITE, CTLTYPE_INT, "maxfrags", SYSCTL_DESCR("Maximum fragments in reassembly queue"), NULL, 0, &ip6_maxfrags, 0, CTL_NET, PF_INET6, IPPROTO_IPV6, IPV6CTL_MAXFRAGS, CTL_EOL); sysctl_createv(clog, 0, NULL, NULL, CTLFLAG_PERMANENT, CTLTYPE_STRUCT, "stats", SYSCTL_DESCR("IPv6 statistics"), sysctl_net_inet6_ip6_stats, 0, NULL, 0, CTL_NET, PF_INET6, IPPROTO_IPV6, IPV6CTL_STATS, CTL_EOL); sysctl_createv(clog, 0, NULL, NULL, CTLFLAG_PERMANENT|CTLFLAG_READWRITE, CTLTYPE_INT, "use_defaultzone", SYSCTL_DESCR("Whether to use the default scope zones"), NULL, 0, &ip6_use_defzone, 0, CTL_NET, PF_INET6, IPPROTO_IPV6, IPV6CTL_USE_DEFAULTZONE, CTL_EOL); sysctl_createv(clog, 0, NULL, NULL, CTLFLAG_PERMANENT|CTLFLAG_READWRITE, CTLTYPE_INT, "mcast_pmtu", SYSCTL_DESCR("Enable pMTU discovery for multicast packet"), NULL, 0, &ip6_mcast_pmtu, 0, CTL_NET, PF_INET6, IPPROTO_IPV6, CTL_CREATE, CTL_EOL); /* anonportalgo RFC6056 subtree */ const struct sysctlnode *portalgo_node; sysctl_createv(clog, 0, NULL, &portalgo_node, CTLFLAG_PERMANENT, CTLTYPE_NODE, "anonportalgo", SYSCTL_DESCR("Anonymous port algorithm selection (RFC 6056)"), NULL, 0, NULL, 0, CTL_NET, PF_INET6, IPPROTO_IPV6, CTL_CREATE, CTL_EOL); sysctl_createv(clog, 0, &portalgo_node, NULL, CTLFLAG_PERMANENT, CTLTYPE_STRING, "available", SYSCTL_DESCR("available algorithms"), sysctl_portalgo_available, 0, NULL, PORTALGO_MAXLEN, CTL_CREATE, CTL_EOL); sysctl_createv(clog, 0, &portalgo_node, NULL, CTLFLAG_PERMANENT|CTLFLAG_READWRITE, CTLTYPE_STRING, "selected", SYSCTL_DESCR("selected algorithm"), sysctl_portalgo_selected6, 0, NULL, PORTALGO_MAXLEN, CTL_CREATE, CTL_EOL); sysctl_createv(clog, 0, &portalgo_node, NULL, CTLFLAG_PERMANENT|CTLFLAG_READWRITE, CTLTYPE_STRUCT, "reserve", SYSCTL_DESCR("bitmap of reserved ports"), sysctl_portalgo_reserve6, 0, NULL, 0, CTL_CREATE, CTL_EOL); sysctl_createv(clog, 0, NULL, NULL, CTLFLAG_PERMANENT|CTLFLAG_READWRITE, CTLTYPE_INT, "neighborgcthresh", SYSCTL_DESCR("Maximum number of entries in neighbor" " cache"), NULL, 1, &ip6_neighborgcthresh, 0, CTL_NET, PF_INET6, IPPROTO_IPV6, CTL_CREATE, CTL_EOL); sysctl_createv(clog, 0, NULL, NULL, CTLFLAG_PERMANENT|CTLFLAG_READWRITE, CTLTYPE_INT, "maxdynroutes", SYSCTL_DESCR("Maximum number of routes created via" " redirect"), NULL, 1, &ip6_maxdynroutes, 0, CTL_NET, PF_INET6, IPPROTO_IPV6, CTL_CREATE, CTL_EOL); } void ip6_statinc(u_int stat) { KASSERT(stat < IP6_NSTATS); IP6_STATINC(stat); } |
| 20 40 125 25 4 3 5 3 4 29 45 26 33 96 96 13 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 197 198 199 200 201 202 203 204 205 206 207 208 209 210 211 212 213 214 215 216 217 218 219 220 221 222 223 224 225 226 227 228 229 230 231 232 233 234 235 236 237 238 239 240 241 242 243 244 245 246 247 248 249 250 251 252 253 254 255 256 257 258 259 260 261 262 263 264 265 | /* $NetBSD: sys_socket.c,v 1.79 2020/11/17 03:22:33 chs Exp $ */ /*- * Copyright (c) 2008, 2009 The NetBSD Foundation, Inc. * All rights reserved. * * This code is derived from software contributed to The NetBSD Foundation * by Andrew Doran. * * Redistribution and use in source and binary forms, with or without * modification, are permitted provided that the following conditions * are met: * 1. Redistributions of source code must retain the above copyright * notice, this list of conditions and the following disclaimer. * 2. Redistributions in binary form must reproduce the above copyright * notice, this list of conditions and the following disclaimer in the * documentation and/or other materials provided with the distribution. * * THIS SOFTWARE IS PROVIDED BY THE NETBSD FOUNDATION, INC. AND CONTRIBUTORS * ``AS IS'' AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED * TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR * PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE FOUNDATION OR CONTRIBUTORS * BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR * CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF * SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS * INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN * CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) * ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE * POSSIBILITY OF SUCH DAMAGE. */ /* * Copyright (c) 1982, 1986, 1990, 1993 * The Regents of the University of California. All rights reserved. * * Redistribution and use in source and binary forms, with or without * modification, are permitted provided that the following conditions * are met: * 1. Redistributions of source code must retain the above copyright * notice, this list of conditions and the following disclaimer. * 2. Redistributions in binary form must reproduce the above copyright * notice, this list of conditions and the following disclaimer in the * documentation and/or other materials provided with the distribution. * 3. Neither the name of the University nor the names of its contributors * may be used to endorse or promote products derived from this software * without specific prior written permission. * * THIS SOFTWARE IS PROVIDED BY THE REGENTS AND CONTRIBUTORS ``AS IS'' AND * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE * ARE DISCLAIMED. IN NO EVENT SHALL THE REGENTS OR CONTRIBUTORS BE LIABLE * FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL * DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS * OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) * HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT * LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY * OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF * SUCH DAMAGE. * * @(#)sys_socket.c 8.3 (Berkeley) 2/14/95 */ #include <sys/cdefs.h> __KERNEL_RCSID(0, "$NetBSD: sys_socket.c,v 1.79 2020/11/17 03:22:33 chs Exp $"); #include <sys/param.h> #include <sys/systm.h> #include <sys/systm.h> #include <sys/file.h> #include <sys/mbuf.h> #include <sys/protosw.h> #include <sys/socket.h> #include <sys/socketvar.h> #include <sys/ioctl.h> #include <sys/stat.h> #include <sys/poll.h> #include <sys/proc.h> #include <sys/kauth.h> #include <net/if.h> #include <net/route.h> const struct fileops socketops = { .fo_name = "socket", .fo_read = soo_read, .fo_write = soo_write, .fo_ioctl = soo_ioctl, .fo_fcntl = fnullop_fcntl, .fo_poll = soo_poll, .fo_stat = soo_stat, .fo_close = soo_close, .fo_kqfilter = soo_kqfilter, .fo_restart = soo_restart, }; int (*ifioctl)(struct socket *, u_long, void *, struct lwp *) = (void *)eopnotsupp; /* ARGSUSED */ int soo_read(file_t *fp, off_t *offset, struct uio *uio, kauth_cred_t cred, int flags) { struct socket *so = fp->f_socket; int error; error = (*so->so_receive)(so, NULL, uio, NULL, NULL, NULL); return error; } /* ARGSUSED */ int soo_write(file_t *fp, off_t *offset, struct uio *uio, kauth_cred_t cred, int flags) { struct socket *so = fp->f_socket; int error; error = (*so->so_send)(so, NULL, uio, NULL, NULL, 0, curlwp); return error; } int soo_ioctl(file_t *fp, u_long cmd, void *data) { struct socket *so = fp->f_socket; int error = 0; switch (cmd) { case FIONBIO: solock(so); if (*(int *)data) so->so_state |= SS_NBIO; else so->so_state &= ~SS_NBIO; sounlock(so); break; case FIOASYNC: solock(so); if (*(int *)data) { so->so_rcv.sb_flags |= SB_ASYNC; so->so_snd.sb_flags |= SB_ASYNC; } else { so->so_rcv.sb_flags &= ~SB_ASYNC; so->so_snd.sb_flags &= ~SB_ASYNC; } sounlock(so); break; case FIONREAD: *(int *)data = so->so_rcv.sb_cc; break; case FIONWRITE: *(int *)data = so->so_snd.sb_cc; break; case FIONSPACE: /* * See the comment around sbspace()'s definition * in sys/socketvar.h in face of counts about maximum * to understand the following test. We detect overflow * and return zero. */ solock(so); if ((so->so_snd.sb_hiwat < so->so_snd.sb_cc) || (so->so_snd.sb_mbmax < so->so_snd.sb_mbcnt)) *(int *)data = 0; else *(int *)data = sbspace(&so->so_snd); sounlock(so); break; case SIOCSPGRP: case FIOSETOWN: case TIOCSPGRP: error = fsetown(&so->so_pgid, cmd, data); break; case SIOCGPGRP: case FIOGETOWN: case TIOCGPGRP: error = fgetown(so->so_pgid, cmd, data); break; case SIOCATMARK: *(int *)data = (so->so_state&SS_RCVATMARK) != 0; break; case SIOCPEELOFF: solock(so); error = do_sys_peeloff(so, data); sounlock(so); break; default: /* * Interface/routing/protocol specific ioctls: * interface and routing ioctls should have a * different entry since a socket's unnecessary */ if (IOCGROUP(cmd) == 'i') /* * KERNEL_LOCK will be held later if if_ioctl() of the * interface isn't MP-safe. */ error = ifioctl(so, cmd, data, curlwp); else { KERNEL_LOCK(1, NULL); error = (*so->so_proto->pr_usrreqs->pr_ioctl)(so, cmd, data, NULL); KERNEL_UNLOCK_ONE(NULL); } break; } return error; } int soo_poll(file_t *fp, int events) { return sopoll(fp->f_socket, events); } int soo_stat(file_t *fp, struct stat *ub) { struct socket *so = fp->f_socket; int error; memset(ub, 0, sizeof(*ub)); ub->st_mode = S_IFSOCK; solock(so); error = (*so->so_proto->pr_usrreqs->pr_stat)(so, ub); sounlock(so); return error; } /* ARGSUSED */ int soo_close(file_t *fp) { int error = 0; if (fp->f_socket) error = soclose(fp->f_socket); fp->f_socket = NULL; return error; } void soo_restart(file_t *fp) { sorestart(fp->f_socket); } |
| 2 2 2 2 2 2 2 1 1 1 1 2 2 1 1 1 1 1 1 1 2 2 2 1 1 1 2 2 2 2 2 2 2 2 2 2 2 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 5 5 3 2 2 2 2 2 2 1 2 1 2 1 1 2 2 2 1 1 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 197 198 199 200 201 202 203 204 205 206 207 208 209 210 211 212 213 214 215 216 217 218 219 220 221 222 223 224 225 226 227 228 229 230 231 232 233 234 235 236 237 238 239 240 241 242 243 244 245 246 247 248 249 250 251 252 253 254 255 256 257 258 259 260 261 262 263 264 265 266 267 268 269 270 271 272 273 274 275 276 277 278 279 280 281 282 283 284 285 286 287 288 289 290 291 292 293 294 295 296 297 298 299 300 301 302 303 304 305 306 307 308 309 310 311 312 313 314 315 316 317 318 319 320 321 322 323 324 325 326 327 328 329 330 331 332 333 334 335 336 337 338 339 340 341 342 343 344 345 346 347 348 349 350 351 352 353 354 355 356 357 358 359 360 361 362 363 364 365 366 367 368 369 370 371 372 373 374 375 376 377 378 379 380 381 382 383 384 385 386 387 388 389 390 391 392 393 394 395 396 397 398 399 400 401 402 403 404 405 406 407 408 409 410 411 412 413 414 415 416 417 418 419 420 421 422 423 424 425 426 427 428 429 430 431 432 433 434 435 436 437 438 439 440 441 442 443 444 445 446 447 448 449 450 451 452 453 454 455 456 457 458 459 460 461 462 463 464 465 466 467 468 469 470 471 472 473 474 475 476 477 478 479 480 481 482 483 484 485 486 487 488 489 490 491 492 493 494 495 496 497 498 499 500 501 502 503 504 505 506 507 508 509 510 511 512 513 514 515 516 517 518 519 520 521 522 523 524 525 526 527 528 529 530 531 532 533 534 535 536 537 538 539 540 541 542 543 544 545 546 547 548 549 550 551 552 553 554 555 556 557 558 559 560 561 562 563 564 565 566 567 568 569 570 571 572 573 574 575 576 577 578 579 580 581 582 583 584 585 586 587 588 589 590 591 592 593 594 595 596 597 598 599 600 601 602 603 604 605 606 607 608 609 610 611 612 613 614 615 616 617 618 619 620 621 622 623 624 625 626 627 628 629 630 631 632 633 634 635 636 637 638 639 640 641 642 643 644 645 646 647 648 649 650 651 652 653 654 655 656 657 658 659 660 661 662 663 664 665 666 667 668 669 670 671 672 673 674 675 676 677 678 679 680 681 682 683 684 685 686 687 688 689 690 691 692 693 694 695 696 697 698 699 700 701 702 703 704 705 706 707 708 709 710 711 712 713 714 715 716 717 718 719 720 721 722 723 724 725 726 727 728 729 730 731 732 733 734 735 736 737 738 739 740 741 742 743 744 745 746 747 748 749 750 751 752 753 754 755 756 757 758 759 760 761 762 763 764 765 766 767 768 769 770 771 772 773 774 775 776 777 778 779 780 781 782 783 784 785 786 787 788 789 790 791 792 793 794 795 796 797 798 799 800 801 802 803 804 805 806 807 808 809 810 811 812 813 814 815 816 817 818 819 820 821 822 823 824 825 826 827 828 829 830 831 832 833 834 835 836 837 838 839 840 841 842 843 844 845 846 847 848 849 850 851 852 853 854 855 856 857 858 859 860 861 862 863 864 865 866 867 868 869 870 871 872 873 874 875 876 877 878 879 880 881 882 883 884 885 886 887 888 889 890 891 892 893 894 895 896 897 898 899 900 901 902 903 904 905 906 907 908 909 910 911 912 913 914 915 916 917 918 919 920 921 922 923 924 925 926 927 928 929 930 931 932 933 934 935 936 937 938 939 940 941 942 943 944 945 946 947 948 949 950 951 952 953 954 955 956 957 958 959 960 961 962 963 964 965 966 967 968 969 970 971 972 973 974 975 976 977 978 979 980 981 982 983 984 985 986 987 988 989 990 991 992 993 994 995 996 997 998 999 1000 1001 1002 1003 1004 1005 1006 1007 1008 1009 1010 1011 1012 1013 1014 1015 1016 1017 1018 1019 1020 1021 1022 1023 1024 1025 1026 1027 1028 1029 1030 1031 1032 1033 1034 1035 1036 1037 1038 1039 1040 1041 1042 1043 1044 1045 1046 1047 1048 1049 1050 1051 1052 1053 1054 1055 1056 1057 1058 1059 1060 1061 1062 1063 1064 1065 1066 1067 1068 1069 1070 1071 1072 1073 1074 1075 1076 1077 1078 1079 1080 1081 1082 1083 1084 1085 1086 1087 1088 1089 1090 1091 1092 1093 1094 1095 1096 1097 1098 1099 1100 1101 1102 1103 1104 1105 1106 1107 1108 1109 1110 1111 1112 1113 1114 1115 1116 1117 1118 1119 1120 1121 1122 1123 1124 1125 1126 1127 1128 1129 1130 1131 1132 1133 1134 1135 1136 1137 1138 1139 1140 1141 1142 1143 1144 1145 1146 1147 1148 1149 1150 1151 1152 1153 1154 1155 1156 1157 1158 1159 1160 1161 1162 1163 1164 1165 1166 1167 1168 1169 1170 1171 1172 1173 1174 1175 1176 1177 1178 1179 1180 1181 1182 1183 1184 1185 1186 1187 1188 1189 1190 1191 1192 1193 1194 1195 1196 1197 1198 1199 1200 1201 1202 1203 1204 1205 1206 1207 1208 1209 1210 1211 1212 1213 1214 1215 1216 1217 1218 1219 1220 1221 1222 1223 1224 1225 1226 1227 1228 1229 1230 1231 1232 1233 1234 1235 1236 1237 1238 1239 1240 1241 1242 1243 1244 1245 1246 1247 1248 1249 1250 1251 1252 1253 1254 1255 1256 1257 1258 1259 1260 1261 1262 1263 1264 1265 1266 1267 1268 1269 1270 1271 1272 1273 1274 1275 1276 1277 1278 1279 1280 1281 1282 1283 1284 1285 1286 1287 1288 1289 1290 1291 1292 1293 1294 1295 1296 1297 1298 1299 1300 1301 1302 1303 1304 1305 1306 1307 1308 1309 1310 1311 1312 1313 1314 1315 1316 1317 1318 1319 1320 1321 1322 1323 1324 1325 1326 1327 1328 1329 1330 1331 1332 1333 1334 1335 1336 1337 1338 1339 1340 1341 1342 1343 1344 1345 1346 1347 1348 1349 1350 1351 1352 1353 1354 1355 1356 1357 1358 1359 1360 1361 1362 1363 1364 1365 1366 1367 1368 1369 1370 1371 1372 1373 1374 1375 1376 1377 1378 1379 1380 1381 1382 1383 1384 1385 1386 1387 1388 1389 1390 1391 1392 1393 1394 1395 1396 1397 1398 1399 1400 1401 1402 1403 1404 1405 1406 1407 1408 1409 1410 1411 1412 1413 1414 1415 1416 1417 1418 1419 1420 1421 1422 1423 1424 1425 1426 1427 1428 1429 1430 1431 1432 1433 1434 1435 1436 1437 1438 1439 1440 1441 1442 1443 1444 1445 1446 1447 1448 1449 1450 1451 1452 1453 1454 1455 1456 1457 1458 1459 1460 1461 1462 1463 1464 1465 1466 1467 1468 1469 1470 1471 1472 1473 1474 1475 1476 1477 1478 1479 1480 1481 1482 1483 1484 1485 1486 1487 1488 1489 1490 1491 1492 1493 1494 1495 1496 1497 1498 1499 1500 1501 1502 1503 1504 1505 1506 1507 1508 1509 1510 1511 1512 1513 1514 1515 1516 1517 1518 1519 1520 1521 1522 1523 1524 1525 1526 1527 1528 1529 1530 1531 1532 1533 1534 1535 1536 1537 1538 1539 1540 1541 1542 1543 1544 1545 1546 1547 1548 1549 1550 1551 1552 1553 1554 1555 1556 1557 1558 1559 1560 1561 1562 1563 1564 1565 1566 1567 1568 1569 1570 1571 1572 1573 1574 1575 1576 1577 1578 1579 1580 1581 1582 1583 1584 1585 1586 1587 1588 1589 1590 1591 1592 1593 1594 1595 1596 1597 1598 1599 1600 1601 1602 1603 1604 1605 1606 1607 1608 1609 1610 1611 1612 1613 1614 1615 1616 1617 1618 1619 1620 1621 1622 1623 1624 1625 1626 1627 1628 1629 1630 1631 1632 1633 1634 1635 1636 1637 1638 1639 1640 1641 1642 1643 1644 1645 1646 1647 1648 1649 1650 1651 1652 1653 1654 1655 1656 1657 1658 1659 1660 1661 1662 1663 1664 1665 1666 1667 1668 1669 1670 1671 1672 1673 1674 1675 1676 1677 1678 1679 1680 1681 1682 1683 1684 1685 1686 1687 1688 1689 1690 1691 1692 1693 1694 1695 1696 1697 1698 1699 1700 1701 1702 1703 1704 1705 1706 1707 1708 1709 1710 1711 1712 1713 1714 1715 1716 1717 1718 1719 1720 1721 1722 1723 1724 1725 1726 1727 1728 1729 1730 1731 1732 1733 1734 1735 1736 1737 1738 1739 1740 1741 1742 1743 1744 1745 1746 1747 1748 1749 1750 1751 1752 1753 1754 1755 1756 1757 1758 1759 1760 1761 1762 1763 1764 1765 1766 1767 1768 1769 1770 1771 1772 1773 1774 1775 1776 1777 1778 1779 1780 1781 1782 1783 1784 1785 1786 1787 1788 1789 1790 1791 1792 1793 1794 1795 1796 1797 1798 1799 1800 1801 1802 1803 1804 1805 1806 1807 1808 1809 1810 1811 1812 1813 1814 1815 1816 1817 1818 1819 1820 1821 1822 1823 1824 1825 1826 1827 1828 1829 1830 1831 1832 1833 1834 1835 1836 1837 1838 1839 1840 1841 1842 1843 1844 1845 1846 1847 1848 1849 1850 1851 1852 1853 1854 1855 1856 1857 1858 1859 1860 1861 1862 1863 1864 1865 1866 1867 1868 1869 1870 1871 1872 1873 1874 1875 1876 1877 1878 1879 1880 1881 1882 1883 1884 1885 1886 1887 1888 1889 1890 1891 1892 1893 1894 1895 1896 1897 1898 1899 1900 1901 1902 1903 1904 1905 1906 1907 1908 1909 1910 1911 1912 1913 1914 1915 1916 1917 1918 1919 1920 1921 1922 1923 1924 1925 1926 1927 1928 1929 1930 1931 1932 1933 1934 1935 1936 1937 1938 1939 1940 1941 1942 1943 1944 1945 1946 1947 1948 1949 1950 1951 1952 1953 1954 1955 1956 1957 1958 1959 1960 1961 1962 1963 1964 1965 1966 1967 1968 1969 1970 1971 1972 1973 1974 1975 1976 1977 1978 1979 1980 1981 1982 1983 1984 1985 1986 1987 1988 1989 1990 1991 1992 1993 1994 1995 1996 1997 1998 1999 2000 2001 2002 2003 2004 2005 2006 2007 2008 2009 2010 2011 2012 2013 2014 2015 2016 2017 2018 2019 2020 2021 2022 2023 2024 2025 2026 2027 2028 2029 2030 2031 2032 2033 2034 2035 2036 2037 2038 2039 2040 2041 2042 2043 2044 2045 2046 2047 2048 2049 2050 2051 2052 2053 2054 2055 2056 2057 2058 2059 2060 2061 2062 2063 2064 2065 2066 2067 2068 2069 2070 2071 2072 2073 2074 2075 2076 2077 2078 2079 2080 2081 2082 2083 2084 2085 2086 2087 2088 2089 2090 2091 2092 | /* $NetBSD: nvlist.c,v 1.8 2019/07/23 00:49:16 rmind Exp $ */ /*- * SPDX-License-Identifier: BSD-2-Clause-FreeBSD * * Copyright (c) 2009-2013 The FreeBSD Foundation * Copyright (c) 2013-2015 Mariusz Zaborski <oshogbo@FreeBSD.org> * All rights reserved. * * This software was developed by Pawel Jakub Dawidek under sponsorship from * the FreeBSD Foundation. * * Redistribution and use in source and binary forms, with or without * modification, are permitted provided that the following conditions * are met: * 1. Redistributions of source code must retain the above copyright * notice, this list of conditions and the following disclaimer. * 2. Redistributions in binary form must reproduce the above copyright * notice, this list of conditions and the following disclaimer in the * documentation and/or other materials provided with the distribution. * * THIS SOFTWARE IS PROVIDED BY THE AUTHORS AND CONTRIBUTORS ``AS IS'' AND * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE * ARE DISCLAIMED. IN NO EVENT SHALL THE AUTHORS OR CONTRIBUTORS BE LIABLE * FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL * DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS * OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) * HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT * LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY * OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF * SUCH DAMAGE. */ #include <sys/cdefs.h> #ifdef __FreeBSD__ __FBSDID("$FreeBSD: head/sys/contrib/libnv/nvlist.c 335347 2018-06-18 22:57:32Z oshogbo $"); #else __RCSID("$NetBSD: nvlist.c,v 1.8 2019/07/23 00:49:16 rmind Exp $"); #endif #include <sys/param.h> #include <sys/endian.h> #include <sys/queue.h> #if defined(_KERNEL) || defined(_STANDALONE) #include <sys/errno.h> #include <sys/kernel.h> #include <sys/lock.h> #include <sys/malloc.h> #include <sys/systm.h> #ifdef __FreeBSD__ #include <machine/stdarg.h> #endif #else #include <sys/socket.h> #include <errno.h> #include <stdarg.h> #include <stdbool.h> #include <stdint.h> #include <stdio.h> #include <stdlib.h> #include <string.h> #include <unistd.h> #include "msgio.h" #endif #ifdef HAVE_PJDLOG #include <pjdlog.h> #endif #ifdef __FreeBSD__ #include <sys/nv.h> #else #include "nv.h" #endif #include "nv_impl.h" #include "nvlist_impl.h" #include "nvpair_impl.h" #ifndef HAVE_PJDLOG #if defined(_KERNEL) || defined(_STANDALONE) #ifdef __FreeBSD__ #define PJDLOG_ASSERT(...) MPASS(__VA_ARGS__) #else #define PJDLOG_ASSERT(...) KASSERT(__VA_ARGS__) #endif #define PJDLOG_RASSERT(expr, ...) KASSERT(expr, (__VA_ARGS__)) #define PJDLOG_ABORT(...) panic(__VA_ARGS__) #else #ifndef __lint__ #include <assert.h> #define PJDLOG_ASSERT(...) assert(__VA_ARGS__) #define PJDLOG_RASSERT(expr, ...) assert(expr) #define PJDLOG_ABORT(...) do { \ fprintf(stderr, "%s:%u: ", __FILE__, __LINE__); \ fprintf(stderr, __VA_ARGS__); \ fprintf(stderr, "\n"); \ abort(); \ } while (/*CONSTCOND*/0) #else #define PJDLOG_ASSERT(...) #define PJDLOG_RASSERT(expr, ...) #define PJDLOG_ABORT(...) #endif #endif #endif #define NV_FLAG_PRIVATE_MASK (NV_FLAG_BIG_ENDIAN | NV_FLAG_IN_ARRAY) #define NV_FLAG_PUBLIC_MASK (NV_FLAG_IGNORE_CASE | NV_FLAG_NO_UNIQUE) #define NV_FLAG_ALL_MASK (NV_FLAG_PRIVATE_MASK | NV_FLAG_PUBLIC_MASK) #define NVLIST_MAGIC 0x6e766c /* "nvl" */ struct nvlist { int nvl_magic; int nvl_error; int nvl_flags; nvpair_t *nvl_parent; nvpair_t *nvl_array_next; struct nvl_head nvl_head; }; #define NVLIST_ASSERT(nvl) do { \ PJDLOG_ASSERT((nvl) != NULL); \ PJDLOG_ASSERT((nvl)->nvl_magic == NVLIST_MAGIC); \ } while (/*CONSTCOND*/0) #ifdef _KERNEL MALLOC_DEFINE(M_NVLIST, "nvlist", "kernel nvlist"); #endif #define NVPAIR_ASSERT(nvp) nvpair_assert(nvp) #define NVLIST_HEADER_MAGIC 0x6c #define NVLIST_HEADER_VERSION 0x00 struct nvlist_header { uint8_t nvlh_magic; uint8_t nvlh_version; uint8_t nvlh_flags; uint64_t nvlh_descriptors; uint64_t nvlh_size; } __packed; nvlist_t * nvlist_create(int flags) { nvlist_t *nvl; PJDLOG_ASSERT((flags & ~(NV_FLAG_PUBLIC_MASK)) == 0); nvl = nv_malloc(sizeof(*nvl)); if (nvl == NULL) return (NULL); nvl->nvl_error = 0; nvl->nvl_flags = flags; nvl->nvl_parent = NULL; nvl->nvl_array_next = NULL; TAILQ_INIT(&nvl->nvl_head); nvl->nvl_magic = NVLIST_MAGIC; return (nvl); } void nvlist_destroy(nvlist_t *nvl) { nvpair_t *nvp; if (nvl == NULL) return; ERRNO_SAVE(); NVLIST_ASSERT(nvl); while ((nvp = nvlist_first_nvpair(nvl)) != NULL) { nvlist_remove_nvpair(nvl, nvp); nvpair_free(nvp); } if (nvl->nvl_array_next != NULL) nvpair_free_structure(nvl->nvl_array_next); nvl->nvl_array_next = NULL; nvl->nvl_parent = NULL; nvl->nvl_magic = 0; nv_free(nvl); ERRNO_RESTORE(); } void nvlist_set_error(nvlist_t *nvl, int error) { PJDLOG_ASSERT(error != 0); /* * Check for error != 0 so that we don't do the wrong thing if somebody * tries to abuse this API when asserts are disabled. */ if (nvl != NULL && error != 0 && nvl->nvl_error == 0) nvl->nvl_error = error; } int nvlist_error(const nvlist_t *nvl) { if (nvl == NULL) return (ENOMEM); NVLIST_ASSERT(nvl); return (nvl->nvl_error); } nvpair_t * nvlist_get_nvpair_parent(const nvlist_t *nvl) { NVLIST_ASSERT(nvl); return (nvl->nvl_parent); } const nvlist_t * nvlist_get_parent(const nvlist_t *nvl, void **cookiep) { nvpair_t *nvp; NVLIST_ASSERT(nvl); nvp = nvl->nvl_parent; if (cookiep != NULL) *cookiep = nvp; if (nvp == NULL) return (NULL); return (nvpair_nvlist(nvp)); } void nvlist_set_parent(nvlist_t *nvl, nvpair_t *parent) { NVLIST_ASSERT(nvl); nvl->nvl_parent = parent; } void nvlist_set_array_next(nvlist_t *nvl, nvpair_t *ele) { NVLIST_ASSERT(nvl); if (ele != NULL) { nvl->nvl_flags |= NV_FLAG_IN_ARRAY; } else { nvl->nvl_flags &= ~NV_FLAG_IN_ARRAY; nv_free(nvl->nvl_array_next); } nvl->nvl_array_next = ele; } nvpair_t * nvlist_get_array_next_nvpair(nvlist_t *nvl) { NVLIST_ASSERT(nvl); return (nvl->nvl_array_next); } bool nvlist_in_array(const nvlist_t *nvl) { NVLIST_ASSERT(nvl); return ((nvl->nvl_flags & NV_FLAG_IN_ARRAY) != 0); } const nvlist_t * nvlist_get_array_next(const nvlist_t *nvl) { nvpair_t *nvp; NVLIST_ASSERT(nvl); nvp = nvl->nvl_array_next; if (nvp == NULL) return (NULL); return (nvpair_get_nvlist(nvp)); } const nvlist_t * nvlist_get_pararr(const nvlist_t *nvl, void **cookiep) { const nvlist_t *ret; ret = nvlist_get_array_next(nvl); if (ret != NULL) { if (cookiep != NULL) *cookiep = NULL; return (ret); } return (nvlist_get_parent(nvl, cookiep)); } bool nvlist_empty(const nvlist_t *nvl) { NVLIST_ASSERT(nvl); PJDLOG_ASSERT(nvl->nvl_error == 0); return (nvlist_first_nvpair(nvl) == NULL); } int nvlist_flags(const nvlist_t *nvl) { NVLIST_ASSERT(nvl); PJDLOG_ASSERT(nvl->nvl_error == 0); return (nvl->nvl_flags & NV_FLAG_PUBLIC_MASK); } void nvlist_set_flags(nvlist_t *nvl, int flags) { NVLIST_ASSERT(nvl); PJDLOG_ASSERT(nvl->nvl_error == 0); nvl->nvl_flags = flags; } __dead void nvlist_report_missing(int type, const char *name) { PJDLOG_ABORT("Element '%s' of type %s doesn't exist.", name, nvpair_type_string(type)); } static nvpair_t * nvlist_find(const nvlist_t *nvl, int type, const char *name) { nvpair_t *nvp; NVLIST_ASSERT(nvl); PJDLOG_ASSERT(nvl->nvl_error == 0); PJDLOG_ASSERT(type == NV_TYPE_NONE || (type >= NV_TYPE_FIRST && type <= NV_TYPE_LAST)); for (nvp = nvlist_first_nvpair(nvl); nvp != NULL; nvp = nvlist_next_nvpair(nvl, nvp)) { if (type != NV_TYPE_NONE && nvpair_type(nvp) != type) continue; if ((nvl->nvl_flags & NV_FLAG_IGNORE_CASE) != 0) { if (strcasecmp(nvpair_name(nvp), name) != 0) continue; } else { if (strcmp(nvpair_name(nvp), name) != 0) continue; } break; } if (nvp == NULL) ERRNO_SET(ENOENT); return (nvp); } bool nvlist_exists_type(const nvlist_t *nvl, const char *name, int type) { NVLIST_ASSERT(nvl); PJDLOG_ASSERT(nvl->nvl_error == 0); PJDLOG_ASSERT(type == NV_TYPE_NONE || (type >= NV_TYPE_FIRST && type <= NV_TYPE_LAST)); return (nvlist_find(nvl, type, name) != NULL); } void nvlist_free_type(nvlist_t *nvl, const char *name, int type) { nvpair_t *nvp; NVLIST_ASSERT(nvl); PJDLOG_ASSERT(nvl->nvl_error == 0); PJDLOG_ASSERT(type == NV_TYPE_NONE || (type >= NV_TYPE_FIRST && type <= NV_TYPE_LAST)); nvp = nvlist_find(nvl, type, name); if (nvp != NULL) nvlist_free_nvpair(nvl, nvp); else nvlist_report_missing(type, name); } nvlist_t * nvlist_clone(const nvlist_t *nvl) { nvlist_t *newnvl; nvpair_t *nvp, *newnvp; NVLIST_ASSERT(nvl); if (nvl->nvl_error != 0) { ERRNO_SET(nvl->nvl_error); return (NULL); } newnvl = nvlist_create(nvl->nvl_flags & NV_FLAG_PUBLIC_MASK); for (nvp = nvlist_first_nvpair(nvl); nvp != NULL; nvp = nvlist_next_nvpair(nvl, nvp)) { newnvp = nvpair_clone(nvp); if (newnvp == NULL) break; (void)nvlist_move_nvpair(newnvl, newnvp); } if (nvp != NULL) { nvlist_destroy(newnvl); return (NULL); } return (newnvl); } #if !defined(_KERNEL) && !defined(_STANDALONE) static bool nvlist_dump_error_check(const nvlist_t *nvl, int fd, int level) { if (nvlist_error(nvl) != 0) { dprintf(fd, "%*serror: %d\n", level * 4, "", nvlist_error(nvl)); return (true); } return (false); } /* * Dump content of nvlist. */ void nvlist_dump(const nvlist_t *nvl, int fd) { const nvlist_t *tmpnvl; nvpair_t *nvp, *tmpnvp; void *cookie; int level; level = 0; if (nvlist_dump_error_check(nvl, fd, level)) return; nvp = nvlist_first_nvpair(nvl); while (nvp != NULL) { dprintf(fd, "%*s%s (%s):", level * 4, "", nvpair_name(nvp), nvpair_type_string(nvpair_type(nvp))); switch (nvpair_type(nvp)) { case NV_TYPE_NULL: dprintf(fd, " null\n"); break; case NV_TYPE_BOOL: dprintf(fd, " %s\n", nvpair_get_bool(nvp) ? "TRUE" : "FALSE"); break; case NV_TYPE_NUMBER: dprintf(fd, " %ju (%jd) (0x%jx)\n", (uintmax_t)nvpair_get_number(nvp), (intmax_t)nvpair_get_number(nvp), (uintmax_t)nvpair_get_number(nvp)); break; case NV_TYPE_STRING: dprintf(fd, " [%s]\n", nvpair_get_string(nvp)); break; case NV_TYPE_NVLIST: dprintf(fd, "\n"); tmpnvl = nvpair_get_nvlist(nvp); if (nvlist_dump_error_check(tmpnvl, fd, level + 1)) break; tmpnvp = nvlist_first_nvpair(tmpnvl); if (tmpnvp != NULL) { nvl = tmpnvl; nvp = tmpnvp; level++; continue; } break; case NV_TYPE_DESCRIPTOR: dprintf(fd, " %d\n", nvpair_get_descriptor(nvp)); break; case NV_TYPE_BINARY: { const unsigned char *binary; unsigned int ii; size_t size; binary = nvpair_get_binary(nvp, &size); dprintf(fd, " %zu ", size); for (ii = 0; ii < size; ii++) dprintf(fd, "%02hhx", binary[ii]); dprintf(fd, "\n"); break; } case NV_TYPE_BOOL_ARRAY: { const bool *value; unsigned int ii; size_t nitems; value = nvpair_get_bool_array(nvp, &nitems); dprintf(fd, " [ "); for (ii = 0; ii < nitems; ii++) { dprintf(fd, "%s", value[ii] ? "TRUE" : "FALSE"); if (ii != nitems - 1) dprintf(fd, ", "); } dprintf(fd, " ]\n"); break; } case NV_TYPE_STRING_ARRAY: { const char * const *value; unsigned int ii; size_t nitems; value = nvpair_get_string_array(nvp, &nitems); dprintf(fd, " [ "); for (ii = 0; ii < nitems; ii++) { if (value[ii] == NULL) dprintf(fd, "NULL"); else dprintf(fd, "\"%s\"", value[ii]); if (ii != nitems - 1) dprintf(fd, ", "); } dprintf(fd, " ]\n"); break; } case NV_TYPE_NUMBER_ARRAY: { const uint64_t *value; unsigned int ii; size_t nitems; value = nvpair_get_number_array(nvp, &nitems); dprintf(fd, " [ "); for (ii = 0; ii < nitems; ii++) { dprintf(fd, "%ju (%jd) (0x%jx)", value[ii], value[ii], value[ii]); if (ii != nitems - 1) dprintf(fd, ", "); } dprintf(fd, " ]\n"); break; } case NV_TYPE_DESCRIPTOR_ARRAY: { const int *value; unsigned int ii; size_t nitems; value = nvpair_get_descriptor_array(nvp, &nitems); dprintf(fd, " [ "); for (ii = 0; ii < nitems; ii++) { dprintf(fd, "%d", value[ii]); if (ii != nitems - 1) dprintf(fd, ", "); } dprintf(fd, " ]\n"); break; } case NV_TYPE_NVLIST_ARRAY: { const nvlist_t * const *value; unsigned int ii; size_t nitems; value = nvpair_get_nvlist_array(nvp, &nitems); dprintf(fd, " %zu\n", nitems); tmpnvl = NULL; tmpnvp = NULL; for (ii = 0; ii < nitems; ii++) { if (nvlist_dump_error_check(value[ii], fd, level + 1)) { break; } if (tmpnvl == NULL) { tmpnvp = nvlist_first_nvpair(value[ii]); if (tmpnvp != NULL) { tmpnvl = value[ii]; } else { dprintf(fd, "%*s,\n", (level + 1) * 4, ""); } } } if (tmpnvp != NULL) { nvl = tmpnvl; nvp = tmpnvp; level++; continue; } break; } default: PJDLOG_ABORT("Unknown type: %d.", nvpair_type(nvp)); } while ((nvp = nvlist_next_nvpair(nvl, nvp)) == NULL) { do { cookie = NULL; if (nvlist_in_array(nvl)) dprintf(fd, "%*s,\n", level * 4, ""); nvl = nvlist_get_pararr(nvl, &cookie); if (nvl == NULL) return; if (nvlist_in_array(nvl) && cookie == NULL) { nvp = nvlist_first_nvpair(nvl); } else { nvp = cookie; level--; } } while (nvp == NULL); if (nvlist_in_array(nvl) && cookie == NULL) break; } } } void nvlist_fdump(const nvlist_t *nvl, FILE *fp) { fflush(fp); nvlist_dump(nvl, fileno(fp)); } #endif /* * The function obtains size of the nvlist after nvlist_pack(). */ size_t nvlist_size(const nvlist_t *nvl) { const nvlist_t *tmpnvl; const nvlist_t * const *nvlarray; const nvpair_t *nvp, *tmpnvp; void *cookie; size_t size, nitems; unsigned int ii; NVLIST_ASSERT(nvl); PJDLOG_ASSERT(nvl->nvl_error == 0); size = sizeof(struct nvlist_header); nvp = nvlist_first_nvpair(nvl); while (nvp != NULL) { size += nvpair_header_size(); size += strlen(nvpair_name(nvp)) + 1; if (nvpair_type(nvp) == NV_TYPE_NVLIST) { size += sizeof(struct nvlist_header); size += nvpair_header_size() + 1; tmpnvl = nvpair_get_nvlist(nvp); PJDLOG_ASSERT(tmpnvl->nvl_error == 0); tmpnvp = nvlist_first_nvpair(tmpnvl); if (tmpnvp != NULL) { nvl = tmpnvl; nvp = tmpnvp; continue; } } else if (nvpair_type(nvp) == NV_TYPE_NVLIST_ARRAY) { nvlarray = nvpair_get_nvlist_array(nvp, &nitems); PJDLOG_ASSERT(nitems > 0); size += (nvpair_header_size() + 1) * nitems; size += sizeof(struct nvlist_header) * nitems; tmpnvl = NULL; tmpnvp = NULL; for (ii = 0; ii < nitems; ii++) { PJDLOG_ASSERT(nvlarray[ii]->nvl_error == 0); tmpnvp = nvlist_first_nvpair(nvlarray[ii]); if (tmpnvp != NULL) { tmpnvl = nvlarray[ii]; break; } } if (tmpnvp != NULL) { nvp = tmpnvp; nvl = tmpnvl; continue; } } else { size += nvpair_size(nvp); } while ((nvp = nvlist_next_nvpair(nvl, nvp)) == NULL) { do { cookie = NULL; nvl = nvlist_get_pararr(nvl, &cookie); if (nvl == NULL) goto out; if (nvlist_in_array(nvl) && cookie == NULL) { nvp = nvlist_first_nvpair(nvl); } else { nvp = cookie; } } while (nvp == NULL); if (nvlist_in_array(nvl) && cookie == NULL) break; } } out: return (size); } #if !defined(_KERNEL) && !defined(_STANDALONE) static int * nvlist_xdescriptors(const nvlist_t *nvl, int *descs) { void *cookie; nvpair_t *nvp; int type; NVLIST_ASSERT(nvl); PJDLOG_ASSERT(nvl->nvl_error == 0); cookie = NULL; do { while (nvlist_next(nvl, &type, &cookie) != NULL) { nvp = cookie; switch (type) { case NV_TYPE_DESCRIPTOR: *descs = nvpair_get_descriptor(nvp); descs++; break; case NV_TYPE_DESCRIPTOR_ARRAY: { const int *value; size_t nitems; unsigned int ii; value = nvpair_get_descriptor_array(nvp, &nitems); for (ii = 0; ii < nitems; ii++) { *descs = value[ii]; descs++; } break; } case NV_TYPE_NVLIST: nvl = nvpair_get_nvlist(nvp); cookie = NULL; break; case NV_TYPE_NVLIST_ARRAY: { const nvlist_t * const *value; size_t nitems; value = nvpair_get_nvlist_array(nvp, &nitems); PJDLOG_ASSERT(value != NULL); PJDLOG_ASSERT(nitems > 0); nvl = value[0]; cookie = NULL; break; } } } } while ((nvl = nvlist_get_pararr(nvl, &cookie)) != NULL); return (descs); } #endif #if !defined(_KERNEL) && !defined(_STANDALONE) int * nvlist_descriptors(const nvlist_t *nvl, size_t *nitemsp) { size_t nitems; int *fds; nitems = nvlist_ndescriptors(nvl); fds = nv_malloc(sizeof(fds[0]) * (nitems + 1)); if (fds == NULL) return (NULL); if (nitems > 0) nvlist_xdescriptors(nvl, fds); fds[nitems] = -1; if (nitemsp != NULL) *nitemsp = nitems; return (fds); } #endif size_t nvlist_ndescriptors(const nvlist_t *nvl) { #if !defined(_KERNEL) && !defined(_STANDALONE) void *cookie; nvpair_t *nvp; size_t ndescs; int type; NVLIST_ASSERT(nvl); PJDLOG_ASSERT(nvl->nvl_error == 0); ndescs = 0; cookie = NULL; do { while (nvlist_next(nvl, &type, &cookie) != NULL) { nvp = cookie; switch (type) { case NV_TYPE_DESCRIPTOR: ndescs++; break; case NV_TYPE_NVLIST: nvl = nvpair_get_nvlist(nvp); cookie = NULL; break; case NV_TYPE_NVLIST_ARRAY: { const nvlist_t * const *value; size_t nitems; value = nvpair_get_nvlist_array(nvp, &nitems); PJDLOG_ASSERT(value != NULL); PJDLOG_ASSERT(nitems > 0); nvl = value[0]; cookie = NULL; break; } case NV_TYPE_DESCRIPTOR_ARRAY: { size_t nitems; (void)nvpair_get_descriptor_array(nvp, &nitems); ndescs += nitems; break; } } } } while ((nvl = nvlist_get_pararr(nvl, &cookie)) != NULL); return (ndescs); #else return (0); #endif } static unsigned char * nvlist_pack_header(const nvlist_t *nvl, unsigned char *ptr, size_t *leftp) { struct nvlist_header nvlhdr; NVLIST_ASSERT(nvl); nvlhdr.nvlh_magic = NVLIST_HEADER_MAGIC; nvlhdr.nvlh_version = NVLIST_HEADER_VERSION; nvlhdr.nvlh_flags = nvl->nvl_flags; #if BYTE_ORDER == BIG_ENDIAN nvlhdr.nvlh_flags |= NV_FLAG_BIG_ENDIAN; #endif nvlhdr.nvlh_descriptors = nvlist_ndescriptors(nvl); nvlhdr.nvlh_size = *leftp - sizeof(nvlhdr); PJDLOG_ASSERT(*leftp >= sizeof(nvlhdr)); memcpy(ptr, &nvlhdr, sizeof(nvlhdr)); ptr += sizeof(nvlhdr); *leftp -= sizeof(nvlhdr); return (ptr); } static void * nvlist_xpack(const nvlist_t *nvl, int64_t *fdidxp, size_t *sizep) { unsigned char *buf, *ptr; size_t left, size; const nvlist_t *tmpnvl; nvpair_t *nvp, *tmpnvp; void *cookie; NVLIST_ASSERT(nvl); if (nvl->nvl_error != 0) { ERRNO_SET(nvl->nvl_error); return (NULL); } size = nvlist_size(nvl); buf = nv_malloc(size); if (buf == NULL) return (NULL); ptr = buf; left = size; ptr = nvlist_pack_header(nvl, ptr, &left); nvp = nvlist_first_nvpair(nvl); while (nvp != NULL) { NVPAIR_ASSERT(nvp); nvpair_init_datasize(nvp); ptr = nvpair_pack_header(nvp, ptr, &left); if (ptr == NULL) goto fail; switch (nvpair_type(nvp)) { case NV_TYPE_NULL: ptr = nvpair_pack_null(nvp, ptr, &left); break; case NV_TYPE_BOOL: ptr = nvpair_pack_bool(nvp, ptr, &left); break; case NV_TYPE_NUMBER: ptr = nvpair_pack_number(nvp, ptr, &left); break; case NV_TYPE_STRING: ptr = nvpair_pack_string(nvp, ptr, &left); break; case NV_TYPE_NVLIST: tmpnvl = nvpair_get_nvlist(nvp); ptr = nvlist_pack_header(tmpnvl, ptr, &left); if (ptr == NULL) goto fail; tmpnvp = nvlist_first_nvpair(tmpnvl); if (tmpnvp != NULL) { nvl = tmpnvl; nvp = tmpnvp; continue; } ptr = nvpair_pack_nvlist_up(ptr, &left); break; #if !defined(_KERNEL) && !defined(_STANDALONE) case NV_TYPE_DESCRIPTOR: ptr = nvpair_pack_descriptor(nvp, ptr, fdidxp, &left); break; case NV_TYPE_DESCRIPTOR_ARRAY: ptr = nvpair_pack_descriptor_array(nvp, ptr, fdidxp, &left); break; #endif case NV_TYPE_BINARY: ptr = nvpair_pack_binary(nvp, ptr, &left); break; case NV_TYPE_BOOL_ARRAY: ptr = nvpair_pack_bool_array(nvp, ptr, &left); break; case NV_TYPE_NUMBER_ARRAY: ptr = nvpair_pack_number_array(nvp, ptr, &left); break; case NV_TYPE_STRING_ARRAY: ptr = nvpair_pack_string_array(nvp, ptr, &left); break; case NV_TYPE_NVLIST_ARRAY: { const nvlist_t * const * value; size_t nitems; unsigned int ii; tmpnvl = NULL; value = nvpair_get_nvlist_array(nvp, &nitems); for (ii = 0; ii < nitems; ii++) { ptr = nvlist_pack_header(value[ii], ptr, &left); if (ptr == NULL) goto out; tmpnvp = nvlist_first_nvpair(value[ii]); if (tmpnvp != NULL) { tmpnvl = value[ii]; break; } ptr = nvpair_pack_nvlist_array_next(ptr, &left); if (ptr == NULL) goto out; } if (tmpnvl != NULL) { nvl = tmpnvl; nvp = tmpnvp; continue; } break; } default: PJDLOG_ABORT("Invalid type (%d).", nvpair_type(nvp)); } if (ptr == NULL) goto fail; while ((nvp = nvlist_next_nvpair(nvl, nvp)) == NULL) { do { cookie = NULL; if (nvlist_in_array(nvl)) { ptr = nvpair_pack_nvlist_array_next(ptr, &left); if (ptr == NULL) goto fail; } nvl = nvlist_get_pararr(nvl, &cookie); if (nvl == NULL) goto out; if (nvlist_in_array(nvl) && cookie == NULL) { nvp = nvlist_first_nvpair(nvl); ptr = nvlist_pack_header(nvl, ptr, &left); if (ptr == NULL) goto fail; } else if (nvpair_type((nvpair_t *)cookie) != NV_TYPE_NVLIST_ARRAY) { ptr = nvpair_pack_nvlist_up(ptr, &left); if (ptr == NULL) goto fail; nvp = cookie; } else { nvp = cookie; } } while (nvp == NULL); if (nvlist_in_array(nvl) && cookie == NULL) break; } } out: if (sizep != NULL) *sizep = size; return (buf); fail: nv_free(buf); return (NULL); } void * nvlist_pack(const nvlist_t *nvl, size_t *sizep) { NVLIST_ASSERT(nvl); if (nvl->nvl_error != 0) { ERRNO_SET(nvl->nvl_error); return (NULL); } if (nvlist_ndescriptors(nvl) > 0) { ERRNO_SET(EOPNOTSUPP); return (NULL); } return (nvlist_xpack(nvl, NULL, sizep)); } static bool nvlist_check_header(struct nvlist_header *nvlhdrp) { if (nvlhdrp->nvlh_magic != NVLIST_HEADER_MAGIC) { ERRNO_SET(EINVAL); return (false); } if ((nvlhdrp->nvlh_flags & ~NV_FLAG_ALL_MASK) != 0) { ERRNO_SET(EINVAL); return (false); } #if BYTE_ORDER == BIG_ENDIAN if ((nvlhdrp->nvlh_flags & NV_FLAG_BIG_ENDIAN) == 0) { nvlhdrp->nvlh_size = le64toh(nvlhdrp->nvlh_size); nvlhdrp->nvlh_descriptors = le64toh(nvlhdrp->nvlh_descriptors); } #else if ((nvlhdrp->nvlh_flags & NV_FLAG_BIG_ENDIAN) != 0) { nvlhdrp->nvlh_size = be64toh(nvlhdrp->nvlh_size); nvlhdrp->nvlh_descriptors = be64toh(nvlhdrp->nvlh_descriptors); } #endif return (true); } const unsigned char * nvlist_unpack_header(nvlist_t *nvl, const unsigned char *ptr, size_t nfds, bool *isbep, size_t *leftp) { struct nvlist_header nvlhdr; int inarrayf; if (*leftp < sizeof(nvlhdr)) goto fail; memcpy(&nvlhdr, ptr, sizeof(nvlhdr)); if (!nvlist_check_header(&nvlhdr)) goto fail; if (nvlhdr.nvlh_size != *leftp - sizeof(nvlhdr)) goto fail; /* * nvlh_descriptors might be smaller than nfds in embedded nvlists. */ if (nvlhdr.nvlh_descriptors > nfds) goto fail; if ((nvlhdr.nvlh_flags & ~NV_FLAG_ALL_MASK) != 0) goto fail; inarrayf = (nvl->nvl_flags & NV_FLAG_IN_ARRAY); nvl->nvl_flags = (nvlhdr.nvlh_flags & NV_FLAG_PUBLIC_MASK) | inarrayf; ptr += sizeof(nvlhdr); if (isbep != NULL) *isbep = (((int)nvlhdr.nvlh_flags & NV_FLAG_BIG_ENDIAN) != 0); *leftp -= sizeof(nvlhdr); return (ptr); fail: ERRNO_SET(EINVAL); return (NULL); } static nvlist_t * nvlist_xunpack(const void *buf, size_t size, const int *fds, size_t nfds, int flags) { const unsigned char *ptr; nvlist_t *nvl, *retnvl, *tmpnvl, *array; nvpair_t *nvp; size_t left; bool isbe; PJDLOG_ASSERT((flags & ~(NV_FLAG_PUBLIC_MASK)) == 0); left = size; ptr = buf; tmpnvl = array = NULL; nvl = retnvl = nvlist_create(0); if (nvl == NULL) goto fail; ptr = nvlist_unpack_header(nvl, ptr, nfds, &isbe, &left); if (ptr == NULL) goto fail; if (nvl->nvl_flags != flags) { ERRNO_SET(EILSEQ); goto fail; } while (left > 0) { ptr = nvpair_unpack(isbe, ptr, &left, &nvp); if (ptr == NULL) goto fail; switch (nvpair_type(nvp)) { case NV_TYPE_NULL: ptr = nvpair_unpack_null(isbe, nvp, ptr, &left); break; case NV_TYPE_BOOL: ptr = nvpair_unpack_bool(isbe, nvp, ptr, &left); break; case NV_TYPE_NUMBER: ptr = nvpair_unpack_number(isbe, nvp, ptr, &left); break; case NV_TYPE_STRING: ptr = nvpair_unpack_string(isbe, nvp, ptr, &left); break; case NV_TYPE_NVLIST: ptr = nvpair_unpack_nvlist(isbe, nvp, ptr, &left, nfds, &tmpnvl); if (tmpnvl == NULL || ptr == NULL) goto fail; nvlist_set_parent(tmpnvl, nvp); break; #if !defined(_KERNEL) && !defined(_STANDALONE) && !defined(__NetBSD__) case NV_TYPE_DESCRIPTOR: ptr = nvpair_unpack_descriptor(isbe, nvp, ptr, &left, fds, nfds); break; case NV_TYPE_DESCRIPTOR_ARRAY: ptr = nvpair_unpack_descriptor_array(isbe, nvp, ptr, &left, fds, nfds); break; #endif case NV_TYPE_BINARY: ptr = nvpair_unpack_binary(isbe, nvp, ptr, &left); break; case NV_TYPE_NVLIST_UP: if (nvl->nvl_parent == NULL) goto fail; nvl = nvpair_nvlist(nvl->nvl_parent); nvpair_free_structure(nvp); continue; case NV_TYPE_NVLIST_ARRAY_NEXT: if (nvl->nvl_array_next == NULL) { if (nvl->nvl_parent == NULL) goto fail; nvl = nvpair_nvlist(nvl->nvl_parent); } else { nvl = __DECONST(nvlist_t *, nvlist_get_array_next(nvl)); ptr = nvlist_unpack_header(nvl, ptr, nfds, &isbe, &left); if (ptr == NULL) goto fail; } nvpair_free_structure(nvp); continue; case NV_TYPE_BOOL_ARRAY: ptr = nvpair_unpack_bool_array(isbe, nvp, ptr, &left); break; case NV_TYPE_NUMBER_ARRAY: ptr = nvpair_unpack_number_array(isbe, nvp, ptr, &left); break; case NV_TYPE_STRING_ARRAY: ptr = nvpair_unpack_string_array(isbe, nvp, ptr, &left); break; case NV_TYPE_NVLIST_ARRAY: ptr = nvpair_unpack_nvlist_array(isbe, nvp, ptr, &left, &array); if (ptr == NULL) goto fail; PJDLOG_ASSERT(array != NULL); tmpnvl = array; do { nvlist_set_parent(array, nvp); array = __DECONST(nvlist_t *, nvlist_get_array_next(array)); } while (array != NULL); ptr = nvlist_unpack_header(tmpnvl, ptr, nfds, &isbe, &left); break; default: PJDLOG_ABORT("Invalid type (%d).", nvpair_type(nvp)); } if (ptr == NULL) goto fail; if (!nvlist_move_nvpair(nvl, nvp)) goto fail; if (tmpnvl != NULL) { nvl = tmpnvl; tmpnvl = NULL; } } return (retnvl); fail: nvlist_destroy(retnvl); return (NULL); } nvlist_t * nvlist_unpack(const void *buf, size_t size, int flags) { return (nvlist_xunpack(buf, size, NULL, 0, flags)); } #if !defined(_KERNEL) && !defined(_STANDALONE) && defined(WITH_MSGIO) int nvlist_send(int sock, const nvlist_t *nvl) { size_t datasize, nfds; int *fds; void *data; int64_t fdidx; int ret; if (nvlist_error(nvl) != 0) { ERRNO_SET(nvlist_error(nvl)); return (-1); } fds = nvlist_descriptors(nvl, &nfds); if (fds == NULL) return (-1); ret = -1; fdidx = 0; data = nvlist_xpack(nvl, &fdidx, &datasize); if (data == NULL) goto out; if (buf_send(sock, data, datasize) == -1) goto out; if (nfds > 0) { if (fd_send(sock, fds, nfds) == -1) goto out; } ret = 0; out: ERRNO_SAVE(); nv_free(fds); nv_free(data); ERRNO_RESTORE(); return (ret); } nvlist_t * nvlist_recv(int sock, int flags) { struct nvlist_header nvlhdr; nvlist_t *nvl, *ret; unsigned char *buf; size_t nfds, size, i; int *fds; if (buf_recv(sock, &nvlhdr, sizeof(nvlhdr)) == -1) return (NULL); if (!nvlist_check_header(&nvlhdr)) return (NULL); nfds = (size_t)nvlhdr.nvlh_descriptors; size = sizeof(nvlhdr) + (size_t)nvlhdr.nvlh_size; buf = nv_malloc(size); if (buf == NULL) return (NULL); memcpy(buf, &nvlhdr, sizeof(nvlhdr)); ret = NULL; fds = NULL; if (buf_recv(sock, buf + sizeof(nvlhdr), size - sizeof(nvlhdr)) == -1) goto out; if (nfds > 0) { fds = nv_malloc(nfds * sizeof(fds[0])); if (fds == NULL) goto out; if (fd_recv(sock, fds, nfds) == -1) goto out; } nvl = nvlist_xunpack(buf, size, fds, nfds, flags); if (nvl == NULL) { ERRNO_SAVE(); for (i = 0; i < nfds; i++) close(fds[i]); ERRNO_RESTORE(); goto out; } ret = nvl; out: ERRNO_SAVE(); nv_free(buf); nv_free(fds); ERRNO_RESTORE(); return (ret); } nvlist_t * nvlist_xfer(int sock, nvlist_t *nvl, int flags) { if (nvlist_send(sock, nvl) < 0) { nvlist_destroy(nvl); return (NULL); } nvlist_destroy(nvl); return (nvlist_recv(sock, flags)); } #endif nvpair_t * nvlist_first_nvpair(const nvlist_t *nvl) { NVLIST_ASSERT(nvl); return (TAILQ_FIRST(&nvl->nvl_head)); } nvpair_t * nvlist_next_nvpair(const nvlist_t *nvl, const nvpair_t *nvp) { nvpair_t *retnvp; NVLIST_ASSERT(nvl); NVPAIR_ASSERT(nvp); PJDLOG_ASSERT(nvpair_nvlist(nvp) == nvl); retnvp = nvpair_next(nvp); PJDLOG_ASSERT(retnvp == NULL || nvpair_nvlist(retnvp) == nvl); return (retnvp); } nvpair_t * nvlist_prev_nvpair(const nvlist_t *nvl, const nvpair_t *nvp) { nvpair_t *retnvp; NVLIST_ASSERT(nvl); NVPAIR_ASSERT(nvp); PJDLOG_ASSERT(nvpair_nvlist(nvp) == nvl); retnvp = nvpair_prev(nvp); PJDLOG_ASSERT(nvpair_nvlist(retnvp) == nvl); return (retnvp); } const char * nvlist_next(const nvlist_t *nvl, int *typep, void **cookiep) { nvpair_t *nvp; NVLIST_ASSERT(nvl); if (cookiep == NULL || *cookiep == NULL) nvp = nvlist_first_nvpair(nvl); else nvp = nvlist_next_nvpair(nvl, *cookiep); if (nvp == NULL) return (NULL); if (typep != NULL) *typep = nvpair_type(nvp); if (cookiep != NULL) *cookiep = nvp; return (nvpair_name(nvp)); } bool nvlist_exists(const nvlist_t *nvl, const char *name) { return (nvlist_find(nvl, NV_TYPE_NONE, name) != NULL); } #define NVLIST_EXISTS(type, TYPE) \ bool \ nvlist_exists_##type(const nvlist_t *nvl, const char *name) \ { \ \ return (nvlist_find(nvl, NV_TYPE_##TYPE, name) != NULL); \ } NVLIST_EXISTS(null, NULL) NVLIST_EXISTS(bool, BOOL) NVLIST_EXISTS(number, NUMBER) NVLIST_EXISTS(string, STRING) NVLIST_EXISTS(nvlist, NVLIST) NVLIST_EXISTS(binary, BINARY) NVLIST_EXISTS(bool_array, BOOL_ARRAY) NVLIST_EXISTS(number_array, NUMBER_ARRAY) NVLIST_EXISTS(string_array, STRING_ARRAY) NVLIST_EXISTS(nvlist_array, NVLIST_ARRAY) #if !defined(_KERNEL) && !defined(_STANDALONE) NVLIST_EXISTS(descriptor, DESCRIPTOR) NVLIST_EXISTS(descriptor_array, DESCRIPTOR_ARRAY) #endif #undef NVLIST_EXISTS void nvlist_add_nvpair(nvlist_t *nvl, const nvpair_t *nvp) { nvpair_t *newnvp; NVPAIR_ASSERT(nvp); if (nvlist_error(nvl) != 0) { ERRNO_SET(nvlist_error(nvl)); return; } if ((nvl->nvl_flags & NV_FLAG_NO_UNIQUE) == 0) { if (nvlist_exists(nvl, nvpair_name(nvp))) { nvl->nvl_error = EEXIST; ERRNO_SET(nvlist_error(nvl)); return; } } newnvp = nvpair_clone(nvp); if (newnvp == NULL) { nvl->nvl_error = ERRNO_OR_DEFAULT(ENOMEM); ERRNO_SET(nvlist_error(nvl)); return; } nvpair_insert(&nvl->nvl_head, newnvp, nvl); } #if !defined(_STANDALONE) void nvlist_add_stringf(nvlist_t *nvl, const char *name, const char *valuefmt, ...) { va_list valueap; va_start(valueap, valuefmt); nvlist_add_stringv(nvl, name, valuefmt, valueap); va_end(valueap); } void nvlist_add_stringv(nvlist_t *nvl, const char *name, const char *valuefmt, va_list valueap) { nvpair_t *nvp; if (nvlist_error(nvl) != 0) { ERRNO_SET(nvlist_error(nvl)); return; } nvp = nvpair_create_stringv(name, valuefmt, valueap); if (nvp == NULL) { nvl->nvl_error = ERRNO_OR_DEFAULT(ENOMEM); ERRNO_SET(nvl->nvl_error); } else { (void)nvlist_move_nvpair(nvl, nvp); } } #endif void nvlist_add_null(nvlist_t *nvl, const char *name) { nvpair_t *nvp; if (nvlist_error(nvl) != 0) { ERRNO_SET(nvlist_error(nvl)); return; } nvp = nvpair_create_null(name); if (nvp == NULL) { nvl->nvl_error = ERRNO_OR_DEFAULT(ENOMEM); ERRNO_SET(nvl->nvl_error); } else { (void)nvlist_move_nvpair(nvl, nvp); } } void nvlist_add_binary(nvlist_t *nvl, const char *name, const void *value, size_t size) { nvpair_t *nvp; if (nvlist_error(nvl) != 0) { ERRNO_SET(nvlist_error(nvl)); return; } nvp = nvpair_create_binary(name, value, size); if (nvp == NULL) { nvl->nvl_error = ERRNO_OR_DEFAULT(ENOMEM); ERRNO_SET(nvl->nvl_error); } else { (void)nvlist_move_nvpair(nvl, nvp); } } #define NVLIST_ADD(vtype, type) \ void \ nvlist_add_##type(nvlist_t *nvl, const char *name, vtype value) \ { \ nvpair_t *nvp; \ \ if (nvlist_error(nvl) != 0) { \ ERRNO_SET(nvlist_error(nvl)); \ return; \ } \ \ nvp = nvpair_create_##type(name, value); \ if (nvp == NULL) { \ nvl->nvl_error = ERRNO_OR_DEFAULT(ENOMEM); \ ERRNO_SET(nvl->nvl_error); \ } else { \ (void)nvlist_move_nvpair(nvl, nvp); \ } \ } NVLIST_ADD(bool, bool) NVLIST_ADD(uint64_t, number) NVLIST_ADD(const char *, string) NVLIST_ADD(const nvlist_t *, nvlist) #if !defined(_KERNEL) && !defined(_STANDALONE) NVLIST_ADD(int, descriptor); #endif #undef NVLIST_ADD #define NVLIST_ADD_ARRAY(vtype, type) \ void \ nvlist_add_##type##_array(nvlist_t *nvl, const char *name, vtype value, \ size_t nitems) \ { \ nvpair_t *nvp; \ \ if (nvlist_error(nvl) != 0) { \ ERRNO_SET(nvlist_error(nvl)); \ return; \ } \ \ nvp = nvpair_create_##type##_array(name, value, nitems); \ if (nvp == NULL) { \ nvl->nvl_error = ERRNO_OR_DEFAULT(ENOMEM); \ ERRNO_SET(nvl->nvl_error); \ } else { \ (void)nvlist_move_nvpair(nvl, nvp); \ } \ } NVLIST_ADD_ARRAY(const bool *, bool) NVLIST_ADD_ARRAY(const uint64_t *, number) NVLIST_ADD_ARRAY(const char * const *, string) NVLIST_ADD_ARRAY(const nvlist_t * const *, nvlist) #if !defined(_KERNEL) && !defined(_STANDALONE) NVLIST_ADD_ARRAY(const int *, descriptor) #endif #undef NVLIST_ADD_ARRAY #define NVLIST_APPEND_ARRAY(vtype, type, TYPE) \ void \ nvlist_append_##type##_array(nvlist_t *nvl, const char *name, vtype value)\ { \ nvpair_t *nvp; \ \ if (nvlist_error(nvl) != 0) { \ ERRNO_SET(nvlist_error(nvl)); \ return; \ } \ nvp = nvlist_find(nvl, NV_TYPE_##TYPE##_ARRAY, name); \ if (nvp == NULL) { \ nvlist_add_##type##_array(nvl, name, &value, 1); \ return; \ } \ if (nvpair_append_##type##_array(nvp, value) == -1) { \ nvl->nvl_error = ERRNO_OR_DEFAULT(ENOMEM); \ ERRNO_SET(nvl->nvl_error); \ } \ } NVLIST_APPEND_ARRAY(const bool, bool, BOOL) NVLIST_APPEND_ARRAY(const uint64_t, number, NUMBER) NVLIST_APPEND_ARRAY(const char *, string, STRING) NVLIST_APPEND_ARRAY(const nvlist_t *, nvlist, NVLIST) #if !defined(_KERNEL) && !defined(_STANDALONE) NVLIST_APPEND_ARRAY(const int, descriptor, DESCRIPTOR) #endif #undef NVLIST_APPEND_ARRAY bool nvlist_move_nvpair(nvlist_t *nvl, nvpair_t *nvp) { NVPAIR_ASSERT(nvp); PJDLOG_ASSERT(nvpair_nvlist(nvp) == NULL); if (nvlist_error(nvl) != 0) { nvpair_free(nvp); ERRNO_SET(nvlist_error(nvl)); return (false); } if ((nvl->nvl_flags & NV_FLAG_NO_UNIQUE) == 0) { if (nvlist_exists(nvl, nvpair_name(nvp))) { nvpair_free(nvp); nvl->nvl_error = EEXIST; ERRNO_SET(nvl->nvl_error); return (false); } } nvpair_insert(&nvl->nvl_head, nvp, nvl); return (true); } void nvlist_move_string(nvlist_t *nvl, const char *name, char *value) { nvpair_t *nvp; if (nvlist_error(nvl) != 0) { nv_free(value); ERRNO_SET(nvlist_error(nvl)); return; } nvp = nvpair_move_string(name, value); if (nvp == NULL) { nvl->nvl_error = ERRNO_OR_DEFAULT(ENOMEM); ERRNO_SET(nvl->nvl_error); } else { (void)nvlist_move_nvpair(nvl, nvp); } } void nvlist_move_nvlist(nvlist_t *nvl, const char *name, nvlist_t *value) { nvpair_t *nvp; if (nvlist_error(nvl) != 0) { if (value != NULL && nvlist_get_nvpair_parent(value) != NULL) nvlist_destroy(value); ERRNO_SET(nvlist_error(nvl)); return; } nvp = nvpair_move_nvlist(name, value); if (nvp == NULL) { nvl->nvl_error = ERRNO_OR_DEFAULT(ENOMEM); ERRNO_SET(nvl->nvl_error); } else { (void)nvlist_move_nvpair(nvl, nvp); } } #if !defined(_KERNEL) && !defined(_STANDALONE) void nvlist_move_descriptor(nvlist_t *nvl, const char *name, int value) { nvpair_t *nvp; if (nvlist_error(nvl) != 0) { close(value); ERRNO_SET(nvlist_error(nvl)); return; } nvp = nvpair_move_descriptor(name, value); if (nvp == NULL) { nvl->nvl_error = ERRNO_OR_DEFAULT(ENOMEM); ERRNO_SET(nvl->nvl_error); } else { (void)nvlist_move_nvpair(nvl, nvp); } } #endif void nvlist_move_binary(nvlist_t *nvl, const char *name, void *value, size_t size) { nvpair_t *nvp; if (nvlist_error(nvl) != 0) { nv_free(value); ERRNO_SET(nvlist_error(nvl)); return; } nvp = nvpair_move_binary(name, value, size); if (nvp == NULL) { nvl->nvl_error = ERRNO_OR_DEFAULT(ENOMEM); ERRNO_SET(nvl->nvl_error); } else { (void)nvlist_move_nvpair(nvl, nvp); } } void nvlist_move_bool_array(nvlist_t *nvl, const char *name, bool *value, size_t nitems) { nvpair_t *nvp; if (nvlist_error(nvl) != 0) { nv_free(value); ERRNO_SET(nvlist_error(nvl)); return; } nvp = nvpair_move_bool_array(name, value, nitems); if (nvp == NULL) { nvl->nvl_error = ERRNO_OR_DEFAULT(ENOMEM); ERRNO_SET(nvl->nvl_error); } else { (void)nvlist_move_nvpair(nvl, nvp); } } void nvlist_move_string_array(nvlist_t *nvl, const char *name, char **value, size_t nitems) { nvpair_t *nvp; size_t i; if (nvlist_error(nvl) != 0) { if (value != NULL) { for (i = 0; i < nitems; i++) nv_free(value[i]); nv_free(value); } ERRNO_SET(nvlist_error(nvl)); return; } nvp = nvpair_move_string_array(name, value, nitems); if (nvp == NULL) { nvl->nvl_error = ERRNO_OR_DEFAULT(ENOMEM); ERRNO_SET(nvl->nvl_error); } else { (void)nvlist_move_nvpair(nvl, nvp); } } void nvlist_move_nvlist_array(nvlist_t *nvl, const char *name, nvlist_t **value, size_t nitems) { nvpair_t *nvp; size_t i; if (nvlist_error(nvl) != 0) { if (value != NULL) { for (i = 0; i < nitems; i++) { if (nvlist_get_pararr(value[i], NULL) == NULL) nvlist_destroy(value[i]); } } nv_free(value); ERRNO_SET(nvlist_error(nvl)); return; } nvp = nvpair_move_nvlist_array(name, value, nitems); if (nvp == NULL) { nvl->nvl_error = ERRNO_OR_DEFAULT(ENOMEM); ERRNO_SET(nvl->nvl_error); } else { (void)nvlist_move_nvpair(nvl, nvp); } } void nvlist_move_number_array(nvlist_t *nvl, const char *name, uint64_t *value, size_t nitems) { nvpair_t *nvp; if (nvlist_error(nvl) != 0) { nv_free(value); ERRNO_SET(nvlist_error(nvl)); return; } nvp = nvpair_move_number_array(name, value, nitems); if (nvp == NULL) { nvl->nvl_error = ERRNO_OR_DEFAULT(ENOMEM); ERRNO_SET(nvl->nvl_error); } else { (void)nvlist_move_nvpair(nvl, nvp); } } #if !defined(_KERNEL) && !defined(_STANDALONE) void nvlist_move_descriptor_array(nvlist_t *nvl, const char *name, int *value, size_t nitems) { nvpair_t *nvp; size_t i; if (nvlist_error(nvl) != 0) { if (value != 0) { for (i = 0; i < nitems; i++) close(value[i]); nv_free(value); } ERRNO_SET(nvlist_error(nvl)); return; } nvp = nvpair_move_descriptor_array(name, value, nitems); if (nvp == NULL) { nvl->nvl_error = ERRNO_OR_DEFAULT(ENOMEM); ERRNO_SET(nvl->nvl_error); } else { (void)nvlist_move_nvpair(nvl, nvp); } } #endif const nvpair_t * nvlist_get_nvpair(const nvlist_t *nvl, const char *name) { return (nvlist_find(nvl, NV_TYPE_NONE, name)); } #define NVLIST_GET(ftype, type, TYPE) \ ftype \ nvlist_get_##type(const nvlist_t *nvl, const char *name) \ { \ const nvpair_t *nvp; \ \ nvp = nvlist_find(nvl, NV_TYPE_##TYPE, name); \ if (nvp == NULL) \ nvlist_report_missing(NV_TYPE_##TYPE, name); \ return (nvpair_get_##type(nvp)); \ } NVLIST_GET(bool, bool, BOOL) NVLIST_GET(uint64_t, number, NUMBER) NVLIST_GET(const char *, string, STRING) NVLIST_GET(const nvlist_t *, nvlist, NVLIST) #if !defined(_KERNEL) && !defined(_STANDALONE) NVLIST_GET(int, descriptor, DESCRIPTOR) #endif #undef NVLIST_GET const void * nvlist_get_binary(const nvlist_t *nvl, const char *name, size_t *sizep) { nvpair_t *nvp; nvp = nvlist_find(nvl, NV_TYPE_BINARY, name); if (nvp == NULL) nvlist_report_missing(NV_TYPE_BINARY, name); return (nvpair_get_binary(nvp, sizep)); } #define NVLIST_GET_ARRAY(ftype, type, TYPE) \ ftype \ nvlist_get_##type##_array(const nvlist_t *nvl, const char *name, \ size_t *nitems) \ { \ const nvpair_t *nvp; \ \ nvp = nvlist_find(nvl, NV_TYPE_##TYPE##_ARRAY, name); \ if (nvp == NULL) \ nvlist_report_missing(NV_TYPE_##TYPE##_ARRAY, name); \ return (nvpair_get_##type##_array(nvp, nitems)); \ } NVLIST_GET_ARRAY(const bool *, bool, BOOL) NVLIST_GET_ARRAY(const uint64_t *, number, NUMBER) NVLIST_GET_ARRAY(const char * const *, string, STRING) NVLIST_GET_ARRAY(const nvlist_t * const *, nvlist, NVLIST) #if !defined(_KERNEL) && !defined(_STANDALONE) NVLIST_GET_ARRAY(const int *, descriptor, DESCRIPTOR) #endif #undef NVLIST_GET_ARRAY #define NVLIST_TAKE(ftype, type, TYPE) \ ftype \ nvlist_take_##type(nvlist_t *nvl, const char *name) \ { \ nvpair_t *nvp; \ ftype value; \ \ nvp = nvlist_find(nvl, NV_TYPE_##TYPE, name); \ if (nvp == NULL) \ nvlist_report_missing(NV_TYPE_##TYPE, name); \ value = (ftype)(intptr_t)nvpair_get_##type(nvp); \ nvlist_remove_nvpair(nvl, nvp); \ nvpair_free_structure(nvp); \ return (value); \ } NVLIST_TAKE(bool, bool, BOOL) NVLIST_TAKE(uint64_t, number, NUMBER) NVLIST_TAKE(char *, string, STRING) NVLIST_TAKE(nvlist_t *, nvlist, NVLIST) #if !defined(_KERNEL) && !defined(_STANDALONE) NVLIST_TAKE(int, descriptor, DESCRIPTOR) #endif #undef NVLIST_TAKE void * nvlist_take_binary(nvlist_t *nvl, const char *name, size_t *sizep) { nvpair_t *nvp; void *value; nvp = nvlist_find(nvl, NV_TYPE_BINARY, name); if (nvp == NULL) nvlist_report_missing(NV_TYPE_BINARY, name); value = (void *)(intptr_t)nvpair_get_binary(nvp, sizep); nvlist_remove_nvpair(nvl, nvp); nvpair_free_structure(nvp); return (value); } #define NVLIST_TAKE_ARRAY(ftype, type, TYPE) \ ftype \ nvlist_take_##type##_array(nvlist_t *nvl, const char *name, \ size_t *nitems) \ { \ nvpair_t *nvp; \ ftype value; \ \ nvp = nvlist_find(nvl, NV_TYPE_##TYPE##_ARRAY, name); \ if (nvp == NULL) \ nvlist_report_missing(NV_TYPE_##TYPE##_ARRAY, name); \ value = (ftype)(intptr_t)nvpair_get_##type##_array(nvp, nitems);\ nvlist_remove_nvpair(nvl, nvp); \ nvpair_free_structure(nvp); \ return (value); \ } NVLIST_TAKE_ARRAY(bool *, bool, BOOL) NVLIST_TAKE_ARRAY(uint64_t *, number, NUMBER) NVLIST_TAKE_ARRAY(char **, string, STRING) NVLIST_TAKE_ARRAY(nvlist_t **, nvlist, NVLIST) #if !defined(_KERNEL) && !defined(_STANDALONE) NVLIST_TAKE_ARRAY(int *, descriptor, DESCRIPTOR) #endif void nvlist_remove_nvpair(nvlist_t *nvl, nvpair_t *nvp) { NVLIST_ASSERT(nvl); NVPAIR_ASSERT(nvp); PJDLOG_ASSERT(nvpair_nvlist(nvp) == nvl); nvpair_remove(&nvl->nvl_head, nvp, nvl); } void nvlist_free(nvlist_t *nvl, const char *name) { nvlist_free_type(nvl, name, NV_TYPE_NONE); } #define NVLIST_FREE(type, TYPE) \ void \ nvlist_free_##type(nvlist_t *nvl, const char *name) \ { \ \ nvlist_free_type(nvl, name, NV_TYPE_##TYPE); \ } NVLIST_FREE(null, NULL) NVLIST_FREE(bool, BOOL) NVLIST_FREE(number, NUMBER) NVLIST_FREE(string, STRING) NVLIST_FREE(nvlist, NVLIST) NVLIST_FREE(binary, BINARY) NVLIST_FREE(bool_array, BOOL_ARRAY) NVLIST_FREE(number_array, NUMBER_ARRAY) NVLIST_FREE(string_array, STRING_ARRAY) NVLIST_FREE(nvlist_array, NVLIST_ARRAY) #if !defined(_KERNEL) && !defined(_STANDALONE) NVLIST_FREE(descriptor, DESCRIPTOR) NVLIST_FREE(descriptor_array, DESCRIPTOR_ARRAY) #endif #undef NVLIST_FREE void nvlist_free_nvpair(nvlist_t *nvl, nvpair_t *nvp) { NVLIST_ASSERT(nvl); NVPAIR_ASSERT(nvp); PJDLOG_ASSERT(nvpair_nvlist(nvp) == nvl); nvlist_remove_nvpair(nvl, nvp); nvpair_free(nvp); } |
| 3051 3 3 1 2 3 3 1 2 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 197 198 199 200 201 202 203 204 205 206 207 208 209 210 211 212 213 214 215 216 217 218 219 220 221 222 223 224 225 226 227 228 229 230 231 232 233 234 235 236 237 238 239 240 241 242 243 244 245 246 247 248 249 250 251 252 253 254 255 256 257 258 259 260 261 262 263 264 265 266 267 268 269 270 271 272 273 274 275 276 277 278 279 280 281 282 283 284 285 286 287 288 289 290 291 292 293 294 295 296 297 298 299 300 301 302 303 304 305 306 307 308 309 310 311 312 313 314 315 316 317 318 319 320 321 322 323 324 325 326 327 328 329 330 331 332 333 334 335 336 337 338 339 340 341 342 343 344 345 346 347 348 349 350 351 352 353 354 355 356 357 358 359 360 361 362 363 364 365 366 367 368 369 370 371 372 373 374 375 376 377 378 379 380 381 382 383 384 385 386 387 388 389 390 391 392 393 394 395 396 397 398 399 400 401 402 403 404 405 406 407 408 409 410 411 412 413 414 415 416 417 418 419 420 421 422 423 424 425 426 427 428 429 430 431 432 433 434 435 436 437 438 439 440 441 442 443 444 445 446 447 448 449 450 451 452 453 454 455 456 457 458 459 460 461 462 463 464 465 466 467 468 469 470 471 472 473 474 475 476 477 478 479 480 481 482 483 484 485 486 487 488 489 490 491 492 493 494 495 496 497 498 499 500 501 502 503 504 505 506 507 508 509 510 511 512 513 514 515 516 517 518 519 520 521 522 523 524 525 526 527 528 529 530 531 532 533 534 535 536 537 538 539 540 541 542 543 544 | /* $NetBSD: kern_clock.c,v 1.148 2022/03/19 14:34:47 riastradh Exp $ */ /*- * Copyright (c) 2000, 2004, 2006, 2007, 2008 The NetBSD Foundation, Inc. * All rights reserved. * * This code is derived from software contributed to The NetBSD Foundation * by Jason R. Thorpe of the Numerical Aerospace Simulation Facility, * NASA Ames Research Center. * This code is derived from software contributed to The NetBSD Foundation * by Charles M. Hannum. * * Redistribution and use in source and binary forms, with or without * modification, are permitted provided that the following conditions * are met: * 1. Redistributions of source code must retain the above copyright * notice, this list of conditions and the following disclaimer. * 2. Redistributions in binary form must reproduce the above copyright * notice, this list of conditions and the following disclaimer in the * documentation and/or other materials provided with the distribution. * * THIS SOFTWARE IS PROVIDED BY THE NETBSD FOUNDATION, INC. AND CONTRIBUTORS * ``AS IS'' AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED * TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR * PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE FOUNDATION OR CONTRIBUTORS * BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR * CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF * SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS * INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN * CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) * ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE * POSSIBILITY OF SUCH DAMAGE. */ /*- * Copyright (c) 1982, 1986, 1991, 1993 * The Regents of the University of California. All rights reserved. * (c) UNIX System Laboratories, Inc. * All or some portions of this file are derived from material licensed * to the University of California by American Telephone and Telegraph * Co. or Unix System Laboratories, Inc. and are reproduced herein with * the permission of UNIX System Laboratories, Inc. * * Redistribution and use in source and binary forms, with or without * modification, are permitted provided that the following conditions * are met: * 1. Redistributions of source code must retain the above copyright * notice, this list of conditions and the following disclaimer. * 2. Redistributions in binary form must reproduce the above copyright * notice, this list of conditions and the following disclaimer in the * documentation and/or other materials provided with the distribution. * 3. Neither the name of the University nor the names of its contributors * may be used to endorse or promote products derived from this software * without specific prior written permission. * * THIS SOFTWARE IS PROVIDED BY THE REGENTS AND CONTRIBUTORS ``AS IS'' AND * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE * ARE DISCLAIMED. IN NO EVENT SHALL THE REGENTS OR CONTRIBUTORS BE LIABLE * FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL * DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS * OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) * HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT * LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY * OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF * SUCH DAMAGE. * * @(#)kern_clock.c 8.5 (Berkeley) 1/21/94 */ #include <sys/cdefs.h> __KERNEL_RCSID(0, "$NetBSD: kern_clock.c,v 1.148 2022/03/19 14:34:47 riastradh Exp $"); #ifdef _KERNEL_OPT #include "opt_dtrace.h" #include "opt_gprof.h" #include "opt_multiprocessor.h" #endif #include <sys/param.h> #include <sys/systm.h> #include <sys/callout.h> #include <sys/kernel.h> #include <sys/proc.h> #include <sys/resourcevar.h> #include <sys/signalvar.h> #include <sys/sysctl.h> #include <sys/timex.h> #include <sys/sched.h> #include <sys/time.h> #include <sys/timetc.h> #include <sys/cpu.h> #include <sys/atomic.h> #include <sys/rndsource.h> #ifdef GPROF #include <sys/gmon.h> #endif #ifdef KDTRACE_HOOKS #include <sys/dtrace_bsd.h> #include <sys/cpu.h> cyclic_clock_func_t cyclic_clock_func[MAXCPUS]; #endif static int sysctl_kern_clockrate(SYSCTLFN_PROTO); /* * Clock handling routines. * * This code is written to operate with two timers that run independently of * each other. The main clock, running hz times per second, is used to keep * track of real time. The second timer handles kernel and user profiling, * and does resource use estimation. If the second timer is programmable, * it is randomized to avoid aliasing between the two clocks. For example, * the randomization prevents an adversary from always giving up the CPU * just before its quantum expires. Otherwise, it would never accumulate * CPU ticks. The mean frequency of the second timer is stathz. * * If no second timer exists, stathz will be zero; in this case we drive * profiling and statistics off the main clock. This WILL NOT be accurate; * do not do it unless absolutely necessary. * * The statistics clock may (or may not) be run at a higher rate while * profiling. This profile clock runs at profhz. We require that profhz * be an integral multiple of stathz. * * If the statistics clock is running fast, it must be divided by the ratio * profhz/stathz for statistics. (For profiling, every tick counts.) */ int stathz; int profhz; int profsrc; int schedhz; int profprocs; static int hardclock_ticks; static int hardscheddiv; /* hard => sched divider (used if schedhz == 0) */ static int psdiv; /* prof => stat divider */ int psratio; /* ratio: prof / stat */ struct clockrnd { struct krndsource source; unsigned needed; }; static struct clockrnd hardclockrnd __aligned(COHERENCY_UNIT); static struct clockrnd statclockrnd __aligned(COHERENCY_UNIT); static void clockrnd_get(size_t needed, void *cookie) { struct clockrnd *C = cookie; /* Start sampling. */ atomic_store_relaxed(&C->needed, 2*NBBY*needed); } static void clockrnd_sample(struct clockrnd *C) { struct cpu_info *ci = curcpu(); /* If there's nothing needed right now, stop here. */ if (__predict_true(atomic_load_relaxed(&C->needed) == 0)) return; /* * If we're not the primary core of a package, we're probably * driven by the same clock as the primary core, so don't * bother. */ if (ci != ci->ci_package1st) return; /* Take a sample and enter it into the pool. */ rnd_add_uint32(&C->source, 0); /* * On the primary CPU, count down. Using an atomic decrement * here isn't really necessary -- on every platform we care * about, stores to unsigned int are atomic, and the only other * memory operation that could happen here is for another CPU * to store a higher value for needed. But using an atomic * decrement avoids giving the impression of data races, and is * unlikely to hurt because only one CPU will ever be writing * to the location. */ if (CPU_IS_PRIMARY(curcpu())) { unsigned needed __diagused; needed = atomic_dec_uint_nv(&C->needed); KASSERT(needed != UINT_MAX); } } static u_int get_intr_timecount(struct timecounter *); static struct timecounter intr_timecounter = { .tc_get_timecount = get_intr_timecount, .tc_poll_pps = NULL, .tc_counter_mask = ~0u, .tc_frequency = 0, .tc_name = "clockinterrupt", /* quality - minimum implementation level for a clock */ .tc_quality = 0, .tc_priv = NULL, }; static u_int get_intr_timecount(struct timecounter *tc) { return (u_int)getticks(); } int getticks(void) { return atomic_load_relaxed(&hardclock_ticks); } /* * Initialize clock frequencies and start both clocks running. */ void initclocks(void) { static struct sysctllog *clog; int i; /* * Set divisors to 1 (normal case) and let the machine-specific * code do its bit. */ psdiv = 1; /* * Call cpu_initclocks() before registering the default * timecounter, in case it needs to adjust hz. */ const int old_hz = hz; cpu_initclocks(); if (old_hz != hz) { tick = 1000000 / hz; tickadj = (240000 / (60 * hz)) ? (240000 / (60 * hz)) : 1; } /* * provide minimum default time counter * will only run at interrupt resolution */ intr_timecounter.tc_frequency = hz; tc_init(&intr_timecounter); /* * Compute profhz and stathz, fix profhz if needed. */ i = stathz ? stathz : hz; if (profhz == 0) profhz = i; psratio = profhz / i; if (schedhz == 0) { /* 16Hz is best */ hardscheddiv = hz / 16; if (hardscheddiv <= 0) panic("hardscheddiv"); } sysctl_createv(&clog, 0, NULL, NULL, CTLFLAG_PERMANENT, CTLTYPE_STRUCT, "clockrate", SYSCTL_DESCR("Kernel clock rates"), sysctl_kern_clockrate, 0, NULL, sizeof(struct clockinfo), CTL_KERN, KERN_CLOCKRATE, CTL_EOL); sysctl_createv(&clog, 0, NULL, NULL, CTLFLAG_PERMANENT, CTLTYPE_INT, "hardclock_ticks", SYSCTL_DESCR("Number of hardclock ticks"), NULL, 0, &hardclock_ticks, sizeof(hardclock_ticks), CTL_KERN, KERN_HARDCLOCK_TICKS, CTL_EOL); rndsource_setcb(&hardclockrnd.source, clockrnd_get, &hardclockrnd); rnd_attach_source(&hardclockrnd.source, "hardclock", RND_TYPE_SKEW, RND_FLAG_COLLECT_TIME|RND_FLAG_HASCB); if (stathz) { rndsource_setcb(&statclockrnd.source, clockrnd_get, &statclockrnd); rnd_attach_source(&statclockrnd.source, "statclock", RND_TYPE_SKEW, RND_FLAG_COLLECT_TIME|RND_FLAG_HASCB); } } /* * The real-time timer, interrupting hz times per second. */ void hardclock(struct clockframe *frame) { struct lwp *l; struct cpu_info *ci; clockrnd_sample(&hardclockrnd); ci = curcpu(); l = ci->ci_onproc; ptimer_tick(l, CLKF_USERMODE(frame)); /* * If no separate statistics clock is available, run it from here. */ if (stathz == 0) statclock(frame); /* * If no separate schedclock is provided, call it here * at about 16 Hz. */ if (schedhz == 0) { if ((int)(--ci->ci_schedstate.spc_schedticks) <= 0) { schedclock(l); ci->ci_schedstate.spc_schedticks = hardscheddiv; } } if ((--ci->ci_schedstate.spc_ticks) <= 0) sched_tick(ci); if (CPU_IS_PRIMARY(ci)) { atomic_store_relaxed(&hardclock_ticks, atomic_load_relaxed(&hardclock_ticks) + 1); tc_ticktock(); } /* * Update real-time timeout queue. */ callout_hardclock(); } /* * Start profiling on a process. * * Kernel profiling passes proc0 which never exits and hence * keeps the profile clock running constantly. */ void startprofclock(struct proc *p) { KASSERT(mutex_owned(&p->p_stmutex)); if ((p->p_stflag & PST_PROFIL) == 0) { p->p_stflag |= PST_PROFIL; /* * This is only necessary if using the clock as the * profiling source. */ if (++profprocs == 1 && stathz != 0) psdiv = psratio; } } /* * Stop profiling on a process. */ void stopprofclock(struct proc *p) { KASSERT(mutex_owned(&p->p_stmutex)); if (p->p_stflag & PST_PROFIL) { p->p_stflag &= ~PST_PROFIL; /* * This is only necessary if using the clock as the * profiling source. */ if (--profprocs == 0 && stathz != 0) psdiv = 1; } } void schedclock(struct lwp *l) { if ((l->l_flag & LW_IDLE) != 0) return; sched_schedclock(l); } /* * Statistics clock. Grab profile sample, and if divider reaches 0, * do process and kernel statistics. */ void statclock(struct clockframe *frame) { #ifdef GPROF struct gmonparam *g; intptr_t i; #endif struct cpu_info *ci = curcpu(); struct schedstate_percpu *spc = &ci->ci_schedstate; struct proc *p; struct lwp *l; if (stathz) clockrnd_sample(&statclockrnd); /* * Notice changes in divisor frequency, and adjust clock * frequency accordingly. */ if (spc->spc_psdiv != psdiv) { spc->spc_psdiv = psdiv; spc->spc_pscnt = psdiv; if (psdiv == 1) { setstatclockrate(stathz); } else { setstatclockrate(profhz); } } l = ci->ci_onproc; if ((l->l_flag & LW_IDLE) != 0) { /* * don't account idle lwps as swapper. */ p = NULL; } else { p = l->l_proc; mutex_spin_enter(&p->p_stmutex); } if (CLKF_USERMODE(frame)) { KASSERT(p != NULL); if ((p->p_stflag & PST_PROFIL) && profsrc == PROFSRC_CLOCK) addupc_intr(l, CLKF_PC(frame)); if (--spc->spc_pscnt > 0) { mutex_spin_exit(&p->p_stmutex); return; } /* * Came from user mode; CPU was in user state. * If this process is being profiled record the tick. */ p->p_uticks++; if (p->p_nice > NZERO) spc->spc_cp_time[CP_NICE]++; else spc->spc_cp_time[CP_USER]++; } else { #ifdef GPROF /* * Kernel statistics are just like addupc_intr, only easier. */ #if defined(MULTIPROCESSOR) && !defined(_RUMPKERNEL) g = curcpu()->ci_gmon; if (g != NULL && profsrc == PROFSRC_CLOCK && g->state == GMON_PROF_ON) { #else g = &_gmonparam; if (profsrc == PROFSRC_CLOCK && g->state == GMON_PROF_ON) { #endif i = CLKF_PC(frame) - g->lowpc; if (i < g->textsize) { i /= HISTFRACTION * sizeof(*g->kcount); g->kcount[i]++; } } #endif #ifdef LWP_PC if (p != NULL && profsrc == PROFSRC_CLOCK && (p->p_stflag & PST_PROFIL)) { addupc_intr(l, LWP_PC(l)); } #endif if (--spc->spc_pscnt > 0) { if (p != NULL) mutex_spin_exit(&p->p_stmutex); return; } /* * Came from kernel mode, so we were: * - handling an interrupt, * - doing syscall or trap work on behalf of the current * user process, or * - spinning in the idle loop. * Whichever it is, charge the time as appropriate. * Note that we charge interrupts to the current process, * regardless of whether they are ``for'' that process, * so that we know how much of its real time was spent * in ``non-process'' (i.e., interrupt) work. */ if (CLKF_INTR(frame) || (curlwp->l_pflag & LP_INTR) != 0) { if (p != NULL) { p->p_iticks++; } spc->spc_cp_time[CP_INTR]++; } else if (p != NULL) { p->p_sticks++; spc->spc_cp_time[CP_SYS]++; } else { spc->spc_cp_time[CP_IDLE]++; } } spc->spc_pscnt = psdiv; if (p != NULL) { atomic_inc_uint(&l->l_cpticks); mutex_spin_exit(&p->p_stmutex); } #ifdef KDTRACE_HOOKS cyclic_clock_func_t func = cyclic_clock_func[cpu_index(ci)]; if (func) { (*func)((struct clockframe *)frame); } #endif } /* * sysctl helper routine for kern.clockrate. Assembles a struct on * the fly to be returned to the caller. */ static int sysctl_kern_clockrate(SYSCTLFN_ARGS) { struct clockinfo clkinfo; struct sysctlnode node; clkinfo.tick = tick; clkinfo.tickadj = tickadj; clkinfo.hz = hz; clkinfo.profhz = profhz; clkinfo.stathz = stathz ? stathz : hz; node = *rnode; node.sysctl_data = &clkinfo; return (sysctl_lookup(SYSCTLFN_CALL(&node))); } |
| 8 626 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 | /* $NetBSD: scsipi_base.h,v 1.24 2017/02/26 10:58:47 maya Exp $ */ /*- * Copyright (c) 1998, 2004 The NetBSD Foundation, Inc. * All rights reserved. * * This code is derived from software contributed to The NetBSD Foundation * by Charles M. Hannum. * * Redistribution and use in source and binary forms, with or without * modification, are permitted provided that the following conditions * are met: * 1. Redistributions of source code must retain the above copyright * notice, this list of conditions and the following disclaimer. * 2. Redistributions in binary form must reproduce the above copyright * notice, this list of conditions and the following disclaimer in the * documentation and/or other materials provided with the distribution. * * THIS SOFTWARE IS PROVIDED BY THE NETBSD FOUNDATION, INC. AND CONTRIBUTORS * ``AS IS'' AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED * TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR * PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE FOUNDATION OR CONTRIBUTORS * BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR * CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF * SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS * INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN * CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) * ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE * POSSIBILITY OF SUCH DAMAGE. */ #ifndef _DEV_SCSIPI_SCSIPI_BASE_H_ #define _DEV_SCSIPI_SCSIPI_BASE_H_ struct scsipi_xfer *scsipi_get_xs(struct scsipi_periph *, int); void scsipi_put_xs(struct scsipi_xfer *); static __inline struct scsipi_xfer *scsipi_make_xs_internal(struct scsipi_periph *, struct scsipi_generic *, int cmdlen, u_char *data_addr, int datalen, int retries, int timeout, struct buf *, int flags) __unused; static __inline struct scsipi_xfer *scsipi_make_xs_unlocked(struct scsipi_periph *, struct scsipi_generic *, int cmdlen, u_char *data_addr, int datalen, int retries, int timeout, struct buf *, int flags) __unused; static __inline struct scsipi_xfer *scsipi_make_xs_locked(struct scsipi_periph *, struct scsipi_generic *, int cmdlen, u_char *data_addr, int datalen, int retries, int timeout, struct buf *, int flags) __unused; /* * Make a scsipi_xfer, and return a pointer to it. */ static __inline struct scsipi_xfer * scsipi_make_xs_internal(struct scsipi_periph *periph, struct scsipi_generic *cmd, int cmdlen, u_char *data_addr, int datalen, int retries, int timeout, struct buf *bp, int flags) { struct scsipi_xfer *xs; if ((xs = scsipi_get_xs(periph, flags)) == NULL) return (NULL); /* * Fill out the scsipi_xfer structure. We don't know whose context * the cmd is in, so copy it. */ memcpy(&xs->cmdstore, cmd, cmdlen); xs->cmd = &xs->cmdstore; xs->cmdlen = cmdlen; xs->data = data_addr; xs->datalen = datalen; xs->xs_retries = retries; xs->timeout = timeout; xs->bp = bp; return (xs); } static __inline struct scsipi_xfer * scsipi_make_xs_unlocked(struct scsipi_periph *periph, struct scsipi_generic *cmd, int cmdlen, u_char *data_addr, int datalen, int retries, int timeout, struct buf *bp, int flags) { return scsipi_make_xs_internal(periph, cmd, cmdlen, data_addr, datalen, retries, timeout, bp, flags & ~XS_CTL_NOSLEEP); } static __inline struct scsipi_xfer * scsipi_make_xs_locked(struct scsipi_periph *periph, struct scsipi_generic *cmd, int cmdlen, u_char *data_addr, int datalen, int retries, int timeout, struct buf *bp, int flags) { KDASSERT(mutex_owned(chan_mtx(periph->periph_channel))); return scsipi_make_xs_internal(periph, cmd, cmdlen, data_addr, datalen, retries, timeout, bp, flags | XS_CTL_NOSLEEP); } #endif /* _DEV_SCSIPI_SCSIPI_BASE_H_ */ |
| 8 8 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 197 198 199 200 201 202 203 204 205 206 207 208 209 210 211 212 213 214 215 216 217 218 219 220 221 222 223 224 225 226 227 228 229 230 231 232 233 234 235 236 237 238 239 240 241 242 243 244 245 246 247 248 249 250 251 252 253 254 255 256 257 258 259 260 261 262 263 264 265 266 267 268 269 270 271 272 273 274 275 276 277 278 279 280 281 282 283 284 285 286 287 288 289 290 291 292 293 294 295 296 297 298 299 300 301 302 303 304 305 306 307 308 309 310 311 312 313 314 315 316 317 318 319 320 321 322 323 324 325 326 327 328 329 330 331 332 333 334 335 336 337 338 339 340 341 342 343 344 345 346 347 348 349 350 351 352 353 354 355 356 357 358 359 360 361 362 363 364 365 366 367 368 369 370 371 372 373 374 375 376 377 378 379 380 381 382 383 384 385 386 387 388 389 390 391 392 393 394 395 396 397 398 399 400 401 402 403 404 405 406 407 408 409 410 411 412 413 414 415 416 417 418 419 420 421 422 423 424 425 426 427 428 429 430 431 432 433 434 435 436 437 438 439 440 441 442 443 444 445 446 447 448 449 450 451 452 453 454 455 456 457 458 459 460 461 462 463 464 465 466 467 468 469 470 471 472 473 474 475 476 477 478 479 480 481 482 483 484 485 486 487 488 489 490 491 492 493 494 495 496 497 498 499 500 501 502 503 504 505 506 507 508 509 510 511 512 513 514 515 516 517 518 519 520 521 522 523 524 525 526 527 528 529 530 531 532 533 534 535 536 537 538 539 540 541 542 543 544 545 546 547 548 549 550 551 552 553 554 555 556 557 558 559 560 561 562 563 564 565 566 567 568 569 570 571 572 573 574 575 576 577 578 579 580 581 582 583 584 585 586 587 588 589 590 591 592 593 594 595 596 597 598 599 600 601 602 603 604 605 606 607 608 609 610 611 612 613 614 615 616 617 618 619 620 621 622 623 624 625 626 627 628 629 630 631 632 633 634 635 636 637 638 | /* $NetBSD: irmce.c,v 1.8 2021/08/07 16:19:16 thorpej Exp $ */ /*- * Copyright (c) 2011 Jared D. McNeill <jmcneill@invisible.ca> * All rights reserved. * * Redistribution and use in source and binary forms, with or without * modification, are permitted provided that the following conditions * are met: * 1. Redistributions of source code must retain the above copyright * notice, this list of conditions and the following disclaimer. * 2. Redistributions in binary form must reproduce the above copyright * notice, this list of conditions and the following disclaimer in the * documentation and/or other materials provided with the distribution. * * THIS SOFTWARE IS PROVIDED BY THE AUTHOR AND CONTRIBUTORS ``AS IS'' * AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED * TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR * PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE FOUNDATION OR CONTRIBUTORS * BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR * CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF * SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS * INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN * CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) * ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE * POSSIBILITY OF SUCH DAMAGE. */ /* * IR receiver/transceiver for Windows Media Center */ #include <sys/cdefs.h> __KERNEL_RCSID(0, "$NetBSD: irmce.c,v 1.8 2021/08/07 16:19:16 thorpej Exp $"); #include <sys/types.h> #include <sys/param.h> #include <sys/systm.h> #include <sys/device.h> #include <sys/conf.h> #include <sys/bus.h> #include <sys/select.h> #include <sys/module.h> #include <dev/usb/usb.h> #include <dev/usb/usbdi.h> #include <dev/usb/usbdi_util.h> #include <dev/usb/usbdevs.h> #include <dev/ir/ir.h> #include <dev/ir/cirio.h> #include <dev/ir/cirvar.h> enum irmce_state { IRMCE_STATE_HEADER, IRMCE_STATE_IRDATA, IRMCE_STATE_CMDHEADER, IRMCE_STATE_CMDDATA, }; struct irmce_softc { device_t sc_dev; device_t sc_cirdev; struct usbd_device * sc_udev; struct usbd_interface * sc_iface; int sc_bulkin_ep; uint16_t sc_bulkin_maxpktsize; struct usbd_pipe * sc_bulkin_pipe; struct usbd_xfer * sc_bulkin_xfer; uint8_t * sc_bulkin_buffer; int sc_bulkout_ep; uint16_t sc_bulkout_maxpktsize; struct usbd_pipe * sc_bulkout_pipe; struct usbd_xfer * sc_bulkout_xfer; uint8_t * sc_bulkout_buffer; bool sc_raw; uint8_t sc_ir_buf[16]; size_t sc_ir_bufused; size_t sc_ir_resid; enum irmce_state sc_ir_state; uint8_t sc_ir_header; bool sc_rc6_hb[256]; size_t sc_rc6_nhb; }; static int irmce_match(device_t, cfdata_t, void *); static void irmce_attach(device_t, device_t, void *); static int irmce_detach(device_t, int); static void irmce_childdet(device_t, device_t); static int irmce_activate(device_t, enum devact); static int irmce_rescan(device_t, const char *, const int *); static int irmce_print(void *, const char *); static int irmce_reset(struct irmce_softc *); static int irmce_open(void *, int, int, struct proc *); static int irmce_close(void *, int, int, struct proc *); static int irmce_read(void *, struct uio *, int); static int irmce_write(void *, struct uio *, int); static int irmce_setparams(void *, struct cir_params *); static const struct cir_methods irmce_cir_methods = { .im_open = irmce_open, .im_close = irmce_close, .im_read = irmce_read, .im_write = irmce_write, .im_setparams = irmce_setparams, }; static const struct { uint16_t vendor; uint16_t product; } irmce_devices[] = { { USB_VENDOR_SMK, USB_PRODUCT_SMK_MCE_IR }, }; CFATTACH_DECL2_NEW(irmce, sizeof(struct irmce_softc), irmce_match, irmce_attach, irmce_detach, irmce_activate, irmce_rescan, irmce_childdet); static int irmce_match(device_t parent, cfdata_t match, void *opaque) { struct usbif_attach_arg *uiaa = opaque; unsigned int i; for (i = 0; i < __arraycount(irmce_devices); i++) { if (irmce_devices[i].vendor == uiaa->uiaa_vendor && irmce_devices[i].product == uiaa->uiaa_product) return UMATCH_VENDOR_PRODUCT; } return UMATCH_NONE; } static void irmce_attach(device_t parent, device_t self, void *opaque) { struct irmce_softc *sc = device_private(self); struct usbif_attach_arg *uiaa = opaque; usb_endpoint_descriptor_t *ed; char *devinfop; unsigned int i; uint8_t nep; if (!pmf_device_register(self, NULL, NULL)) aprint_error_dev(self, "couldn't establish power handler\n"); aprint_naive("\n"); devinfop = usbd_devinfo_alloc(uiaa->uiaa_device, 0); aprint_normal(": %s\n", devinfop); usbd_devinfo_free(devinfop); sc->sc_dev = self; sc->sc_udev = uiaa->uiaa_device; sc->sc_iface = uiaa->uiaa_iface; nep = 0; usbd_endpoint_count(sc->sc_iface, &nep); sc->sc_bulkin_ep = sc->sc_bulkout_ep = -1; for (i = 0; i < nep; i++) { int dir, type; ed = usbd_interface2endpoint_descriptor(sc->sc_iface, i); if (ed == NULL) { aprint_error_dev(self, "couldn't read endpoint descriptor %d\n", i); continue; } dir = UE_GET_DIR(ed->bEndpointAddress); type = UE_GET_XFERTYPE(ed->bmAttributes); if (type != UE_BULK) continue; if (dir == UE_DIR_IN && sc->sc_bulkin_ep == -1) { sc->sc_bulkin_ep = ed->bEndpointAddress; sc->sc_bulkin_maxpktsize = UE_GET_SIZE(UGETW(ed->wMaxPacketSize)) * (UE_GET_TRANS(UGETW(ed->wMaxPacketSize)) + 1); } if (dir == UE_DIR_OUT && sc->sc_bulkout_ep == -1) { sc->sc_bulkout_ep = ed->bEndpointAddress; sc->sc_bulkout_maxpktsize = UE_GET_SIZE(UGETW(ed->wMaxPacketSize)) * (UE_GET_TRANS(UGETW(ed->wMaxPacketSize)) + 1); } } aprint_debug_dev(self, "in 0x%02x/%d out 0x%02x/%d\n", sc->sc_bulkin_ep, sc->sc_bulkin_maxpktsize, sc->sc_bulkout_ep, sc->sc_bulkout_maxpktsize); if (sc->sc_bulkin_maxpktsize < 16 || sc->sc_bulkout_maxpktsize < 16) { aprint_error_dev(self, "bad maxpktsize\n"); return; } usbd_status err; err = usbd_open_pipe(sc->sc_iface, sc->sc_bulkin_ep, USBD_EXCLUSIVE_USE, &sc->sc_bulkin_pipe); if (err) { aprint_error_dev(sc->sc_dev, "couldn't open bulk-in pipe: %s\n", usbd_errstr(err)); return; } err = usbd_open_pipe(sc->sc_iface, sc->sc_bulkout_ep, USBD_EXCLUSIVE_USE, &sc->sc_bulkout_pipe); if (err) { aprint_error_dev(sc->sc_dev, "couldn't open bulk-out pipe: %s\n", usbd_errstr(err)); usbd_close_pipe(sc->sc_bulkin_pipe); sc->sc_bulkin_pipe = NULL; return; } int error; error = usbd_create_xfer(sc->sc_bulkin_pipe, sc->sc_bulkin_maxpktsize, 0, 0, &sc->sc_bulkin_xfer); if (error) { goto fail; } error = usbd_create_xfer(sc->sc_bulkout_pipe, sc->sc_bulkout_maxpktsize, USBD_FORCE_SHORT_XFER, 0, &sc->sc_bulkout_xfer); if (error) { goto fail; } sc->sc_bulkin_buffer = usbd_get_buffer(sc->sc_bulkin_xfer); sc->sc_bulkout_buffer = usbd_get_buffer(sc->sc_bulkout_xfer); irmce_rescan(self, NULL, NULL); return; fail: if (sc->sc_bulkin_xfer) usbd_destroy_xfer(sc->sc_bulkin_xfer); if (sc->sc_bulkout_xfer) usbd_destroy_xfer(sc->sc_bulkout_xfer); } static int irmce_detach(device_t self, int flags) { struct irmce_softc *sc = device_private(self); int error; if (sc->sc_cirdev) { error = config_detach(sc->sc_cirdev, flags); if (error) return error; } if (sc->sc_bulkin_pipe) { usbd_abort_pipe(sc->sc_bulkin_pipe); } if (sc->sc_bulkout_pipe) { usbd_abort_pipe(sc->sc_bulkout_pipe); } if (sc->sc_bulkin_xfer) { usbd_destroy_xfer(sc->sc_bulkin_xfer); sc->sc_bulkin_buffer = NULL; sc->sc_bulkin_xfer = NULL; } if (sc->sc_bulkout_xfer) { usbd_destroy_xfer(sc->sc_bulkout_xfer); sc->sc_bulkout_buffer = NULL; sc->sc_bulkout_xfer = NULL; } if (sc->sc_bulkin_pipe) { usbd_close_pipe(sc->sc_bulkin_pipe); sc->sc_bulkin_pipe = NULL; } if (sc->sc_bulkout_pipe) { usbd_close_pipe(sc->sc_bulkout_pipe); sc->sc_bulkout_pipe = NULL; } pmf_device_deregister(self); return 0; } static int irmce_activate(device_t self, enum devact act) { return 0; } static int irmce_rescan(device_t self, const char *ifattr, const int *locators) { struct irmce_softc *sc = device_private(self); struct ir_attach_args iaa; if (sc->sc_cirdev == NULL) { iaa.ia_type = IR_TYPE_CIR; iaa.ia_methods = &irmce_cir_methods; iaa.ia_handle = sc; sc->sc_cirdev = config_found(self, &iaa, irmce_print, CFARGS_NONE); } return 0; } static int irmce_print(void *priv, const char *pnp) { if (pnp) aprint_normal("cir at %s", pnp); return UNCONF; } static void irmce_childdet(device_t self, device_t child) { struct irmce_softc *sc = device_private(self); if (sc->sc_cirdev == child) sc->sc_cirdev = NULL; } static int irmce_reset(struct irmce_softc *sc) { static const uint8_t reset_cmd[] = { 0x00, 0xff, 0xaa }; uint8_t *p = sc->sc_bulkout_buffer; usbd_status err; uint32_t wlen; unsigned int n; for (n = 0; n < __arraycount(reset_cmd); n++) *p++ = reset_cmd[n]; wlen = sizeof(reset_cmd); err = usbd_bulk_transfer(sc->sc_bulkout_xfer, sc->sc_bulkout_pipe, USBD_FORCE_SHORT_XFER, USBD_DEFAULT_TIMEOUT, sc->sc_bulkout_buffer, &wlen); if (err != USBD_NORMAL_COMPLETION) { if (err == USBD_INTERRUPTED) return EINTR; else if (err == USBD_TIMEOUT) return ETIMEDOUT; else return EIO; } return 0; } static int irmce_open(void *priv, int flag, int mode, struct proc *p) { struct irmce_softc *sc = priv; int err = irmce_reset(sc); if (err) { aprint_error_dev(sc->sc_dev, "couldn't reset device: %s\n", usbd_errstr(err)); } sc->sc_ir_state = IRMCE_STATE_HEADER; sc->sc_rc6_nhb = 0; return 0; } static int irmce_close(void *priv, int flag, int mode, struct proc *p) { struct irmce_softc *sc = priv; if (sc->sc_bulkin_pipe) { usbd_abort_pipe(sc->sc_bulkin_pipe); } if (sc->sc_bulkout_pipe) { usbd_abort_pipe(sc->sc_bulkout_pipe); } return 0; } static int irmce_rc6_decode(struct irmce_softc *sc, uint8_t *buf, size_t buflen, struct uio *uio) { bool *hb = &sc->sc_rc6_hb[0]; unsigned int n; int state, pulse; uint32_t data; uint8_t mode; bool idle = false; for (n = 0; n < buflen; n++) { state = (buf[n] & 0x80) ? 1 : 0; pulse = (buf[n] & 0x7f) * 50; if (pulse >= 300 && pulse <= 600) { hb[sc->sc_rc6_nhb++] = state; } else if (pulse >= 680 && pulse <= 1080) { hb[sc->sc_rc6_nhb++] = state; hb[sc->sc_rc6_nhb++] = state; } else if (pulse >= 1150 && pulse <= 1450) { hb[sc->sc_rc6_nhb++] = state; hb[sc->sc_rc6_nhb++] = state; hb[sc->sc_rc6_nhb++] = state; } else if (pulse >= 2400 && pulse <= 2800) { hb[sc->sc_rc6_nhb++] = state; hb[sc->sc_rc6_nhb++] = state; hb[sc->sc_rc6_nhb++] = state; hb[sc->sc_rc6_nhb++] = state; hb[sc->sc_rc6_nhb++] = state; hb[sc->sc_rc6_nhb++] = state; } else if (pulse > 3000) { if (sc->sc_rc6_nhb & 1) hb[sc->sc_rc6_nhb++] = state; idle = true; break; } else { aprint_debug_dev(sc->sc_dev, "error parsing RC6 stream (pulse=%d)\n", pulse); return EIO; } } if (!idle) return 0; if (sc->sc_rc6_nhb < 20) { aprint_debug_dev(sc->sc_dev, "not enough RC6 data\n"); return EIO; } /* RC6 leader 11111100 */ if (!hb[0] || !hb[1] || !hb[2] || !hb[3] || !hb[4] || !hb[5] || hb[6] || hb[7]) { aprint_debug_dev(sc->sc_dev, "bad RC6 leader\n"); return EIO; } /* start bit 10 */ if (!hb[8] || hb[9]) { aprint_debug_dev(sc->sc_dev, "missing RC6 start bit\n"); return EIO; } /* mode info */ mode = 0x00; for (n = 10; n < 15; n += 2) { if (hb[n] && !hb[n + 1]) mode = (mode << 1) | 1; else if (!hb[n] && hb[n + 1]) mode = (mode << 1) | 0; else { aprint_debug_dev(sc->sc_dev, "bad RC6 mode bits\n"); return EIO; } } data = 0; for (n = 20; n < sc->sc_rc6_nhb; n += 2) { if (hb[n] && !hb[n + 1]) data = (data << 1) | 1; else if (!hb[n] && hb[n + 1]) data = (data << 1) | 0; else { aprint_debug_dev(sc->sc_dev, "bad RC6 data bits\n"); return EIO; } } sc->sc_rc6_nhb = 0; return uiomove(&data, sizeof(data), uio); } static int irmce_process(struct irmce_softc *sc, uint8_t *buf, size_t buflen, struct uio *uio) { uint8_t *p = buf; uint8_t data, cmd; int error; while (p - buf < (ssize_t)buflen) { switch (sc->sc_ir_state) { case IRMCE_STATE_HEADER: sc->sc_ir_header = data = *p++; if ((data & 0xe0) == 0x80 && (data & 0x1f) != 0x1f) { sc->sc_ir_bufused = 0; sc->sc_ir_resid = data & 0x1f; sc->sc_ir_state = IRMCE_STATE_IRDATA; if (sc->sc_ir_resid > sizeof(sc->sc_ir_buf)) return EIO; if (sc->sc_ir_resid == 0) sc->sc_ir_state = IRMCE_STATE_HEADER; } else { sc->sc_ir_state = IRMCE_STATE_CMDHEADER; } break; case IRMCE_STATE_CMDHEADER: cmd = *p++; data = sc->sc_ir_header; if (data == 0x00 && cmd == 0x9f) sc->sc_ir_resid = 1; else if (data == 0xff && cmd == 0x0b) sc->sc_ir_resid = 2; else if (data == 0x9f) { if (cmd == 0x04 || cmd == 0x06 || cmd == 0x0c || cmd == 0x15) { sc->sc_ir_resid = 2; } else if (cmd == 0x01 || cmd == 0x08 || cmd == 0x14) { sc->sc_ir_resid = 1; } } if (sc->sc_ir_resid > 0) sc->sc_ir_state = IRMCE_STATE_CMDDATA; else sc->sc_ir_state = IRMCE_STATE_HEADER; break; case IRMCE_STATE_IRDATA: sc->sc_ir_resid--; sc->sc_ir_buf[sc->sc_ir_bufused++] = *p; p++; if (sc->sc_ir_resid == 0) { sc->sc_ir_state = IRMCE_STATE_HEADER; error = irmce_rc6_decode(sc, sc->sc_ir_buf, sc->sc_ir_bufused, uio); if (error) sc->sc_rc6_nhb = 0; } break; case IRMCE_STATE_CMDDATA: p++; sc->sc_ir_resid--; if (sc->sc_ir_resid == 0) sc->sc_ir_state = IRMCE_STATE_HEADER; break; } } return 0; } static int irmce_read(void *priv, struct uio *uio, int flag) { struct irmce_softc *sc = priv; usbd_status err; uint32_t rlen; int error = 0; while (uio->uio_resid > 0) { rlen = sc->sc_bulkin_maxpktsize; err = usbd_bulk_transfer(sc->sc_bulkin_xfer, sc->sc_bulkin_pipe, USBD_SHORT_XFER_OK, USBD_DEFAULT_TIMEOUT, sc->sc_bulkin_buffer, &rlen); if (err != USBD_NORMAL_COMPLETION) { if (err == USBD_INTERRUPTED) return EINTR; else if (err == USBD_TIMEOUT) continue; else return EIO; } if (sc->sc_raw) { error = uiomove(sc->sc_bulkin_buffer, rlen, uio); break; } else { error = irmce_process(sc, sc->sc_bulkin_buffer, rlen, uio); if (error) break; } } return error; } static int irmce_write(void *priv, struct uio *uio, int flag) { return EIO; } static int irmce_setparams(void *priv, struct cir_params *params) { struct irmce_softc *sc = priv; if (params->raw > 1) return EINVAL; sc->sc_raw = params->raw; return 0; } MODULE(MODULE_CLASS_DRIVER, irmce, NULL); #ifdef _MODULE #include "ioconf.c" #endif static int irmce_modcmd(modcmd_t cmd, void *opaque) { switch (cmd) { case MODULE_CMD_INIT: #ifdef _MODULE return config_init_component(cfdriver_ioconf_irmce, cfattach_ioconf_irmce, cfdata_ioconf_irmce); #else return 0; #endif case MODULE_CMD_FINI: #ifdef _MODULE return config_fini_component(cfdriver_ioconf_irmce, cfattach_ioconf_irmce, cfdata_ioconf_irmce); #else return 0; #endif default: return ENOTTY; } } |
| 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 | /* $NetBSD: msgbuf.h,v 1.17 2018/04/19 21:19:07 christos Exp $ */ /* * Copyright (c) 1981, 1984, 1993 * The Regents of the University of California. All rights reserved. * * Redistribution and use in source and binary forms, with or without * modification, are permitted provided that the following conditions * are met: * 1. Redistributions of source code must retain the above copyright * notice, this list of conditions and the following disclaimer. * 2. Redistributions in binary form must reproduce the above copyright * notice, this list of conditions and the following disclaimer in the * documentation and/or other materials provided with the distribution. * 3. Neither the name of the University nor the names of its contributors * may be used to endorse or promote products derived from this software * without specific prior written permission. * * THIS SOFTWARE IS PROVIDED BY THE REGENTS AND CONTRIBUTORS ``AS IS'' AND * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE * ARE DISCLAIMED. IN NO EVENT SHALL THE REGENTS OR CONTRIBUTORS BE LIABLE * FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL * DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS * OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) * HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT * LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY * OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF * SUCH DAMAGE. * * @(#)msgbuf.h 8.1 (Berkeley) 6/2/93 */ #ifndef _SYS_MSGBUF_H_ #define _SYS_MSGBUF_H_ struct kern_msgbuf { #define MSG_MAGIC 0x063061 long msg_magic; long msg_bufx; /* write pointer */ long msg_bufr; /* read pointer */ long msg_bufs; /* real msg_bufc size (bytes) */ char msg_bufc[1]; /* buffer */ }; #ifdef _KERNEL extern int msgbufmapped; /* is the message buffer mapped */ extern int msgbufenabled; /* is logging to the buffer enabled */ extern struct kern_msgbuf *msgbufp; /* the mapped buffer, itself. */ void initmsgbuf(void *, size_t); void loginit(void); void logputchar(int); static __inline int logenabled(const struct kern_msgbuf *mbp) { return msgbufenabled && mbp->msg_magic == MSG_MAGIC; } #endif #endif /* !_SYS_MSGBUF_H_ */ |
| 12 12 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 197 198 199 200 201 202 203 204 205 206 207 208 209 210 211 212 213 214 215 216 217 218 219 220 221 222 223 224 225 226 227 228 229 230 231 232 233 234 235 236 237 238 239 240 241 242 243 244 245 246 247 248 249 250 251 252 253 254 255 256 257 258 259 260 261 262 263 264 265 266 267 268 269 270 271 272 273 274 275 276 277 278 279 280 281 282 283 284 285 286 287 288 289 290 291 292 293 294 295 296 297 298 299 300 301 302 303 304 305 306 307 308 309 310 311 312 313 314 315 316 317 318 319 320 321 322 323 324 325 326 327 328 329 330 331 332 333 334 335 336 337 338 339 340 341 342 343 344 345 346 347 348 349 350 351 352 353 354 355 356 357 358 359 360 361 362 363 364 365 366 367 368 369 370 371 372 373 374 375 376 377 378 379 380 381 382 383 384 385 386 387 388 389 390 391 392 393 394 395 396 397 398 399 400 401 402 403 404 405 406 407 408 409 410 411 412 413 414 415 416 417 418 419 420 421 422 423 424 425 426 427 428 429 430 431 432 433 434 435 436 437 | /* $NetBSD: if_ether.h,v 1.89 2022/06/20 08:14:48 yamaguchi Exp $ */ /* * Copyright (c) 1982, 1986, 1993 * The Regents of the University of California. All rights reserved. * * Redistribution and use in source and binary forms, with or without * modification, are permitted provided that the following conditions * are met: * 1. Redistributions of source code must retain the above copyright * notice, this list of conditions and the following disclaimer. * 2. Redistributions in binary form must reproduce the above copyright * notice, this list of conditions and the following disclaimer in the * documentation and/or other materials provided with the distribution. * 3. Neither the name of the University nor the names of its contributors * may be used to endorse or promote products derived from this software * without specific prior written permission. * * THIS SOFTWARE IS PROVIDED BY THE REGENTS AND CONTRIBUTORS ``AS IS'' AND * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE * ARE DISCLAIMED. IN NO EVENT SHALL THE REGENTS OR CONTRIBUTORS BE LIABLE * FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL * DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS * OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) * HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT * LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY * OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF * SUCH DAMAGE. * * @(#)if_ether.h 8.1 (Berkeley) 6/10/93 */ #ifndef _NET_IF_ETHER_H_ #define _NET_IF_ETHER_H_ #ifdef _KERNEL #ifdef _KERNEL_OPT #include "opt_mbuftrace.h" #endif #include <sys/mbuf.h> #endif #ifndef _STANDALONE #include <net/if.h> #endif /* * Some basic Ethernet constants. */ #define ETHER_ADDR_LEN 6 /* length of an Ethernet address */ #define ETHER_TYPE_LEN 2 /* length of the Ethernet type field */ #define ETHER_CRC_LEN 4 /* length of the Ethernet CRC */ #define ETHER_HDR_LEN ((ETHER_ADDR_LEN * 2) + ETHER_TYPE_LEN) #define ETHER_MIN_LEN 64 /* minimum frame length, including CRC */ #define ETHER_MAX_LEN 1518 /* maximum frame length, including CRC */ #define ETHER_MAX_LEN_JUMBO 9018 /* maximum jumbo frame len, including CRC */ /* * Some Ethernet extensions. */ #define ETHER_VLAN_ENCAP_LEN 4 /* length of 802.1Q VLAN encapsulation */ #define EVL_VLANOFTAG(tag) ((tag) & 4095) /* VLAN ID */ #define EVL_PRIOFTAG(tag) (((tag) >> 13) & 7) /* Priority */ #define EVL_CFIOFTAG(tag) (((tag) >> 12) & 1) /* CFI */ #define ETHER_PPPOE_ENCAP_LEN 8 /* length of PPPoE encapsulation */ /* * Mbuf adjust factor to force 32-bit alignment of IP header. * Drivers should do m_adj(m, ETHER_ALIGN) when setting up a * receive so the upper layers get the IP header properly aligned * past the 14-byte Ethernet header. */ #define ETHER_ALIGN 2 /* driver adjust for IP hdr alignment */ /* * Ethernet address - 6 octets * this is only used by the ethers(3) functions. */ struct ether_addr { uint8_t ether_addr_octet[ETHER_ADDR_LEN]; }; /* * Structure of a 10Mb/s Ethernet header. */ struct ether_header { uint8_t ether_dhost[ETHER_ADDR_LEN]; uint8_t ether_shost[ETHER_ADDR_LEN]; uint16_t ether_type; }; #include <net/ethertypes.h> #define ETHER_IS_MULTICAST(addr) (*(addr) & 0x01) /* is address mcast/bcast? */ #define ETHER_IS_LOCAL(addr) (*(addr) & 0x02) /* is address local? */ #define ETHERMTU_JUMBO (ETHER_MAX_LEN_JUMBO - ETHER_HDR_LEN - ETHER_CRC_LEN) #define ETHERMTU (ETHER_MAX_LEN - ETHER_HDR_LEN - ETHER_CRC_LEN) #define ETHERMIN (ETHER_MIN_LEN - ETHER_HDR_LEN - ETHER_CRC_LEN) /* * Compute the maximum frame size based on ethertype (i.e. possible * encapsulation) and whether or not an FCS is present. */ #define ETHER_MAX_FRAME(ifp, etype, hasfcs) \ ((ifp)->if_mtu + ETHER_HDR_LEN + \ ((hasfcs) ? ETHER_CRC_LEN : 0) + \ (((etype) == ETHERTYPE_VLAN) ? ETHER_VLAN_ENCAP_LEN : 0) + \ (((etype) == ETHERTYPE_PPPOE) ? ETHER_PPPOE_ENCAP_LEN : 0)) /* * Ethernet CRC32 polynomials (big- and little-endian verions). */ #define ETHER_CRC_POLY_LE 0xedb88320 #define ETHER_CRC_POLY_BE 0x04c11db6 #ifndef _STANDALONE /* * Ethernet-specific mbuf flags. */ #define M_HASFCS M_LINK0 /* FCS included at end of frame */ #define M_PROMISC M_LINK1 /* this packet is not for us */ #ifdef _KERNEL /* * Macro to map an IP multicast address to an Ethernet multicast address. * The high-order 25 bits of the Ethernet address are statically assigned, * and the low-order 23 bits are taken from the low end of the IP address. */ #define ETHER_MAP_IP_MULTICAST(ipaddr, enaddr) \ /* const struct in_addr *ipaddr; */ \ /* uint8_t enaddr[ETHER_ADDR_LEN]; */ \ do { \ (enaddr)[0] = 0x01; \ (enaddr)[1] = 0x00; \ (enaddr)[2] = 0x5e; \ (enaddr)[3] = ((const uint8_t *)ipaddr)[1] & 0x7f; \ (enaddr)[4] = ((const uint8_t *)ipaddr)[2]; \ (enaddr)[5] = ((const uint8_t *)ipaddr)[3]; \ } while (/*CONSTCOND*/0) /* * Macro to map an IP6 multicast address to an Ethernet multicast address. * The high-order 16 bits of the Ethernet address are statically assigned, * and the low-order 32 bits are taken from the low end of the IP6 address. */ #define ETHER_MAP_IPV6_MULTICAST(ip6addr, enaddr) \ /* struct in6_addr *ip6addr; */ \ /* uint8_t enaddr[ETHER_ADDR_LEN]; */ \ { \ (enaddr)[0] = 0x33; \ (enaddr)[1] = 0x33; \ (enaddr)[2] = ((const uint8_t *)ip6addr)[12]; \ (enaddr)[3] = ((const uint8_t *)ip6addr)[13]; \ (enaddr)[4] = ((const uint8_t *)ip6addr)[14]; \ (enaddr)[5] = ((const uint8_t *)ip6addr)[15]; \ } #endif struct mii_data; struct ethercom; typedef int (*ether_cb_t)(struct ethercom *); typedef int (*ether_vlancb_t)(struct ethercom *, uint16_t, bool); /* * Structure shared between the ethernet driver modules and * the multicast list code. For example, each ec_softc or il_softc * begins with this structure. */ struct ethercom { struct ifnet ec_if; /* network-visible interface */ LIST_HEAD(, ether_multi) ec_multiaddrs; /* list of ether multicast addrs */ int ec_multicnt; /* length of ec_multiaddrs list */ int ec_capabilities; /* capabilities, provided by driver */ int ec_capenable; /* tells hardware which capabilities to enable */ int ec_nvlans; /* # VLANs on this interface */ SIMPLEQ_HEAD(, vlanid_list) ec_vids; /* list of VLAN IDs */ /* The device handle for the MII bus child device. */ struct mii_data *ec_mii; struct ifmedia *ec_ifmedia; /* * Called after a change to ec_if.if_flags. Returns * ENETRESET if the device should be reinitialized with * ec_if.if_init, 0 on success, not 0 on failure. */ ether_cb_t ec_ifflags_cb; /* * Called whenever a vlan interface is configured or unconfigured. * Args include the vlan tag and a flag indicating whether the tag is * being added or removed. */ ether_vlancb_t ec_vlan_cb; /* Hooks called at the beginning of detach of this interface */ khook_list_t *ec_ifdetach_hooks; kmutex_t *ec_lock; /* Flags used only by the kernel */ int ec_flags; #ifdef MBUFTRACE struct mowner ec_rx_mowner; /* mbufs received */ struct mowner ec_tx_mowner; /* mbufs transmitted */ #endif }; #define ETHERCAP_VLAN_MTU 0x00000001 /* VLAN-compatible MTU */ #define ETHERCAP_VLAN_HWTAGGING 0x00000002 /* hardware VLAN tag support */ #define ETHERCAP_JUMBO_MTU 0x00000004 /* 9000 byte MTU supported */ #define ETHERCAP_VLAN_HWFILTER 0x00000008 /* iface hw can filter vlan tag */ #define ETHERCAP_EEE 0x00000010 /* Energy Efficiency Ethernet */ #define ETHERCAP_MASK 0x0000001f #define ECCAPBITS \ "\020" \ "\1VLAN_MTU" \ "\2VLAN_HWTAGGING" \ "\3JUMBO_MTU" \ "\4VLAN_HWFILTER" \ "\5EEE" /* ioctl() for Ethernet capabilities */ struct eccapreq { char eccr_name[IFNAMSIZ]; /* if name, e.g. "en0" */ int eccr_capabilities; /* supported capabiliites */ int eccr_capenable; /* capabilities enabled */ }; /* sysctl for Ethernet multicast addresses */ struct ether_multi_sysctl { u_int enm_refcount; uint8_t enm_addrlo[ETHER_ADDR_LEN]; uint8_t enm_addrhi[ETHER_ADDR_LEN]; }; #ifdef _KERNEL /* * Flags for ec_flags */ /* Store IFF_ALLMULTI in ec_flags instead of if_flags to avoid data races. */ #define ETHER_F_ALLMULTI __BIT(0) extern const uint8_t etherbroadcastaddr[ETHER_ADDR_LEN]; extern const uint8_t ethermulticastaddr_slowprotocols[ETHER_ADDR_LEN]; extern const uint8_t ether_ipmulticast_min[ETHER_ADDR_LEN]; extern const uint8_t ether_ipmulticast_max[ETHER_ADDR_LEN]; void ether_set_ifflags_cb(struct ethercom *, ether_cb_t); void ether_set_vlan_cb(struct ethercom *, ether_vlancb_t); int ether_ioctl(struct ifnet *, u_long, void *); int ether_addmulti(const struct sockaddr *, struct ethercom *); int ether_delmulti(const struct sockaddr *, struct ethercom *); int ether_multiaddr(const struct sockaddr *, uint8_t[], uint8_t[]); void ether_input(struct ifnet *, struct mbuf *); /* * Ethernet multicast address structure. There is one of these for each * multicast address or range of multicast addresses that we are supposed * to listen to on a particular interface. They are kept in a linked list, * rooted in the interface's ethercom structure. */ struct ether_multi { uint8_t enm_addrlo[ETHER_ADDR_LEN]; /* low or only address of range */ uint8_t enm_addrhi[ETHER_ADDR_LEN]; /* high or only address of range */ u_int enm_refcount; /* no. claims to this addr/range */ LIST_ENTRY(ether_multi) enm_list; }; /* * Structure used by macros below to remember position when stepping through * all of the ether_multi records. */ struct ether_multistep { struct ether_multi *e_enm; }; /* * lookup the ether_multi record for a given range of Ethernet * multicast addresses connected to a given ethercom structure. * If no matching record is found, NULL is returned. */ static __inline struct ether_multi * ether_lookup_multi(const uint8_t *addrlo, const uint8_t *addrhi, const struct ethercom *ec) { struct ether_multi *enm; LIST_FOREACH(enm, &ec->ec_multiaddrs, enm_list) { if (memcmp(enm->enm_addrlo, addrlo, ETHER_ADDR_LEN) != 0) continue; if (memcmp(enm->enm_addrhi, addrhi, ETHER_ADDR_LEN) != 0) continue; break; } return enm; } /* * step through all of the ether_multi records, one at a time. * The current position is remembered in "step", which the caller must * provide. ether_first_multi(), below, must be called to initialize "step" * and get the first record. Both functions return a NULL when there * are no remaining records. */ static __inline struct ether_multi * ether_next_multi(struct ether_multistep *step) { struct ether_multi *enm; enm = step->e_enm; if (enm != NULL) step->e_enm = LIST_NEXT(enm, enm_list); return enm; } #define ETHER_NEXT_MULTI(step, enm) \ /* struct ether_multistep step; */ \ /* struct ether_multi *enm; */ \ (enm) = ether_next_multi(&(step)) static __inline struct ether_multi * ether_first_multi(struct ether_multistep *step, const struct ethercom *ec) { step->e_enm = LIST_FIRST(&ec->ec_multiaddrs); return ether_next_multi(step); } #define ETHER_FIRST_MULTI(step, ec, enm) \ /* struct ether_multistep step; */ \ /* struct ethercom *ec; */ \ /* struct ether_multi *enm; */ \ (enm) = ether_first_multi(&(step), (ec)) #define ETHER_LOCK(ec) mutex_enter((ec)->ec_lock) #define ETHER_UNLOCK(ec) mutex_exit((ec)->ec_lock) /* * Ethernet 802.1Q VLAN structures. */ /* for ethercom */ struct vlanid_list { uint16_t vid; SIMPLEQ_ENTRY(vlanid_list) vid_list; }; /* add VLAN tag to input/received packet */ static __inline void vlan_set_tag(struct mbuf *m, uint16_t vlantag) { /* VLAN tag contains priority, CFI and VLAN ID */ KASSERT((m->m_flags & M_PKTHDR) != 0); m->m_pkthdr.ether_vtag = vlantag; m->m_flags |= M_VLANTAG; return; } /* extract VLAN ID value from a VLAN tag */ static __inline uint16_t vlan_get_tag(struct mbuf *m) { KASSERT((m->m_flags & M_PKTHDR) != 0); KASSERT(m->m_flags & M_VLANTAG); return m->m_pkthdr.ether_vtag; } static __inline bool vlan_has_tag(struct mbuf *m) { return (m->m_flags & M_VLANTAG) != 0; } static __inline bool vlan_is_hwtag_enabled(struct ifnet *_ifp) { struct ethercom *ec = (void *)_ifp; if (ec->ec_capenable & ETHERCAP_VLAN_HWTAGGING) return true; return false; } /* test if any VLAN is configured for this interface */ #define VLAN_ATTACHED(ec) ((ec)->ec_nvlans > 0) void etherinit(void); void ether_ifattach(struct ifnet *, const uint8_t *); void ether_ifdetach(struct ifnet *); int ether_mediachange(struct ifnet *); void ether_mediastatus(struct ifnet *, struct ifmediareq *); void * ether_ifdetachhook_establish(struct ifnet *, void (*)(void *), void *arg); void ether_ifdetachhook_disestablish(struct ifnet *, void *, kmutex_t *); char *ether_sprintf(const uint8_t *); char *ether_snprintf(char *, size_t, const uint8_t *); uint32_t ether_crc32_le(const uint8_t *, size_t); uint32_t ether_crc32_be(const uint8_t *, size_t); int ether_aton_r(u_char *, size_t, const char *); int ether_enable_vlan_mtu(struct ifnet *); int ether_disable_vlan_mtu(struct ifnet *); int ether_add_vlantag(struct ifnet *, uint16_t, bool *); int ether_del_vlantag(struct ifnet *, uint16_t); int ether_inject_vlantag(struct mbuf **, uint16_t, uint16_t); struct mbuf * ether_strip_vlantag(struct mbuf *); #else /* * Prototype ethers(3) functions. */ #include <sys/cdefs.h> __BEGIN_DECLS char * ether_ntoa(const struct ether_addr *); struct ether_addr * ether_aton(const char *); int ether_ntohost(char *, const struct ether_addr *); int ether_hostton(const char *, struct ether_addr *); int ether_line(const char *, struct ether_addr *, char *); __END_DECLS #endif #endif /* _STANDALONE */ #endif /* !_NET_IF_ETHER_H_ */ |
| 2099 2098 2096 130 1776 923 922 921 23 922 1487 1301 474 5907 4895 4898 4894 832 833 832 832 1 833 832 5776 5782 5778 1487 1486 4 1329 1328 1117 1309 1306 33 5 6 5767 5768 5769 5859 4884 4888 4882 5730 5724 5730 5703 3 759 5560 5372 4413 444 445 232 231 638 627 627 628 32 29 7 29 29 29 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 197 198 199 200 201 202 203 204 205 206 207 208 209 210 211 212 213 214 215 216 217 218 219 220 221 222 223 224 225 226 227 228 229 230 231 232 233 234 235 236 237 238 239 240 241 242 243 244 245 246 247 248 249 250 251 252 253 254 255 256 257 258 259 260 261 262 263 264 265 266 267 268 269 270 271 272 273 274 275 276 277 278 279 280 281 282 283 284 285 286 287 288 289 290 291 292 293 294 295 296 297 298 299 300 301 302 303 304 305 306 307 308 309 310 311 312 313 314 315 316 317 318 319 320 321 322 323 324 325 326 327 328 329 330 331 332 333 334 335 336 337 338 339 340 341 342 343 344 345 346 347 348 349 350 351 352 353 354 355 356 357 358 359 360 361 362 363 364 365 366 367 368 369 370 371 372 373 374 375 376 377 378 379 380 381 382 383 384 385 386 387 388 389 390 391 392 393 394 395 396 397 398 399 400 401 402 403 404 405 406 407 408 409 410 411 412 413 414 415 416 417 418 419 420 421 422 423 424 425 426 427 428 429 430 431 432 433 434 435 436 437 438 439 440 441 442 443 444 445 446 447 448 449 450 451 452 453 454 455 456 457 458 459 460 461 462 463 464 465 466 467 468 469 470 471 472 473 474 475 476 477 478 479 480 481 482 483 484 485 486 487 488 489 490 491 492 493 494 495 496 497 498 499 500 501 502 503 504 505 506 507 508 509 510 511 512 513 514 515 516 517 518 519 520 521 522 523 524 525 526 527 528 529 530 531 532 533 534 535 536 537 538 539 540 541 542 543 544 545 546 547 548 549 550 551 552 553 554 555 556 557 558 559 560 561 562 563 564 565 566 567 568 569 570 571 572 573 574 575 576 577 578 579 580 581 582 583 584 585 586 587 588 589 590 591 592 593 594 595 596 597 598 599 600 601 602 603 604 605 606 607 608 609 610 611 612 613 614 615 616 617 618 619 620 621 622 623 624 625 626 627 628 629 630 631 632 633 634 635 636 637 638 639 640 641 642 643 644 645 646 647 648 649 650 651 652 653 654 655 656 657 658 659 660 661 662 663 664 665 666 667 668 669 670 671 672 673 674 675 676 677 678 679 680 681 682 683 684 685 686 687 688 689 690 691 692 693 694 695 696 697 698 699 700 701 702 703 704 705 706 707 708 709 710 711 712 713 714 715 716 717 718 719 720 721 722 723 724 725 726 727 728 729 730 731 732 733 734 735 736 737 738 739 740 741 742 743 744 745 746 747 748 749 750 751 752 753 754 755 756 757 758 759 760 761 762 763 764 765 766 767 768 769 770 771 772 773 774 775 776 777 778 779 780 781 782 783 784 785 786 787 788 789 790 791 792 793 794 795 796 797 798 799 800 801 802 803 804 805 806 807 808 809 810 811 812 813 814 815 816 817 818 819 820 821 822 823 824 825 826 827 828 829 830 831 832 833 834 835 836 837 838 839 840 841 842 843 844 845 846 847 848 849 850 851 852 853 854 855 856 857 858 859 860 861 862 863 864 865 866 867 868 869 870 871 872 873 874 875 876 877 878 879 880 881 882 883 884 885 886 887 888 889 890 891 892 893 894 895 896 897 898 899 900 901 902 903 904 905 906 907 908 909 910 911 912 913 914 915 916 917 918 919 920 921 922 923 924 925 926 927 928 929 930 931 932 933 934 935 936 937 938 939 940 941 942 943 944 945 946 947 948 949 950 951 952 953 954 955 956 957 958 959 960 961 962 963 964 965 966 967 968 969 970 971 972 973 974 975 976 977 978 979 980 981 982 983 984 985 | /* $NetBSD: kern_mutex.c,v 1.99 2022/04/09 23:46:10 riastradh Exp $ */ /*- * Copyright (c) 2002, 2006, 2007, 2008, 2019 The NetBSD Foundation, Inc. * All rights reserved. * * This code is derived from software contributed to The NetBSD Foundation * by Jason R. Thorpe and Andrew Doran. * * Redistribution and use in source and binary forms, with or without * modification, are permitted provided that the following conditions * are met: * 1. Redistributions of source code must retain the above copyright * notice, this list of conditions and the following disclaimer. * 2. Redistributions in binary form must reproduce the above copyright * notice, this list of conditions and the following disclaimer in the * documentation and/or other materials provided with the distribution. * * THIS SOFTWARE IS PROVIDED BY THE NETBSD FOUNDATION, INC. AND CONTRIBUTORS * ``AS IS'' AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED * TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR * PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE FOUNDATION OR CONTRIBUTORS * BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR * CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF * SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS * INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN * CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) * ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE * POSSIBILITY OF SUCH DAMAGE. */ /* * Kernel mutex implementation, modeled after those found in Solaris, * a description of which can be found in: * * Solaris Internals: Core Kernel Architecture, Jim Mauro and * Richard McDougall. */ #define __MUTEX_PRIVATE #include <sys/cdefs.h> __KERNEL_RCSID(0, "$NetBSD: kern_mutex.c,v 1.99 2022/04/09 23:46:10 riastradh Exp $"); #include <sys/param.h> #include <sys/atomic.h> #include <sys/proc.h> #include <sys/mutex.h> #include <sys/sched.h> #include <sys/sleepq.h> #include <sys/systm.h> #include <sys/lockdebug.h> #include <sys/kernel.h> #include <sys/intr.h> #include <sys/lock.h> #include <sys/types.h> #include <sys/cpu.h> #include <sys/pserialize.h> #include <dev/lockstat.h> #include <machine/lock.h> /* * When not running a debug kernel, spin mutexes are not much * more than an splraiseipl() and splx() pair. */ #if defined(DIAGNOSTIC) || defined(MULTIPROCESSOR) || defined(LOCKDEBUG) #define FULL #endif /* * Debugging support. */ #define MUTEX_WANTLOCK(mtx) \ LOCKDEBUG_WANTLOCK(MUTEX_DEBUG_P(mtx), (mtx), \ (uintptr_t)__builtin_return_address(0), 0) #define MUTEX_TESTLOCK(mtx) \ LOCKDEBUG_WANTLOCK(MUTEX_DEBUG_P(mtx), (mtx), \ (uintptr_t)__builtin_return_address(0), -1) #define MUTEX_LOCKED(mtx) \ LOCKDEBUG_LOCKED(MUTEX_DEBUG_P(mtx), (mtx), NULL, \ (uintptr_t)__builtin_return_address(0), 0) #define MUTEX_UNLOCKED(mtx) \ LOCKDEBUG_UNLOCKED(MUTEX_DEBUG_P(mtx), (mtx), \ (uintptr_t)__builtin_return_address(0), 0) #define MUTEX_ABORT(mtx, msg) \ mutex_abort(__func__, __LINE__, mtx, msg) #if defined(LOCKDEBUG) #define MUTEX_DASSERT(mtx, cond) \ do { \ if (__predict_false(!(cond))) \ MUTEX_ABORT(mtx, "assertion failed: " #cond); \ } while (/* CONSTCOND */ 0) #else /* LOCKDEBUG */ #define MUTEX_DASSERT(mtx, cond) /* nothing */ #endif /* LOCKDEBUG */ #if defined(DIAGNOSTIC) #define MUTEX_ASSERT(mtx, cond) \ do { \ if (__predict_false(!(cond))) \ MUTEX_ABORT(mtx, "assertion failed: " #cond); \ } while (/* CONSTCOND */ 0) #else /* DIAGNOSTIC */ #define MUTEX_ASSERT(mtx, cond) /* nothing */ #endif /* DIAGNOSTIC */ /* * Some architectures can't use __cpu_simple_lock as is so allow a way * for them to use an alternate definition. */ #ifndef MUTEX_SPINBIT_LOCK_INIT #define MUTEX_SPINBIT_LOCK_INIT(mtx) __cpu_simple_lock_init(&(mtx)->mtx_lock) #endif #ifndef MUTEX_SPINBIT_LOCKED_P #define MUTEX_SPINBIT_LOCKED_P(mtx) __SIMPLELOCK_LOCKED_P(&(mtx)->mtx_lock) #endif #ifndef MUTEX_SPINBIT_LOCK_TRY #define MUTEX_SPINBIT_LOCK_TRY(mtx) __cpu_simple_lock_try(&(mtx)->mtx_lock) #endif #ifndef MUTEX_SPINBIT_LOCK_UNLOCK #define MUTEX_SPINBIT_LOCK_UNLOCK(mtx) __cpu_simple_unlock(&(mtx)->mtx_lock) #endif #ifndef MUTEX_INITIALIZE_SPIN_IPL #define MUTEX_INITIALIZE_SPIN_IPL(mtx, ipl) \ ((mtx)->mtx_ipl = makeiplcookie((ipl))) #endif /* * Spin mutex SPL save / restore. */ #define MUTEX_SPIN_SPLRAISE(mtx) \ do { \ struct cpu_info *x__ci; \ int x__cnt, s; \ s = splraiseipl(MUTEX_SPIN_IPL(mtx)); \ x__ci = curcpu(); \ x__cnt = x__ci->ci_mtx_count--; \ __insn_barrier(); \ if (x__cnt == 0) \ x__ci->ci_mtx_oldspl = (s); \ } while (/* CONSTCOND */ 0) #define MUTEX_SPIN_SPLRESTORE(mtx) \ do { \ struct cpu_info *x__ci = curcpu(); \ int s = x__ci->ci_mtx_oldspl; \ __insn_barrier(); \ if (++(x__ci->ci_mtx_count) == 0) \ splx(s); \ } while (/* CONSTCOND */ 0) /* * Memory barriers. */ #ifdef __HAVE_ATOMIC_AS_MEMBAR #define MUTEX_MEMBAR_ENTER() #define MUTEX_MEMBAR_ACQUIRE() #define MUTEX_MEMBAR_RELEASE() #else #define MUTEX_MEMBAR_ENTER() membar_enter() #define MUTEX_MEMBAR_ACQUIRE() membar_acquire() #define MUTEX_MEMBAR_RELEASE() membar_release() #endif /* * For architectures that provide 'simple' mutexes: they provide a * CAS function that is either MP-safe, or does not need to be MP * safe. Adaptive mutexes on these architectures do not require an * additional interlock. */ #ifdef __HAVE_SIMPLE_MUTEXES #define MUTEX_OWNER(owner) \ (owner & MUTEX_THREAD) #define MUTEX_HAS_WAITERS(mtx) \ (((int)(mtx)->mtx_owner & MUTEX_BIT_WAITERS) != 0) #define MUTEX_INITIALIZE_ADAPTIVE(mtx, dodebug) \ do { \ if (!dodebug) \ (mtx)->mtx_owner |= MUTEX_BIT_NODEBUG; \ } while (/* CONSTCOND */ 0) #define MUTEX_INITIALIZE_SPIN(mtx, dodebug, ipl) \ do { \ (mtx)->mtx_owner = MUTEX_BIT_SPIN; \ if (!dodebug) \ (mtx)->mtx_owner |= MUTEX_BIT_NODEBUG; \ MUTEX_INITIALIZE_SPIN_IPL((mtx), (ipl)); \ MUTEX_SPINBIT_LOCK_INIT((mtx)); \ } while (/* CONSTCOND */ 0) #define MUTEX_DESTROY(mtx) \ do { \ (mtx)->mtx_owner = MUTEX_THREAD; \ } while (/* CONSTCOND */ 0) #define MUTEX_SPIN_P(owner) \ (((owner) & MUTEX_BIT_SPIN) != 0) #define MUTEX_ADAPTIVE_P(owner) \ (((owner) & MUTEX_BIT_SPIN) == 0) #ifndef MUTEX_CAS #define MUTEX_CAS(p, o, n) \ (atomic_cas_ulong((volatile unsigned long *)(p), (o), (n)) == (o)) #endif /* MUTEX_CAS */ #define MUTEX_DEBUG_P(mtx) (((mtx)->mtx_owner & MUTEX_BIT_NODEBUG) == 0) #if defined(LOCKDEBUG) #define MUTEX_OWNED(owner) (((owner) & ~MUTEX_BIT_NODEBUG) != 0) #define MUTEX_INHERITDEBUG(n, o) (n) |= (o) & MUTEX_BIT_NODEBUG #else /* defined(LOCKDEBUG) */ #define MUTEX_OWNED(owner) ((owner) != 0) #define MUTEX_INHERITDEBUG(n, o) /* nothing */ #endif /* defined(LOCKDEBUG) */ static inline int MUTEX_ACQUIRE(kmutex_t *mtx, uintptr_t curthread) { int rv; uintptr_t oldown = 0; uintptr_t newown = curthread; MUTEX_INHERITDEBUG(oldown, mtx->mtx_owner); MUTEX_INHERITDEBUG(newown, oldown); rv = MUTEX_CAS(&mtx->mtx_owner, oldown, newown); MUTEX_MEMBAR_ACQUIRE(); return rv; } static inline int MUTEX_SET_WAITERS(kmutex_t *mtx, uintptr_t owner) { int rv; rv = MUTEX_CAS(&mtx->mtx_owner, owner, owner | MUTEX_BIT_WAITERS); MUTEX_MEMBAR_ENTER(); return rv; } static inline void MUTEX_RELEASE(kmutex_t *mtx) { uintptr_t newown; MUTEX_MEMBAR_RELEASE(); newown = 0; MUTEX_INHERITDEBUG(newown, mtx->mtx_owner); mtx->mtx_owner = newown; } #endif /* __HAVE_SIMPLE_MUTEXES */ /* * Patch in stubs via strong alias where they are not available. */ #if defined(LOCKDEBUG) #undef __HAVE_MUTEX_STUBS #undef __HAVE_SPIN_MUTEX_STUBS #endif #ifndef __HAVE_MUTEX_STUBS __strong_alias(mutex_enter,mutex_vector_enter); __strong_alias(mutex_exit,mutex_vector_exit); #endif #ifndef __HAVE_SPIN_MUTEX_STUBS __strong_alias(mutex_spin_enter,mutex_vector_enter); __strong_alias(mutex_spin_exit,mutex_vector_exit); #endif static void mutex_abort(const char *, size_t, const kmutex_t *, const char *); static void mutex_dump(const volatile void *, lockop_printer_t); lockops_t mutex_spin_lockops = { .lo_name = "Mutex", .lo_type = LOCKOPS_SPIN, .lo_dump = mutex_dump, }; lockops_t mutex_adaptive_lockops = { .lo_name = "Mutex", .lo_type = LOCKOPS_SLEEP, .lo_dump = mutex_dump, }; syncobj_t mutex_syncobj = { .sobj_flag = SOBJ_SLEEPQ_SORTED, .sobj_unsleep = turnstile_unsleep, .sobj_changepri = turnstile_changepri, .sobj_lendpri = sleepq_lendpri, .sobj_owner = (void *)mutex_owner, }; /* * mutex_dump: * * Dump the contents of a mutex structure. */ static void mutex_dump(const volatile void *cookie, lockop_printer_t pr) { const volatile kmutex_t *mtx = cookie; uintptr_t owner = mtx->mtx_owner; pr("owner field : %#018lx wait/spin: %16d/%d\n", (long)MUTEX_OWNER(owner), MUTEX_HAS_WAITERS(mtx), MUTEX_SPIN_P(owner)); } /* * mutex_abort: * * Dump information about an error and panic the system. This * generates a lot of machine code in the DIAGNOSTIC case, so * we ask the compiler to not inline it. */ static void __noinline mutex_abort(const char *func, size_t line, const kmutex_t *mtx, const char *msg) { LOCKDEBUG_ABORT(func, line, mtx, (MUTEX_SPIN_P(mtx->mtx_owner) ? &mutex_spin_lockops : &mutex_adaptive_lockops), msg); } /* * mutex_init: * * Initialize a mutex for use. Note that adaptive mutexes are in * essence spin mutexes that can sleep to avoid deadlock and wasting * CPU time. We can't easily provide a type of mutex that always * sleeps - see comments in mutex_vector_enter() about releasing * mutexes unlocked. */ void _mutex_init(kmutex_t *, kmutex_type_t, int, uintptr_t); void _mutex_init(kmutex_t *mtx, kmutex_type_t type, int ipl, uintptr_t return_address) { lockops_t *lockops __unused; bool dodebug; memset(mtx, 0, sizeof(*mtx)); if (ipl == IPL_NONE || ipl == IPL_SOFTCLOCK || ipl == IPL_SOFTBIO || ipl == IPL_SOFTNET || ipl == IPL_SOFTSERIAL) { lockops = (type == MUTEX_NODEBUG ? NULL : &mutex_adaptive_lockops); dodebug = LOCKDEBUG_ALLOC(mtx, lockops, return_address); MUTEX_INITIALIZE_ADAPTIVE(mtx, dodebug); } else { lockops = (type == MUTEX_NODEBUG ? NULL : &mutex_spin_lockops); dodebug = LOCKDEBUG_ALLOC(mtx, lockops, return_address); MUTEX_INITIALIZE_SPIN(mtx, dodebug, ipl); } } void mutex_init(kmutex_t *mtx, kmutex_type_t type, int ipl) { _mutex_init(mtx, type, ipl, (uintptr_t)__builtin_return_address(0)); } /* * mutex_destroy: * * Tear down a mutex. */ void mutex_destroy(kmutex_t *mtx) { uintptr_t owner = mtx->mtx_owner; if (MUTEX_ADAPTIVE_P(owner)) { MUTEX_ASSERT(mtx, !MUTEX_OWNED(owner)); MUTEX_ASSERT(mtx, !MUTEX_HAS_WAITERS(mtx)); } else { MUTEX_ASSERT(mtx, !MUTEX_SPINBIT_LOCKED_P(mtx)); } LOCKDEBUG_FREE(MUTEX_DEBUG_P(mtx), mtx); MUTEX_DESTROY(mtx); } #ifdef MULTIPROCESSOR /* * mutex_oncpu: * * Return true if an adaptive mutex owner is running on a CPU in the * system. If the target is waiting on the kernel big lock, then we * must release it. This is necessary to avoid deadlock. */ static bool mutex_oncpu(uintptr_t owner) { struct cpu_info *ci; lwp_t *l; KASSERT(kpreempt_disabled()); if (!MUTEX_OWNED(owner)) { return false; } /* * See lwp_dtor() why dereference of the LWP pointer is safe. * We must have kernel preemption disabled for that. */ l = (lwp_t *)MUTEX_OWNER(owner); ci = l->l_cpu; if (ci && ci->ci_curlwp == l) { /* Target is running; do we need to block? */ return (ci->ci_biglock_wanted != l); } /* Not running. It may be safe to block now. */ return false; } #endif /* MULTIPROCESSOR */ /* * mutex_vector_enter: * * Support routine for mutex_enter() that must handle all cases. In * the LOCKDEBUG case, mutex_enter() is always aliased here, even if * fast-path stubs are available. If a mutex_spin_enter() stub is * not available, then it is also aliased directly here. */ void mutex_vector_enter(kmutex_t *mtx) { uintptr_t owner, curthread; turnstile_t *ts; #ifdef MULTIPROCESSOR u_int count; #endif LOCKSTAT_COUNTER(spincnt); LOCKSTAT_COUNTER(slpcnt); LOCKSTAT_TIMER(spintime); LOCKSTAT_TIMER(slptime); LOCKSTAT_FLAG(lsflag); /* * Handle spin mutexes. */ KPREEMPT_DISABLE(curlwp); owner = mtx->mtx_owner; if (MUTEX_SPIN_P(owner)) { #if defined(LOCKDEBUG) && defined(MULTIPROCESSOR) u_int spins = 0; #endif KPREEMPT_ENABLE(curlwp); MUTEX_SPIN_SPLRAISE(mtx); MUTEX_WANTLOCK(mtx); #ifdef FULL if (MUTEX_SPINBIT_LOCK_TRY(mtx)) { MUTEX_LOCKED(mtx); return; } #if !defined(MULTIPROCESSOR) MUTEX_ABORT(mtx, "locking against myself"); #else /* !MULTIPROCESSOR */ LOCKSTAT_ENTER(lsflag); LOCKSTAT_START_TIMER(lsflag, spintime); count = SPINLOCK_BACKOFF_MIN; /* * Spin testing the lock word and do exponential backoff * to reduce cache line ping-ponging between CPUs. */ do { while (MUTEX_SPINBIT_LOCKED_P(mtx)) { SPINLOCK_SPIN_HOOK; SPINLOCK_BACKOFF(count); #ifdef LOCKDEBUG if (SPINLOCK_SPINOUT(spins)) MUTEX_ABORT(mtx, "spinout"); #endif /* LOCKDEBUG */ } } while (!MUTEX_SPINBIT_LOCK_TRY(mtx)); if (count != SPINLOCK_BACKOFF_MIN) { LOCKSTAT_STOP_TIMER(lsflag, spintime); LOCKSTAT_EVENT(lsflag, mtx, LB_SPIN_MUTEX | LB_SPIN, 1, spintime); } LOCKSTAT_EXIT(lsflag); #endif /* !MULTIPROCESSOR */ #endif /* FULL */ MUTEX_LOCKED(mtx); return; } curthread = (uintptr_t)curlwp; MUTEX_DASSERT(mtx, MUTEX_ADAPTIVE_P(owner)); MUTEX_ASSERT(mtx, curthread != 0); MUTEX_ASSERT(mtx, !cpu_intr_p()); MUTEX_WANTLOCK(mtx); if (panicstr == NULL) { KDASSERT(pserialize_not_in_read_section()); LOCKDEBUG_BARRIER(&kernel_lock, 1); } LOCKSTAT_ENTER(lsflag); /* * Adaptive mutex; spin trying to acquire the mutex. If we * determine that the owner is not running on a processor, * then we stop spinning, and sleep instead. */ for (;;) { if (!MUTEX_OWNED(owner)) { /* * Mutex owner clear could mean two things: * * * The mutex has been released. * * The owner field hasn't been set yet. * * Try to acquire it again. If that fails, * we'll just loop again. */ if (MUTEX_ACQUIRE(mtx, curthread)) break; owner = mtx->mtx_owner; continue; } if (__predict_false(MUTEX_OWNER(owner) == curthread)) { MUTEX_ABORT(mtx, "locking against myself"); } #ifdef MULTIPROCESSOR /* * Check to see if the owner is running on a processor. * If so, then we should just spin, as the owner will * likely release the lock very soon. */ if (mutex_oncpu(owner)) { LOCKSTAT_START_TIMER(lsflag, spintime); count = SPINLOCK_BACKOFF_MIN; do { KPREEMPT_ENABLE(curlwp); SPINLOCK_BACKOFF(count); KPREEMPT_DISABLE(curlwp); owner = mtx->mtx_owner; } while (mutex_oncpu(owner)); LOCKSTAT_STOP_TIMER(lsflag, spintime); LOCKSTAT_COUNT(spincnt, 1); if (!MUTEX_OWNED(owner)) continue; } #endif ts = turnstile_lookup(mtx); /* * Once we have the turnstile chain interlock, mark the * mutex as having waiters. If that fails, spin again: * chances are that the mutex has been released. */ if (!MUTEX_SET_WAITERS(mtx, owner)) { turnstile_exit(mtx); owner = mtx->mtx_owner; continue; } #ifdef MULTIPROCESSOR /* * mutex_exit() is permitted to release the mutex without * any interlocking instructions, and the following can * occur as a result: * * CPU 1: MUTEX_SET_WAITERS() CPU2: mutex_exit() * ---------------------------- ---------------------------- * .. acquire cache line * .. test for waiters * acquire cache line <- lose cache line * lock cache line .. * verify mutex is held .. * set waiters .. * unlock cache line .. * lose cache line -> acquire cache line * .. clear lock word, waiters * return success * * There is another race that can occur: a third CPU could * acquire the mutex as soon as it is released. Since * adaptive mutexes are primarily spin mutexes, this is not * something that we need to worry about too much. What we * do need to ensure is that the waiters bit gets set. * * To allow the unlocked release, we need to make some * assumptions here: * * o Release is the only non-atomic/unlocked operation * that can be performed on the mutex. (It must still * be atomic on the local CPU, e.g. in case interrupted * or preempted). * * o At any given time, MUTEX_SET_WAITERS() can only ever * be in progress on one CPU in the system - guaranteed * by the turnstile chain lock. * * o No other operations other than MUTEX_SET_WAITERS() * and release can modify a mutex with a non-zero * owner field. * * o The result of a successful MUTEX_SET_WAITERS() call * is an unbuffered write that is immediately visible * to all other processors in the system. * * o If the holding LWP switches away, it posts a store * fence before changing curlwp, ensuring that any * overwrite of the mutex waiters flag by mutex_exit() * completes before the modification of curlwp becomes * visible to this CPU. * * o cpu_switchto() posts a store fence after setting curlwp * and before resuming execution of an LWP. * * o _kernel_lock() posts a store fence before setting * curcpu()->ci_biglock_wanted, and after clearing it. * This ensures that any overwrite of the mutex waiters * flag by mutex_exit() completes before the modification * of ci_biglock_wanted becomes visible. * * We now post a read memory barrier (after setting the * waiters field) and check the lock holder's status again. * Some of the possible outcomes (not an exhaustive list): * * 1. The on-CPU check returns true: the holding LWP is * running again. The lock may be released soon and * we should spin. Importantly, we can't trust the * value of the waiters flag. * * 2. The on-CPU check returns false: the holding LWP is * not running. We now have the opportunity to check * if mutex_exit() has blatted the modifications made * by MUTEX_SET_WAITERS(). * * 3. The on-CPU check returns false: the holding LWP may * or may not be running. It has context switched at * some point during our check. Again, we have the * chance to see if the waiters bit is still set or * has been overwritten. * * 4. The on-CPU check returns false: the holding LWP is * running on a CPU, but wants the big lock. It's OK * to check the waiters field in this case. * * 5. The has-waiters check fails: the mutex has been * released, the waiters flag cleared and another LWP * now owns the mutex. * * 6. The has-waiters check fails: the mutex has been * released. * * If the waiters bit is not set it's unsafe to go asleep, * as we might never be awoken. */ membar_consumer(); if (mutex_oncpu(owner)) { turnstile_exit(mtx); owner = mtx->mtx_owner; continue; } membar_consumer(); if (!MUTEX_HAS_WAITERS(mtx)) { turnstile_exit(mtx); owner = mtx->mtx_owner; continue; } #endif /* MULTIPROCESSOR */ LOCKSTAT_START_TIMER(lsflag, slptime); turnstile_block(ts, TS_WRITER_Q, mtx, &mutex_syncobj); LOCKSTAT_STOP_TIMER(lsflag, slptime); LOCKSTAT_COUNT(slpcnt, 1); owner = mtx->mtx_owner; } KPREEMPT_ENABLE(curlwp); LOCKSTAT_EVENT(lsflag, mtx, LB_ADAPTIVE_MUTEX | LB_SLEEP1, slpcnt, slptime); LOCKSTAT_EVENT(lsflag, mtx, LB_ADAPTIVE_MUTEX | LB_SPIN, spincnt, spintime); LOCKSTAT_EXIT(lsflag); MUTEX_DASSERT(mtx, MUTEX_OWNER(mtx->mtx_owner) == curthread); MUTEX_LOCKED(mtx); } /* * mutex_vector_exit: * * Support routine for mutex_exit() that handles all cases. */ void mutex_vector_exit(kmutex_t *mtx) { turnstile_t *ts; uintptr_t curthread; if (MUTEX_SPIN_P(mtx->mtx_owner)) { #ifdef FULL if (__predict_false(!MUTEX_SPINBIT_LOCKED_P(mtx))) { MUTEX_ABORT(mtx, "exiting unheld spin mutex"); } MUTEX_UNLOCKED(mtx); MUTEX_SPINBIT_LOCK_UNLOCK(mtx); #endif MUTEX_SPIN_SPLRESTORE(mtx); return; } #ifndef __HAVE_MUTEX_STUBS /* * On some architectures without mutex stubs, we can enter here to * release mutexes before interrupts and whatnot are up and running. * We need this hack to keep them sweet. */ if (__predict_false(cold)) { MUTEX_UNLOCKED(mtx); MUTEX_RELEASE(mtx); return; } #endif curthread = (uintptr_t)curlwp; MUTEX_DASSERT(mtx, curthread != 0); MUTEX_ASSERT(mtx, MUTEX_OWNER(mtx->mtx_owner) == curthread); MUTEX_UNLOCKED(mtx); #if !defined(LOCKDEBUG) __USE(curthread); #endif #ifdef LOCKDEBUG /* * Avoid having to take the turnstile chain lock every time * around. Raise the priority level to splhigh() in order * to disable preemption and so make the following atomic. */ { int s = splhigh(); if (!MUTEX_HAS_WAITERS(mtx)) { MUTEX_RELEASE(mtx); splx(s); return; } splx(s); } #endif /* * Get this lock's turnstile. This gets the interlock on * the sleep queue. Once we have that, we can clear the * lock. If there was no turnstile for the lock, there * were no waiters remaining. */ ts = turnstile_lookup(mtx); if (ts == NULL) { MUTEX_RELEASE(mtx); turnstile_exit(mtx); } else { MUTEX_RELEASE(mtx); turnstile_wakeup(ts, TS_WRITER_Q, TS_WAITERS(ts, TS_WRITER_Q), NULL); } } #ifndef __HAVE_SIMPLE_MUTEXES /* * mutex_wakeup: * * Support routine for mutex_exit() that wakes up all waiters. * We assume that the mutex has been released, but it need not * be. */ void mutex_wakeup(kmutex_t *mtx) { turnstile_t *ts; ts = turnstile_lookup(mtx); if (ts == NULL) { turnstile_exit(mtx); return; } MUTEX_CLEAR_WAITERS(mtx); turnstile_wakeup(ts, TS_WRITER_Q, TS_WAITERS(ts, TS_WRITER_Q), NULL); } #endif /* !__HAVE_SIMPLE_MUTEXES */ /* * mutex_owned: * * Return true if the current LWP (adaptive) or CPU (spin) * holds the mutex. */ int mutex_owned(const kmutex_t *mtx) { if (mtx == NULL) return 0; if (MUTEX_ADAPTIVE_P(mtx->mtx_owner)) return MUTEX_OWNER(mtx->mtx_owner) == (uintptr_t)curlwp; #ifdef FULL return MUTEX_SPINBIT_LOCKED_P(mtx); #else return 1; #endif } /* * mutex_owner: * * Return the current owner of an adaptive mutex. Used for * priority inheritance. */ lwp_t * mutex_owner(const kmutex_t *mtx) { MUTEX_ASSERT(mtx, MUTEX_ADAPTIVE_P(mtx->mtx_owner)); return (struct lwp *)MUTEX_OWNER(mtx->mtx_owner); } /* * mutex_owner_running: * * Return true if an adaptive mutex is unheld, or held and the owner is * running on a CPU. For the pagedaemon only - do not document or use * in other code. */ bool mutex_owner_running(const kmutex_t *mtx) { #ifdef MULTIPROCESSOR uintptr_t owner; bool rv; MUTEX_ASSERT(mtx, MUTEX_ADAPTIVE_P(mtx->mtx_owner)); kpreempt_disable(); owner = mtx->mtx_owner; rv = !MUTEX_OWNED(owner) || mutex_oncpu(MUTEX_OWNER(owner)); kpreempt_enable(); return rv; #else return mutex_owner(mtx) == curlwp; #endif } /* * mutex_ownable: * * When compiled with DEBUG and LOCKDEBUG defined, ensure that * the mutex is available. We cannot use !mutex_owned() since * that won't work correctly for spin mutexes. */ int mutex_ownable(const kmutex_t *mtx) { #ifdef LOCKDEBUG MUTEX_TESTLOCK(mtx); #endif return 1; } /* * mutex_tryenter: * * Try to acquire the mutex; return non-zero if we did. */ int mutex_tryenter(kmutex_t *mtx) { uintptr_t curthread; /* * Handle spin mutexes. */ if (MUTEX_SPIN_P(mtx->mtx_owner)) { MUTEX_SPIN_SPLRAISE(mtx); #ifdef FULL if (MUTEX_SPINBIT_LOCK_TRY(mtx)) { MUTEX_WANTLOCK(mtx); MUTEX_LOCKED(mtx); return 1; } MUTEX_SPIN_SPLRESTORE(mtx); #else MUTEX_WANTLOCK(mtx); MUTEX_LOCKED(mtx); return 1; #endif } else { curthread = (uintptr_t)curlwp; MUTEX_ASSERT(mtx, curthread != 0); if (MUTEX_ACQUIRE(mtx, curthread)) { MUTEX_WANTLOCK(mtx); MUTEX_LOCKED(mtx); MUTEX_DASSERT(mtx, MUTEX_OWNER(mtx->mtx_owner) == curthread); return 1; } } return 0; } #if defined(__HAVE_SPIN_MUTEX_STUBS) || defined(FULL) /* * mutex_spin_retry: * * Support routine for mutex_spin_enter(). Assumes that the caller * has already raised the SPL, and adjusted counters. */ void mutex_spin_retry(kmutex_t *mtx) { #ifdef MULTIPROCESSOR u_int count; LOCKSTAT_TIMER(spintime); LOCKSTAT_FLAG(lsflag); #ifdef LOCKDEBUG u_int spins = 0; #endif /* LOCKDEBUG */ MUTEX_WANTLOCK(mtx); LOCKSTAT_ENTER(lsflag); LOCKSTAT_START_TIMER(lsflag, spintime); count = SPINLOCK_BACKOFF_MIN; /* * Spin testing the lock word and do exponential backoff * to reduce cache line ping-ponging between CPUs. */ do { while (MUTEX_SPINBIT_LOCKED_P(mtx)) { SPINLOCK_BACKOFF(count); #ifdef LOCKDEBUG if (SPINLOCK_SPINOUT(spins)) MUTEX_ABORT(mtx, "spinout"); #endif /* LOCKDEBUG */ } } while (!MUTEX_SPINBIT_LOCK_TRY(mtx)); LOCKSTAT_STOP_TIMER(lsflag, spintime); LOCKSTAT_EVENT(lsflag, mtx, LB_SPIN_MUTEX | LB_SPIN, 1, spintime); LOCKSTAT_EXIT(lsflag); MUTEX_LOCKED(mtx); #else /* MULTIPROCESSOR */ MUTEX_ABORT(mtx, "locking against myself"); #endif /* MULTIPROCESSOR */ } #endif /* defined(__HAVE_SPIN_MUTEX_STUBS) || defined(FULL) */ |
| 8 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 197 198 199 200 201 202 203 204 205 206 207 208 209 210 211 212 213 214 215 216 217 218 219 220 221 222 223 224 225 226 227 228 229 230 231 232 233 234 235 236 237 238 239 240 241 242 243 244 245 246 247 248 249 250 251 252 253 254 255 256 257 258 259 260 261 262 263 264 265 266 267 268 269 270 271 272 273 274 275 276 277 278 279 280 281 282 283 284 285 286 287 288 289 290 291 292 293 294 295 296 297 298 299 300 301 302 303 304 305 306 307 308 309 310 311 312 313 314 315 316 317 318 319 320 321 322 323 324 325 326 327 328 329 330 331 332 333 334 335 336 337 338 339 340 341 342 343 344 345 346 347 348 349 350 351 352 353 354 355 356 357 358 359 360 361 362 363 364 365 366 367 368 369 370 371 372 373 374 375 376 377 378 379 380 381 382 383 384 385 386 387 388 389 390 391 392 393 394 395 396 397 398 399 400 401 402 403 404 405 406 407 408 409 410 411 412 413 414 415 416 417 418 419 420 421 422 423 424 425 426 427 428 429 430 431 432 433 434 435 436 437 438 439 440 441 442 443 444 445 446 447 448 449 450 451 452 453 454 455 456 457 458 459 460 461 462 463 464 465 466 467 468 469 470 471 472 473 474 475 476 477 478 479 480 481 482 483 484 485 486 487 488 489 490 491 492 493 494 495 496 497 498 499 500 501 502 503 504 505 506 507 508 509 510 511 512 513 514 515 516 517 518 519 520 521 522 523 524 525 526 527 528 529 530 531 532 533 534 535 536 537 538 539 540 541 542 543 544 545 546 547 548 549 550 551 552 553 554 555 556 557 558 559 560 561 562 563 564 565 566 567 568 569 570 571 572 573 574 575 576 577 578 579 580 581 582 583 584 585 586 587 588 589 590 591 592 593 594 595 596 597 598 599 600 601 602 603 604 605 606 607 608 609 610 611 612 613 614 615 616 617 618 619 620 621 622 623 624 625 626 627 628 629 630 631 632 633 634 635 636 637 638 639 640 641 642 643 644 645 646 647 648 649 650 651 652 653 654 655 656 657 658 659 660 661 662 663 664 665 666 667 668 669 670 671 672 673 674 675 676 677 678 679 680 681 682 683 684 685 686 687 688 689 690 691 692 693 694 695 696 697 698 699 700 701 702 703 704 705 706 707 708 709 710 711 712 713 714 715 716 717 718 719 720 721 722 723 724 725 726 727 728 729 730 731 732 733 734 735 736 737 738 739 740 741 742 743 744 745 746 747 748 749 750 751 752 753 754 755 756 757 758 759 760 761 762 763 764 765 766 767 768 769 770 771 772 773 774 775 776 777 778 779 780 781 782 783 784 785 786 787 788 789 790 791 792 793 794 795 796 797 798 799 800 801 802 803 804 805 806 807 808 809 810 811 812 813 814 815 816 817 | /* $NetBSD: if_udav.c,v 1.99 2022/08/20 14:09:20 riastradh Exp $ */ /* $nabe: if_udav.c,v 1.3 2003/08/21 16:57:19 nabe Exp $ */ /* * Copyright (c) 2003 * Shingo WATANABE <nabe@nabechan.org>. All rights reserved. * * Redistribution and use in source and binary forms, with or without * modification, are permitted provided that the following conditions * are met: * 1. Redistributions of source code must retain the above copyright * notice, this list of conditions and the following disclaimer. * 2. Redistributions in binary form must reproduce the above copyright * notice, this list of conditions and the following disclaimer in the * documentation and/or other materials provided with the distribution. * 3. Neither the name of the author nor the names of any co-contributors * may be used to endorse or promote products derived from this software * without specific prior written permission. * * THIS SOFTWARE IS PROVIDED BY THE AUTHOR AND CONTRIBUTORS ``AS IS'' AND * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE * ARE DISCLAIMED. IN NO EVENT SHALL THE AUTHOR OR CONTRIBUTORS BE LIABLE * FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL * DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS * OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) * HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT * LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY * OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF * SUCH DAMAGE. * */ /* * DM9601(DAVICOM USB to Ethernet MAC Controller with Integrated 10/100 PHY) * The spec can be found at the following url. * http://www.davicom.com.tw/big5/download/Data%20Sheet/DM9601-DS-F01-062202s.pdf */ /* * TODO: * Interrupt Endpoint support * External PHYs * powerhook() support? */ #include <sys/cdefs.h> __KERNEL_RCSID(0, "$NetBSD: if_udav.c,v 1.99 2022/08/20 14:09:20 riastradh Exp $"); #ifdef _KERNEL_OPT #include "opt_usb.h" #endif #include <sys/param.h> #include <dev/usb/usbnet.h> #include <dev/usb/if_udavreg.h> /* Function declarations */ static int udav_match(device_t, cfdata_t, void *); static void udav_attach(device_t, device_t, void *); CFATTACH_DECL_NEW(udav, sizeof(struct usbnet), udav_match, udav_attach, usbnet_detach, usbnet_activate); static void udav_chip_init(struct usbnet *); static unsigned udav_uno_tx_prepare(struct usbnet *, struct mbuf *, struct usbnet_chain *); static void udav_uno_rx_loop(struct usbnet *, struct usbnet_chain *, uint32_t); static void udav_uno_stop(struct ifnet *, int); static void udav_uno_mcast(struct ifnet *); static int udav_uno_mii_read_reg(struct usbnet *, int, int, uint16_t *); static int udav_uno_mii_write_reg(struct usbnet *, int, int, uint16_t); static void udav_uno_mii_statchg(struct ifnet *); static int udav_uno_init(struct ifnet *); static void udav_reset(struct usbnet *); static int udav_csr_read(struct usbnet *, int, void *, int); static int udav_csr_write(struct usbnet *, int, void *, int); static int udav_csr_read1(struct usbnet *, int); static int udav_csr_write1(struct usbnet *, int, unsigned char); #if 0 static int udav_mem_read(struct usbnet *, int, void *, int); static int udav_mem_write(struct usbnet *, int, void *, int); static int udav_mem_write1(struct usbnet *, int, unsigned char); #endif /* Macros */ #ifdef UDAV_DEBUG #define DPRINTF(x) if (udavdebug) printf x #define DPRINTFN(n, x) if (udavdebug >= (n)) printf x int udavdebug = 0; #else #define DPRINTF(x) #define DPRINTFN(n, x) #endif #define UDAV_SETBIT(un, reg, x) \ udav_csr_write1(un, reg, udav_csr_read1(un, reg) | (x)) #define UDAV_CLRBIT(un, reg, x) \ udav_csr_write1(un, reg, udav_csr_read1(un, reg) & ~(x)) static const struct udav_type { struct usb_devno udav_dev; uint16_t udav_flags; #define UDAV_EXT_PHY 0x0001 #define UDAV_NO_PHY 0x0002 } udav_devs [] = { /* Corega USB-TXC */ {{ USB_VENDOR_COREGA, USB_PRODUCT_COREGA_FETHER_USB_TXC }, 0}, /* ShanTou ST268 USB NIC */ {{ USB_VENDOR_SHANTOU, USB_PRODUCT_SHANTOU_ST268_USB_NIC }, 0}, /* ShanTou ADM8515 */ {{ USB_VENDOR_SHANTOU, USB_PRODUCT_SHANTOU_ADM8515 }, 0}, /* SUNRISING SR9600 */ {{ USB_VENDOR_SUNRISING, USB_PRODUCT_SUNRISING_SR9600 }, 0 }, /* SUNRISING QF9700 */ {{ USB_VENDOR_SUNRISING, USB_PRODUCT_SUNRISING_QF9700 }, UDAV_NO_PHY }, /* QUAN DM9601 */ {{USB_VENDOR_QUAN, USB_PRODUCT_QUAN_DM9601 }, 0}, #if 0 /* DAVICOM DM9601 Generic? */ /* XXX: The following ids was obtained from the data sheet. */ {{ 0x0a46, 0x9601 }, 0}, #endif }; #define udav_lookup(v, p) ((const struct udav_type *)usb_lookup(udav_devs, v, p)) static const struct usbnet_ops udav_ops = { .uno_stop = udav_uno_stop, .uno_mcast = udav_uno_mcast, .uno_read_reg = udav_uno_mii_read_reg, .uno_write_reg = udav_uno_mii_write_reg, .uno_statchg = udav_uno_mii_statchg, .uno_tx_prepare = udav_uno_tx_prepare, .uno_rx_loop = udav_uno_rx_loop, .uno_init = udav_uno_init, }; /* Probe */ static int udav_match(device_t parent, cfdata_t match, void *aux) { struct usb_attach_arg *uaa = aux; return udav_lookup(uaa->uaa_vendor, uaa->uaa_product) != NULL ? UMATCH_VENDOR_PRODUCT : UMATCH_NONE; } /* Attach */ static void udav_attach(device_t parent, device_t self, void *aux) { USBNET_MII_DECL_DEFAULT(unm); struct usbnet_mii *unmp; struct usbnet * const un = device_private(self); struct usb_attach_arg *uaa = aux; struct usbd_device *dev = uaa->uaa_device; struct usbd_interface *iface; usbd_status err; usb_interface_descriptor_t *id; usb_endpoint_descriptor_t *ed; char *devinfop; int i; aprint_naive("\n"); aprint_normal("\n"); devinfop = usbd_devinfo_alloc(dev, 0); aprint_normal_dev(self, "%s\n", devinfop); usbd_devinfo_free(devinfop); un->un_dev = self; un->un_udev = dev; un->un_sc = un; un->un_ops = &udav_ops; un->un_rx_xfer_flags = USBD_SHORT_XFER_OK; un->un_tx_xfer_flags = USBD_FORCE_SHORT_XFER; un->un_rx_list_cnt = UDAV_RX_LIST_CNT; un->un_tx_list_cnt = UDAV_TX_LIST_CNT; un->un_rx_bufsz = UDAV_BUFSZ; un->un_tx_bufsz = UDAV_BUFSZ; /* Move the device into the configured state. */ err = usbd_set_config_no(dev, UDAV_CONFIG_NO, 1); /* idx 0 */ if (err) { aprint_error_dev(self, "failed to set configuration" ", err=%s\n", usbd_errstr(err)); return; } /* get control interface */ err = usbd_device2interface_handle(dev, UDAV_IFACE_INDEX, &iface); if (err) { aprint_error_dev(self, "failed to get interface, err=%s\n", usbd_errstr(err)); return; } un->un_iface = iface; un->un_flags = udav_lookup(uaa->uaa_vendor, uaa->uaa_product)->udav_flags; /* get interface descriptor */ id = usbd_get_interface_descriptor(un->un_iface); /* find endpoints */ un->un_ed[USBNET_ENDPT_RX] = un->un_ed[USBNET_ENDPT_TX] = un->un_ed[USBNET_ENDPT_INTR] = -1; for (i = 0; i < id->bNumEndpoints; i++) { ed = usbd_interface2endpoint_descriptor(un->un_iface, i); if (ed == NULL) { aprint_error_dev(self, "couldn't get endpoint %d\n", i); return; } if ((ed->bmAttributes & UE_XFERTYPE) == UE_BULK && UE_GET_DIR(ed->bEndpointAddress) == UE_DIR_IN) un->un_ed[USBNET_ENDPT_RX] = ed->bEndpointAddress; else if ((ed->bmAttributes & UE_XFERTYPE) == UE_BULK && UE_GET_DIR(ed->bEndpointAddress) == UE_DIR_OUT) un->un_ed[USBNET_ENDPT_TX] = ed->bEndpointAddress; else if ((ed->bmAttributes & UE_XFERTYPE) == UE_INTERRUPT && UE_GET_DIR(ed->bEndpointAddress) == UE_DIR_IN) un->un_ed[USBNET_ENDPT_INTR] = ed->bEndpointAddress; } if (un->un_ed[USBNET_ENDPT_RX] == 0 || un->un_ed[USBNET_ENDPT_TX] == 0 || un->un_ed[USBNET_ENDPT_INTR] == 0) { aprint_error_dev(self, "missing endpoint\n"); return; } /* Not supported yet. */ un->un_ed[USBNET_ENDPT_INTR] = 0; usbnet_attach(un); // /* reset the adapter */ // udav_reset(un); /* Get Ethernet Address */ err = udav_csr_read(un, UDAV_PAR, un->un_eaddr, ETHER_ADDR_LEN); if (err) { aprint_error_dev(self, "read MAC address failed\n"); return; } if (ISSET(un->un_flags, UDAV_NO_PHY)) unmp = NULL; else unmp = &unm; /* initialize interface information */ usbnet_attach_ifp(un, IFF_SIMPLEX | IFF_BROADCAST | IFF_MULTICAST, 0, unmp); return; } #if 0 /* read memory */ static int udav_mem_read(struct usbnet *un, int offset, void *buf, int len) { usb_device_request_t req; usbd_status err; DPRINTFN(0x200, ("%s: %s: enter\n", device_xname(un->un_dev), __func__)); if (usbnet_isdying(un)) return 0; offset &= 0xffff; len &= 0xff; req.bmRequestType = UT_READ_VENDOR_DEVICE; req.bRequest = UDAV_REQ_MEM_READ; USETW(req.wValue, 0x0000); USETW(req.wIndex, offset); USETW(req.wLength, len); err = usbd_do_request(un->un_udev, &req, buf); if (err) { DPRINTF(("%s: %s: read failed. off=%04x, err=%d\n", device_xname(un->un_dev), __func__, offset, err)); } return err; } /* write memory */ static int udav_mem_write(struct usbnet *un, int offset, void *buf, int len) { usb_device_request_t req; usbd_status err; DPRINTFN(0x200, ("%s: %s: enter\n", device_xname(un->un_dev), __func__)); if (usbnet_isdying(un)) return 0; offset &= 0xffff; len &= 0xff; req.bmRequestType = UT_WRITE_VENDOR_DEVICE; req.bRequest = UDAV_REQ_MEM_WRITE; USETW(req.wValue, 0x0000); USETW(req.wIndex, offset); USETW(req.wLength, len); err = usbd_do_request(un->un_udev, &req, buf); if (err) { DPRINTF(("%s: %s: write failed. off=%04x, err=%d\n", device_xname(un->un_dev), __func__, offset, err)); } return err; } /* write memory */ static int udav_mem_write1(struct usbnet *un, int offset, unsigned char ch) { usb_device_request_t req; usbd_status err; DPRINTFN(0x200, ("%s: %s: enter\n", device_xname(un->un_dev), __func__)); if (usbnet_isdying(un)) return 0; offset &= 0xffff; req.bmRequestType = UT_WRITE_VENDOR_DEVICE; req.bRequest = UDAV_REQ_MEM_WRITE1; USETW(req.wValue, ch); USETW(req.wIndex, offset); USETW(req.wLength, 0x0000); err = usbd_do_request(un->un_udev, &req, NULL); if (err) { DPRINTF(("%s: %s: write failed. off=%04x, err=%d\n", device_xname(un->un_dev), __func__, offset, err)); } return err; } #endif /* read register(s) */ static int udav_csr_read(struct usbnet *un, int offset, void *buf, int len) { usb_device_request_t req; usbd_status err; if (usbnet_isdying(un)) return USBD_IOERROR; DPRINTFN(0x200, ("%s: %s: enter\n", device_xname(un->un_dev), __func__)); offset &= 0xff; len &= 0xff; req.bmRequestType = UT_READ_VENDOR_DEVICE; req.bRequest = UDAV_REQ_REG_READ; USETW(req.wValue, 0x0000); USETW(req.wIndex, offset); USETW(req.wLength, len); err = usbd_do_request(un->un_udev, &req, buf); if (err) { DPRINTF(("%s: %s: read failed. off=%04x, err=%d\n", device_xname(un->un_dev), __func__, offset, err)); memset(buf, 0, len); } return err; } /* write register(s) */ static int udav_csr_write(struct usbnet *un, int offset, void *buf, int len) { usb_device_request_t req; usbd_status err; if (usbnet_isdying(un)) return USBD_IOERROR; DPRINTFN(0x200, ("%s: %s: enter\n", device_xname(un->un_dev), __func__)); offset &= 0xff; len &= 0xff; req.bmRequestType = UT_WRITE_VENDOR_DEVICE; req.bRequest = UDAV_REQ_REG_WRITE; USETW(req.wValue, 0x0000); USETW(req.wIndex, offset); USETW(req.wLength, len); err = usbd_do_request(un->un_udev, &req, buf); if (err) { DPRINTF(("%s: %s: write failed. off=%04x, err=%d\n", device_xname(un->un_dev), __func__, offset, err)); } return err; } static int udav_csr_read1(struct usbnet *un, int offset) { uint8_t val = 0; DPRINTFN(0x200, ("%s: %s: enter\n", device_xname(un->un_dev), __func__)); if (usbnet_isdying(un)) return 0; return udav_csr_read(un, offset, &val, 1) ? 0 : val; } /* write a register */ static int udav_csr_write1(struct usbnet *un, int offset, unsigned char ch) { usb_device_request_t req; usbd_status err; if (usbnet_isdying(un)) return USBD_IOERROR; DPRINTFN(0x200, ("%s: %s: enter\n", device_xname(un->un_dev), __func__)); offset &= 0xff; req.bmRequestType = UT_WRITE_VENDOR_DEVICE; req.bRequest = UDAV_REQ_REG_WRITE1; USETW(req.wValue, ch); USETW(req.wIndex, offset); USETW(req.wLength, 0x0000); err = usbd_do_request(un->un_udev, &req, NULL); if (err) { DPRINTF(("%s: %s: write failed. off=%04x, err=%d\n", device_xname(un->un_dev), __func__, offset, err)); } return err; } static int udav_uno_init(struct ifnet *ifp) { struct usbnet * const un = ifp->if_softc; struct mii_data * const mii = usbnet_mii(un); uint8_t eaddr[ETHER_ADDR_LEN]; int rc = 0; DPRINTF(("%s: %s: enter\n", device_xname(un->un_dev), __func__)); memcpy(eaddr, CLLADDR(ifp->if_sadl), sizeof(eaddr)); udav_csr_write(un, UDAV_PAR, eaddr, ETHER_ADDR_LEN); /* Initialize network control register */ /* Disable loopback */ UDAV_CLRBIT(un, UDAV_NCR, UDAV_NCR_LBK0 | UDAV_NCR_LBK1); /* Initialize RX control register */ UDAV_SETBIT(un, UDAV_RCR, UDAV_RCR_DIS_LONG | UDAV_RCR_DIS_CRC); /* If we want promiscuous mode, accept all physical frames. */ if (usbnet_ispromisc(un)) UDAV_SETBIT(un, UDAV_RCR, UDAV_RCR_ALL | UDAV_RCR_PRMSC); else UDAV_CLRBIT(un, UDAV_RCR, UDAV_RCR_ALL | UDAV_RCR_PRMSC); /* Enable RX */ UDAV_SETBIT(un, UDAV_RCR, UDAV_RCR_RXEN); /* clear POWER_DOWN state of internal PHY */ UDAV_SETBIT(un, UDAV_GPCR, UDAV_GPCR_GEP_CNTL0); UDAV_CLRBIT(un, UDAV_GPR, UDAV_GPR_GEPIO0); if (mii && (rc = mii_mediachg(mii)) == ENXIO) rc = 0; if (rc != 0) { return rc; } if (usbnet_isdying(un)) return EIO; return 0; } static void udav_reset(struct usbnet *un) { if (usbnet_isdying(un)) return; DPRINTF(("%s: %s: enter\n", device_xname(un->un_dev), __func__)); udav_chip_init(un); } static void udav_chip_init(struct usbnet *un) { /* Select PHY */ #if 1 /* * XXX: force select internal phy. * external phy routines are not tested. */ UDAV_CLRBIT(un, UDAV_NCR, UDAV_NCR_EXT_PHY); #else if (un->un_flags & UDAV_EXT_PHY) { UDAV_SETBIT(un, UDAV_NCR, UDAV_NCR_EXT_PHY); } else { UDAV_CLRBIT(un, UDAV_NCR, UDAV_NCR_EXT_PHY); } #endif UDAV_SETBIT(un, UDAV_NCR, UDAV_NCR_RST); for (int i = 0; i < UDAV_TX_TIMEOUT; i++) { if (usbnet_isdying(un)) return; if (!(udav_csr_read1(un, UDAV_NCR) & UDAV_NCR_RST)) break; delay(10); /* XXX */ } delay(10000); /* XXX */ } #define UDAV_BITS 6 #define UDAV_CALCHASH(addr) \ (ether_crc32_le((addr), ETHER_ADDR_LEN) & ((1 << UDAV_BITS) - 1)) static void udav_uno_mcast(struct ifnet *ifp) { struct usbnet * const un = ifp->if_softc; struct ethercom *ec = usbnet_ec(un); struct ether_multi *enm; struct ether_multistep step; uint8_t hashes[8]; int h = 0; DPRINTF(("%s: %s: enter\n", device_xname(un->un_dev), __func__)); if (usbnet_isdying(un)) return; if (ISSET(un->un_flags, UDAV_NO_PHY)) { UDAV_SETBIT(un, UDAV_RCR, UDAV_RCR_ALL); UDAV_SETBIT(un, UDAV_RCR, UDAV_RCR_PRMSC); return; } if (usbnet_ispromisc(un)) { ETHER_LOCK(ec); ec->ec_flags |= ETHER_F_ALLMULTI; ETHER_UNLOCK(ec); UDAV_SETBIT(un, UDAV_RCR, UDAV_RCR_ALL | UDAV_RCR_PRMSC); return; } /* first, zot all the existing hash bits */ memset(hashes, 0x00, sizeof(hashes)); hashes[7] |= 0x80; /* broadcast address */ udav_csr_write(un, UDAV_MAR, hashes, sizeof(hashes)); /* now program new ones */ ETHER_LOCK(ec); ETHER_FIRST_MULTI(step, ec, enm); while (enm != NULL) { if (memcmp(enm->enm_addrlo, enm->enm_addrhi, ETHER_ADDR_LEN) != 0) { ec->ec_flags |= ETHER_F_ALLMULTI; ETHER_UNLOCK(ec); UDAV_SETBIT(un, UDAV_RCR, UDAV_RCR_ALL); UDAV_CLRBIT(un, UDAV_RCR, UDAV_RCR_PRMSC); return; } h = UDAV_CALCHASH(enm->enm_addrlo); hashes[h>>3] |= 1 << (h & 0x7); ETHER_NEXT_MULTI(step, enm); } ec->ec_flags &= ~ETHER_F_ALLMULTI; ETHER_UNLOCK(ec); /* disable all multicast */ UDAV_CLRBIT(un, UDAV_RCR, UDAV_RCR_ALL); /* write hash value to the register */ udav_csr_write(un, UDAV_MAR, hashes, sizeof(hashes)); } static unsigned udav_uno_tx_prepare(struct usbnet *un, struct mbuf *m, struct usbnet_chain *c) { int total_len; uint8_t *buf = c->unc_buf; DPRINTF(("%s: %s: enter\n", device_xname(un->un_dev), __func__)); if ((unsigned)m->m_pkthdr.len > un->un_tx_bufsz - 2) return 0; /* Copy the mbuf data into a contiguous buffer */ m_copydata(m, 0, m->m_pkthdr.len, buf + 2); total_len = m->m_pkthdr.len; if (total_len < UDAV_MIN_FRAME_LEN) { memset(buf + 2 + total_len, 0, UDAV_MIN_FRAME_LEN - total_len); total_len = UDAV_MIN_FRAME_LEN; } /* Frame length is specified in the first 2bytes of the buffer */ buf[0] = (uint8_t)total_len; buf[1] = (uint8_t)(total_len >> 8); total_len += 2; DPRINTF(("%s: %s: send %d bytes\n", device_xname(un->un_dev), __func__, total_len)); return total_len; } static void udav_uno_rx_loop(struct usbnet *un, struct usbnet_chain *c, uint32_t total_len) { struct ifnet *ifp = usbnet_ifp(un); uint8_t *buf = c->unc_buf; uint16_t pkt_len; uint8_t pktstat; DPRINTF(("%s: %s: enter\n", device_xname(un->un_dev), __func__)); /* first byte in received data */ pktstat = *buf; total_len -= sizeof(pktstat); buf += sizeof(pktstat); DPRINTF(("%s: RX Status: 0x%02x\n", device_xname(un->un_dev), pktstat)); pkt_len = UGETW(buf); total_len -= sizeof(pkt_len); buf += sizeof(pkt_len); DPRINTF(("%s: RX Length: 0x%02x\n", device_xname(un->un_dev), pkt_len)); if (pktstat & UDAV_RSR_LCS) { if_statinc(ifp, if_collisions); return; } if (pkt_len < sizeof(struct ether_header) || pkt_len > total_len || (pktstat & UDAV_RSR_ERR)) { if_statinc(ifp, if_ierrors); return; } pkt_len -= ETHER_CRC_LEN; DPRINTF(("%s: Rx deliver: 0x%02x\n", device_xname(un->un_dev), pkt_len)); usbnet_enqueue(un, buf, pkt_len, 0, 0, 0); } /* Stop the adapter and free any mbufs allocated to the RX and TX lists. */ static void udav_uno_stop(struct ifnet *ifp, int disable) { struct usbnet * const un = ifp->if_softc; DPRINTF(("%s: %s: enter\n", device_xname(un->un_dev), __func__)); udav_reset(un); } static int udav_uno_mii_read_reg(struct usbnet *un, int phy, int reg, uint16_t *val) { uint8_t data[2]; DPRINTFN(0xff, ("%s: %s: enter, phy=%d reg=0x%04x\n", device_xname(un->un_dev), __func__, phy, reg)); if (usbnet_isdying(un)) { #ifdef DIAGNOSTIC printf("%s: %s: dying\n", device_xname(un->un_dev), __func__); #endif *val = 0; return EINVAL; } /* XXX: one PHY only for the internal PHY */ if (phy != 0) { DPRINTFN(0xff, ("%s: %s: phy=%d is not supported\n", device_xname(un->un_dev), __func__, phy)); *val = 0; return EINVAL; } /* select internal PHY and set PHY register address */ udav_csr_write1(un, UDAV_EPAR, UDAV_EPAR_PHY_ADR0 | (reg & UDAV_EPAR_EROA_MASK)); /* select PHY operation and start read command */ udav_csr_write1(un, UDAV_EPCR, UDAV_EPCR_EPOS | UDAV_EPCR_ERPRR); /* XXX: should be wait? */ /* end read command */ UDAV_CLRBIT(un, UDAV_EPCR, UDAV_EPCR_ERPRR); /* retrieve the result from data registers */ udav_csr_read(un, UDAV_EPDRL, data, 2); *val = data[0] | (data[1] << 8); DPRINTFN(0xff, ("%s: %s: phy=%d reg=0x%04x => 0x%04hx\n", device_xname(un->un_dev), __func__, phy, reg, *val)); return 0; } static int udav_uno_mii_write_reg(struct usbnet *un, int phy, int reg, uint16_t val) { uint8_t data[2]; DPRINTFN(0xff, ("%s: %s: enter, phy=%d reg=0x%04x val=0x%04hx\n", device_xname(un->un_dev), __func__, phy, reg, val)); if (usbnet_isdying(un)) { #ifdef DIAGNOSTIC printf("%s: %s: dying\n", device_xname(un->un_dev), __func__); #endif return EIO; } /* XXX: one PHY only for the internal PHY */ if (phy != 0) { DPRINTFN(0xff, ("%s: %s: phy=%d is not supported\n", device_xname(un->un_dev), __func__, phy)); return EIO; } /* select internal PHY and set PHY register address */ udav_csr_write1(un, UDAV_EPAR, UDAV_EPAR_PHY_ADR0 | (reg & UDAV_EPAR_EROA_MASK)); /* put the value to the data registers */ data[0] = val & 0xff; data[1] = (val >> 8) & 0xff; udav_csr_write(un, UDAV_EPDRL, data, 2); /* select PHY operation and start write command */ udav_csr_write1(un, UDAV_EPCR, UDAV_EPCR_EPOS | UDAV_EPCR_ERPRW); /* XXX: should be wait? */ /* end write command */ UDAV_CLRBIT(un, UDAV_EPCR, UDAV_EPCR_ERPRW); return 0; } static void udav_uno_mii_statchg(struct ifnet *ifp) { struct usbnet * const un = ifp->if_softc; struct mii_data * const mii = usbnet_mii(un); DPRINTF(("%s: %s: enter\n", ifp->if_xname, __func__)); if (usbnet_isdying(un)) return; if ((mii->mii_media_status & IFM_ACTIVE) && IFM_SUBTYPE(mii->mii_media_active) != IFM_NONE) { DPRINTF(("%s: %s: got link\n", device_xname(un->un_dev), __func__)); usbnet_set_link(un, true); } } #ifdef _MODULE #include "ioconf.c" #endif USBNET_MODULE(udav) |
| 956 955 27 945 27 27 27 744 954 28 28 28 92 92 92 1 29 74 44 50 43 42 27 26 17 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 197 198 199 200 201 202 203 204 205 206 207 208 209 210 211 212 213 214 215 216 217 218 219 220 221 222 223 224 225 226 227 228 229 230 231 232 233 234 235 236 237 238 239 240 241 242 243 244 245 246 247 248 249 250 251 252 253 254 255 256 257 258 259 260 261 262 263 264 265 266 267 268 269 270 271 272 273 274 275 276 277 278 279 280 281 282 283 284 285 286 287 288 289 290 291 292 293 294 295 296 297 298 299 300 301 302 303 304 305 306 307 308 309 310 311 312 313 314 315 316 317 318 319 320 321 322 323 324 325 326 327 328 329 330 331 332 333 334 335 336 | /* $NetBSD: ufs_inode.c,v 1.112 2020/09/05 16:30:13 riastradh Exp $ */ /* * Copyright (c) 1991, 1993 * The Regents of the University of California. All rights reserved. * (c) UNIX System Laboratories, Inc. * All or some portions of this file are derived from material licensed * to the University of California by American Telephone and Telegraph * Co. or Unix System Laboratories, Inc. and are reproduced herein with * the permission of UNIX System Laboratories, Inc. * * Redistribution and use in source and binary forms, with or without * modification, are permitted provided that the following conditions * are met: * 1. Redistributions of source code must retain the above copyright * notice, this list of conditions and the following disclaimer. * 2. Redistributions in binary form must reproduce the above copyright * notice, this list of conditions and the following disclaimer in the * documentation and/or other materials provided with the distribution. * 3. Neither the name of the University nor the names of its contributors * may be used to endorse or promote products derived from this software * without specific prior written permission. * * THIS SOFTWARE IS PROVIDED BY THE REGENTS AND CONTRIBUTORS ``AS IS'' AND * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE * ARE DISCLAIMED. IN NO EVENT SHALL THE REGENTS OR CONTRIBUTORS BE LIABLE * FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL * DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS * OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) * HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT * LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY * OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF * SUCH DAMAGE. * * @(#)ufs_inode.c 8.9 (Berkeley) 5/14/95 */ #include <sys/cdefs.h> __KERNEL_RCSID(0, "$NetBSD: ufs_inode.c,v 1.112 2020/09/05 16:30:13 riastradh Exp $"); #if defined(_KERNEL_OPT) #include "opt_ffs.h" #include "opt_quota.h" #include "opt_wapbl.h" #include "opt_uvmhist.h" #endif #include <sys/param.h> #include <sys/systm.h> #include <sys/proc.h> #include <sys/vnode.h> #include <sys/mount.h> #include <sys/kernel.h> #include <sys/namei.h> #include <sys/kauth.h> #include <sys/wapbl.h> #include <sys/kmem.h> #include <ufs/ufs/inode.h> #include <ufs/ufs/ufsmount.h> #include <ufs/ufs/ufs_extern.h> #include <ufs/ufs/ufs_wapbl.h> #ifdef UFS_DIRHASH #include <ufs/ufs/dirhash.h> #endif #ifdef UFS_EXTATTR #include <ufs/ufs/extattr.h> #endif #ifdef UVMHIST #include <uvm/uvm.h> #endif #include <uvm/uvm_page.h> #include <uvm/uvm_stat.h> /* * Last reference to an inode. If necessary, write or delete it. */ int ufs_inactive(void *v) { struct vop_inactive_v2_args /* { struct vnode *a_vp; struct bool *a_recycle; } */ *ap = v; struct vnode *vp = ap->a_vp; struct inode *ip = VTOI(vp); struct mount *mp = vp->v_mount; mode_t mode; int allerror = 0, error; bool wapbl_locked = false; UFS_WAPBL_JUNLOCK_ASSERT(mp); /* * Ignore inodes related to stale file handles. */ if (ip->i_mode == 0) goto out; if (ip->i_nlink <= 0 && (mp->mnt_flag & MNT_RDONLY) == 0) { #ifdef UFS_EXTATTR ufs_extattr_vnode_inactive(vp, curlwp); #endif /* * All file blocks must be freed before we can let the vnode * be reclaimed, so can't postpone full truncating any further. */ ufs_truncate_all(vp); #if defined(QUOTA) || defined(QUOTA2) error = UFS_WAPBL_BEGIN(mp); if (error) { allerror = error; } else { wapbl_locked = true; (void)chkiq(ip, -1, NOCRED, 0); } #endif DIP_ASSIGN(ip, rdev, 0); mode = ip->i_mode; ip->i_mode = 0; ip->i_omode = mode; DIP_ASSIGN(ip, mode, 0); ip->i_flag |= IN_CHANGE | IN_UPDATE; /* * Defer final inode free and update to ufs_reclaim(). */ } if (ip->i_flag & (IN_CHANGE | IN_UPDATE | IN_MODIFIED)) { if (! wapbl_locked) { error = UFS_WAPBL_BEGIN(mp); if (error) { allerror = error; goto out; } wapbl_locked = true; } UFS_UPDATE(vp, NULL, NULL, 0); } out: if (wapbl_locked) UFS_WAPBL_END(mp); /* * If we are done with the inode, reclaim it * so that it can be reused immediately. */ *ap->a_recycle = (ip->i_mode == 0); if (ip->i_mode == 0 && (DIP(ip, size) != 0 || DIP(ip, blocks) != 0)) { printf("%s: unlinked ino %" PRId64 " on \"%s\" has" " non zero size %" PRIx64 " or blocks %" PRIx64 " with allerror %d\n", __func__, ip->i_number, mp->mnt_stat.f_mntonname, DIP(ip, size), DIP(ip, blocks), allerror); panic("%s: dirty filesystem?", __func__); } return (allerror); } /* * Reclaim an inode so that it can be used for other purposes. */ int ufs_reclaim(struct vnode *vp) { struct inode *ip = VTOI(vp); if (!UFS_WAPBL_BEGIN(vp->v_mount)) { UFS_UPDATE(vp, NULL, NULL, UPDATE_CLOSE); UFS_WAPBL_END(vp->v_mount); } UFS_UPDATE(vp, NULL, NULL, UPDATE_CLOSE); if (ip->i_devvp) { vrele(ip->i_devvp); ip->i_devvp = 0; } #if defined(QUOTA) || defined(QUOTA2) ufsquota_free(ip); #endif #ifdef UFS_DIRHASH if (ip->i_dirhash != NULL) ufsdirhash_free(ip); #endif return (0); } /* * allocate a range of blocks in a file. * after this function returns, any page entirely contained within the range * will map to invalid data and thus must be overwritten before it is made * accessible to others. */ int ufs_balloc_range(struct vnode *vp, off_t off, off_t len, kauth_cred_t cred, int flags) { off_t neweof; /* file size after the operation */ off_t neweob; /* offset next to the last block after the operation */ off_t pagestart; /* starting offset of range covered by pgs */ off_t eob; /* offset next to allocated blocks */ struct uvm_object *uobj; int i, delta, error, npages; int bshift = vp->v_mount->mnt_fs_bshift; int bsize = 1 << bshift; int ppb = MAX(bsize >> PAGE_SHIFT, 1); struct vm_page **pgs; size_t pgssize; UVMHIST_FUNC("ufs_balloc_range"); UVMHIST_CALLED(ubchist); UVMHIST_LOG(ubchist, "vp %#jx off 0x%jx len 0x%jx u_size 0x%jx", (uintptr_t)vp, off, len, vp->v_size); neweof = MAX(vp->v_size, off + len); GOP_SIZE(vp, neweof, &neweob, 0); error = 0; uobj = &vp->v_uobj; /* * read or create pages covering the range of the allocation and * keep them locked until the new block is allocated, so there * will be no window where the old contents of the new block are * visible to racing threads. */ pagestart = trunc_page(off) & ~(bsize - 1); npages = MIN(ppb, (round_page(neweob) - pagestart) >> PAGE_SHIFT); pgssize = npages * sizeof(struct vm_page *); pgs = kmem_zalloc(pgssize, KM_SLEEP); /* * adjust off to be block-aligned. */ delta = off & (bsize - 1); off -= delta; len += delta; genfs_node_wrlock(vp); rw_enter(uobj->vmobjlock, RW_WRITER); error = VOP_GETPAGES(vp, pagestart, pgs, &npages, 0, VM_PROT_WRITE, 0, PGO_SYNCIO | PGO_PASTEOF | PGO_NOBLOCKALLOC | PGO_NOTIMESTAMP | PGO_GLOCKHELD); if (error) { genfs_node_unlock(vp); goto out; } /* * now allocate the range. */ error = GOP_ALLOC(vp, off, len, flags, cred); genfs_node_unlock(vp); /* * if the allocation succeeded, mark all the pages dirty * and clear PG_RDONLY on any pages that are now fully backed * by disk blocks. if the allocation failed, we do not invalidate * the pages since they might have already existed and been dirty, * in which case we need to keep them around. if we created the pages, * they will be clean and read-only, and leaving such pages * in the cache won't cause any problems. */ GOP_SIZE(vp, off + len, &eob, 0); rw_enter(uobj->vmobjlock, RW_WRITER); for (i = 0; i < npages; i++) { KASSERT((pgs[i]->flags & PG_RELEASED) == 0); if (!error) { if (off <= pagestart + (i << PAGE_SHIFT) && pagestart + ((i + 1) << PAGE_SHIFT) <= eob) { pgs[i]->flags &= ~PG_RDONLY; } uvm_pagemarkdirty(pgs[i], UVM_PAGE_STATUS_DIRTY); } uvm_pagelock(pgs[i]); uvm_pageactivate(pgs[i]); uvm_pageunlock(pgs[i]); } uvm_page_unbusy(pgs, npages); rw_exit(uobj->vmobjlock); out: kmem_free(pgs, pgssize); return error; } int ufs_truncate_retry(struct vnode *vp, int ioflag, uint64_t newsize, kauth_cred_t cred) { struct inode *ip = VTOI(vp); struct mount *mp = vp->v_mount; int error = 0; UFS_WAPBL_JUNLOCK_ASSERT(mp); /* * Truncate might temporarily fail, loop until done. */ do { error = UFS_WAPBL_BEGIN(mp); if (error) goto out; error = UFS_TRUNCATE(vp, newsize, ioflag, cred); UFS_WAPBL_END(mp); if (error != 0 && error != EAGAIN) goto out; } while (ip->i_size != newsize); out: return error; } /* truncate all the data of the inode including extended attributes */ int ufs_truncate_all(struct vnode *vp) { struct inode *ip = VTOI(vp); off_t isize = ip->i_size; if (ip->i_ump->um_fstype == UFS2) isize += ip->i_ffs2_extsize; if (isize == 0) return 0; return ufs_truncate_retry(vp, IO_NORMAL | IO_EXT, 0, NOCRED); } |
| 151 150 150 84 78 151 46 44 17 16 1 45 24 9 15 132 130 3 3 13 117 117 4 15 225 225 225 225 225 225 225 216 11 225 119 118 119 119 119 119 119 117 119 118 565 565 565 565 565 136 565 314 117 10 381 2 16 18 565 22 6 6 6 6 107 107 107 107 91 107 35 35 35 89 89 89 86 26 6 89 18 18 18 827 182 178 182 128 92 77 104 5 99 39 3 66 689 688 1614 92 92 1615 179 92 7 85 7 85 21 152 20 137 19 136 9 8 1 6 1 120 63 91 63 91 28 15 14 3 14 4 4 4 13 12 2 12 2 5 113 45 70 45 70 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 197 198 199 200 201 202 203 204 205 206 207 208 209 210 211 212 213 214 215 216 217 218 219 220 221 222 223 224 225 226 227 228 229 230 231 232 233 234 235 236 237 238 239 240 241 242 243 244 245 246 247 248 249 250 251 252 253 254 255 256 257 258 259 260 261 262 263 264 265 266 267 268 269 270 271 272 273 274 275 276 277 278 279 280 281 282 283 284 285 286 287 288 289 290 291 292 293 294 295 296 297 298 299 300 301 302 303 304 305 306 307 308 309 310 311 312 313 314 315 316 317 318 319 320 321 322 323 324 325 326 327 328 329 330 331 332 333 334 335 336 337 338 339 340 341 342 343 344 345 346 347 348 349 350 351 352 353 354 355 356 357 358 359 360 361 362 363 364 365 366 367 368 369 370 371 372 373 374 375 376 377 378 379 380 381 382 383 384 385 386 387 388 389 390 391 392 393 394 395 396 397 398 399 400 401 402 403 404 405 406 407 408 409 410 411 412 413 414 415 416 417 418 419 420 421 422 423 424 425 426 427 428 429 430 431 432 433 434 435 436 437 438 439 440 441 442 443 444 445 446 447 448 449 450 451 452 453 454 455 456 457 458 459 460 461 462 463 464 465 466 467 468 469 470 471 472 473 474 475 476 477 478 479 480 481 482 483 484 485 486 487 488 489 490 491 492 493 494 495 496 497 498 499 500 501 502 503 504 505 506 507 508 509 510 511 512 513 514 515 516 517 518 519 520 521 522 523 524 525 526 527 528 529 530 531 532 533 534 535 536 537 538 539 540 541 542 543 544 545 546 547 548 549 550 551 552 553 554 555 556 557 558 559 560 561 562 563 564 565 566 567 568 569 570 571 572 573 574 575 576 577 578 579 580 581 582 583 584 585 586 587 588 589 590 591 592 593 594 595 596 597 598 599 600 601 602 603 604 605 606 607 608 609 610 611 612 613 614 615 616 617 618 619 620 621 622 623 624 625 626 627 628 629 630 631 632 633 634 635 636 637 638 639 640 641 642 643 644 645 646 647 648 649 650 651 652 653 654 655 656 657 658 659 660 661 662 663 664 665 666 667 668 669 670 671 672 673 674 675 676 677 678 679 680 681 682 683 684 685 686 687 688 689 690 691 692 693 694 695 696 697 698 699 700 701 702 703 704 705 706 707 708 709 710 711 712 713 714 715 716 717 718 719 720 721 722 723 724 725 726 727 728 729 730 731 732 733 734 735 736 737 738 739 740 741 742 743 744 745 746 747 748 749 750 751 752 753 754 755 756 757 758 759 760 761 762 763 764 765 766 767 768 769 770 771 772 773 774 775 776 777 778 779 780 781 782 783 784 785 786 787 788 789 790 791 792 793 794 795 796 797 798 799 800 801 802 803 804 805 806 807 808 809 810 811 812 813 814 815 816 817 818 819 820 821 822 823 824 825 826 827 828 829 830 831 832 833 834 835 836 837 838 839 840 841 842 843 844 845 846 847 848 849 850 851 852 853 854 855 856 857 858 859 860 861 862 863 864 865 866 867 868 869 870 871 872 873 874 875 876 877 878 879 880 881 882 883 884 885 886 887 888 889 890 891 892 893 894 895 896 897 898 899 900 901 902 903 904 905 906 907 908 909 910 911 912 913 914 915 916 917 918 919 920 921 922 923 924 925 926 927 928 929 930 931 932 933 934 935 936 937 938 939 940 941 942 943 944 945 946 947 948 949 950 951 952 953 954 955 956 957 958 959 960 961 962 963 964 965 966 967 968 969 970 971 972 973 974 975 976 977 978 979 980 981 982 983 984 985 986 987 988 989 990 991 992 993 994 995 996 997 998 999 1000 1001 1002 1003 1004 1005 1006 1007 1008 1009 1010 1011 1012 1013 1014 1015 1016 1017 1018 1019 1020 1021 1022 1023 1024 1025 1026 1027 1028 1029 1030 1031 1032 1033 1034 1035 1036 1037 1038 1039 1040 1041 1042 1043 1044 1045 1046 1047 1048 1049 1050 1051 1052 1053 1054 1055 1056 1057 1058 1059 1060 1061 1062 1063 1064 1065 1066 1067 1068 1069 1070 1071 1072 1073 1074 1075 1076 1077 1078 1079 1080 1081 1082 1083 1084 1085 1086 1087 1088 1089 1090 1091 1092 1093 1094 1095 1096 1097 1098 1099 1100 1101 1102 1103 1104 1105 1106 1107 1108 1109 1110 1111 1112 1113 1114 1115 1116 1117 1118 1119 1120 1121 1122 1123 1124 1125 1126 1127 1128 1129 1130 1131 1132 1133 1134 1135 1136 1137 1138 1139 1140 1141 1142 1143 1144 1145 1146 1147 1148 1149 1150 1151 1152 1153 1154 1155 1156 1157 1158 1159 1160 1161 1162 1163 1164 1165 1166 1167 1168 1169 1170 1171 1172 1173 1174 1175 1176 1177 1178 1179 1180 1181 1182 1183 1184 1185 1186 1187 1188 1189 1190 1191 1192 1193 1194 1195 1196 1197 1198 1199 1200 1201 1202 1203 1204 1205 1206 1207 1208 1209 1210 1211 1212 1213 1214 1215 1216 1217 1218 1219 1220 1221 1222 1223 1224 1225 1226 1227 1228 1229 1230 1231 1232 1233 1234 1235 1236 1237 1238 1239 1240 1241 1242 1243 1244 1245 1246 1247 1248 1249 1250 1251 1252 1253 1254 1255 1256 1257 1258 1259 1260 1261 1262 1263 1264 1265 1266 1267 1268 1269 1270 1271 1272 1273 1274 1275 1276 1277 1278 1279 1280 1281 1282 1283 1284 1285 1286 1287 1288 1289 1290 1291 1292 1293 1294 1295 1296 1297 1298 1299 1300 1301 1302 1303 1304 1305 1306 1307 1308 1309 1310 1311 1312 1313 1314 1315 1316 1317 1318 1319 1320 1321 1322 1323 1324 1325 1326 1327 1328 1329 1330 1331 1332 1333 1334 1335 1336 1337 1338 1339 1340 1341 1342 1343 1344 1345 1346 1347 1348 1349 1350 1351 1352 1353 1354 1355 1356 1357 1358 1359 1360 1361 1362 1363 1364 1365 1366 1367 1368 1369 1370 1371 1372 1373 1374 1375 1376 1377 1378 1379 1380 1381 1382 1383 1384 1385 1386 1387 1388 1389 1390 1391 1392 1393 1394 1395 1396 1397 1398 1399 1400 1401 1402 1403 1404 1405 1406 1407 1408 1409 1410 1411 1412 1413 1414 1415 1416 1417 1418 1419 1420 1421 1422 1423 1424 1425 1426 1427 1428 1429 1430 1431 1432 1433 1434 1435 1436 1437 1438 1439 1440 1441 1442 1443 1444 1445 1446 1447 1448 1449 1450 1451 1452 1453 1454 1455 1456 1457 1458 1459 1460 1461 1462 1463 1464 1465 1466 1467 1468 1469 1470 1471 1472 1473 1474 1475 1476 1477 1478 1479 1480 1481 1482 1483 1484 1485 1486 1487 1488 1489 1490 1491 1492 1493 1494 1495 1496 1497 1498 1499 1500 1501 1502 1503 1504 1505 1506 1507 1508 1509 1510 1511 1512 1513 1514 1515 1516 1517 1518 1519 1520 1521 1522 1523 1524 1525 1526 1527 1528 1529 1530 1531 1532 1533 1534 1535 1536 1537 1538 1539 1540 1541 1542 1543 1544 1545 1546 1547 1548 1549 1550 1551 1552 1553 1554 1555 1556 1557 1558 1559 1560 1561 1562 1563 1564 1565 1566 1567 1568 1569 1570 1571 1572 1573 1574 1575 1576 1577 1578 1579 1580 1581 1582 1583 1584 1585 1586 1587 1588 1589 1590 1591 1592 1593 1594 1595 1596 1597 1598 1599 1600 1601 1602 1603 1604 1605 1606 1607 1608 1609 1610 1611 1612 1613 1614 1615 1616 1617 1618 1619 1620 1621 1622 1623 1624 1625 1626 1627 1628 1629 1630 1631 1632 1633 1634 1635 1636 1637 1638 1639 1640 1641 1642 1643 1644 1645 1646 1647 1648 1649 1650 1651 1652 1653 1654 1655 1656 1657 1658 1659 1660 1661 1662 1663 1664 1665 1666 1667 1668 1669 1670 1671 1672 1673 1674 1675 1676 1677 1678 1679 1680 1681 1682 1683 1684 1685 1686 1687 1688 1689 1690 1691 1692 1693 1694 1695 1696 1697 1698 1699 1700 1701 1702 1703 1704 1705 1706 1707 1708 1709 1710 1711 1712 1713 1714 1715 1716 1717 1718 1719 1720 1721 1722 1723 1724 1725 1726 1727 1728 1729 1730 1731 1732 1733 1734 1735 1736 1737 1738 1739 1740 1741 1742 1743 1744 1745 1746 | /* $NetBSD: vfs_subr.c,v 1.493 2022/03/28 12:38:33 riastradh Exp $ */ /*- * Copyright (c) 1997, 1998, 2004, 2005, 2007, 2008, 2019, 2020 * The NetBSD Foundation, Inc. * All rights reserved. * * This code is derived from software contributed to The NetBSD Foundation * by Jason R. Thorpe of the Numerical Aerospace Simulation Facility, * NASA Ames Research Center, by Charles M. Hannum, by Andrew Doran, * by Marshall Kirk McKusick and Greg Ganger at the University of Michigan. * * Redistribution and use in source and binary forms, with or without * modification, are permitted provided that the following conditions * are met: * 1. Redistributions of source code must retain the above copyright * notice, this list of conditions and the following disclaimer. * 2. Redistributions in binary form must reproduce the above copyright * notice, this list of conditions and the following disclaimer in the * documentation and/or other materials provided with the distribution. * * THIS SOFTWARE IS PROVIDED BY THE NETBSD FOUNDATION, INC. AND CONTRIBUTORS * ``AS IS'' AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED * TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR * PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE FOUNDATION OR CONTRIBUTORS * BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR * CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF * SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS * INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN * CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) * ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE * POSSIBILITY OF SUCH DAMAGE. */ /* * Copyright (c) 1989, 1993 * The Regents of the University of California. All rights reserved. * (c) UNIX System Laboratories, Inc. * All or some portions of this file are derived from material licensed * to the University of California by American Telephone and Telegraph * Co. or Unix System Laboratories, Inc. and are reproduced herein with * the permission of UNIX System Laboratories, Inc. * * Redistribution and use in source and binary forms, with or without * modification, are permitted provided that the following conditions * are met: * 1. Redistributions of source code must retain the above copyright * notice, this list of conditions and the following disclaimer. * 2. Redistributions in binary form must reproduce the above copyright * notice, this list of conditions and the following disclaimer in the * documentation and/or other materials provided with the distribution. * 3. Neither the name of the University nor the names of its contributors * may be used to endorse or promote products derived from this software * without specific prior written permission. * * THIS SOFTWARE IS PROVIDED BY THE REGENTS AND CONTRIBUTORS ``AS IS'' AND * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE * ARE DISCLAIMED. IN NO EVENT SHALL THE REGENTS OR CONTRIBUTORS BE LIABLE * FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL * DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS * OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) * HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT * LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY * OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF * SUCH DAMAGE. * * @(#)vfs_subr.c 8.13 (Berkeley) 4/18/94 */ #include <sys/cdefs.h> __KERNEL_RCSID(0, "$NetBSD: vfs_subr.c,v 1.493 2022/03/28 12:38:33 riastradh Exp $"); #ifdef _KERNEL_OPT #include "opt_ddb.h" #include "opt_compat_netbsd.h" #include "opt_compat_43.h" #endif #include <sys/param.h> #include <sys/systm.h> #include <sys/conf.h> #include <sys/dirent.h> #include <sys/filedesc.h> #include <sys/kernel.h> #include <sys/mount.h> #include <sys/fstrans.h> #include <sys/vnode_impl.h> #include <sys/stat.h> #include <sys/sysctl.h> #include <sys/namei.h> #include <sys/buf.h> #include <sys/errno.h> #include <sys/kmem.h> #include <sys/syscallargs.h> #include <sys/kauth.h> #include <sys/module.h> #include <miscfs/genfs/genfs.h> #include <miscfs/specfs/specdev.h> #include <uvm/uvm_ddb.h> const enum vtype iftovt_tab[16] = { VNON, VFIFO, VCHR, VNON, VDIR, VNON, VBLK, VNON, VREG, VNON, VLNK, VNON, VSOCK, VNON, VNON, VBAD, }; const int vttoif_tab[9] = { 0, S_IFREG, S_IFDIR, S_IFBLK, S_IFCHR, S_IFLNK, S_IFSOCK, S_IFIFO, S_IFMT, }; /* * Insq/Remq for the vnode usage lists. */ #define bufinsvn(bp, dp) LIST_INSERT_HEAD(dp, bp, b_vnbufs) #define bufremvn(bp) { \ LIST_REMOVE(bp, b_vnbufs); \ (bp)->b_vnbufs.le_next = NOLIST; \ } int doforce = 1; /* 1 => permit forcible unmounting */ extern struct mount *dead_rootmount; /* * Local declarations. */ static void vn_initialize_syncerd(void); /* * Initialize the vnode management data structures. */ void vntblinit(void) { vn_initialize_syncerd(); vfs_mount_sysinit(); vfs_vnode_sysinit(); } /* * Flush out and invalidate all buffers associated with a vnode. * Called with the underlying vnode locked, which should prevent new dirty * buffers from being queued. */ int vinvalbuf(struct vnode *vp, int flags, kauth_cred_t cred, struct lwp *l, bool catch_p, int slptimeo) { struct buf *bp, *nbp; int error; int flushflags = PGO_ALLPAGES | PGO_FREE | PGO_SYNCIO | (flags & V_SAVE ? PGO_CLEANIT | PGO_RECLAIM : 0); /* XXXUBC this doesn't look at flags or slp* */ rw_enter(vp->v_uobj.vmobjlock, RW_WRITER); error = VOP_PUTPAGES(vp, 0, 0, flushflags); if (error) { return error; } if (flags & V_SAVE) { error = VOP_FSYNC(vp, cred, FSYNC_WAIT|FSYNC_RECLAIM, 0, 0); if (error) return (error); KASSERT(LIST_EMPTY(&vp->v_dirtyblkhd)); } mutex_enter(&bufcache_lock); restart: for (bp = LIST_FIRST(&vp->v_dirtyblkhd); bp; bp = nbp) { KASSERT(bp->b_vp == vp); nbp = LIST_NEXT(bp, b_vnbufs); error = bbusy(bp, catch_p, slptimeo, NULL); if (error != 0) { if (error == EPASSTHROUGH) goto restart; mutex_exit(&bufcache_lock); return (error); } brelsel(bp, BC_INVAL | BC_VFLUSH); } for (bp = LIST_FIRST(&vp->v_cleanblkhd); bp; bp = nbp) { KASSERT(bp->b_vp == vp); nbp = LIST_NEXT(bp, b_vnbufs); error = bbusy(bp, catch_p, slptimeo, NULL); if (error != 0) { if (error == EPASSTHROUGH) goto restart; mutex_exit(&bufcache_lock); return (error); } /* * XXX Since there are no node locks for NFS, I believe * there is a slight chance that a delayed write will * occur while sleeping just above, so check for it. */ if ((bp->b_oflags & BO_DELWRI) && (flags & V_SAVE)) { #ifdef DEBUG printf("buffer still DELWRI\n"); #endif bp->b_cflags |= BC_BUSY | BC_VFLUSH; mutex_exit(&bufcache_lock); VOP_BWRITE(bp->b_vp, bp); mutex_enter(&bufcache_lock); goto restart; } brelsel(bp, BC_INVAL | BC_VFLUSH); } #ifdef DIAGNOSTIC if (!LIST_EMPTY(&vp->v_cleanblkhd) || !LIST_EMPTY(&vp->v_dirtyblkhd)) panic("vinvalbuf: flush failed, vp %p", vp); #endif mutex_exit(&bufcache_lock); return (0); } /* * Destroy any in core blocks past the truncation length. * Called with the underlying vnode locked, which should prevent new dirty * buffers from being queued. */ int vtruncbuf(struct vnode *vp, daddr_t lbn, bool catch_p, int slptimeo) { struct buf *bp, *nbp; int error; voff_t off; off = round_page((voff_t)lbn << vp->v_mount->mnt_fs_bshift); rw_enter(vp->v_uobj.vmobjlock, RW_WRITER); error = VOP_PUTPAGES(vp, off, 0, PGO_FREE | PGO_SYNCIO); if (error) { return error; } mutex_enter(&bufcache_lock); restart: for (bp = LIST_FIRST(&vp->v_dirtyblkhd); bp; bp = nbp) { KASSERT(bp->b_vp == vp); nbp = LIST_NEXT(bp, b_vnbufs); if (bp->b_lblkno < lbn) continue; error = bbusy(bp, catch_p, slptimeo, NULL); if (error != 0) { if (error == EPASSTHROUGH) goto restart; mutex_exit(&bufcache_lock); return (error); } brelsel(bp, BC_INVAL | BC_VFLUSH); } for (bp = LIST_FIRST(&vp->v_cleanblkhd); bp; bp = nbp) { KASSERT(bp->b_vp == vp); nbp = LIST_NEXT(bp, b_vnbufs); if (bp->b_lblkno < lbn) continue; error = bbusy(bp, catch_p, slptimeo, NULL); if (error != 0) { if (error == EPASSTHROUGH) goto restart; mutex_exit(&bufcache_lock); return (error); } brelsel(bp, BC_INVAL | BC_VFLUSH); } mutex_exit(&bufcache_lock); return (0); } /* * Flush all dirty buffers from a vnode. * Called with the underlying vnode locked, which should prevent new dirty * buffers from being queued. */ int vflushbuf(struct vnode *vp, int flags) { struct buf *bp, *nbp; int error, pflags; bool dirty, sync; sync = (flags & FSYNC_WAIT) != 0; pflags = PGO_CLEANIT | PGO_ALLPAGES | (sync ? PGO_SYNCIO : 0) | ((flags & FSYNC_LAZY) ? PGO_LAZY : 0); rw_enter(vp->v_uobj.vmobjlock, RW_WRITER); (void) VOP_PUTPAGES(vp, 0, 0, pflags); loop: mutex_enter(&bufcache_lock); for (bp = LIST_FIRST(&vp->v_dirtyblkhd); bp; bp = nbp) { KASSERT(bp->b_vp == vp); nbp = LIST_NEXT(bp, b_vnbufs); if ((bp->b_cflags & BC_BUSY)) continue; if ((bp->b_oflags & BO_DELWRI) == 0) panic("vflushbuf: not dirty, bp %p", bp); bp->b_cflags |= BC_BUSY | BC_VFLUSH; mutex_exit(&bufcache_lock); /* * Wait for I/O associated with indirect blocks to complete, * since there is no way to quickly wait for them below. */ if (bp->b_vp == vp || !sync) (void) bawrite(bp); else { error = bwrite(bp); if (error) return error; } goto loop; } mutex_exit(&bufcache_lock); if (!sync) return 0; mutex_enter(vp->v_interlock); while (vp->v_numoutput != 0) cv_wait(&vp->v_cv, vp->v_interlock); dirty = !LIST_EMPTY(&vp->v_dirtyblkhd); mutex_exit(vp->v_interlock); if (dirty) { vprint("vflushbuf: dirty", vp); goto loop; } return 0; } /* * Create a vnode for a block device. * Used for root filesystem and swap areas. * Also used for memory file system special devices. */ int bdevvp(dev_t dev, vnode_t **vpp) { struct vattr va; vattr_null(&va); va.va_type = VBLK; va.va_rdev = dev; return vcache_new(dead_rootmount, NULL, &va, NOCRED, NULL, vpp); } /* * Create a vnode for a character device. * Used for kernfs and some console handling. */ int cdevvp(dev_t dev, vnode_t **vpp) { struct vattr va; vattr_null(&va); va.va_type = VCHR; va.va_rdev = dev; return vcache_new(dead_rootmount, NULL, &va, NOCRED, NULL, vpp); } /* * Associate a buffer with a vnode. There must already be a hold on * the vnode. */ void bgetvp(struct vnode *vp, struct buf *bp) { KASSERT(bp->b_vp == NULL); KASSERT(bp->b_objlock == &buffer_lock); KASSERT(mutex_owned(vp->v_interlock)); KASSERT(mutex_owned(&bufcache_lock)); KASSERT((bp->b_cflags & BC_BUSY) != 0); KASSERT(!cv_has_waiters(&bp->b_done)); vholdl(vp); bp->b_vp = vp; if (vp->v_type == VBLK || vp->v_type == VCHR) bp->b_dev = vp->v_rdev; else bp->b_dev = NODEV; /* * Insert onto list for new vnode. */ bufinsvn(bp, &vp->v_cleanblkhd); bp->b_objlock = vp->v_interlock; } /* * Disassociate a buffer from a vnode. */ void brelvp(struct buf *bp) { struct vnode *vp = bp->b_vp; KASSERT(vp != NULL); KASSERT(bp->b_objlock == vp->v_interlock); KASSERT(mutex_owned(vp->v_interlock)); KASSERT(mutex_owned(&bufcache_lock)); KASSERT((bp->b_cflags & BC_BUSY) != 0); KASSERT(!cv_has_waiters(&bp->b_done)); /* * Delete from old vnode list, if on one. */ if (LIST_NEXT(bp, b_vnbufs) != NOLIST) bufremvn(bp); if ((vp->v_iflag & (VI_ONWORKLST | VI_PAGES)) == VI_ONWORKLST && LIST_FIRST(&vp->v_dirtyblkhd) == NULL) vn_syncer_remove_from_worklist(vp); bp->b_objlock = &buffer_lock; bp->b_vp = NULL; holdrelel(vp); } /* * Reassign a buffer from one vnode list to another. * The list reassignment must be within the same vnode. * Used to assign file specific control information * (indirect blocks) to the list to which they belong. */ void reassignbuf(struct buf *bp, struct vnode *vp) { struct buflists *listheadp; int delayx; KASSERT(mutex_owned(&bufcache_lock)); KASSERT(bp->b_objlock == vp->v_interlock); KASSERT(mutex_owned(vp->v_interlock)); KASSERT((bp->b_cflags & BC_BUSY) != 0); /* * Delete from old vnode list, if on one. */ if (LIST_NEXT(bp, b_vnbufs) != NOLIST) bufremvn(bp); /* * If dirty, put on list of dirty buffers; * otherwise insert onto list of clean buffers. */ if ((bp->b_oflags & BO_DELWRI) == 0) { listheadp = &vp->v_cleanblkhd; if ((vp->v_iflag & (VI_ONWORKLST | VI_PAGES)) == VI_ONWORKLST && LIST_FIRST(&vp->v_dirtyblkhd) == NULL) vn_syncer_remove_from_worklist(vp); } else { listheadp = &vp->v_dirtyblkhd; if ((vp->v_iflag & VI_ONWORKLST) == 0) { switch (vp->v_type) { case VDIR: delayx = dirdelay; break; case VBLK: if (spec_node_getmountedfs(vp) != NULL) { delayx = metadelay; break; } /* fall through */ default: delayx = filedelay; break; } if (!vp->v_mount || (vp->v_mount->mnt_flag & MNT_ASYNC) == 0) vn_syncer_add_to_worklist(vp, delayx); } } bufinsvn(bp, listheadp); } /* * Lookup a vnode by device number and return it referenced. */ int vfinddev(dev_t dev, enum vtype type, vnode_t **vpp) { return (spec_node_lookup_by_dev(type, dev, VDEAD_NOWAIT, vpp) == 0); } /* * Revoke all the vnodes corresponding to the specified minor number * range (endpoints inclusive) of the specified major. */ void vdevgone(int maj, int minl, int minh, enum vtype type) { vnode_t *vp; dev_t dev; int mn; for (mn = minl; mn <= minh; mn++) { dev = makedev(maj, mn); /* * Notify anyone trying to get at this device that it * has been detached, and then revoke it. */ switch (type) { case VBLK: bdev_detached(dev); break; case VCHR: cdev_detached(dev); break; default: panic("invalid specnode type: %d", type); } /* * Passing 0 as flags, instead of VDEAD_NOWAIT, means * spec_node_lookup_by_dev will wait for vnodes it * finds concurrently being revoked before returning. */ while (spec_node_lookup_by_dev(type, dev, 0, &vp) == 0) { VOP_REVOKE(vp, REVOKEALL); vrele(vp); } } } /* * The filesystem synchronizer mechanism - syncer. * * It is useful to delay writes of file data and filesystem metadata for * a certain amount of time so that quickly created and deleted files need * not waste disk bandwidth being created and removed. To implement this, * vnodes are appended to a "workitem" queue. * * Most pending metadata should not wait for more than ten seconds. Thus, * mounted on block devices are delayed only about a half the time that file * data is delayed. Similarly, directory updates are more critical, so are * only delayed about a third the time that file data is delayed. * * There are SYNCER_MAXDELAY queues that are processed in a round-robin * manner at a rate of one each second (driven off the filesystem syner * thread). The syncer_delayno variable indicates the next queue that is * to be processed. Items that need to be processed soon are placed in * this queue: * * syncer_workitem_pending[syncer_delayno] * * A delay of e.g. fifteen seconds is done by placing the request fifteen * entries later in the queue: * * syncer_workitem_pending[(syncer_delayno + 15) & syncer_mask] * * Flag VI_ONWORKLST indicates that vnode is added into the queue. */ #define SYNCER_MAXDELAY 32 typedef TAILQ_HEAD(synclist, vnode_impl) synclist_t; static void vn_syncer_add1(struct vnode *, int); static void sysctl_vfs_syncfs_setup(struct sysctllog **); /* * Defines and variables for the syncer process. */ int syncer_maxdelay = SYNCER_MAXDELAY; /* maximum delay time */ time_t syncdelay = 30; /* max time to delay syncing data */ time_t filedelay = 30; /* time to delay syncing files */ time_t dirdelay = 15; /* time to delay syncing directories */ time_t metadelay = 10; /* time to delay syncing metadata */ time_t lockdelay = 1; /* time to delay if locking fails */ static kmutex_t syncer_data_lock; /* short term lock on data structs */ static int syncer_delayno = 0; static long syncer_last; static synclist_t * syncer_workitem_pending; static void vn_initialize_syncerd(void) { int i; syncer_last = SYNCER_MAXDELAY + 2; sysctl_vfs_syncfs_setup(NULL); syncer_workitem_pending = kmem_alloc(syncer_last * sizeof (struct synclist), KM_SLEEP); for (i = 0; i < syncer_last; i++) TAILQ_INIT(&syncer_workitem_pending[i]); mutex_init(&syncer_data_lock, MUTEX_DEFAULT, IPL_NONE); } /* * Return delay factor appropriate for the given file system. For * WAPBL we use the sync vnode to burst out metadata updates: sync * those file systems more frequently. */ static inline int sync_delay(struct mount *mp) { return mp->mnt_wapbl != NULL ? metadelay : syncdelay; } /* * Compute the next slot index from delay. */ static inline int sync_delay_slot(int delayx) { if (delayx > syncer_maxdelay - 2) delayx = syncer_maxdelay - 2; return (syncer_delayno + delayx) % syncer_last; } /* * Add an item to the syncer work queue. */ static void vn_syncer_add1(struct vnode *vp, int delayx) { synclist_t *slp; vnode_impl_t *vip = VNODE_TO_VIMPL(vp); KASSERT(mutex_owned(&syncer_data_lock)); if (vp->v_iflag & VI_ONWORKLST) { /* * Remove in order to adjust the position of the vnode. * Note: called from sched_sync(), which will not hold * interlock, therefore we cannot modify v_iflag here. */ slp = &syncer_workitem_pending[vip->vi_synclist_slot]; TAILQ_REMOVE(slp, vip, vi_synclist); } else { KASSERT(mutex_owned(vp->v_interlock)); vp->v_iflag |= VI_ONWORKLST; } vip->vi_synclist_slot = sync_delay_slot(delayx); slp = &syncer_workitem_pending[vip->vi_synclist_slot]; TAILQ_INSERT_TAIL(slp, vip, vi_synclist); } void vn_syncer_add_to_worklist(struct vnode *vp, int delayx) { KASSERT(mutex_owned(vp->v_interlock)); mutex_enter(&syncer_data_lock); vn_syncer_add1(vp, delayx); mutex_exit(&syncer_data_lock); } /* * Remove an item from the syncer work queue. */ void vn_syncer_remove_from_worklist(struct vnode *vp) { synclist_t *slp; vnode_impl_t *vip = VNODE_TO_VIMPL(vp); KASSERT(mutex_owned(vp->v_interlock)); if (vp->v_iflag & VI_ONWORKLST) { mutex_enter(&syncer_data_lock); vp->v_iflag &= ~VI_ONWORKLST; slp = &syncer_workitem_pending[vip->vi_synclist_slot]; TAILQ_REMOVE(slp, vip, vi_synclist); mutex_exit(&syncer_data_lock); } } /* * Add this mount point to the syncer. */ void vfs_syncer_add_to_worklist(struct mount *mp) { static int start, incr, next; int vdelay; KASSERT(mutex_owned(mp->mnt_updating)); KASSERT((mp->mnt_iflag & IMNT_ONWORKLIST) == 0); /* * We attempt to scatter the mount points on the list * so that they will go off at evenly distributed times * even if all the filesystems are mounted at once. */ next += incr; if (next == 0 || next > syncer_maxdelay) { start /= 2; incr /= 2; if (start == 0) { start = syncer_maxdelay / 2; incr = syncer_maxdelay; } next = start; } mp->mnt_iflag |= IMNT_ONWORKLIST; vdelay = sync_delay(mp); mp->mnt_synclist_slot = vdelay > 0 ? next % vdelay : 0; } /* * Remove the mount point from the syncer. */ void vfs_syncer_remove_from_worklist(struct mount *mp) { KASSERT(mutex_owned(mp->mnt_updating)); KASSERT((mp->mnt_iflag & IMNT_ONWORKLIST) != 0); mp->mnt_iflag &= ~IMNT_ONWORKLIST; } /* * Try lazy sync, return true on success. */ static bool lazy_sync_vnode(struct vnode *vp) { bool synced; KASSERT(mutex_owned(&syncer_data_lock)); synced = false; if (vcache_tryvget(vp) == 0) { mutex_exit(&syncer_data_lock); if (vn_lock(vp, LK_EXCLUSIVE | LK_NOWAIT) == 0) { synced = true; (void) VOP_FSYNC(vp, curlwp->l_cred, FSYNC_LAZY, 0, 0); vput(vp); } else vrele(vp); mutex_enter(&syncer_data_lock); } return synced; } /* * System filesystem synchronizer daemon. */ void sched_sync(void *arg) { mount_iterator_t *iter; synclist_t *slp; struct vnode_impl *vi; struct vnode *vp; struct mount *mp; time_t starttime; bool synced; for (;;) { starttime = time_second; /* * Sync mounts whose dirty time has expired. */ mountlist_iterator_init(&iter); while ((mp = mountlist_iterator_trynext(iter)) != NULL) { if ((mp->mnt_iflag & IMNT_ONWORKLIST) == 0 || mp->mnt_synclist_slot != syncer_delayno) { continue; } mp->mnt_synclist_slot = sync_delay_slot(sync_delay(mp)); VFS_SYNC(mp, MNT_LAZY, curlwp->l_cred); } mountlist_iterator_destroy(iter); mutex_enter(&syncer_data_lock); /* * Push files whose dirty time has expired. */ slp = &syncer_workitem_pending[syncer_delayno]; syncer_delayno += 1; if (syncer_delayno >= syncer_last) syncer_delayno = 0; while ((vi = TAILQ_FIRST(slp)) != NULL) { vp = VIMPL_TO_VNODE(vi); synced = lazy_sync_vnode(vp); /* * XXX The vnode may have been recycled, in which * case it may have a new identity. */ vi = TAILQ_FIRST(slp); if (vi != NULL && VIMPL_TO_VNODE(vi) == vp) { /* * Put us back on the worklist. The worklist * routine will remove us from our current * position and then add us back in at a later * position. * * Try again sooner rather than later if * we were unable to lock the vnode. Lock * failure should not prevent us from doing * the sync "soon". * * If we locked it yet arrive here, it's * likely that lazy sync is in progress and * so the vnode still has dirty metadata. * syncdelay is mainly to get this vnode out * of the way so we do not consider it again * "soon" in this loop, so the delay time is * not critical as long as it is not "soon". * While write-back strategy is the file * system's domain, we expect write-back to * occur no later than syncdelay seconds * into the future. */ vn_syncer_add1(vp, synced ? syncdelay : lockdelay); } } /* * If it has taken us less than a second to process the * current work, then wait. Otherwise start right over * again. We can still lose time if any single round * takes more than two seconds, but it does not really * matter as we are just trying to generally pace the * filesystem activity. */ if (time_second == starttime) { kpause("syncer", false, hz, &syncer_data_lock); } mutex_exit(&syncer_data_lock); } } static void sysctl_vfs_syncfs_setup(struct sysctllog **clog) { const struct sysctlnode *rnode, *cnode; sysctl_createv(clog, 0, NULL, &rnode, CTLFLAG_PERMANENT, CTLTYPE_NODE, "sync", SYSCTL_DESCR("syncer options"), NULL, 0, NULL, 0, CTL_VFS, CTL_CREATE, CTL_EOL); sysctl_createv(clog, 0, &rnode, &cnode, CTLFLAG_PERMANENT|CTLFLAG_READWRITE, CTLTYPE_QUAD, "delay", SYSCTL_DESCR("max time to delay syncing data"), NULL, 0, &syncdelay, 0, CTL_CREATE, CTL_EOL); sysctl_createv(clog, 0, &rnode, &cnode, CTLFLAG_PERMANENT|CTLFLAG_READWRITE, CTLTYPE_QUAD, "filedelay", SYSCTL_DESCR("time to delay syncing files"), NULL, 0, &filedelay, 0, CTL_CREATE, CTL_EOL); sysctl_createv(clog, 0, &rnode, &cnode, CTLFLAG_PERMANENT|CTLFLAG_READWRITE, CTLTYPE_QUAD, "dirdelay", SYSCTL_DESCR("time to delay syncing directories"), NULL, 0, &dirdelay, 0, CTL_CREATE, CTL_EOL); sysctl_createv(clog, 0, &rnode, &cnode, CTLFLAG_PERMANENT|CTLFLAG_READWRITE, CTLTYPE_QUAD, "metadelay", SYSCTL_DESCR("time to delay syncing metadata"), NULL, 0, &metadelay, 0, CTL_CREATE, CTL_EOL); } /* * sysctl helper routine to return list of supported fstypes */ int sysctl_vfs_generic_fstypes(SYSCTLFN_ARGS) { char bf[sizeof(((struct statvfs *)NULL)->f_fstypename)]; char *where = oldp; struct vfsops *v; size_t needed, left, slen; int error, first; if (newp != NULL) return (EPERM); if (namelen != 0) return (EINVAL); first = 1; error = 0; needed = 0; left = *oldlenp; sysctl_unlock(); mutex_enter(&vfs_list_lock); LIST_FOREACH(v, &vfs_list, vfs_list) { if (where == NULL) needed += strlen(v->vfs_name) + 1; else { memset(bf, 0, sizeof(bf)); if (first) { strncpy(bf, v->vfs_name, sizeof(bf)); first = 0; } else { bf[0] = ' '; strncpy(bf + 1, v->vfs_name, sizeof(bf) - 1); } bf[sizeof(bf)-1] = '\0'; slen = strlen(bf); if (left < slen + 1) break; v->vfs_refcount++; mutex_exit(&vfs_list_lock); /* +1 to copy out the trailing NUL byte */ error = copyout(bf, where, slen + 1); mutex_enter(&vfs_list_lock); v->vfs_refcount--; if (error) break; where += slen; needed += slen; left -= slen; } } mutex_exit(&vfs_list_lock); sysctl_relock(); *oldlenp = needed; return (error); } int kinfo_vdebug = 1; int kinfo_vgetfailed; #define KINFO_VNODESLOP 10 /* * Dump vnode list (via sysctl). * Copyout address of vnode followed by vnode. */ int sysctl_kern_vnode(SYSCTLFN_ARGS) { char *where = oldp; size_t *sizep = oldlenp; struct mount *mp; vnode_t *vp, vbuf; mount_iterator_t *iter; struct vnode_iterator *marker; char *bp = where; char *ewhere; int error; if (namelen != 0) return (EOPNOTSUPP); if (newp != NULL) return (EPERM); #define VPTRSZ sizeof(vnode_t *) #define VNODESZ sizeof(vnode_t) if (where == NULL) { *sizep = (numvnodes + KINFO_VNODESLOP) * (VPTRSZ + VNODESZ); return (0); } ewhere = where + *sizep; sysctl_unlock(); mountlist_iterator_init(&iter); while ((mp = mountlist_iterator_next(iter)) != NULL) { vfs_vnode_iterator_init(mp, &marker); while ((vp = vfs_vnode_iterator_next(marker, NULL, NULL))) { if (bp + VPTRSZ + VNODESZ > ewhere) { vrele(vp); vfs_vnode_iterator_destroy(marker); mountlist_iterator_destroy(iter); sysctl_relock(); *sizep = bp - where; return (ENOMEM); } memcpy(&vbuf, vp, VNODESZ); if ((error = copyout(&vp, bp, VPTRSZ)) || (error = copyout(&vbuf, bp + VPTRSZ, VNODESZ))) { vrele(vp); vfs_vnode_iterator_destroy(marker); mountlist_iterator_destroy(iter); sysctl_relock(); return (error); } vrele(vp); bp += VPTRSZ + VNODESZ; } vfs_vnode_iterator_destroy(marker); } mountlist_iterator_destroy(iter); sysctl_relock(); *sizep = bp - where; return (0); } /* * Set vnode attributes to VNOVAL */ void vattr_null(struct vattr *vap) { memset(vap, 0, sizeof(*vap)); vap->va_type = VNON; /* * Assign individually so that it is safe even if size and * sign of each member are varied. */ vap->va_mode = VNOVAL; vap->va_nlink = VNOVAL; vap->va_uid = VNOVAL; vap->va_gid = VNOVAL; vap->va_fsid = VNOVAL; vap->va_fileid = VNOVAL; vap->va_size = VNOVAL; vap->va_blocksize = VNOVAL; vap->va_atime.tv_sec = vap->va_mtime.tv_sec = vap->va_ctime.tv_sec = vap->va_birthtime.tv_sec = VNOVAL; vap->va_atime.tv_nsec = vap->va_mtime.tv_nsec = vap->va_ctime.tv_nsec = vap->va_birthtime.tv_nsec = VNOVAL; vap->va_gen = VNOVAL; vap->va_flags = VNOVAL; vap->va_rdev = VNOVAL; vap->va_bytes = VNOVAL; } /* * Vnode state to string. */ const char * vstate_name(enum vnode_state state) { switch (state) { case VS_ACTIVE: return "ACTIVE"; case VS_MARKER: return "MARKER"; case VS_LOADING: return "LOADING"; case VS_LOADED: return "LOADED"; case VS_BLOCKED: return "BLOCKED"; case VS_RECLAIMING: return "RECLAIMING"; case VS_RECLAIMED: return "RECLAIMED"; default: return "ILLEGAL"; } } /* * Print a description of a vnode (common part). */ static void vprint_common(struct vnode *vp, const char *prefix, void (*pr)(const char *, ...) __printflike(1, 2)) { int n; char bf[96]; const uint8_t *cp; vnode_impl_t *vip; const char * const vnode_tags[] = { VNODE_TAGS }; const char * const vnode_types[] = { VNODE_TYPES }; const char vnode_flagbits[] = VNODE_FLAGBITS; #define ARRAY_SIZE(arr) (sizeof(arr) / sizeof(arr[0])) #define ARRAY_PRINT(idx, arr) \ ((unsigned int)(idx) < ARRAY_SIZE(arr) ? (arr)[(idx)] : "UNKNOWN") vip = VNODE_TO_VIMPL(vp); snprintb(bf, sizeof(bf), vnode_flagbits, vp->v_iflag | vp->v_vflag | vp->v_uflag); (*pr)("vnode %p flags %s\n", vp, bf); (*pr)("%stag %s(%d) type %s(%d) mount %p typedata %p\n", prefix, ARRAY_PRINT(vp->v_tag, vnode_tags), vp->v_tag, ARRAY_PRINT(vp->v_type, vnode_types), vp->v_type, vp->v_mount, vp->v_mountedhere); (*pr)("%susecount %d writecount %d holdcount %d\n", prefix, vrefcnt(vp), vp->v_writecount, vp->v_holdcnt); (*pr)("%ssize %" PRIx64 " writesize %" PRIx64 " numoutput %d\n", prefix, vp->v_size, vp->v_writesize, vp->v_numoutput); (*pr)("%sdata %p lock %p\n", prefix, vp->v_data, &vip->vi_lock); (*pr)("%sstate %s key(%p %zd)", prefix, vstate_name(vip->vi_state), vip->vi_key.vk_mount, vip->vi_key.vk_key_len); n = vip->vi_key.vk_key_len; cp = vip->vi_key.vk_key; while (n-- > 0) (*pr)(" %02x", *cp++); (*pr)("\n"); (*pr)("%slrulisthd %p\n", prefix, vip->vi_lrulisthd); #undef ARRAY_PRINT #undef ARRAY_SIZE } /* * Print out a description of a vnode. */ void vprint(const char *label, struct vnode *vp) { if (label != NULL) printf("%s: ", label); vprint_common(vp, "\t", printf); if (vp->v_data != NULL) { printf("\t"); VOP_PRINT(vp); } } /* * Given a file system name, look up the vfsops for that * file system, or return NULL if file system isn't present * in the kernel. */ struct vfsops * vfs_getopsbyname(const char *name) { struct vfsops *v; mutex_enter(&vfs_list_lock); LIST_FOREACH(v, &vfs_list, vfs_list) { if (strcmp(v->vfs_name, name) == 0) break; } if (v != NULL) v->vfs_refcount++; mutex_exit(&vfs_list_lock); return (v); } void copy_statvfs_info(struct statvfs *sbp, const struct mount *mp) { const struct statvfs *mbp; if (sbp == (mbp = &mp->mnt_stat)) return; (void)memcpy(&sbp->f_fsidx, &mbp->f_fsidx, sizeof(sbp->f_fsidx)); sbp->f_fsid = mbp->f_fsid; sbp->f_owner = mbp->f_owner; sbp->f_flag = mbp->f_flag; sbp->f_syncwrites = mbp->f_syncwrites; sbp->f_asyncwrites = mbp->f_asyncwrites; sbp->f_syncreads = mbp->f_syncreads; sbp->f_asyncreads = mbp->f_asyncreads; (void)memcpy(sbp->f_spare, mbp->f_spare, sizeof(mbp->f_spare)); (void)memcpy(sbp->f_fstypename, mbp->f_fstypename, sizeof(sbp->f_fstypename)); (void)memcpy(sbp->f_mntonname, mbp->f_mntonname, sizeof(sbp->f_mntonname)); (void)memcpy(sbp->f_mntfromname, mp->mnt_stat.f_mntfromname, sizeof(sbp->f_mntfromname)); (void)memcpy(sbp->f_mntfromlabel, mp->mnt_stat.f_mntfromlabel, sizeof(sbp->f_mntfromlabel)); sbp->f_namemax = mbp->f_namemax; } int set_statvfs_info(const char *onp, int ukon, const char *fromp, int ukfrom, const char *vfsname, struct mount *mp, struct lwp *l) { int error; size_t size; struct statvfs *sfs = &mp->mnt_stat; int (*fun)(const void *, void *, size_t, size_t *); (void)strlcpy(mp->mnt_stat.f_fstypename, vfsname, sizeof(mp->mnt_stat.f_fstypename)); if (onp) { struct cwdinfo *cwdi = l->l_proc->p_cwdi; fun = (ukon == UIO_SYSSPACE) ? copystr : copyinstr; if (cwdi->cwdi_rdir != NULL) { size_t len; char *bp; char *path = PNBUF_GET(); bp = path + MAXPATHLEN; *--bp = '\0'; rw_enter(&cwdi->cwdi_lock, RW_READER); error = getcwd_common(cwdi->cwdi_rdir, rootvnode, &bp, path, MAXPATHLEN / 2, 0, l); rw_exit(&cwdi->cwdi_lock); if (error) { PNBUF_PUT(path); return error; } len = strlen(bp); if (len > sizeof(sfs->f_mntonname) - 1) len = sizeof(sfs->f_mntonname) - 1; (void)strncpy(sfs->f_mntonname, bp, len); PNBUF_PUT(path); if (len < sizeof(sfs->f_mntonname) - 1) { error = (*fun)(onp, &sfs->f_mntonname[len], sizeof(sfs->f_mntonname) - len - 1, &size); if (error) return error; size += len; } else { size = len; } } else { error = (*fun)(onp, &sfs->f_mntonname, sizeof(sfs->f_mntonname) - 1, &size); if (error) return error; } (void)memset(sfs->f_mntonname + size, 0, sizeof(sfs->f_mntonname) - size); } if (fromp) { fun = (ukfrom == UIO_SYSSPACE) ? copystr : copyinstr; error = (*fun)(fromp, sfs->f_mntfromname, sizeof(sfs->f_mntfromname) - 1, &size); if (error) return error; (void)memset(sfs->f_mntfromname + size, 0, sizeof(sfs->f_mntfromname) - size); } return 0; } /* * Knob to control the precision of file timestamps: * * 0 = seconds only; nanoseconds zeroed. * 1 = seconds and nanoseconds, accurate within 1/HZ. * 2 = seconds and nanoseconds, truncated to microseconds. * >=3 = seconds and nanoseconds, maximum precision. */ enum { TSP_SEC, TSP_HZ, TSP_USEC, TSP_NSEC }; int vfs_timestamp_precision __read_mostly = TSP_NSEC; void vfs_timestamp(struct timespec *tsp) { struct timeval tv; switch (vfs_timestamp_precision) { case TSP_SEC: tsp->tv_sec = time_second; tsp->tv_nsec = 0; break; case TSP_HZ: getnanotime(tsp); break; case TSP_USEC: microtime(&tv); TIMEVAL_TO_TIMESPEC(&tv, tsp); break; case TSP_NSEC: default: nanotime(tsp); break; } } /* * The purpose of this routine is to remove granularity from accmode_t, * reducing it into standard unix access bits - VEXEC, VREAD, VWRITE, * VADMIN and VAPPEND. * * If it returns 0, the caller is supposed to continue with the usual * access checks using 'accmode' as modified by this routine. If it * returns nonzero value, the caller is supposed to return that value * as errno. * * Note that after this routine runs, accmode may be zero. */ int vfs_unixify_accmode(accmode_t *accmode) { /* * There is no way to specify explicit "deny" rule using * file mode or POSIX.1e ACLs. */ if (*accmode & VEXPLICIT_DENY) { *accmode = 0; return (0); } /* * None of these can be translated into usual access bits. * Also, the common case for NFSv4 ACLs is to not contain * either of these bits. Caller should check for VWRITE * on the containing directory instead. */ if (*accmode & (VDELETE_CHILD | VDELETE)) return (EPERM); if (*accmode & VADMIN_PERMS) { *accmode &= ~VADMIN_PERMS; *accmode |= VADMIN; } /* * There is no way to deny VREAD_ATTRIBUTES, VREAD_ACL * or VSYNCHRONIZE using file mode or POSIX.1e ACL. */ *accmode &= ~(VSTAT_PERMS | VSYNCHRONIZE); return (0); } time_t rootfstime; /* recorded root fs time, if known */ void setrootfstime(time_t t) { rootfstime = t; } static const uint8_t vttodt_tab[ ] = { [VNON] = DT_UNKNOWN, [VREG] = DT_REG, [VDIR] = DT_DIR, [VBLK] = DT_BLK, [VCHR] = DT_CHR, [VLNK] = DT_LNK, [VSOCK] = DT_SOCK, [VFIFO] = DT_FIFO, [VBAD] = DT_UNKNOWN }; uint8_t vtype2dt(enum vtype vt) { CTASSERT(VBAD == __arraycount(vttodt_tab) - 1); return vttodt_tab[vt]; } int VFS_MOUNT(struct mount *mp, const char *a, void *b, size_t *c) { int error; KERNEL_LOCK(1, NULL); error = (*(mp->mnt_op->vfs_mount))(mp, a, b, c); KERNEL_UNLOCK_ONE(NULL); return error; } int VFS_START(struct mount *mp, int a) { int error; if ((mp->mnt_iflag & IMNT_MPSAFE) == 0) { KERNEL_LOCK(1, NULL); } error = (*(mp->mnt_op->vfs_start))(mp, a); if ((mp->mnt_iflag & IMNT_MPSAFE) == 0) { KERNEL_UNLOCK_ONE(NULL); } return error; } int VFS_UNMOUNT(struct mount *mp, int a) { int error; KERNEL_LOCK(1, NULL); error = (*(mp->mnt_op->vfs_unmount))(mp, a); KERNEL_UNLOCK_ONE(NULL); return error; } int VFS_ROOT(struct mount *mp, int lktype, struct vnode **a) { int error; if ((mp->mnt_iflag & IMNT_MPSAFE) == 0) { KERNEL_LOCK(1, NULL); } error = (*(mp->mnt_op->vfs_root))(mp, lktype, a); if ((mp->mnt_iflag & IMNT_MPSAFE) == 0) { KERNEL_UNLOCK_ONE(NULL); } return error; } int VFS_QUOTACTL(struct mount *mp, struct quotactl_args *args) { int error; if ((mp->mnt_iflag & IMNT_MPSAFE) == 0) { KERNEL_LOCK(1, NULL); } error = (*(mp->mnt_op->vfs_quotactl))(mp, args); if ((mp->mnt_iflag & IMNT_MPSAFE) == 0) { KERNEL_UNLOCK_ONE(NULL); } return error; } int VFS_STATVFS(struct mount *mp, struct statvfs *a) { int error; if ((mp->mnt_iflag & IMNT_MPSAFE) == 0) { KERNEL_LOCK(1, NULL); } error = (*(mp->mnt_op->vfs_statvfs))(mp, a); if ((mp->mnt_iflag & IMNT_MPSAFE) == 0) { KERNEL_UNLOCK_ONE(NULL); } return error; } int VFS_SYNC(struct mount *mp, int a, struct kauth_cred *b) { int error; if ((mp->mnt_iflag & IMNT_MPSAFE) == 0) { KERNEL_LOCK(1, NULL); } error = (*(mp->mnt_op->vfs_sync))(mp, a, b); if ((mp->mnt_iflag & IMNT_MPSAFE) == 0) { KERNEL_UNLOCK_ONE(NULL); } return error; } int VFS_FHTOVP(struct mount *mp, struct fid *a, int b, struct vnode **c) { int error; if ((mp->mnt_iflag & IMNT_MPSAFE) == 0) { KERNEL_LOCK(1, NULL); } error = (*(mp->mnt_op->vfs_fhtovp))(mp, a, b, c); if ((mp->mnt_iflag & IMNT_MPSAFE) == 0) { KERNEL_UNLOCK_ONE(NULL); } return error; } int VFS_VPTOFH(struct vnode *vp, struct fid *a, size_t *b) { int error; if ((vp->v_vflag & VV_MPSAFE) == 0) { KERNEL_LOCK(1, NULL); } error = (*(vp->v_mount->mnt_op->vfs_vptofh))(vp, a, b); if ((vp->v_vflag & VV_MPSAFE) == 0) { KERNEL_UNLOCK_ONE(NULL); } return error; } int VFS_SNAPSHOT(struct mount *mp, struct vnode *a, struct timespec *b) { int error; if ((mp->mnt_iflag & IMNT_MPSAFE) == 0) { KERNEL_LOCK(1, NULL); } error = (*(mp->mnt_op->vfs_snapshot))(mp, a, b); if ((mp->mnt_iflag & IMNT_MPSAFE) == 0) { KERNEL_UNLOCK_ONE(NULL); } return error; } int VFS_EXTATTRCTL(struct mount *mp, int a, struct vnode *b, int c, const char *d) { int error; KERNEL_LOCK(1, NULL); /* XXXSMP check ffs */ error = (*(mp->mnt_op->vfs_extattrctl))(mp, a, b, c, d); KERNEL_UNLOCK_ONE(NULL); /* XXX */ return error; } int VFS_SUSPENDCTL(struct mount *mp, int a) { int error; if ((mp->mnt_iflag & IMNT_MPSAFE) == 0) { KERNEL_LOCK(1, NULL); } error = (*(mp->mnt_op->vfs_suspendctl))(mp, a); if ((mp->mnt_iflag & IMNT_MPSAFE) == 0) { KERNEL_UNLOCK_ONE(NULL); } return error; } #if defined(DDB) || defined(DEBUGPRINT) static const char buf_flagbits[] = BUF_FLAGBITS; void vfs_buf_print(struct buf *bp, int full, void (*pr)(const char *, ...)) { char bf[1024]; (*pr)(" vp %p lblkno 0x%"PRIx64" blkno 0x%"PRIx64" rawblkno 0x%" PRIx64 " dev 0x%x\n", bp->b_vp, bp->b_lblkno, bp->b_blkno, bp->b_rawblkno, bp->b_dev); snprintb(bf, sizeof(bf), buf_flagbits, bp->b_flags | bp->b_oflags | bp->b_cflags); (*pr)(" error %d flags %s\n", bp->b_error, bf); (*pr)(" bufsize 0x%lx bcount 0x%lx resid 0x%lx\n", bp->b_bufsize, bp->b_bcount, bp->b_resid); (*pr)(" data %p saveaddr %p\n", bp->b_data, bp->b_saveaddr); (*pr)(" iodone %p objlock %p\n", bp->b_iodone, bp->b_objlock); } void vfs_vnode_print(struct vnode *vp, int full, void (*pr)(const char *, ...)) { uvm_object_printit(&vp->v_uobj, full, pr); (*pr)("\n"); vprint_common(vp, "", pr); if (full) { struct buf *bp; (*pr)("clean bufs:\n"); LIST_FOREACH(bp, &vp->v_cleanblkhd, b_vnbufs) { (*pr)(" bp %p\n", bp); vfs_buf_print(bp, full, pr); } (*pr)("dirty bufs:\n"); LIST_FOREACH(bp, &vp->v_dirtyblkhd, b_vnbufs) { (*pr)(" bp %p\n", bp); vfs_buf_print(bp, full, pr); } } } void vfs_vnode_lock_print(void *vlock, int full, void (*pr)(const char *, ...)) { struct mount *mp; vnode_impl_t *vip; for (mp = _mountlist_next(NULL); mp; mp = _mountlist_next(mp)) { TAILQ_FOREACH(vip, &mp->mnt_vnodelist, vi_mntvnodes) { if (&vip->vi_lock == vlock || VIMPL_TO_VNODE(vip)->v_interlock == vlock) vfs_vnode_print(VIMPL_TO_VNODE(vip), full, pr); } } } void vfs_mount_print_all(int full, void (*pr)(const char *, ...)) { struct mount *mp; for (mp = _mountlist_next(NULL); mp; mp = _mountlist_next(mp)) vfs_mount_print(mp, full, pr); } void vfs_mount_print(struct mount *mp, int full, void (*pr)(const char *, ...)) { char sbuf[256]; (*pr)("vnodecovered = %p data = %p\n", mp->mnt_vnodecovered, mp->mnt_data); (*pr)("fs_bshift %d dev_bshift = %d\n", mp->mnt_fs_bshift, mp->mnt_dev_bshift); snprintb(sbuf, sizeof(sbuf), __MNT_FLAG_BITS, mp->mnt_flag); (*pr)("flag = %s\n", sbuf); snprintb(sbuf, sizeof(sbuf), __IMNT_FLAG_BITS, mp->mnt_iflag); (*pr)("iflag = %s\n", sbuf); (*pr)("refcnt = %d updating @ %p\n", mp->mnt_refcnt, mp->mnt_updating); (*pr)("statvfs cache:\n"); (*pr)("\tbsize = %lu\n", mp->mnt_stat.f_bsize); (*pr)("\tfrsize = %lu\n", mp->mnt_stat.f_frsize); (*pr)("\tiosize = %lu\n", mp->mnt_stat.f_iosize); (*pr)("\tblocks = %"PRIu64"\n", mp->mnt_stat.f_blocks); (*pr)("\tbfree = %"PRIu64"\n", mp->mnt_stat.f_bfree); (*pr)("\tbavail = %"PRIu64"\n", mp->mnt_stat.f_bavail); (*pr)("\tbresvd = %"PRIu64"\n", mp->mnt_stat.f_bresvd); (*pr)("\tfiles = %"PRIu64"\n", mp->mnt_stat.f_files); (*pr)("\tffree = %"PRIu64"\n", mp->mnt_stat.f_ffree); (*pr)("\tfavail = %"PRIu64"\n", mp->mnt_stat.f_favail); (*pr)("\tfresvd = %"PRIu64"\n", mp->mnt_stat.f_fresvd); (*pr)("\tf_fsidx = { 0x%"PRIx32", 0x%"PRIx32" }\n", mp->mnt_stat.f_fsidx.__fsid_val[0], mp->mnt_stat.f_fsidx.__fsid_val[1]); (*pr)("\towner = %"PRIu32"\n", mp->mnt_stat.f_owner); (*pr)("\tnamemax = %lu\n", mp->mnt_stat.f_namemax); snprintb(sbuf, sizeof(sbuf), __MNT_FLAG_BITS, mp->mnt_stat.f_flag); (*pr)("\tflag = %s\n", sbuf); (*pr)("\tsyncwrites = %" PRIu64 "\n", mp->mnt_stat.f_syncwrites); (*pr)("\tasyncwrites = %" PRIu64 "\n", mp->mnt_stat.f_asyncwrites); (*pr)("\tsyncreads = %" PRIu64 "\n", mp->mnt_stat.f_syncreads); (*pr)("\tasyncreads = %" PRIu64 "\n", mp->mnt_stat.f_asyncreads); (*pr)("\tfstypename = %s\n", mp->mnt_stat.f_fstypename); (*pr)("\tmntonname = %s\n", mp->mnt_stat.f_mntonname); (*pr)("\tmntfromname = %s\n", mp->mnt_stat.f_mntfromname); { int cnt = 0; vnode_t *vp; vnode_impl_t *vip; (*pr)("locked vnodes ="); TAILQ_FOREACH(vip, &mp->mnt_vnodelist, vi_mntvnodes) { vp = VIMPL_TO_VNODE(vip); if (VOP_ISLOCKED(vp)) { if ((++cnt % 6) == 0) { (*pr)(" %p,\n\t", vp); } else { (*pr)(" %p,", vp); } } } (*pr)("\n"); } if (full) { int cnt = 0; vnode_t *vp; vnode_impl_t *vip; (*pr)("all vnodes ="); TAILQ_FOREACH(vip, &mp->mnt_vnodelist, vi_mntvnodes) { vp = VIMPL_TO_VNODE(vip); if (!TAILQ_NEXT(vip, vi_mntvnodes)) { (*pr)(" %p", vp); } else if ((++cnt % 6) == 0) { (*pr)(" %p,\n\t", vp); } else { (*pr)(" %p,", vp); } } (*pr)("\n"); } } /* * List all of the locked vnodes in the system. */ void printlockedvnodes(void); void printlockedvnodes(void) { struct mount *mp; vnode_t *vp; vnode_impl_t *vip; printf("Locked vnodes\n"); for (mp = _mountlist_next(NULL); mp; mp = _mountlist_next(mp)) { TAILQ_FOREACH(vip, &mp->mnt_vnodelist, vi_mntvnodes) { vp = VIMPL_TO_VNODE(vip); if (VOP_ISLOCKED(vp)) vprint(NULL, vp); } } } #endif /* DDB || DEBUGPRINT */ |
| 1 2327 2602 30 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 197 198 199 200 201 202 203 204 205 206 207 208 209 210 211 212 213 214 215 216 217 218 219 220 221 222 223 224 225 226 227 228 229 230 231 232 233 234 235 236 237 238 239 240 241 242 243 244 245 246 247 248 249 250 251 252 253 254 255 256 257 258 259 260 261 262 263 264 265 266 267 268 269 270 271 272 273 274 275 276 277 278 279 280 281 282 283 284 285 286 287 288 289 290 291 292 293 294 295 296 297 298 299 300 301 302 303 304 305 306 307 308 309 310 311 312 313 314 315 316 317 318 319 320 321 322 323 324 325 326 327 328 329 330 331 332 333 334 335 336 337 338 339 340 341 342 343 344 345 346 347 348 349 350 351 352 353 354 355 356 357 358 359 360 361 362 363 364 365 366 367 368 369 370 371 372 373 374 375 376 377 378 379 380 381 382 383 384 385 386 387 388 389 390 391 392 393 394 395 396 397 398 399 400 401 402 403 404 405 406 407 408 409 410 411 412 413 414 415 416 417 418 419 420 421 422 423 424 425 426 427 428 429 430 431 432 433 434 435 436 437 438 439 440 441 442 443 444 445 446 447 448 449 450 451 452 453 454 455 456 457 458 459 460 461 462 463 464 465 466 467 468 469 470 471 472 473 474 475 476 477 478 479 480 481 482 483 484 485 486 487 488 489 490 491 492 493 494 495 496 497 498 499 500 501 502 503 504 505 506 507 508 509 510 511 512 513 514 515 516 517 518 519 520 521 522 523 524 525 526 527 528 529 530 531 532 533 534 535 536 537 538 539 540 541 542 543 544 545 546 547 548 549 550 551 552 553 554 555 556 557 558 559 560 561 562 563 564 565 566 567 568 569 570 571 572 573 574 575 576 577 578 579 580 581 582 583 584 585 586 587 588 589 590 591 592 593 594 595 596 597 598 599 600 601 602 603 604 605 606 607 608 609 610 611 612 613 614 615 616 617 618 619 620 621 622 623 624 625 626 627 628 629 630 631 632 633 634 635 636 637 638 639 640 641 642 643 644 645 646 647 648 649 650 651 652 653 654 655 656 657 658 659 660 661 662 663 | /* $NetBSD: cpufunc.h,v 1.42 2020/10/24 07:14:29 mgorny Exp $ */ /* * Copyright (c) 1998, 2007, 2019 The NetBSD Foundation, Inc. * All rights reserved. * * This code is derived from software contributed to The NetBSD Foundation * by Charles M. Hannum, and by Andrew Doran. * * Redistribution and use in source and binary forms, with or without * modification, are permitted provided that the following conditions * are met: * 1. Redistributions of source code must retain the above copyright * notice, this list of conditions and the following disclaimer. * 2. Redistributions in binary form must reproduce the above copyright * notice, this list of conditions and the following disclaimer in the * documentation and/or other materials provided with the distribution. * * THIS SOFTWARE IS PROVIDED BY THE NETBSD FOUNDATION, INC. AND CONTRIBUTORS * ``AS IS'' AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED * TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR * PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE FOUNDATION OR CONTRIBUTORS * BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR * CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF * SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS * INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN * CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) * ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE * POSSIBILITY OF SUCH DAMAGE. */ #ifndef _X86_CPUFUNC_H_ #define _X86_CPUFUNC_H_ /* * Functions to provide access to x86-specific instructions. */ #include <sys/cdefs.h> #include <sys/types.h> #include <machine/segments.h> #include <machine/specialreg.h> #ifdef _KERNEL #if defined(_KERNEL_OPT) #include "opt_xen.h" #endif static inline void x86_pause(void) { __asm volatile ("pause"); } void x86_lfence(void); void x86_sfence(void); void x86_mfence(void); void x86_flush(void); void x86_hlt(void); void x86_stihlt(void); void tlbflush(void); void tlbflushg(void); void invlpg(vaddr_t); void wbinvd(void); void breakpoint(void); #define INVPCID_ADDRESS 0 #define INVPCID_CONTEXT 1 #define INVPCID_ALL 2 #define INVPCID_ALL_NONGLOBAL 3 static inline void invpcid(register_t op, uint64_t pcid, vaddr_t va) { struct { uint64_t pcid; uint64_t addr; } desc = { .pcid = pcid, .addr = va }; __asm volatile ( "invpcid %[desc],%[op]" : : [desc] "m" (desc), [op] "r" (op) : "memory" ); } extern uint64_t (*rdtsc)(void); #define _SERIALIZE_lfence __asm volatile ("lfence") #define _SERIALIZE_mfence __asm volatile ("mfence") #define _SERIALIZE_cpuid __asm volatile ("xor %%eax, %%eax;cpuid" ::: \ "eax", "ebx", "ecx", "edx"); #define RDTSCFUNC(fence) \ static inline uint64_t \ rdtsc_##fence(void) \ { \ uint32_t low, high; \ \ _SERIALIZE_##fence; \ __asm volatile ( \ "rdtsc" \ : "=a" (low), "=d" (high) \ : \ ); \ \ return (low | ((uint64_t)high << 32)); \ } RDTSCFUNC(lfence) RDTSCFUNC(mfence) RDTSCFUNC(cpuid) #undef _SERIALIZE_LFENCE #undef _SERIALIZE_MFENCE #undef _SERIALIZE_CPUID #ifndef XENPV struct x86_hotpatch_source { uint8_t *saddr; uint8_t *eaddr; }; struct x86_hotpatch_descriptor { uint8_t name; uint8_t nsrc; const struct x86_hotpatch_source *srcs[]; }; void x86_hotpatch(uint8_t, uint8_t); void x86_patch(bool); #endif void x86_monitor(const void *, uint32_t, uint32_t); void x86_mwait(uint32_t, uint32_t); static inline void x86_cpuid2(uint32_t eax, uint32_t ecx, uint32_t *regs) { uint32_t ebx, edx; __asm volatile ( "cpuid" : "=a" (eax), "=b" (ebx), "=c" (ecx), "=d" (edx) : "a" (eax), "c" (ecx) ); regs[0] = eax; regs[1] = ebx; regs[2] = ecx; regs[3] = edx; } #define x86_cpuid(a,b) x86_cpuid2((a), 0, (b)) /* -------------------------------------------------------------------------- */ void lidt(struct region_descriptor *); void lldt(u_short); void ltr(u_short); static inline uint16_t x86_getss(void) { uint16_t val; __asm volatile ( "mov %%ss,%[val]" : [val] "=r" (val) : ); return val; } static inline void setds(uint16_t val) { __asm volatile ( "mov %[val],%%ds" : : [val] "r" (val) ); } static inline void setes(uint16_t val) { __asm volatile ( "mov %[val],%%es" : : [val] "r" (val) ); } static inline void setfs(uint16_t val) { __asm volatile ( "mov %[val],%%fs" : : [val] "r" (val) ); } void setusergs(int); /* -------------------------------------------------------------------------- */ #define FUNC_CR(crnum) \ static inline void lcr##crnum(register_t val) \ { \ __asm volatile ( \ "mov %[val],%%cr" #crnum \ : \ : [val] "r" (val) \ : "memory" \ ); \ } \ static inline register_t rcr##crnum(void) \ { \ register_t val; \ __asm volatile ( \ "mov %%cr" #crnum ",%[val]" \ : [val] "=r" (val) \ : \ ); \ return val; \ } #define PROTO_CR(crnum) \ void lcr##crnum(register_t); \ register_t rcr##crnum(void); #ifndef XENPV FUNC_CR(0) FUNC_CR(2) FUNC_CR(3) #else PROTO_CR(0) PROTO_CR(2) PROTO_CR(3) #endif FUNC_CR(4) FUNC_CR(8) /* -------------------------------------------------------------------------- */ #define FUNC_DR(drnum) \ static inline void ldr##drnum(register_t val) \ { \ __asm volatile ( \ "mov %[val],%%dr" #drnum \ : \ : [val] "r" (val) \ ); \ } \ static inline register_t rdr##drnum(void) \ { \ register_t val; \ __asm volatile ( \ "mov %%dr" #drnum ",%[val]" \ : [val] "=r" (val) \ : \ ); \ return val; \ } #define PROTO_DR(drnum) \ register_t rdr##drnum(void); \ void ldr##drnum(register_t); #ifndef XENPV FUNC_DR(0) FUNC_DR(1) FUNC_DR(2) FUNC_DR(3) FUNC_DR(6) FUNC_DR(7) #else PROTO_DR(0) PROTO_DR(1) PROTO_DR(2) PROTO_DR(3) PROTO_DR(6) PROTO_DR(7) #endif /* -------------------------------------------------------------------------- */ union savefpu; static inline void fninit(void) { __asm volatile ("fninit" ::: "memory"); } static inline void fnclex(void) { __asm volatile ("fnclex"); } static inline void fnstcw(uint16_t *val) { __asm volatile ( "fnstcw %[val]" : [val] "=m" (*val) : ); } static inline void fnstsw(uint16_t *val) { __asm volatile ( "fnstsw %[val]" : [val] "=m" (*val) : ); } static inline void clts(void) { __asm volatile ("clts" ::: "memory"); } void stts(void); static inline void x86_stmxcsr(uint32_t *val) { __asm volatile ( "stmxcsr %[val]" : [val] "=m" (*val) : ); } static inline void x86_ldmxcsr(uint32_t *val) { __asm volatile ( "ldmxcsr %[val]" : : [val] "m" (*val) ); } void fldummy(void); static inline uint64_t rdxcr(uint32_t xcr) { uint32_t low, high; __asm volatile ( "xgetbv" : "=a" (low), "=d" (high) : "c" (xcr) ); return (low | ((uint64_t)high << 32)); } static inline void wrxcr(uint32_t xcr, uint64_t val) { uint32_t low, high; low = val; high = val >> 32; __asm volatile ( "xsetbv" : : "a" (low), "d" (high), "c" (xcr) ); } static inline void fnsave(void *addr) { uint8_t *area = addr; __asm volatile ( "fnsave %[area]" : [area] "=m" (*area) : : "memory" ); } static inline void frstor(const void *addr) { const uint8_t *area = addr; __asm volatile ( "frstor %[area]" : : [area] "m" (*area) : "memory" ); } static inline void fxsave(void *addr) { uint8_t *area = addr; __asm volatile ( "fxsave %[area]" : [area] "=m" (*area) : : "memory" ); } static inline void fxrstor(const void *addr) { const uint8_t *area = addr; __asm volatile ( "fxrstor %[area]" : : [area] "m" (*area) : "memory" ); } static inline void xsave(void *addr, uint64_t mask) { uint8_t *area = addr; uint32_t low, high; low = mask; high = mask >> 32; __asm volatile ( "xsave %[area]" : [area] "=m" (*area) : "a" (low), "d" (high) : "memory" ); } static inline void xsaveopt(void *addr, uint64_t mask) { uint8_t *area = addr; uint32_t low, high; low = mask; high = mask >> 32; __asm volatile ( "xsaveopt %[area]" : [area] "=m" (*area) : "a" (low), "d" (high) : "memory" ); } static inline void xrstor(const void *addr, uint64_t mask) { const uint8_t *area = addr; uint32_t low, high; low = mask; high = mask >> 32; __asm volatile ( "xrstor %[area]" : : [area] "m" (*area), "a" (low), "d" (high) : "memory" ); } #ifdef __x86_64__ static inline void fxsave64(void *addr) { uint8_t *area = addr; __asm volatile ( "fxsave64 %[area]" : [area] "=m" (*area) : : "memory" ); } static inline void fxrstor64(const void *addr) { const uint8_t *area = addr; __asm volatile ( "fxrstor64 %[area]" : : [area] "m" (*area) : "memory" ); } static inline void xsave64(void *addr, uint64_t mask) { uint8_t *area = addr; uint32_t low, high; low = mask; high = mask >> 32; __asm volatile ( "xsave64 %[area]" : [area] "=m" (*area) : "a" (low), "d" (high) : "memory" ); } static inline void xsaveopt64(void *addr, uint64_t mask) { uint8_t *area = addr; uint32_t low, high; low = mask; high = mask >> 32; __asm volatile ( "xsaveopt64 %[area]" : [area] "=m" (*area) : "a" (low), "d" (high) : "memory" ); } static inline void xrstor64(const void *addr, uint64_t mask) { const uint8_t *area = addr; uint32_t low, high; low = mask; high = mask >> 32; __asm volatile ( "xrstor64 %[area]" : : [area] "m" (*area), "a" (low), "d" (high) : "memory" ); } #endif /* -------------------------------------------------------------------------- */ #ifdef XENPV void x86_disable_intr(void); void x86_enable_intr(void); #else static inline void x86_disable_intr(void) { __asm volatile ("cli" ::: "memory"); } static inline void x86_enable_intr(void) { __asm volatile ("sti" ::: "memory"); } #endif /* XENPV */ /* Use read_psl, write_psl when saving and restoring interrupt state. */ u_long x86_read_psl(void); void x86_write_psl(u_long); /* Use read_flags, write_flags to adjust other members of %eflags. */ u_long x86_read_flags(void); void x86_write_flags(u_long); void x86_reset(void); /* -------------------------------------------------------------------------- */ /* * Some of the undocumented AMD64 MSRs need a 'passcode' to access. * See LinuxBIOSv2: src/cpu/amd/model_fxx/model_fxx_init.c */ #define OPTERON_MSR_PASSCODE 0x9c5a203aU static inline uint64_t rdmsr(u_int msr) { uint32_t low, high; __asm volatile ( "rdmsr" : "=a" (low), "=d" (high) : "c" (msr) ); return (low | ((uint64_t)high << 32)); } static inline uint64_t rdmsr_locked(u_int msr) { uint32_t low, high, pass = OPTERON_MSR_PASSCODE; __asm volatile ( "rdmsr" : "=a" (low), "=d" (high) : "c" (msr), "D" (pass) ); return (low | ((uint64_t)high << 32)); } int rdmsr_safe(u_int, uint64_t *); static inline void wrmsr(u_int msr, uint64_t val) { uint32_t low, high; low = val; high = val >> 32; __asm volatile ( "wrmsr" : : "a" (low), "d" (high), "c" (msr) : "memory" ); } static inline void wrmsr_locked(u_int msr, uint64_t val) { uint32_t low, high, pass = OPTERON_MSR_PASSCODE; low = val; high = val >> 32; __asm volatile ( "wrmsr" : : "a" (low), "d" (high), "c" (msr), "D" (pass) : "memory" ); } #endif /* _KERNEL */ #endif /* !_X86_CPUFUNC_H_ */ |
| 33 28 6 51 46 5 5 5 5 51 50 50 51 50 50 49 50 50 33 18 51 13 51 5 5 48 3 47 2 1 2 1 5 47 47 47 47 3 47 29 28 28 1 1 1 11 35 46 46 10 46 24 9 34 34 34 23 11 33 34 48 13 49 5 45 3 14 33 232 232 230 8 1 2 14 129 5 4 1 1 2 2 2 1 2 2 2 1 2 2 2 1 1 1 1 1 2 2 1 2 7 1 1 1 2 5 2 39 1 1 2 2 1 1 1 1 1 2 2 2 1 1 1 1 6 6 1 1 1 1 1 1 18 1 18 2 10 5 5 41 1 2 3 2 1 2 1 2 1 1 2 21 21 16 1 17 19 17 20 1 17 18 18 19 17 20 25 4 2 12 2 5 9 1 8 4 8 3 44 26 27 3 2 1 4 3 4 1 6 8 1 7 3 7 4 10 3 4 4 5 3 11 8 6 17 2 1 1 1 31 29 2 1 21 21 9 2 3 4 57 43 2 11 2 1 10 4 3 5 6 2 2 1 1 1 1 2 1 1 1 1 1 4 1 1 1 2 1 2 1 2 1 1 2 6 5 1 1 1 1 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 197 198 199 200 201 202 203 204 205 206 207 208 209 210 211 212 213 214 215 216 217 218 219 220 221 222 223 224 225 226 227 228 229 230 231 232 233 234 235 236 237 238 239 240 241 242 243 244 245 246 247 248 249 250 251 252 253 254 255 256 257 258 259 260 261 262 263 264 265 266 267 268 269 270 271 272 273 274 275 276 277 278 279 280 281 282 283 284 285 286 287 288 289 290 291 292 293 294 295 296 297 298 299 300 301 302 303 304 305 306 307 308 309 310 311 312 313 314 315 316 317 318 319 320 321 322 323 324 325 326 327 328 329 330 331 332 333 334 335 336 337 338 339 340 341 342 343 344 345 346 347 348 349 350 351 352 353 354 355 356 357 358 359 360 361 362 363 364 365 366 367 368 369 370 371 372 373 374 375 376 377 378 379 380 381 382 383 384 385 386 387 388 389 390 391 392 393 394 395 396 397 398 399 400 401 402 403 404 405 406 407 408 409 410 411 412 413 414 415 416 417 418 419 420 421 422 423 424 425 426 427 428 429 430 431 432 433 434 435 436 437 438 439 440 441 442 443 444 445 446 447 448 449 450 451 452 453 454 455 456 457 458 459 460 461 462 463 464 465 466 467 468 469 470 471 472 473 474 475 476 477 478 479 480 481 482 483 484 485 486 487 488 489 490 491 492 493 494 495 496 497 498 499 500 501 502 503 504 505 506 507 508 509 510 511 512 513 514 515 516 517 518 519 520 521 522 523 524 525 526 527 528 529 530 531 532 533 534 535 536 537 538 539 540 541 542 543 544 545 546 547 548 549 550 551 552 553 554 555 556 557 558 559 560 561 562 563 564 565 566 567 568 569 570 571 572 573 574 575 576 577 578 579 580 581 582 583 584 585 586 587 588 589 590 591 592 593 594 595 596 597 598 599 600 601 602 603 604 605 606 607 608 609 610 611 612 613 614 615 616 617 618 619 620 621 622 623 624 625 626 627 628 629 630 631 632 633 634 635 636 637 638 639 640 641 642 643 644 645 646 647 648 649 650 651 652 653 654 655 656 657 658 659 660 661 662 663 664 665 666 667 668 669 670 671 672 673 674 675 676 677 678 679 680 681 682 683 684 685 686 687 688 689 690 691 692 693 694 695 696 697 698 699 700 701 702 703 704 705 706 707 708 709 710 711 712 713 714 715 716 717 718 719 720 721 722 723 724 725 726 727 728 729 730 731 732 733 734 735 736 737 738 739 740 741 742 743 744 745 746 747 748 749 750 751 752 753 754 755 756 757 758 759 760 761 762 763 764 765 766 767 768 769 770 771 772 773 774 775 776 777 778 779 780 781 782 783 784 785 786 787 788 789 790 791 792 793 794 795 796 797 798 799 800 801 802 803 804 805 806 807 808 809 810 811 812 813 814 815 816 817 818 819 820 821 822 823 824 825 826 827 828 829 830 831 832 833 834 835 836 837 838 839 840 841 842 843 844 845 846 847 848 849 850 851 852 853 854 855 856 857 858 859 860 861 862 863 864 865 866 867 868 869 870 871 872 873 874 875 876 877 878 879 880 881 882 883 884 885 886 887 888 889 890 891 892 893 894 895 896 897 898 899 900 901 902 903 904 905 906 907 908 909 910 911 912 913 914 915 916 917 918 919 920 921 922 923 924 925 926 927 928 929 930 931 932 933 934 935 936 937 938 939 940 941 942 943 944 945 946 947 948 949 950 951 952 953 954 955 956 957 958 959 960 961 962 963 964 965 966 967 968 969 970 971 972 973 974 975 976 977 978 979 980 981 982 983 984 985 986 987 988 989 990 991 992 993 994 995 996 997 998 999 1000 1001 1002 1003 1004 1005 1006 1007 1008 1009 1010 1011 1012 1013 1014 1015 1016 1017 1018 1019 1020 1021 1022 1023 1024 1025 1026 1027 1028 1029 1030 1031 1032 1033 1034 1035 1036 1037 1038 1039 1040 1041 1042 1043 1044 1045 1046 1047 1048 1049 1050 1051 1052 1053 1054 1055 1056 1057 1058 1059 1060 1061 1062 1063 1064 1065 1066 1067 1068 1069 1070 1071 1072 1073 1074 1075 1076 1077 1078 1079 1080 1081 1082 1083 1084 1085 1086 1087 1088 1089 1090 1091 1092 1093 1094 1095 1096 1097 1098 1099 1100 1101 1102 1103 1104 1105 1106 1107 1108 1109 1110 1111 1112 1113 1114 1115 1116 1117 1118 1119 1120 1121 1122 1123 1124 1125 1126 1127 1128 1129 1130 1131 1132 1133 1134 1135 1136 1137 1138 1139 1140 1141 1142 1143 1144 1145 1146 1147 1148 1149 1150 1151 1152 1153 1154 1155 1156 1157 1158 1159 1160 1161 1162 1163 1164 1165 1166 1167 1168 1169 1170 1171 1172 1173 1174 1175 1176 1177 1178 1179 1180 1181 1182 1183 1184 1185 1186 1187 1188 1189 1190 1191 1192 1193 1194 1195 1196 1197 1198 1199 1200 1201 1202 1203 1204 1205 1206 1207 1208 1209 1210 1211 1212 1213 1214 1215 1216 1217 1218 1219 1220 1221 1222 1223 1224 1225 1226 1227 1228 1229 1230 1231 1232 1233 1234 1235 1236 1237 1238 1239 1240 1241 1242 1243 1244 1245 1246 1247 1248 1249 1250 1251 1252 1253 1254 1255 1256 1257 1258 1259 1260 1261 1262 1263 1264 1265 1266 1267 1268 1269 1270 1271 1272 1273 1274 1275 1276 1277 1278 1279 1280 1281 1282 1283 1284 1285 1286 1287 1288 1289 1290 1291 1292 1293 1294 1295 1296 1297 1298 1299 1300 1301 1302 1303 1304 1305 1306 1307 1308 1309 1310 1311 1312 1313 1314 1315 1316 1317 1318 1319 1320 1321 1322 1323 1324 1325 1326 1327 1328 1329 1330 1331 1332 1333 1334 1335 1336 1337 1338 1339 1340 1341 1342 1343 1344 1345 1346 1347 1348 1349 1350 1351 1352 1353 1354 1355 1356 1357 1358 1359 1360 1361 1362 1363 1364 1365 1366 1367 1368 1369 1370 1371 1372 1373 1374 1375 1376 1377 1378 1379 1380 1381 1382 1383 1384 1385 1386 1387 1388 1389 1390 1391 1392 1393 1394 1395 1396 1397 1398 1399 1400 1401 1402 1403 1404 1405 1406 1407 1408 1409 1410 1411 1412 1413 1414 1415 1416 1417 1418 1419 1420 1421 1422 1423 1424 1425 1426 1427 1428 1429 1430 1431 1432 1433 1434 1435 1436 1437 1438 1439 1440 1441 1442 1443 1444 1445 1446 1447 1448 1449 1450 1451 1452 1453 1454 1455 1456 1457 1458 1459 1460 1461 1462 1463 1464 1465 1466 1467 1468 1469 1470 1471 1472 1473 1474 1475 1476 1477 1478 1479 1480 1481 1482 1483 1484 1485 1486 1487 1488 1489 1490 1491 1492 1493 1494 1495 1496 1497 1498 1499 1500 1501 1502 1503 1504 1505 1506 1507 1508 1509 1510 1511 1512 1513 1514 1515 1516 1517 1518 1519 1520 1521 1522 1523 1524 1525 1526 1527 1528 1529 1530 1531 1532 1533 1534 1535 1536 1537 1538 1539 1540 1541 1542 1543 1544 1545 1546 1547 1548 1549 1550 1551 1552 1553 1554 1555 1556 1557 1558 1559 1560 1561 1562 1563 1564 1565 1566 1567 1568 1569 1570 1571 1572 1573 1574 1575 1576 1577 1578 1579 1580 1581 1582 1583 1584 1585 1586 1587 1588 1589 1590 1591 1592 1593 1594 1595 1596 1597 1598 1599 1600 1601 1602 1603 1604 1605 1606 1607 1608 1609 1610 1611 1612 1613 1614 1615 1616 1617 1618 1619 1620 1621 1622 1623 1624 1625 1626 1627 1628 1629 1630 1631 1632 1633 1634 1635 1636 1637 1638 1639 1640 1641 1642 1643 1644 1645 1646 1647 1648 1649 1650 1651 1652 1653 1654 1655 1656 1657 1658 1659 1660 1661 1662 1663 1664 1665 1666 1667 1668 1669 1670 1671 1672 1673 1674 1675 1676 1677 1678 1679 1680 1681 1682 1683 1684 1685 1686 1687 1688 1689 1690 1691 1692 1693 1694 1695 1696 1697 1698 1699 1700 1701 1702 1703 1704 1705 1706 1707 1708 1709 1710 1711 1712 1713 1714 1715 1716 1717 1718 1719 1720 1721 1722 1723 1724 1725 1726 1727 1728 1729 1730 1731 1732 1733 1734 1735 1736 1737 1738 1739 1740 1741 1742 1743 1744 1745 1746 1747 1748 1749 1750 1751 1752 1753 1754 1755 1756 1757 1758 1759 1760 1761 1762 1763 1764 1765 1766 1767 1768 1769 1770 1771 1772 1773 1774 1775 1776 1777 1778 1779 1780 1781 1782 1783 1784 1785 1786 1787 1788 1789 1790 1791 1792 1793 1794 1795 1796 1797 1798 1799 1800 1801 1802 1803 1804 1805 1806 1807 1808 1809 1810 1811 1812 1813 1814 1815 1816 1817 1818 1819 1820 1821 1822 1823 1824 1825 1826 1827 1828 1829 1830 1831 1832 1833 1834 1835 1836 1837 1838 1839 1840 1841 1842 1843 1844 1845 1846 1847 1848 1849 1850 1851 1852 1853 1854 1855 1856 1857 1858 1859 1860 1861 1862 1863 1864 1865 1866 1867 1868 1869 1870 1871 1872 1873 1874 1875 1876 1877 1878 1879 1880 1881 1882 1883 1884 1885 1886 1887 1888 1889 1890 1891 1892 1893 1894 1895 1896 1897 1898 1899 1900 1901 1902 1903 1904 1905 1906 1907 1908 1909 1910 1911 1912 1913 1914 1915 1916 1917 1918 1919 1920 1921 1922 1923 1924 1925 1926 1927 1928 1929 1930 1931 1932 1933 1934 1935 1936 1937 1938 1939 1940 1941 1942 1943 1944 1945 1946 1947 1948 1949 1950 1951 1952 1953 1954 1955 1956 1957 1958 1959 1960 1961 1962 1963 1964 1965 1966 1967 1968 1969 1970 1971 1972 1973 1974 1975 1976 1977 1978 1979 1980 1981 1982 1983 1984 1985 1986 1987 1988 1989 1990 1991 1992 1993 1994 1995 1996 1997 1998 1999 2000 2001 2002 2003 2004 2005 2006 2007 2008 2009 2010 2011 2012 2013 2014 2015 2016 2017 2018 2019 2020 2021 2022 2023 2024 2025 2026 2027 2028 2029 2030 2031 2032 2033 2034 2035 2036 2037 2038 2039 2040 2041 2042 2043 2044 2045 2046 2047 2048 2049 2050 2051 2052 2053 2054 2055 2056 2057 2058 2059 2060 2061 2062 2063 2064 2065 2066 2067 2068 2069 2070 2071 2072 2073 2074 2075 2076 2077 2078 2079 2080 2081 2082 2083 2084 2085 2086 2087 2088 2089 2090 2091 2092 2093 2094 2095 2096 2097 2098 2099 2100 2101 2102 2103 2104 2105 2106 2107 2108 2109 2110 2111 2112 2113 2114 2115 2116 2117 2118 2119 2120 2121 2122 2123 2124 2125 2126 2127 2128 2129 2130 2131 2132 2133 2134 2135 2136 2137 2138 2139 2140 2141 2142 2143 2144 2145 2146 2147 2148 2149 2150 2151 2152 2153 2154 2155 2156 2157 2158 2159 2160 2161 2162 2163 2164 2165 2166 2167 2168 2169 2170 2171 2172 2173 2174 2175 2176 2177 2178 2179 2180 2181 2182 2183 2184 2185 2186 2187 2188 2189 2190 2191 2192 2193 2194 2195 2196 2197 2198 2199 2200 2201 2202 2203 2204 2205 2206 2207 2208 2209 2210 2211 2212 2213 2214 2215 2216 2217 2218 2219 2220 2221 2222 2223 2224 2225 2226 2227 2228 2229 2230 2231 2232 2233 2234 2235 2236 2237 2238 2239 2240 2241 2242 2243 2244 2245 2246 2247 2248 2249 2250 2251 2252 2253 2254 2255 2256 2257 2258 2259 2260 2261 2262 2263 2264 2265 2266 2267 2268 2269 2270 2271 2272 2273 2274 2275 2276 2277 2278 2279 2280 2281 2282 2283 2284 2285 2286 2287 2288 2289 2290 2291 2292 2293 2294 2295 2296 2297 2298 2299 2300 2301 2302 2303 2304 2305 2306 2307 2308 2309 2310 2311 2312 2313 2314 2315 2316 2317 2318 2319 2320 2321 2322 2323 2324 2325 2326 2327 2328 2329 2330 2331 2332 2333 2334 2335 2336 2337 2338 2339 2340 2341 2342 2343 2344 2345 2346 2347 2348 2349 2350 2351 2352 2353 2354 2355 2356 2357 2358 2359 2360 2361 2362 2363 2364 2365 2366 2367 2368 2369 2370 2371 2372 2373 2374 2375 2376 2377 2378 2379 2380 2381 2382 2383 2384 2385 2386 2387 2388 2389 2390 2391 2392 2393 2394 2395 2396 2397 2398 2399 2400 2401 2402 2403 2404 2405 2406 2407 2408 2409 2410 2411 2412 2413 2414 2415 2416 2417 2418 2419 2420 2421 2422 2423 2424 2425 2426 2427 2428 2429 2430 2431 2432 2433 2434 2435 2436 2437 2438 2439 2440 2441 2442 2443 2444 2445 2446 2447 2448 2449 2450 2451 2452 2453 2454 2455 2456 2457 2458 2459 2460 2461 2462 2463 2464 2465 2466 2467 2468 2469 2470 2471 2472 2473 2474 2475 2476 2477 2478 2479 2480 2481 2482 2483 2484 2485 2486 2487 2488 2489 2490 2491 2492 2493 2494 2495 2496 2497 2498 2499 2500 2501 2502 2503 2504 2505 2506 2507 2508 2509 2510 2511 2512 2513 2514 2515 2516 2517 2518 2519 2520 2521 2522 2523 2524 2525 2526 2527 2528 2529 2530 2531 2532 2533 2534 2535 2536 2537 2538 2539 2540 2541 2542 2543 2544 2545 2546 2547 2548 2549 2550 2551 2552 2553 2554 2555 2556 2557 2558 2559 2560 2561 2562 2563 2564 2565 2566 2567 2568 2569 2570 2571 2572 2573 2574 2575 2576 2577 2578 2579 2580 2581 2582 2583 2584 2585 2586 2587 2588 2589 2590 2591 2592 2593 2594 2595 2596 2597 2598 2599 2600 2601 2602 2603 2604 2605 2606 2607 2608 2609 2610 2611 2612 2613 2614 2615 2616 2617 2618 2619 2620 2621 2622 2623 2624 2625 2626 2627 2628 2629 2630 2631 2632 2633 2634 2635 2636 2637 2638 2639 2640 2641 2642 2643 2644 2645 2646 2647 2648 2649 2650 2651 2652 2653 2654 2655 2656 2657 2658 2659 2660 2661 2662 2663 2664 2665 2666 2667 2668 2669 2670 2671 2672 2673 2674 2675 2676 2677 2678 2679 2680 2681 2682 2683 2684 2685 2686 2687 2688 2689 2690 2691 2692 2693 2694 2695 2696 2697 2698 2699 2700 2701 2702 2703 2704 2705 2706 2707 2708 2709 2710 2711 2712 2713 2714 2715 2716 2717 2718 2719 2720 2721 2722 2723 2724 2725 2726 2727 2728 2729 2730 2731 2732 2733 2734 2735 2736 2737 2738 2739 2740 2741 2742 2743 2744 2745 2746 2747 2748 2749 2750 2751 2752 2753 2754 2755 2756 2757 2758 2759 2760 2761 2762 2763 2764 2765 2766 2767 2768 2769 2770 2771 2772 2773 2774 2775 2776 2777 2778 2779 2780 2781 2782 2783 2784 2785 2786 2787 2788 2789 2790 2791 2792 2793 2794 2795 2796 2797 2798 2799 2800 2801 2802 2803 2804 2805 2806 2807 2808 2809 2810 2811 2812 2813 2814 2815 2816 2817 2818 2819 2820 2821 2822 2823 2824 2825 2826 2827 2828 2829 2830 2831 2832 2833 2834 2835 2836 2837 2838 2839 2840 2841 2842 2843 2844 2845 2846 2847 2848 2849 2850 2851 2852 2853 2854 2855 2856 2857 2858 2859 2860 2861 2862 2863 2864 2865 2866 2867 2868 2869 2870 2871 2872 2873 2874 2875 2876 2877 2878 2879 2880 2881 2882 2883 2884 2885 2886 2887 2888 2889 2890 2891 2892 2893 2894 2895 2896 2897 2898 2899 2900 2901 2902 2903 2904 2905 2906 2907 2908 2909 2910 2911 2912 2913 2914 2915 2916 2917 2918 2919 2920 2921 2922 2923 2924 2925 2926 2927 2928 2929 2930 2931 2932 2933 2934 2935 2936 2937 2938 2939 2940 2941 2942 2943 2944 2945 2946 2947 2948 2949 2950 2951 2952 2953 2954 2955 2956 2957 2958 2959 2960 2961 2962 2963 2964 2965 2966 2967 2968 2969 2970 2971 2972 2973 2974 2975 2976 2977 2978 2979 2980 2981 2982 2983 2984 2985 2986 2987 2988 2989 2990 2991 2992 2993 2994 2995 2996 2997 2998 2999 3000 3001 3002 3003 3004 3005 3006 3007 3008 3009 3010 3011 3012 3013 3014 3015 3016 3017 3018 3019 3020 3021 3022 3023 3024 3025 3026 3027 3028 3029 3030 3031 3032 3033 3034 3035 3036 3037 3038 3039 3040 3041 3042 3043 3044 3045 3046 3047 3048 3049 3050 3051 3052 3053 3054 3055 3056 3057 3058 3059 3060 3061 3062 3063 3064 3065 3066 3067 3068 3069 3070 3071 3072 3073 3074 3075 3076 3077 3078 3079 3080 3081 3082 3083 3084 3085 3086 3087 3088 3089 3090 3091 3092 3093 3094 3095 3096 3097 3098 3099 3100 3101 3102 3103 3104 3105 3106 3107 3108 3109 3110 3111 3112 3113 3114 3115 3116 3117 3118 3119 3120 3121 3122 3123 3124 3125 3126 3127 3128 3129 3130 3131 3132 3133 3134 3135 3136 3137 3138 3139 3140 3141 3142 3143 3144 3145 3146 3147 3148 3149 3150 3151 3152 3153 3154 3155 3156 3157 3158 3159 3160 3161 3162 3163 3164 3165 3166 3167 3168 3169 3170 3171 3172 3173 3174 3175 3176 3177 3178 3179 3180 3181 3182 3183 3184 3185 3186 3187 3188 3189 3190 3191 3192 3193 3194 3195 3196 3197 3198 3199 3200 3201 3202 3203 3204 3205 3206 3207 3208 3209 3210 3211 3212 3213 3214 3215 3216 3217 3218 3219 3220 3221 3222 3223 3224 3225 3226 3227 3228 3229 3230 3231 3232 3233 3234 3235 3236 3237 3238 3239 3240 3241 3242 3243 3244 3245 3246 3247 3248 3249 3250 3251 3252 3253 3254 3255 3256 3257 3258 3259 3260 3261 3262 3263 3264 3265 3266 3267 3268 3269 3270 3271 3272 3273 3274 3275 3276 3277 3278 3279 3280 3281 3282 3283 3284 3285 3286 3287 3288 3289 3290 3291 3292 3293 3294 3295 3296 3297 3298 3299 3300 3301 3302 3303 3304 3305 3306 3307 3308 3309 3310 3311 3312 3313 3314 3315 3316 3317 3318 3319 3320 3321 3322 3323 3324 3325 3326 3327 3328 3329 3330 3331 3332 3333 3334 3335 3336 3337 3338 3339 3340 3341 3342 3343 3344 3345 3346 3347 3348 3349 3350 3351 3352 3353 3354 3355 3356 3357 3358 | /* $NetBSD: ip6_output.c,v 1.229 2021/09/21 15:07:43 christos Exp $ */ /* $KAME: ip6_output.c,v 1.172 2001/03/25 09:55:56 itojun Exp $ */ /* * Copyright (C) 1995, 1996, 1997, and 1998 WIDE Project. * All rights reserved. * * Redistribution and use in source and binary forms, with or without * modification, are permitted provided that the following conditions * are met: * 1. Redistributions of source code must retain the above copyright * notice, this list of conditions and the following disclaimer. * 2. Redistributions in binary form must reproduce the above copyright * notice, this list of conditions and the following disclaimer in the * documentation and/or other materials provided with the distribution. * 3. Neither the name of the project nor the names of its contributors * may be used to endorse or promote products derived from this software * without specific prior written permission. * * THIS SOFTWARE IS PROVIDED BY THE PROJECT AND CONTRIBUTORS ``AS IS'' AND * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE * ARE DISCLAIMED. IN NO EVENT SHALL THE PROJECT OR CONTRIBUTORS BE LIABLE * FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL * DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS * OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) * HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT * LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY * OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF * SUCH DAMAGE. */ /* * Copyright (c) 1982, 1986, 1988, 1990, 1993 * The Regents of the University of California. All rights reserved. * * Redistribution and use in source and binary forms, with or without * modification, are permitted provided that the following conditions * are met: * 1. Redistributions of source code must retain the above copyright * notice, this list of conditions and the following disclaimer. * 2. Redistributions in binary form must reproduce the above copyright * notice, this list of conditions and the following disclaimer in the * documentation and/or other materials provided with the distribution. * 3. Neither the name of the University nor the names of its contributors * may be used to endorse or promote products derived from this software * without specific prior written permission. * * THIS SOFTWARE IS PROVIDED BY THE REGENTS AND CONTRIBUTORS ``AS IS'' AND * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE * ARE DISCLAIMED. IN NO EVENT SHALL THE REGENTS OR CONTRIBUTORS BE LIABLE * FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL * DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS * OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) * HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT * LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY * OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF * SUCH DAMAGE. * * @(#)ip_output.c 8.3 (Berkeley) 1/21/94 */ #include <sys/cdefs.h> __KERNEL_RCSID(0, "$NetBSD: ip6_output.c,v 1.229 2021/09/21 15:07:43 christos Exp $"); #ifdef _KERNEL_OPT #include "opt_inet.h" #include "opt_inet6.h" #include "opt_ipsec.h" #endif #include <sys/param.h> #include <sys/malloc.h> #include <sys/mbuf.h> #include <sys/errno.h> #include <sys/socket.h> #include <sys/socketvar.h> #include <sys/syslog.h> #include <sys/systm.h> #include <sys/proc.h> #include <sys/kauth.h> #include <net/if.h> #include <net/route.h> #include <net/pfil.h> #include <netinet/in.h> #include <netinet/in_var.h> #include <netinet/ip6.h> #include <netinet/ip_var.h> #include <netinet/icmp6.h> #include <netinet/in_offload.h> #include <netinet/portalgo.h> #include <netinet6/in6_offload.h> #include <netinet6/ip6_var.h> #include <netinet6/ip6_private.h> #include <netinet6/in6_pcb.h> #include <netinet6/nd6.h> #include <netinet6/ip6protosw.h> #include <netinet6/scope6_var.h> #ifdef IPSEC #include <netipsec/ipsec.h> #include <netipsec/ipsec6.h> #include <netipsec/key.h> #endif extern pfil_head_t *inet6_pfil_hook; /* XXX */ struct ip6_exthdrs { struct mbuf *ip6e_ip6; struct mbuf *ip6e_hbh; struct mbuf *ip6e_dest1; struct mbuf *ip6e_rthdr; struct mbuf *ip6e_dest2; }; static int ip6_pcbopt(int, u_char *, int, struct ip6_pktopts **, kauth_cred_t, int); static int ip6_getpcbopt(struct ip6_pktopts *, int, struct sockopt *); static int ip6_setpktopt(int, u_char *, int, struct ip6_pktopts *, kauth_cred_t, int, int, int); static int ip6_setmoptions(const struct sockopt *, struct in6pcb *); static int ip6_getmoptions(struct sockopt *, struct in6pcb *); static int ip6_copyexthdr(struct mbuf **, void *, int); static int ip6_insertfraghdr(struct mbuf *, struct mbuf *, int, struct ip6_frag **); static int ip6_insert_jumboopt(struct ip6_exthdrs *, u_int32_t); static int ip6_splithdr(struct mbuf *, struct ip6_exthdrs *); static int ip6_getpmtu(struct rtentry *, struct ifnet *, u_long *, int *); static int copypktopts(struct ip6_pktopts *, struct ip6_pktopts *, int); static int ip6_ifaddrvalid(const struct in6_addr *, const struct in6_addr *); static int ip6_handle_rthdr(struct ip6_rthdr *, struct ip6_hdr *); #ifdef RFC2292 static int ip6_pcbopts(struct ip6_pktopts **, struct socket *, struct sockopt *); #endif static int ip6_handle_rthdr(struct ip6_rthdr *rh, struct ip6_hdr *ip6) { int error = 0; switch (rh->ip6r_type) { case IPV6_RTHDR_TYPE_0: /* Dropped, RFC5095. */ default: /* is it possible? */ error = EINVAL; } return error; } /* * Send an IP packet to a host. */ int ip6_if_output(struct ifnet * const ifp, struct ifnet * const origifp, struct mbuf * const m, const struct sockaddr_in6 * const dst, const struct rtentry *rt) { int error = 0; if (rt != NULL) { error = rt_check_reject_route(rt, ifp); if (error != 0) { IP6_STATINC(IP6_STAT_RTREJECT); m_freem(m); return error; } } if ((ifp->if_flags & IFF_LOOPBACK) != 0) error = if_output_lock(ifp, origifp, m, sin6tocsa(dst), rt); else error = if_output_lock(ifp, ifp, m, sin6tocsa(dst), rt); return error; } /* * IP6 output. The packet in mbuf chain m contains a skeletal IP6 * header (with pri, len, nxt, hlim, src, dst). * * This function may modify ver and hlim only. The mbuf chain containing the * packet will be freed. The mbuf opt, if present, will not be freed. * * Type of "mtu": rt_rmx.rmx_mtu is u_long, ifnet.ifr_mtu is int, and * nd_ifinfo.linkmtu is u_int32_t. So we use u_long to hold largest one, * which is rt_rmx.rmx_mtu. */ int ip6_output( struct mbuf *m0, struct ip6_pktopts *opt, struct route *ro, int flags, struct ip6_moptions *im6o, struct in6pcb *in6p, struct ifnet **ifpp /* XXX: just for statistics */ ) { struct ip6_hdr *ip6, *mhip6; struct ifnet *ifp = NULL, *origifp = NULL; struct mbuf *m = m0; int tlen, len, off; bool tso; struct route ip6route; struct rtentry *rt = NULL, *rt_pmtu; const struct sockaddr_in6 *dst; struct sockaddr_in6 src_sa, dst_sa; int error = 0; struct in6_ifaddr *ia = NULL; u_long mtu; int alwaysfrag, dontfrag; u_int32_t optlen = 0, plen = 0, unfragpartlen = 0; struct ip6_exthdrs exthdrs; struct in6_addr finaldst, src0, dst0; u_int32_t zone; struct route *ro_pmtu = NULL; int hdrsplit = 0; int needipsec = 0; #ifdef IPSEC struct secpolicy *sp = NULL; #endif struct psref psref, psref_ia; int bound = curlwp_bind(); bool release_psref_ia = false; #ifdef DIAGNOSTIC if ((m->m_flags & M_PKTHDR) == 0) panic("ip6_output: no HDR"); if ((m->m_pkthdr.csum_flags & (M_CSUM_TCPv4|M_CSUM_UDPv4|M_CSUM_TSOv4)) != 0) { panic("ip6_output: IPv4 checksum offload flags: %d", m->m_pkthdr.csum_flags); } if ((m->m_pkthdr.csum_flags & (M_CSUM_TCPv6|M_CSUM_UDPv6)) == (M_CSUM_TCPv6|M_CSUM_UDPv6)) { panic("ip6_output: conflicting checksum offload flags: %d", m->m_pkthdr.csum_flags); } #endif M_CSUM_DATA_IPv6_SET(m->m_pkthdr.csum_data, sizeof(struct ip6_hdr)); #define MAKE_EXTHDR(hp, mp) \ do { \ if (hp) { \ struct ip6_ext *eh = (struct ip6_ext *)(hp); \ error = ip6_copyexthdr((mp), (void *)(hp), \ ((eh)->ip6e_len + 1) << 3); \ if (error) \ goto freehdrs; \ } \ } while (/*CONSTCOND*/ 0) memset(&exthdrs, 0, sizeof(exthdrs)); if (opt) { /* Hop-by-Hop options header */ MAKE_EXTHDR(opt->ip6po_hbh, &exthdrs.ip6e_hbh); /* Destination options header (1st part) */ MAKE_EXTHDR(opt->ip6po_dest1, &exthdrs.ip6e_dest1); /* Routing header */ MAKE_EXTHDR(opt->ip6po_rthdr, &exthdrs.ip6e_rthdr); /* Destination options header (2nd part) */ MAKE_EXTHDR(opt->ip6po_dest2, &exthdrs.ip6e_dest2); } /* * Calculate the total length of the extension header chain. * Keep the length of the unfragmentable part for fragmentation. */ optlen = 0; if (exthdrs.ip6e_hbh) optlen += exthdrs.ip6e_hbh->m_len; if (exthdrs.ip6e_dest1) optlen += exthdrs.ip6e_dest1->m_len; if (exthdrs.ip6e_rthdr) optlen += exthdrs.ip6e_rthdr->m_len; unfragpartlen = optlen + sizeof(struct ip6_hdr); /* NOTE: we don't add AH/ESP length here. do that later. */ if (exthdrs.ip6e_dest2) optlen += exthdrs.ip6e_dest2->m_len; #ifdef IPSEC if (ipsec_used) { /* Check the security policy (SP) for the packet */ sp = ipsec6_check_policy(m, in6p, flags, &needipsec, &error); if (error != 0) { /* * Hack: -EINVAL is used to signal that a packet * should be silently discarded. This is typically * because we asked key management for an SA and * it was delayed (e.g. kicked up to IKE). */ if (error == -EINVAL) error = 0; IP6_STATINC(IP6_STAT_IPSECDROP_OUT); goto freehdrs; } } #endif if (needipsec && (m->m_pkthdr.csum_flags & (M_CSUM_UDPv6|M_CSUM_TCPv6)) != 0) { in6_undefer_cksum_tcpudp(m); m->m_pkthdr.csum_flags &= ~(M_CSUM_UDPv6|M_CSUM_TCPv6); } /* * If we need IPsec, or there is at least one extension header, * separate IP6 header from the payload. */ if ((needipsec || optlen) && !hdrsplit) { if ((error = ip6_splithdr(m, &exthdrs)) != 0) { IP6_STATINC(IP6_STAT_ODROPPED); m = NULL; goto freehdrs; } m = exthdrs.ip6e_ip6; hdrsplit++; } /* adjust pointer */ ip6 = mtod(m, struct ip6_hdr *); /* adjust mbuf packet header length */ m->m_pkthdr.len += optlen; plen = m->m_pkthdr.len - sizeof(*ip6); /* If this is a jumbo payload, insert a jumbo payload option. */ if (plen > IPV6_MAXPACKET) { if (!hdrsplit) { if ((error = ip6_splithdr(m, &exthdrs)) != 0) { IP6_STATINC(IP6_STAT_ODROPPED); m = NULL; goto freehdrs; } m = exthdrs.ip6e_ip6; hdrsplit++; } /* adjust pointer */ ip6 = mtod(m, struct ip6_hdr *); if ((error = ip6_insert_jumboopt(&exthdrs, plen)) != 0) { IP6_STATINC(IP6_STAT_ODROPPED); goto freehdrs; } optlen += 8; /* XXX JUMBOOPTLEN */ ip6->ip6_plen = 0; } else ip6->ip6_plen = htons(plen); /* * Concatenate headers and fill in next header fields. * Here we have, on "m" * IPv6 payload * and we insert headers accordingly. Finally, we should be getting: * IPv6 hbh dest1 rthdr ah* [esp* dest2 payload] * * during the header composing process, "m" points to IPv6 header. * "mprev" points to an extension header prior to esp. */ { u_char *nexthdrp = &ip6->ip6_nxt; struct mbuf *mprev = m; /* * we treat dest2 specially. this makes IPsec processing * much easier. the goal here is to make mprev point the * mbuf prior to dest2. * * result: IPv6 dest2 payload * m and mprev will point to IPv6 header. */ if (exthdrs.ip6e_dest2) { if (!hdrsplit) panic("assumption failed: hdr not split"); exthdrs.ip6e_dest2->m_next = m->m_next; m->m_next = exthdrs.ip6e_dest2; *mtod(exthdrs.ip6e_dest2, u_char *) = ip6->ip6_nxt; ip6->ip6_nxt = IPPROTO_DSTOPTS; } #define MAKE_CHAIN(m, mp, p, i)\ do {\ if (m) {\ if (!hdrsplit) \ panic("assumption failed: hdr not split"); \ *mtod((m), u_char *) = *(p);\ *(p) = (i);\ p = mtod((m), u_char *);\ (m)->m_next = (mp)->m_next;\ (mp)->m_next = (m);\ (mp) = (m);\ }\ } while (/*CONSTCOND*/ 0) /* * result: IPv6 hbh dest1 rthdr dest2 payload * m will point to IPv6 header. mprev will point to the * extension header prior to dest2 (rthdr in the above case). */ MAKE_CHAIN(exthdrs.ip6e_hbh, mprev, nexthdrp, IPPROTO_HOPOPTS); MAKE_CHAIN(exthdrs.ip6e_dest1, mprev, nexthdrp, IPPROTO_DSTOPTS); MAKE_CHAIN(exthdrs.ip6e_rthdr, mprev, nexthdrp, IPPROTO_ROUTING); M_CSUM_DATA_IPv6_SET(m->m_pkthdr.csum_data, sizeof(struct ip6_hdr) + optlen); } /* Need to save for pmtu */ finaldst = ip6->ip6_dst; /* * If there is a routing header, replace destination address field * with the first hop of the routing header. */ if (exthdrs.ip6e_rthdr) { struct ip6_rthdr *rh; rh = mtod(exthdrs.ip6e_rthdr, struct ip6_rthdr *); error = ip6_handle_rthdr(rh, ip6); if (error != 0) { IP6_STATINC(IP6_STAT_ODROPPED); goto bad; } } /* Source address validation */ if (IN6_IS_ADDR_UNSPECIFIED(&ip6->ip6_src) && (flags & IPV6_UNSPECSRC) == 0) { error = EOPNOTSUPP; IP6_STATINC(IP6_STAT_BADSCOPE); goto bad; } if (IN6_IS_ADDR_MULTICAST(&ip6->ip6_src)) { error = EOPNOTSUPP; IP6_STATINC(IP6_STAT_BADSCOPE); goto bad; } IP6_STATINC(IP6_STAT_LOCALOUT); /* * Route packet. */ /* initialize cached route */ if (ro == NULL) { memset(&ip6route, 0, sizeof(ip6route)); ro = &ip6route; } ro_pmtu = ro; if (opt && opt->ip6po_rthdr) ro = &opt->ip6po_route; /* * if specified, try to fill in the traffic class field. * do not override if a non-zero value is already set. * we check the diffserv field and the ecn field separately. */ if (opt && opt->ip6po_tclass >= 0) { int mask = 0; if ((ip6->ip6_flow & htonl(0xfc << 20)) == 0) mask |= 0xfc; if ((ip6->ip6_flow & htonl(0x03 << 20)) == 0) mask |= 0x03; if (mask != 0) ip6->ip6_flow |= htonl((opt->ip6po_tclass & mask) << 20); } /* fill in or override the hop limit field, if necessary. */ if (opt && opt->ip6po_hlim != -1) ip6->ip6_hlim = opt->ip6po_hlim & 0xff; else if (IN6_IS_ADDR_MULTICAST(&ip6->ip6_dst)) { if (im6o != NULL) ip6->ip6_hlim = im6o->im6o_multicast_hlim; else ip6->ip6_hlim = ip6_defmcasthlim; } #ifdef IPSEC if (needipsec) { int s = splsoftnet(); error = ipsec6_process_packet(m, sp->req, flags); splx(s); /* * Preserve KAME behaviour: ENOENT can be returned * when an SA acquire is in progress. Don't propagate * this to user-level; it confuses applications. * XXX this will go away when the SADB is redone. */ if (error == ENOENT) error = 0; goto done; } #endif /* adjust pointer */ ip6 = mtod(m, struct ip6_hdr *); sockaddr_in6_init(&dst_sa, &ip6->ip6_dst, 0, 0, 0); /* We do not need a route for multicast */ if (IN6_IS_ADDR_MULTICAST(&ip6->ip6_dst)) { struct in6_pktinfo *pi = NULL; /* * If the outgoing interface for the address is specified by * the caller, use it. */ if (opt && (pi = opt->ip6po_pktinfo) != NULL) { /* XXX boundary check is assumed to be already done. */ ifp = if_get_byindex(pi->ipi6_ifindex, &psref); } else if (im6o != NULL) { ifp = if_get_byindex(im6o->im6o_multicast_if_index, &psref); } } if (ifp == NULL) { error = in6_selectroute(&dst_sa, opt, &ro, &rt, true); if (error != 0) goto bad; ifp = if_get_byindex(rt->rt_ifp->if_index, &psref); } if (rt == NULL) { /* * If in6_selectroute() does not return a route entry, * dst may not have been updated. */ error = rtcache_setdst(ro, sin6tosa(&dst_sa)); if (error) { IP6_STATINC(IP6_STAT_ODROPPED); goto bad; } } /* * then rt (for unicast) and ifp must be non-NULL valid values. */ if ((flags & IPV6_FORWARDING) == 0) { /* XXX: the FORWARDING flag can be set for mrouting. */ in6_ifstat_inc(ifp, ifs6_out_request); } if (rt != NULL) { ia = (struct in6_ifaddr *)(rt->rt_ifa); rt->rt_use++; } /* * The outgoing interface must be in the zone of source and * destination addresses. We should use ia_ifp to support the * case of sending packets to an address of our own. */ if (ia != NULL) { origifp = ia->ia_ifp; if (if_is_deactivated(origifp)) { IP6_STATINC(IP6_STAT_ODROPPED); goto bad; } if_acquire(origifp, &psref_ia); release_psref_ia = true; } else origifp = ifp; src0 = ip6->ip6_src; if (in6_setscope(&src0, origifp, &zone)) goto badscope; sockaddr_in6_init(&src_sa, &ip6->ip6_src, 0, 0, 0); if (sa6_recoverscope(&src_sa) || zone != src_sa.sin6_scope_id) goto badscope; dst0 = ip6->ip6_dst; if (in6_setscope(&dst0, origifp, &zone)) goto badscope; /* re-initialize to be sure */ sockaddr_in6_init(&dst_sa, &ip6->ip6_dst, 0, 0, 0); if (sa6_recoverscope(&dst_sa) || zone != dst_sa.sin6_scope_id) goto badscope; /* scope check is done. */ /* Ensure we only send from a valid address. */ if ((ifp->if_flags & IFF_LOOPBACK) == 0 && (flags & IPV6_FORWARDING) == 0 && (error = ip6_ifaddrvalid(&src0, &dst0)) != 0) { char ip6buf[INET6_ADDRSTRLEN]; nd6log(LOG_ERR, "refusing to send from invalid address %s (pid %d)\n", IN6_PRINT(ip6buf, &src0), curproc->p_pid); IP6_STATINC(IP6_STAT_ODROPPED); in6_ifstat_inc(origifp, ifs6_out_discard); if (error == 1) /* * Address exists, but is tentative or detached. * We can't send from it because it's invalid, * so we drop the packet. */ error = 0; else error = EADDRNOTAVAIL; goto bad; } if (rt != NULL && (rt->rt_flags & RTF_GATEWAY) && !IN6_IS_ADDR_MULTICAST(&ip6->ip6_dst)) dst = satocsin6(rt->rt_gateway); else dst = satocsin6(rtcache_getdst(ro)); /* * XXXXXX: original code follows: */ if (!IN6_IS_ADDR_MULTICAST(&ip6->ip6_dst)) m->m_flags &= ~(M_BCAST | M_MCAST); /* just in case */ else { bool ingroup; m->m_flags = (m->m_flags & ~M_BCAST) | M_MCAST; in6_ifstat_inc(ifp, ifs6_out_mcast); /* * Confirm that the outgoing interface supports multicast. */ if (!(ifp->if_flags & IFF_MULTICAST)) { IP6_STATINC(IP6_STAT_NOROUTE); in6_ifstat_inc(ifp, ifs6_out_discard); error = ENETUNREACH; goto bad; } ingroup = in6_multi_group(&ip6->ip6_dst, ifp); if (ingroup && (im6o == NULL || im6o->im6o_multicast_loop)) { /* * If we belong to the destination multicast group * on the outgoing interface, and the caller did not * forbid loopback, loop back a copy. */ KASSERT(dst != NULL); ip6_mloopback(ifp, m, dst); } else { /* * If we are acting as a multicast router, perform * multicast forwarding as if the packet had just * arrived on the interface to which we are about * to send. The multicast forwarding function * recursively calls this function, using the * IPV6_FORWARDING flag to prevent infinite recursion. * * Multicasts that are looped back by ip6_mloopback(), * above, will be forwarded by the ip6_input() routine, * if necessary. */ if (ip6_mrouter && (flags & IPV6_FORWARDING) == 0) { if (ip6_mforward(ip6, ifp, m) != 0) { m_freem(m); goto done; } } } /* * Multicasts with a hoplimit of zero may be looped back, * above, but must not be transmitted on a network. * Also, multicasts addressed to the loopback interface * are not sent -- the above call to ip6_mloopback() will * loop back a copy if this host actually belongs to the * destination group on the loopback interface. */ if (ip6->ip6_hlim == 0 || (ifp->if_flags & IFF_LOOPBACK) || IN6_IS_ADDR_MC_INTFACELOCAL(&ip6->ip6_dst)) { m_freem(m); goto done; } } /* * Fill the outgoing interface to tell the upper layer * to increment per-interface statistics. */ if (ifpp) *ifpp = ifp; /* Determine path MTU. */ /* * ro_pmtu represent final destination while * ro might represent immediate destination. * Use ro_pmtu destination since MTU might differ. */ if (ro_pmtu != ro) { union { struct sockaddr dst; struct sockaddr_in6 dst6; } u; /* ro_pmtu may not have a cache */ sockaddr_in6_init(&u.dst6, &finaldst, 0, 0, 0); rt_pmtu = rtcache_lookup(ro_pmtu, &u.dst); } else rt_pmtu = rt; error = ip6_getpmtu(rt_pmtu, ifp, &mtu, &alwaysfrag); if (rt_pmtu != NULL && rt_pmtu != rt) rtcache_unref(rt_pmtu, ro_pmtu); KASSERT(error == 0); /* ip6_getpmtu never fail if ifp is passed */ /* * The caller of this function may specify to use the minimum MTU * in some cases. * An advanced API option (IPV6_USE_MIN_MTU) can also override MTU * setting. The logic is a bit complicated; by default, unicast * packets will follow path MTU while multicast packets will be sent at * the minimum MTU. If IP6PO_MINMTU_ALL is specified, all packets * including unicast ones will be sent at the minimum MTU. Multicast * packets will always be sent at the minimum MTU unless * IP6PO_MINMTU_DISABLE is explicitly specified. * See RFC 3542 for more details. */ if (mtu > IPV6_MMTU) { if ((flags & IPV6_MINMTU)) mtu = IPV6_MMTU; else if (opt && opt->ip6po_minmtu == IP6PO_MINMTU_ALL) mtu = IPV6_MMTU; else if (IN6_IS_ADDR_MULTICAST(&ip6->ip6_dst) && (opt == NULL || opt->ip6po_minmtu != IP6PO_MINMTU_DISABLE)) { mtu = IPV6_MMTU; } } /* * clear embedded scope identifiers if necessary. * in6_clearscope will touch the addresses only when necessary. */ in6_clearscope(&ip6->ip6_src); in6_clearscope(&ip6->ip6_dst); /* * If the outgoing packet contains a hop-by-hop options header, * it must be examined and processed even by the source node. * (RFC 2460, section 4.) * * XXX Is this really necessary? */ if (ip6->ip6_nxt == IPPROTO_HOPOPTS) { u_int32_t dummy1; /* XXX unused */ u_int32_t dummy2; /* XXX unused */ int hoff = sizeof(struct ip6_hdr); if (ip6_hopopts_input(&dummy1, &dummy2, &m, &hoff)) { /* m was already freed at this point */ error = EINVAL; goto done; } ip6 = mtod(m, struct ip6_hdr *); } /* * Run through list of hooks for output packets. */ error = pfil_run_hooks(inet6_pfil_hook, &m, ifp, PFIL_OUT); if (error != 0 || m == NULL) { IP6_STATINC(IP6_STAT_PFILDROP_OUT); goto done; } ip6 = mtod(m, struct ip6_hdr *); /* * Send the packet to the outgoing interface. * If necessary, do IPv6 fragmentation before sending. * * the logic here is rather complex: * 1: normal case (dontfrag == 0, alwaysfrag == 0) * 1-a: send as is if tlen <= path mtu * 1-b: fragment if tlen > path mtu * * 2: if user asks us not to fragment (dontfrag == 1) * 2-a: send as is if tlen <= interface mtu * 2-b: error if tlen > interface mtu * * 3: if we always need to attach fragment header (alwaysfrag == 1) * always fragment * * 4: if dontfrag == 1 && alwaysfrag == 1 * error, as we cannot handle this conflicting request */ tlen = m->m_pkthdr.len; tso = (m->m_pkthdr.csum_flags & M_CSUM_TSOv6) != 0; if (opt && (opt->ip6po_flags & IP6PO_DONTFRAG)) dontfrag = 1; else dontfrag = 0; if (dontfrag && alwaysfrag) { /* case 4 */ /* conflicting request - can't transmit */ IP6_STATINC(IP6_STAT_CANTFRAG); error = EMSGSIZE; goto bad; } if (dontfrag && (!tso && tlen > ifp->if_mtu)) { /* case 2-b */ /* * Even if the DONTFRAG option is specified, we cannot send the * packet when the data length is larger than the MTU of the * outgoing interface. * Notify the error by sending IPV6_PATHMTU ancillary data as * well as returning an error code (the latter is not described * in the API spec.) */ u_int32_t mtu32; struct ip6ctlparam ip6cp; mtu32 = (u_int32_t)mtu; memset(&ip6cp, 0, sizeof(ip6cp)); ip6cp.ip6c_cmdarg = (void *)&mtu32; pfctlinput2(PRC_MSGSIZE, rtcache_getdst(ro_pmtu), &ip6cp); IP6_STATINC(IP6_STAT_CANTFRAG); error = EMSGSIZE; goto bad; } /* * transmit packet without fragmentation */ if (dontfrag || (!alwaysfrag && (tlen <= mtu || tso))) { /* case 1-a and 2-a */ struct in6_ifaddr *ia6; int sw_csum; int s; ip6 = mtod(m, struct ip6_hdr *); s = pserialize_read_enter(); ia6 = in6_ifawithifp(ifp, &ip6->ip6_src); if (ia6) { /* Record statistics for this interface address. */ ia6->ia_ifa.ifa_data.ifad_outbytes += m->m_pkthdr.len; } pserialize_read_exit(s); sw_csum = m->m_pkthdr.csum_flags & ~ifp->if_csum_flags_tx; if ((sw_csum & (M_CSUM_UDPv6|M_CSUM_TCPv6)) != 0) { if (IN6_NEED_CHECKSUM(ifp, sw_csum & (M_CSUM_UDPv6|M_CSUM_TCPv6))) { in6_undefer_cksum_tcpudp(m); } m->m_pkthdr.csum_flags &= ~(M_CSUM_UDPv6|M_CSUM_TCPv6); } KASSERT(dst != NULL); if (__predict_false(sw_csum & M_CSUM_TSOv6)) { /* * TSO6 is required by a packet, but disabled for * the interface. */ error = ip6_tso_output(ifp, origifp, m, dst, rt); } else error = ip6_if_output(ifp, origifp, m, dst, rt); goto done; } if (tso) { IP6_STATINC(IP6_STAT_CANTFRAG); /* XXX */ error = EINVAL; /* XXX */ goto bad; } /* * try to fragment the packet. case 1-b and 3 */ if (mtu < IPV6_MMTU) { /* path MTU cannot be less than IPV6_MMTU */ IP6_STATINC(IP6_STAT_CANTFRAG); error = EMSGSIZE; in6_ifstat_inc(ifp, ifs6_out_fragfail); goto bad; } else if (ip6->ip6_plen == 0) { /* jumbo payload cannot be fragmented */ IP6_STATINC(IP6_STAT_CANTFRAG); error = EMSGSIZE; in6_ifstat_inc(ifp, ifs6_out_fragfail); goto bad; } else { const uint32_t id = ip6_randomid(); struct mbuf **mnext, *m_frgpart; const int hlen = unfragpartlen; struct ip6_frag *ip6f; u_char nextproto; if (mtu > IPV6_MAXPACKET) mtu = IPV6_MAXPACKET; /* * Must be able to put at least 8 bytes per fragment. */ len = (mtu - hlen - sizeof(struct ip6_frag)) & ~7; if (len < 8) { IP6_STATINC(IP6_STAT_CANTFRAG); error = EMSGSIZE; in6_ifstat_inc(ifp, ifs6_out_fragfail); goto bad; } mnext = &m->m_nextpkt; /* * Change the next header field of the last header in the * unfragmentable part. */ if (exthdrs.ip6e_rthdr) { nextproto = *mtod(exthdrs.ip6e_rthdr, u_char *); *mtod(exthdrs.ip6e_rthdr, u_char *) = IPPROTO_FRAGMENT; } else if (exthdrs.ip6e_dest1) { nextproto = *mtod(exthdrs.ip6e_dest1, u_char *); *mtod(exthdrs.ip6e_dest1, u_char *) = IPPROTO_FRAGMENT; } else if (exthdrs.ip6e_hbh) { nextproto = *mtod(exthdrs.ip6e_hbh, u_char *); *mtod(exthdrs.ip6e_hbh, u_char *) = IPPROTO_FRAGMENT; } else { nextproto = ip6->ip6_nxt; ip6->ip6_nxt = IPPROTO_FRAGMENT; } if ((m->m_pkthdr.csum_flags & (M_CSUM_UDPv6|M_CSUM_TCPv6)) != 0) { if (IN6_NEED_CHECKSUM(ifp, m->m_pkthdr.csum_flags & (M_CSUM_UDPv6|M_CSUM_TCPv6))) { in6_undefer_cksum_tcpudp(m); } m->m_pkthdr.csum_flags &= ~(M_CSUM_UDPv6|M_CSUM_TCPv6); } /* * Loop through length of segment after first fragment, * make new header and copy data of each part and link onto * chain. */ m0 = m; for (off = hlen; off < tlen; off += len) { struct mbuf *mlast; MGETHDR(m, M_DONTWAIT, MT_HEADER); if (!m) { error = ENOBUFS; IP6_STATINC(IP6_STAT_ODROPPED); goto sendorfree; } m_reset_rcvif(m); m->m_flags = m0->m_flags & M_COPYFLAGS; *mnext = m; mnext = &m->m_nextpkt; m->m_data += max_linkhdr; mhip6 = mtod(m, struct ip6_hdr *); *mhip6 = *ip6; m->m_len = sizeof(*mhip6); ip6f = NULL; error = ip6_insertfraghdr(m0, m, hlen, &ip6f); if (error) { IP6_STATINC(IP6_STAT_ODROPPED); goto sendorfree; } /* Fill in the Frag6 Header */ ip6f->ip6f_offlg = htons((u_int16_t)((off - hlen) & ~7)); if (off + len >= tlen) len = tlen - off; else ip6f->ip6f_offlg |= IP6F_MORE_FRAG; ip6f->ip6f_reserved = 0; ip6f->ip6f_ident = id; ip6f->ip6f_nxt = nextproto; mhip6->ip6_plen = htons((u_int16_t)(len + hlen + sizeof(*ip6f) - sizeof(struct ip6_hdr))); if ((m_frgpart = m_copym(m0, off, len, M_DONTWAIT)) == NULL) { error = ENOBUFS; IP6_STATINC(IP6_STAT_ODROPPED); goto sendorfree; } for (mlast = m; mlast->m_next; mlast = mlast->m_next) ; mlast->m_next = m_frgpart; m->m_pkthdr.len = len + hlen + sizeof(*ip6f); m_reset_rcvif(m); IP6_STATINC(IP6_STAT_OFRAGMENTS); in6_ifstat_inc(ifp, ifs6_out_fragcreat); } in6_ifstat_inc(ifp, ifs6_out_fragok); } sendorfree: m = m0->m_nextpkt; m0->m_nextpkt = 0; m_freem(m0); for (m0 = m; m; m = m0) { m0 = m->m_nextpkt; m->m_nextpkt = 0; if (error == 0) { struct in6_ifaddr *ia6; int s; ip6 = mtod(m, struct ip6_hdr *); s = pserialize_read_enter(); ia6 = in6_ifawithifp(ifp, &ip6->ip6_src); if (ia6) { /* * Record statistics for this interface * address. */ ia6->ia_ifa.ifa_data.ifad_outbytes += m->m_pkthdr.len; } pserialize_read_exit(s); KASSERT(dst != NULL); error = ip6_if_output(ifp, origifp, m, dst, rt); } else m_freem(m); } if (error == 0) IP6_STATINC(IP6_STAT_FRAGMENTED); done: rtcache_unref(rt, ro); if (ro == &ip6route) rtcache_free(&ip6route); #ifdef IPSEC if (sp != NULL) KEY_SP_UNREF(&sp); #endif if_put(ifp, &psref); if (release_psref_ia) if_put(origifp, &psref_ia); curlwp_bindx(bound); return error; freehdrs: m_freem(exthdrs.ip6e_hbh); m_freem(exthdrs.ip6e_dest1); m_freem(exthdrs.ip6e_rthdr); m_freem(exthdrs.ip6e_dest2); /* FALLTHROUGH */ bad: m_freem(m); goto done; badscope: IP6_STATINC(IP6_STAT_BADSCOPE); in6_ifstat_inc(origifp, ifs6_out_discard); if (error == 0) error = EHOSTUNREACH; /* XXX */ goto bad; } static int ip6_copyexthdr(struct mbuf **mp, void *hdr, int hlen) { struct mbuf *m; if (hlen > MCLBYTES) return ENOBUFS; /* XXX */ MGET(m, M_DONTWAIT, MT_DATA); if (!m) return ENOBUFS; if (hlen > MLEN) { MCLGET(m, M_DONTWAIT); if ((m->m_flags & M_EXT) == 0) { m_free(m); return ENOBUFS; } } m->m_len = hlen; if (hdr) memcpy(mtod(m, void *), hdr, hlen); *mp = m; return 0; } /* * Insert jumbo payload option. */ static int ip6_insert_jumboopt(struct ip6_exthdrs *exthdrs, u_int32_t plen) { struct mbuf *mopt; u_int8_t *optbuf; u_int32_t v; #define JUMBOOPTLEN 8 /* length of jumbo payload option and padding */ /* * If there is no hop-by-hop options header, allocate new one. * If there is one but it doesn't have enough space to store the * jumbo payload option, allocate a cluster to store the whole options. * Otherwise, use it to store the options. */ if (exthdrs->ip6e_hbh == NULL) { MGET(mopt, M_DONTWAIT, MT_DATA); if (mopt == 0) return (ENOBUFS); mopt->m_len = JUMBOOPTLEN; optbuf = mtod(mopt, u_int8_t *); optbuf[1] = 0; /* = ((JUMBOOPTLEN) >> 3) - 1 */ exthdrs->ip6e_hbh = mopt; } else { struct ip6_hbh *hbh; mopt = exthdrs->ip6e_hbh; if (M_TRAILINGSPACE(mopt) < JUMBOOPTLEN) { const int oldoptlen = mopt->m_len; struct mbuf *n; /* * Assumptions: * - exthdrs->ip6e_hbh is not referenced from places * other than exthdrs. * - exthdrs->ip6e_hbh is not an mbuf chain. */ KASSERT(mopt->m_next == NULL); /* * Give up if the whole (new) hbh header does not fit * even in an mbuf cluster. */ if (oldoptlen + JUMBOOPTLEN > MCLBYTES) return ENOBUFS; /* * At this point, we must always prepare a cluster. */ MGET(n, M_DONTWAIT, MT_DATA); if (n) { MCLGET(n, M_DONTWAIT); if ((n->m_flags & M_EXT) == 0) { m_freem(n); n = NULL; } } if (!n) return ENOBUFS; n->m_len = oldoptlen + JUMBOOPTLEN; bcopy(mtod(mopt, void *), mtod(n, void *), oldoptlen); optbuf = mtod(n, u_int8_t *) + oldoptlen; m_freem(mopt); mopt = exthdrs->ip6e_hbh = n; } else { optbuf = mtod(mopt, u_int8_t *) + mopt->m_len; mopt->m_len += JUMBOOPTLEN; } optbuf[0] = IP6OPT_PADN; optbuf[1] = 0; /* * Adjust the header length according to the pad and * the jumbo payload option. */ hbh = mtod(mopt, struct ip6_hbh *); hbh->ip6h_len += (JUMBOOPTLEN >> 3); } /* fill in the option. */ optbuf[2] = IP6OPT_JUMBO; optbuf[3] = 4; v = (u_int32_t)htonl(plen + JUMBOOPTLEN); memcpy(&optbuf[4], &v, sizeof(u_int32_t)); /* finally, adjust the packet header length */ exthdrs->ip6e_ip6->m_pkthdr.len += JUMBOOPTLEN; return 0; #undef JUMBOOPTLEN } /* * Insert fragment header and copy unfragmentable header portions. * * *frghdrp will not be read, and it is guaranteed that either an * error is returned or that *frghdrp will point to space allocated * for the fragment header. * * On entry, m contains: * IPv6 Header * On exit, it contains: * IPv6 Header -> Unfragmentable Part -> Frag6 Header */ static int ip6_insertfraghdr(struct mbuf *m0, struct mbuf *m, int hlen, struct ip6_frag **frghdrp) { struct mbuf *n, *mlast; if (hlen > sizeof(struct ip6_hdr)) { n = m_copym(m0, sizeof(struct ip6_hdr), hlen - sizeof(struct ip6_hdr), M_DONTWAIT); if (n == NULL) return ENOBUFS; m->m_next = n; } else n = m; /* Search for the last mbuf of unfragmentable part. */ for (mlast = n; mlast->m_next; mlast = mlast->m_next) ; if ((mlast->m_flags & M_EXT) == 0 && M_TRAILINGSPACE(mlast) >= sizeof(struct ip6_frag)) { /* use the trailing space of the last mbuf for the fragment hdr */ *frghdrp = (struct ip6_frag *)(mtod(mlast, char *) + mlast->m_len); mlast->m_len += sizeof(struct ip6_frag); } else { /* allocate a new mbuf for the fragment header */ struct mbuf *mfrg; MGET(mfrg, M_DONTWAIT, MT_DATA); if (mfrg == NULL) return ENOBUFS; mfrg->m_len = sizeof(struct ip6_frag); *frghdrp = mtod(mfrg, struct ip6_frag *); mlast->m_next = mfrg; } return 0; } static int ip6_getpmtu(struct rtentry *rt, struct ifnet *ifp, u_long *mtup, int *alwaysfragp) { u_int32_t mtu = 0; int alwaysfrag = 0; int error = 0; if (rt != NULL) { if (ifp == NULL) ifp = rt->rt_ifp; mtu = rt->rt_rmx.rmx_mtu; if (mtu == 0) mtu = ifp->if_mtu; else if (mtu < IPV6_MMTU) { /* * RFC2460 section 5, last paragraph: * if we record ICMPv6 too big message with * mtu < IPV6_MMTU, transmit packets sized IPV6_MMTU * or smaller, with fragment header attached. * (fragment header is needed regardless from the * packet size, for translators to identify packets) */ alwaysfrag = 1; mtu = IPV6_MMTU; } else if (mtu > ifp->if_mtu) { /* * The MTU on the route is larger than the MTU on * the interface! This shouldn't happen, unless the * MTU of the interface has been changed after the * interface was brought up. Change the MTU in the * route to match the interface MTU (as long as the * field isn't locked). */ mtu = ifp->if_mtu; if (!(rt->rt_rmx.rmx_locks & RTV_MTU)) rt->rt_rmx.rmx_mtu = mtu; } } else if (ifp) { mtu = ifp->if_mtu; } else error = EHOSTUNREACH; /* XXX */ *mtup = mtu; if (alwaysfragp) *alwaysfragp = alwaysfrag; return (error); } /* * IP6 socket option processing. */ int ip6_ctloutput(int op, struct socket *so, struct sockopt *sopt) { int optdatalen, uproto; void *optdata; struct in6pcb *in6p = sotoin6pcb(so); struct ip_moptions **mopts; int error, optval; int level, optname; KASSERT(solocked(so)); KASSERT(sopt != NULL); level = sopt->sopt_level; optname = sopt->sopt_name; error = optval = 0; uproto = (int)so->so_proto->pr_protocol; switch (level) { case IPPROTO_IP: switch (optname) { case IP_ADD_MEMBERSHIP: case IP_DROP_MEMBERSHIP: case IP_MULTICAST_IF: case IP_MULTICAST_LOOP: case IP_MULTICAST_TTL: mopts = &in6p->in6p_v4moptions; switch (op) { case PRCO_GETOPT: return ip_getmoptions(*mopts, sopt); case PRCO_SETOPT: return ip_setmoptions(mopts, sopt); default: return EINVAL; } default: return ENOPROTOOPT; } case IPPROTO_IPV6: break; default: return ENOPROTOOPT; } switch (op) { case PRCO_SETOPT: switch (optname) { #ifdef RFC2292 case IPV6_2292PKTOPTIONS: error = ip6_pcbopts(&in6p->in6p_outputopts, so, sopt); break; #endif /* * Use of some Hop-by-Hop options or some * Destination options, might require special * privilege. That is, normal applications * (without special privilege) might be forbidden * from setting certain options in outgoing packets, * and might never see certain options in received * packets. [RFC 2292 Section 6] * KAME specific note: * KAME prevents non-privileged users from sending or * receiving ANY hbh/dst options in order to avoid * overhead of parsing options in the kernel. */ case IPV6_RECVHOPOPTS: case IPV6_RECVDSTOPTS: case IPV6_RECVRTHDRDSTOPTS: error = kauth_authorize_network( kauth_cred_get(), KAUTH_NETWORK_IPV6, KAUTH_REQ_NETWORK_IPV6_HOPBYHOP, NULL, NULL, NULL); if (error) break; /* FALLTHROUGH */ case IPV6_UNICAST_HOPS: case IPV6_HOPLIMIT: case IPV6_FAITH: case IPV6_RECVPKTINFO: case IPV6_RECVHOPLIMIT: case IPV6_RECVRTHDR: case IPV6_RECVPATHMTU: case IPV6_RECVTCLASS: case IPV6_V6ONLY: case IPV6_BINDANY: error = sockopt_getint(sopt, &optval); if (error) break; switch (optname) { case IPV6_UNICAST_HOPS: if (optval < -1 || optval >= 256) error = EINVAL; else { /* -1 = kernel default */ in6p->in6p_hops = optval; } break; #define OPTSET(bit) \ do { \ if (optval) \ in6p->in6p_flags |= (bit); \ else \ in6p->in6p_flags &= ~(bit); \ } while (/*CONSTCOND*/ 0) #ifdef RFC2292 #define OPTSET2292(bit) \ do { \ in6p->in6p_flags |= IN6P_RFC2292; \ if (optval) \ in6p->in6p_flags |= (bit); \ else \ in6p->in6p_flags &= ~(bit); \ } while (/*CONSTCOND*/ 0) #endif #define OPTBIT(bit) (in6p->in6p_flags & (bit) ? 1 : 0) case IPV6_RECVPKTINFO: #ifdef RFC2292 /* cannot mix with RFC2292 */ if (OPTBIT(IN6P_RFC2292)) { error = EINVAL; break; } #endif OPTSET(IN6P_PKTINFO); break; case IPV6_HOPLIMIT: { struct ip6_pktopts **optp; #ifdef RFC2292 /* cannot mix with RFC2292 */ if (OPTBIT(IN6P_RFC2292)) { error = EINVAL; break; } #endif optp = &in6p->in6p_outputopts; error = ip6_pcbopt(IPV6_HOPLIMIT, (u_char *)&optval, sizeof(optval), optp, kauth_cred_get(), uproto); break; } case IPV6_RECVHOPLIMIT: #ifdef RFC2292 /* cannot mix with RFC2292 */ if (OPTBIT(IN6P_RFC2292)) { error = EINVAL; break; } #endif OPTSET(IN6P_HOPLIMIT); break; case IPV6_RECVHOPOPTS: #ifdef RFC2292 /* cannot mix with RFC2292 */ if (OPTBIT(IN6P_RFC2292)) { error = EINVAL; break; } #endif OPTSET(IN6P_HOPOPTS); break; case IPV6_RECVDSTOPTS: #ifdef RFC2292 /* cannot mix with RFC2292 */ if (OPTBIT(IN6P_RFC2292)) { error = EINVAL; break; } #endif OPTSET(IN6P_DSTOPTS); break; case IPV6_RECVRTHDRDSTOPTS: #ifdef RFC2292 /* cannot mix with RFC2292 */ if (OPTBIT(IN6P_RFC2292)) { error = EINVAL; break; } #endif OPTSET(IN6P_RTHDRDSTOPTS); break; case IPV6_RECVRTHDR: #ifdef RFC2292 /* cannot mix with RFC2292 */ if (OPTBIT(IN6P_RFC2292)) { error = EINVAL; break; } #endif OPTSET(IN6P_RTHDR); break; case IPV6_FAITH: OPTSET(IN6P_FAITH); break; case IPV6_RECVPATHMTU: /* * We ignore this option for TCP * sockets. * (RFC3542 leaves this case * unspecified.) */ if (uproto != IPPROTO_TCP) OPTSET(IN6P_MTU); break; case IPV6_V6ONLY: /* * make setsockopt(IPV6_V6ONLY) * available only prior to bind(2). * see ipng mailing list, Jun 22 2001. */ if (in6p->in6p_lport || !IN6_IS_ADDR_UNSPECIFIED(&in6p->in6p_laddr)) { error = EINVAL; break; } #ifdef INET6_BINDV6ONLY if (!optval) error = EINVAL; #else OPTSET(IN6P_IPV6_V6ONLY); #endif break; case IPV6_RECVTCLASS: #ifdef RFC2292 /* cannot mix with RFC2292 XXX */ if (OPTBIT(IN6P_RFC2292)) { error = EINVAL; break; } #endif OPTSET(IN6P_TCLASS); break; case IPV6_BINDANY: error = kauth_authorize_network( kauth_cred_get(), KAUTH_NETWORK_BIND, KAUTH_REQ_NETWORK_BIND_ANYADDR, so, NULL, NULL); if (error) break; OPTSET(IN6P_BINDANY); break; } break; case IPV6_OTCLASS: { struct ip6_pktopts **optp; u_int8_t tclass; error = sockopt_get(sopt, &tclass, sizeof(tclass)); if (error) break; optp = &in6p->in6p_outputopts; error = ip6_pcbopt(optname, (u_char *)&tclass, sizeof(tclass), optp, kauth_cred_get(), uproto); break; } case IPV6_TCLASS: case IPV6_DONTFRAG: case IPV6_USE_MIN_MTU: case IPV6_PREFER_TEMPADDR: error = sockopt_getint(sopt, &optval); if (error) break; { struct ip6_pktopts **optp; optp = &in6p->in6p_outputopts; error = ip6_pcbopt(optname, (u_char *)&optval, sizeof(optval), optp, kauth_cred_get(), uproto); break; } #ifdef RFC2292 case IPV6_2292PKTINFO: case IPV6_2292HOPLIMIT: case IPV6_2292HOPOPTS: case IPV6_2292DSTOPTS: case IPV6_2292RTHDR: /* RFC 2292 */ error = sockopt_getint(sopt, &optval); if (error) break; switch (optname) { case IPV6_2292PKTINFO: OPTSET2292(IN6P_PKTINFO); break; case IPV6_2292HOPLIMIT: OPTSET2292(IN6P_HOPLIMIT); break; case IPV6_2292HOPOPTS: /* * Check super-user privilege. * See comments for IPV6_RECVHOPOPTS. */ error = kauth_authorize_network( kauth_cred_get(), KAUTH_NETWORK_IPV6, KAUTH_REQ_NETWORK_IPV6_HOPBYHOP, NULL, NULL, NULL); if (error) return (error); OPTSET2292(IN6P_HOPOPTS); break; case IPV6_2292DSTOPTS: error = kauth_authorize_network( kauth_cred_get(), KAUTH_NETWORK_IPV6, KAUTH_REQ_NETWORK_IPV6_HOPBYHOP, NULL, NULL, NULL); if (error) return (error); OPTSET2292(IN6P_DSTOPTS|IN6P_RTHDRDSTOPTS); /* XXX */ break; case IPV6_2292RTHDR: OPTSET2292(IN6P_RTHDR); break; } break; #endif case IPV6_PKTINFO: case IPV6_HOPOPTS: case IPV6_RTHDR: case IPV6_DSTOPTS: case IPV6_RTHDRDSTOPTS: case IPV6_NEXTHOP: { /* new advanced API (RFC3542) */ void *optbuf; int optbuflen; struct ip6_pktopts **optp; #ifdef RFC2292 /* cannot mix with RFC2292 */ if (OPTBIT(IN6P_RFC2292)) { error = EINVAL; break; } #endif optbuflen = sopt->sopt_size; optbuf = malloc(optbuflen, M_IP6OPT, M_NOWAIT); if (optbuf == NULL) { error = ENOBUFS; break; } error = sockopt_get(sopt, optbuf, optbuflen); if (error) { free(optbuf, M_IP6OPT); break; } optp = &in6p->in6p_outputopts; error = ip6_pcbopt(optname, optbuf, optbuflen, optp, kauth_cred_get(), uproto); free(optbuf, M_IP6OPT); break; } #undef OPTSET case IPV6_MULTICAST_IF: case IPV6_MULTICAST_HOPS: case IPV6_MULTICAST_LOOP: case IPV6_JOIN_GROUP: case IPV6_LEAVE_GROUP: error = ip6_setmoptions(sopt, in6p); break; case IPV6_PORTRANGE: error = sockopt_getint(sopt, &optval); if (error) break; switch (optval) { case IPV6_PORTRANGE_DEFAULT: in6p->in6p_flags &= ~(IN6P_LOWPORT); in6p->in6p_flags &= ~(IN6P_HIGHPORT); break; case IPV6_PORTRANGE_HIGH: in6p->in6p_flags &= ~(IN6P_LOWPORT); in6p->in6p_flags |= IN6P_HIGHPORT; break; case IPV6_PORTRANGE_LOW: in6p->in6p_flags &= ~(IN6P_HIGHPORT); in6p->in6p_flags |= IN6P_LOWPORT; break; default: error = EINVAL; break; } break; case IPV6_PORTALGO: error = sockopt_getint(sopt, &optval); if (error) break; error = portalgo_algo_index_select( (struct inpcb_hdr *)in6p, optval); break; #if defined(IPSEC) case IPV6_IPSEC_POLICY: if (ipsec_enabled) { error = ipsec_set_policy(in6p, sopt->sopt_data, sopt->sopt_size, kauth_cred_get()); } else error = ENOPROTOOPT; break; #endif /* IPSEC */ default: error = ENOPROTOOPT; break; } break; case PRCO_GETOPT: switch (optname) { #ifdef RFC2292 case IPV6_2292PKTOPTIONS: /* * RFC3542 (effectively) deprecated the * semantics of the 2292-style pktoptions. * Since it was not reliable in nature (i.e., * applications had to expect the lack of some * information after all), it would make sense * to simplify this part by always returning * empty data. */ break; #endif case IPV6_RECVHOPOPTS: case IPV6_RECVDSTOPTS: case IPV6_RECVRTHDRDSTOPTS: case IPV6_UNICAST_HOPS: case IPV6_RECVPKTINFO: case IPV6_RECVHOPLIMIT: case IPV6_RECVRTHDR: case IPV6_RECVPATHMTU: case IPV6_FAITH: case IPV6_V6ONLY: case IPV6_PORTRANGE: case IPV6_RECVTCLASS: case IPV6_BINDANY: switch (optname) { case IPV6_RECVHOPOPTS: optval = OPTBIT(IN6P_HOPOPTS); break; case IPV6_RECVDSTOPTS: optval = OPTBIT(IN6P_DSTOPTS); break; case IPV6_RECVRTHDRDSTOPTS: optval = OPTBIT(IN6P_RTHDRDSTOPTS); break; case IPV6_UNICAST_HOPS: optval = in6p->in6p_hops; break; case IPV6_RECVPKTINFO: optval = OPTBIT(IN6P_PKTINFO); break; case IPV6_RECVHOPLIMIT: optval = OPTBIT(IN6P_HOPLIMIT); break; case IPV6_RECVRTHDR: optval = OPTBIT(IN6P_RTHDR); break; case IPV6_RECVPATHMTU: optval = OPTBIT(IN6P_MTU); break; case IPV6_FAITH: optval = OPTBIT(IN6P_FAITH); break; case IPV6_V6ONLY: optval = OPTBIT(IN6P_IPV6_V6ONLY); break; case IPV6_PORTRANGE: { int flags; flags = in6p->in6p_flags; if (flags & IN6P_HIGHPORT) optval = IPV6_PORTRANGE_HIGH; else if (flags & IN6P_LOWPORT) optval = IPV6_PORTRANGE_LOW; else optval = 0; break; } case IPV6_RECVTCLASS: optval = OPTBIT(IN6P_TCLASS); break; case IPV6_BINDANY: optval = OPTBIT(IN6P_BINDANY); break; } if (error) break; error = sockopt_setint(sopt, optval); break; case IPV6_PATHMTU: { u_long pmtu = 0; struct ip6_mtuinfo mtuinfo; struct route *ro = &in6p->in6p_route; struct rtentry *rt; union { struct sockaddr dst; struct sockaddr_in6 dst6; } u; if (!(so->so_state & SS_ISCONNECTED)) return (ENOTCONN); /* * XXX: we dot not consider the case of source * routing, or optional information to specify * the outgoing interface. */ sockaddr_in6_init(&u.dst6, &in6p->in6p_faddr, 0, 0, 0); rt = rtcache_lookup(ro, &u.dst); error = ip6_getpmtu(rt, NULL, &pmtu, NULL); rtcache_unref(rt, ro); if (error) break; if (pmtu > IPV6_MAXPACKET) pmtu = IPV6_MAXPACKET; memset(&mtuinfo, 0, sizeof(mtuinfo)); mtuinfo.ip6m_mtu = (u_int32_t)pmtu; optdata = (void *)&mtuinfo; optdatalen = sizeof(mtuinfo); if (optdatalen > MCLBYTES) return (EMSGSIZE); /* XXX */ error = sockopt_set(sopt, optdata, optdatalen); break; } #ifdef RFC2292 case IPV6_2292PKTINFO: case IPV6_2292HOPLIMIT: case IPV6_2292HOPOPTS: case IPV6_2292RTHDR: case IPV6_2292DSTOPTS: switch (optname) { case IPV6_2292PKTINFO: optval = OPTBIT(IN6P_PKTINFO); break; case IPV6_2292HOPLIMIT: optval = OPTBIT(IN6P_HOPLIMIT); break; case IPV6_2292HOPOPTS: optval = OPTBIT(IN6P_HOPOPTS); break; case IPV6_2292RTHDR: optval = OPTBIT(IN6P_RTHDR); break; case IPV6_2292DSTOPTS: optval = OPTBIT(IN6P_DSTOPTS|IN6P_RTHDRDSTOPTS); break; } error = sockopt_setint(sopt, optval); break; #endif case IPV6_PKTINFO: case IPV6_HOPOPTS: case IPV6_RTHDR: case IPV6_DSTOPTS: case IPV6_RTHDRDSTOPTS: case IPV6_NEXTHOP: case IPV6_OTCLASS: case IPV6_TCLASS: case IPV6_DONTFRAG: case IPV6_USE_MIN_MTU: case IPV6_PREFER_TEMPADDR: error = ip6_getpcbopt(in6p->in6p_outputopts, optname, sopt); break; case IPV6_MULTICAST_IF: case IPV6_MULTICAST_HOPS: case IPV6_MULTICAST_LOOP: case IPV6_JOIN_GROUP: case IPV6_LEAVE_GROUP: error = ip6_getmoptions(sopt, in6p); break; case IPV6_PORTALGO: optval = ((struct inpcb_hdr *)in6p)->inph_portalgo; error = sockopt_setint(sopt, optval); break; #if defined(IPSEC) case IPV6_IPSEC_POLICY: if (ipsec_used) { struct mbuf *m = NULL; /* * XXX: this will return EINVAL as sopt is * empty */ error = ipsec_get_policy(in6p, sopt->sopt_data, sopt->sopt_size, &m); if (!error) error = sockopt_setmbuf(sopt, m); } else error = ENOPROTOOPT; break; #endif /* IPSEC */ default: error = ENOPROTOOPT; break; } break; } return (error); } int ip6_raw_ctloutput(int op, struct socket *so, struct sockopt *sopt) { int error = 0, optval; const int icmp6off = offsetof(struct icmp6_hdr, icmp6_cksum); struct in6pcb *in6p = sotoin6pcb(so); int level, optname; KASSERT(sopt != NULL); level = sopt->sopt_level; optname = sopt->sopt_name; if (level != IPPROTO_IPV6) { return ENOPROTOOPT; } switch (optname) { case IPV6_CHECKSUM: /* * For ICMPv6 sockets, no modification allowed for checksum * offset, permit "no change" values to help existing apps. * * XXX RFC3542 says: "An attempt to set IPV6_CHECKSUM * for an ICMPv6 socket will fail." The current * behavior does not meet RFC3542. */ switch (op) { case PRCO_SETOPT: error = sockopt_getint(sopt, &optval); if (error) break; if ((optval % 2) != 0) { /* the API assumes even offset values */ error = EINVAL; } else if (so->so_proto->pr_protocol == IPPROTO_ICMPV6) { if (optval != icmp6off) error = EINVAL; } else in6p->in6p_cksum = optval; break; case PRCO_GETOPT: if (so->so_proto->pr_protocol == IPPROTO_ICMPV6) optval = icmp6off; else optval = in6p->in6p_cksum; error = sockopt_setint(sopt, optval); break; default: error = EINVAL; break; } break; default: error = ENOPROTOOPT; break; } return (error); } #ifdef RFC2292 /* * Set up IP6 options in pcb for insertion in output packets or * specifying behavior of outgoing packets. */ static int ip6_pcbopts(struct ip6_pktopts **pktopt, struct socket *so, struct sockopt *sopt) { struct ip6_pktopts *opt = *pktopt; struct mbuf *m; int error = 0; KASSERT(solocked(so)); /* turn off any old options. */ if (opt) { #ifdef DIAGNOSTIC if (opt->ip6po_pktinfo || opt->ip6po_nexthop || opt->ip6po_hbh || opt->ip6po_dest1 || opt->ip6po_dest2 || opt->ip6po_rhinfo.ip6po_rhi_rthdr) printf("ip6_pcbopts: all specified options are cleared.\n"); #endif ip6_clearpktopts(opt, -1); } else { opt = malloc(sizeof(*opt), M_IP6OPT, M_NOWAIT); if (opt == NULL) return (ENOBUFS); } *pktopt = NULL; if (sopt == NULL || sopt->sopt_size == 0) { /* * Only turning off any previous options, regardless of * whether the opt is just created or given. */ free(opt, M_IP6OPT); return (0); } /* set options specified by user. */ m = sockopt_getmbuf(sopt); if (m == NULL) { free(opt, M_IP6OPT); return (ENOBUFS); } error = ip6_setpktopts(m, opt, NULL, kauth_cred_get(), so->so_proto->pr_protocol); m_freem(m); if (error != 0) { ip6_clearpktopts(opt, -1); /* XXX: discard all options */ free(opt, M_IP6OPT); return (error); } *pktopt = opt; return (0); } #endif /* * initialize ip6_pktopts. beware that there are non-zero default values in * the struct. */ void ip6_initpktopts(struct ip6_pktopts *opt) { memset(opt, 0, sizeof(*opt)); opt->ip6po_hlim = -1; /* -1 means default hop limit */ opt->ip6po_tclass = -1; /* -1 means default traffic class */ opt->ip6po_minmtu = IP6PO_MINMTU_MCASTONLY; opt->ip6po_prefer_tempaddr = IP6PO_TEMPADDR_SYSTEM; } #define sin6tosa(sin6) ((struct sockaddr *)(sin6)) /* XXX */ static int ip6_pcbopt(int optname, u_char *buf, int len, struct ip6_pktopts **pktopt, kauth_cred_t cred, int uproto) { struct ip6_pktopts *opt; if (*pktopt == NULL) { *pktopt = malloc(sizeof(struct ip6_pktopts), M_IP6OPT, M_NOWAIT); if (*pktopt == NULL) return (ENOBUFS); ip6_initpktopts(*pktopt); } opt = *pktopt; return (ip6_setpktopt(optname, buf, len, opt, cred, 1, 0, uproto)); } static int ip6_getpcbopt(struct ip6_pktopts *pktopt, int optname, struct sockopt *sopt) { void *optdata = NULL; int optdatalen = 0; struct ip6_ext *ip6e; int error = 0; struct in6_pktinfo null_pktinfo; int deftclass = 0, on; int defminmtu = IP6PO_MINMTU_MCASTONLY; int defpreftemp = IP6PO_TEMPADDR_SYSTEM; switch (optname) { case IPV6_PKTINFO: if (pktopt && pktopt->ip6po_pktinfo) optdata = (void *)pktopt->ip6po_pktinfo; else { /* XXX: we don't have to do this every time... */ memset(&null_pktinfo, 0, sizeof(null_pktinfo)); optdata = (void *)&null_pktinfo; } optdatalen = sizeof(struct in6_pktinfo); break; case IPV6_OTCLASS: /* XXX */ return (EINVAL); case IPV6_TCLASS: if (pktopt && pktopt->ip6po_tclass >= 0) optdata = (void *)&pktopt->ip6po_tclass; else optdata = (void *)&deftclass; optdatalen = sizeof(int); break; case IPV6_HOPOPTS: if (pktopt && pktopt->ip6po_hbh) { optdata = (void *)pktopt->ip6po_hbh; ip6e = (struct ip6_ext *)pktopt->ip6po_hbh; optdatalen = (ip6e->ip6e_len + 1) << 3; } break; case IPV6_RTHDR: if (pktopt && pktopt->ip6po_rthdr) { optdata = (void *)pktopt->ip6po_rthdr; ip6e = (struct ip6_ext *)pktopt->ip6po_rthdr; optdatalen = (ip6e->ip6e_len + 1) << 3; } break; case IPV6_RTHDRDSTOPTS: if (pktopt && pktopt->ip6po_dest1) { optdata = (void *)pktopt->ip6po_dest1; ip6e = (struct ip6_ext *)pktopt->ip6po_dest1; optdatalen = (ip6e->ip6e_len + 1) << 3; } break; case IPV6_DSTOPTS: if (pktopt && pktopt->ip6po_dest2) { optdata = (void *)pktopt->ip6po_dest2; ip6e = (struct ip6_ext *)pktopt->ip6po_dest2; optdatalen = (ip6e->ip6e_len + 1) << 3; } break; case IPV6_NEXTHOP: if (pktopt && pktopt->ip6po_nexthop) { optdata = (void *)pktopt->ip6po_nexthop; optdatalen = pktopt->ip6po_nexthop->sa_len; } break; case IPV6_USE_MIN_MTU: if (pktopt) optdata = (void *)&pktopt->ip6po_minmtu; else optdata = (void *)&defminmtu; optdatalen = sizeof(int); break; case IPV6_DONTFRAG: if (pktopt && ((pktopt->ip6po_flags) & IP6PO_DONTFRAG)) on = 1; else on = 0; optdata = (void *)&on; optdatalen = sizeof(on); break; case IPV6_PREFER_TEMPADDR: if (pktopt) optdata = (void *)&pktopt->ip6po_prefer_tempaddr; else optdata = (void *)&defpreftemp; optdatalen = sizeof(int); break; default: /* should not happen */ #ifdef DIAGNOSTIC panic("ip6_getpcbopt: unexpected option\n"); #endif return (ENOPROTOOPT); } error = sockopt_set(sopt, optdata, optdatalen); return (error); } void ip6_clearpktopts(struct ip6_pktopts *pktopt, int optname) { if (optname == -1 || optname == IPV6_PKTINFO) { if (pktopt->ip6po_pktinfo) free(pktopt->ip6po_pktinfo, M_IP6OPT); pktopt->ip6po_pktinfo = NULL; } if (optname == -1 || optname == IPV6_HOPLIMIT) pktopt->ip6po_hlim = -1; if (optname == -1 || optname == IPV6_TCLASS) pktopt->ip6po_tclass = -1; if (optname == -1 || optname == IPV6_NEXTHOP) { rtcache_free(&pktopt->ip6po_nextroute); if (pktopt->ip6po_nexthop) free(pktopt->ip6po_nexthop, M_IP6OPT); pktopt->ip6po_nexthop = NULL; } if (optname == -1 || optname == IPV6_HOPOPTS) { if (pktopt->ip6po_hbh) free(pktopt->ip6po_hbh, M_IP6OPT); pktopt->ip6po_hbh = NULL; } if (optname == -1 || optname == IPV6_RTHDRDSTOPTS) { if (pktopt->ip6po_dest1) free(pktopt->ip6po_dest1, M_IP6OPT); pktopt->ip6po_dest1 = NULL; } if (optname == -1 || optname == IPV6_RTHDR) { if (pktopt->ip6po_rhinfo.ip6po_rhi_rthdr) free(pktopt->ip6po_rhinfo.ip6po_rhi_rthdr, M_IP6OPT); pktopt->ip6po_rhinfo.ip6po_rhi_rthdr = NULL; rtcache_free(&pktopt->ip6po_route); } if (optname == -1 || optname == IPV6_DSTOPTS) { if (pktopt->ip6po_dest2) free(pktopt->ip6po_dest2, M_IP6OPT); pktopt->ip6po_dest2 = NULL; } } #define PKTOPT_EXTHDRCPY(type) \ do { \ if (src->type) { \ int hlen = (((struct ip6_ext *)src->type)->ip6e_len + 1) << 3;\ dst->type = malloc(hlen, M_IP6OPT, canwait); \ if (dst->type == NULL) \ goto bad; \ memcpy(dst->type, src->type, hlen); \ } \ } while (/*CONSTCOND*/ 0) static int copypktopts(struct ip6_pktopts *dst, struct ip6_pktopts *src, int canwait) { dst->ip6po_hlim = src->ip6po_hlim; dst->ip6po_tclass = src->ip6po_tclass; dst->ip6po_flags = src->ip6po_flags; dst->ip6po_minmtu = src->ip6po_minmtu; dst->ip6po_prefer_tempaddr = src->ip6po_prefer_tempaddr; if (src->ip6po_pktinfo) { dst->ip6po_pktinfo = malloc(sizeof(*dst->ip6po_pktinfo), M_IP6OPT, canwait); if (dst->ip6po_pktinfo == NULL) goto bad; *dst->ip6po_pktinfo = *src->ip6po_pktinfo; } if (src->ip6po_nexthop) { dst->ip6po_nexthop = malloc(src->ip6po_nexthop->sa_len, M_IP6OPT, canwait); if (dst->ip6po_nexthop == NULL) goto bad; memcpy(dst->ip6po_nexthop, src->ip6po_nexthop, src->ip6po_nexthop->sa_len); } PKTOPT_EXTHDRCPY(ip6po_hbh); PKTOPT_EXTHDRCPY(ip6po_dest1); PKTOPT_EXTHDRCPY(ip6po_dest2); PKTOPT_EXTHDRCPY(ip6po_rthdr); /* not copy the cached route */ return (0); bad: if (dst->ip6po_pktinfo) free(dst->ip6po_pktinfo, M_IP6OPT); if (dst->ip6po_nexthop) free(dst->ip6po_nexthop, M_IP6OPT); if (dst->ip6po_hbh) free(dst->ip6po_hbh, M_IP6OPT); if (dst->ip6po_dest1) free(dst->ip6po_dest1, M_IP6OPT); if (dst->ip6po_dest2) free(dst->ip6po_dest2, M_IP6OPT); if (dst->ip6po_rthdr) free(dst->ip6po_rthdr, M_IP6OPT); return (ENOBUFS); } #undef PKTOPT_EXTHDRCPY struct ip6_pktopts * ip6_copypktopts(struct ip6_pktopts *src, int canwait) { int error; struct ip6_pktopts *dst; dst = malloc(sizeof(*dst), M_IP6OPT, canwait); if (dst == NULL) return (NULL); ip6_initpktopts(dst); if ((error = copypktopts(dst, src, canwait)) != 0) { free(dst, M_IP6OPT); return (NULL); } return (dst); } void ip6_freepcbopts(struct ip6_pktopts *pktopt) { if (pktopt == NULL) return; ip6_clearpktopts(pktopt, -1); free(pktopt, M_IP6OPT); } int ip6_get_membership(const struct sockopt *sopt, struct ifnet **ifp, struct psref *psref, void *v, size_t l) { struct ipv6_mreq mreq; int error; struct in6_addr *ia = &mreq.ipv6mr_multiaddr; struct in_addr *ia4 = (void *)&ia->s6_addr32[3]; error = sockopt_get(sopt, &mreq, sizeof(mreq)); if (error != 0) return error; if (IN6_IS_ADDR_UNSPECIFIED(ia)) { /* * We use the unspecified address to specify to accept * all multicast addresses. Only super user is allowed * to do this. */ if (kauth_authorize_network(kauth_cred_get(), KAUTH_NETWORK_IPV6, KAUTH_REQ_NETWORK_IPV6_JOIN_MULTICAST, NULL, NULL, NULL)) return EACCES; } else if (IN6_IS_ADDR_V4MAPPED(ia)) { // Don't bother if we are not going to use ifp. if (l == sizeof(*ia)) { memcpy(v, ia, l); return 0; } } else if (!IN6_IS_ADDR_MULTICAST(ia)) { return EINVAL; } /* * If no interface was explicitly specified, choose an * appropriate one according to the given multicast address. */ if (mreq.ipv6mr_interface == 0) { struct rtentry *rt; union { struct sockaddr dst; struct sockaddr_in dst4; struct sockaddr_in6 dst6; } u; struct route ro; /* * Look up the routing table for the * address, and choose the outgoing interface. * XXX: is it a good approach? */ memset(&ro, 0, sizeof(ro)); if (IN6_IS_ADDR_V4MAPPED(ia)) sockaddr_in_init(&u.dst4, ia4, 0); else sockaddr_in6_init(&u.dst6, ia, 0, 0, 0); error = rtcache_setdst(&ro, &u.dst); if (error != 0) return error; rt = rtcache_init(&ro); *ifp = rt != NULL ? if_get_byindex(rt->rt_ifp->if_index, psref) : NULL; rtcache_unref(rt, &ro); rtcache_free(&ro); } else { /* * If the interface is specified, validate it. */ *ifp = if_get_byindex(mreq.ipv6mr_interface, psref); if (*ifp == NULL) return ENXIO; /* XXX EINVAL? */ } if (sizeof(*ia) == l) memcpy(v, ia, l); else memcpy(v, ia4, l); return 0; } /* * Set the IP6 multicast options in response to user setsockopt(). */ static int ip6_setmoptions(const struct sockopt *sopt, struct in6pcb *in6p) { int error = 0; u_int loop, ifindex; struct ipv6_mreq mreq; struct in6_addr ia; struct ifnet *ifp; struct ip6_moptions *im6o = in6p->in6p_moptions; struct in6_multi_mship *imm; KASSERT(in6p_locked(in6p)); if (im6o == NULL) { /* * No multicast option buffer attached to the pcb; * allocate one and initialize to default values. */ im6o = malloc(sizeof(*im6o), M_IPMOPTS, M_NOWAIT); if (im6o == NULL) return (ENOBUFS); in6p->in6p_moptions = im6o; im6o->im6o_multicast_if_index = 0; im6o->im6o_multicast_hlim = ip6_defmcasthlim; im6o->im6o_multicast_loop = IPV6_DEFAULT_MULTICAST_LOOP; LIST_INIT(&im6o->im6o_memberships); } switch (sopt->sopt_name) { case IPV6_MULTICAST_IF: { int s; /* * Select the interface for outgoing multicast packets. */ error = sockopt_get(sopt, &ifindex, sizeof(ifindex)); if (error != 0) break; s = pserialize_read_enter(); if (ifindex != 0) { if ((ifp = if_byindex(ifindex)) == NULL) { pserialize_read_exit(s); error = ENXIO; /* XXX EINVAL? */ break; } if ((ifp->if_flags & IFF_MULTICAST) == 0) { pserialize_read_exit(s); error = EADDRNOTAVAIL; break; } } else ifp = NULL; im6o->im6o_multicast_if_index = if_get_index(ifp); pserialize_read_exit(s); break; } case IPV6_MULTICAST_HOPS: { /* * Set the IP6 hoplimit for outgoing multicast packets. */ int optval; error = sockopt_getint(sopt, &optval); if (error != 0) break; if (optval < -1 || optval >= 256) error = EINVAL; else if (optval == -1) im6o->im6o_multicast_hlim = ip6_defmcasthlim; else im6o->im6o_multicast_hlim = optval; break; } case IPV6_MULTICAST_LOOP: /* * Set the loopback flag for outgoing multicast packets. * Must be zero or one. */ error = sockopt_get(sopt, &loop, sizeof(loop)); if (error != 0) break; if (loop > 1) { error = EINVAL; break; } im6o->im6o_multicast_loop = loop; break; case IPV6_JOIN_GROUP: { int bound; struct psref psref; /* * Add a multicast group membership. * Group must be a valid IP6 multicast address. */ bound = curlwp_bind(); ifp = NULL; error = ip6_get_membership(sopt, &ifp, &psref, &ia, sizeof(ia)); if (error != 0) { KASSERT(ifp == NULL); curlwp_bindx(bound); return error; } if (IN6_IS_ADDR_V4MAPPED(&ia)) { error = ip_setmoptions(&in6p->in6p_v4moptions, sopt); goto put_break; } /* * See if we found an interface, and confirm that it * supports multicast */ if (ifp == NULL || (ifp->if_flags & IFF_MULTICAST) == 0) { error = EADDRNOTAVAIL; goto put_break; } if (in6_setscope(&ia, ifp, NULL)) { error = EADDRNOTAVAIL; /* XXX: should not happen */ goto put_break; } /* * See if the membership already exists. */ LIST_FOREACH(imm, &im6o->im6o_memberships, i6mm_chain) { if (imm->i6mm_maddr->in6m_ifp == ifp && IN6_ARE_ADDR_EQUAL(&imm->i6mm_maddr->in6m_addr, &ia)) goto put_break; } if (imm != NULL) { error = EADDRINUSE; goto put_break; } /* * Everything looks good; add a new record to the multicast * address list for the given interface. */ imm = in6_joingroup(ifp, &ia, &error, 0); if (imm == NULL) goto put_break; LIST_INSERT_HEAD(&im6o->im6o_memberships, imm, i6mm_chain); put_break: if_put(ifp, &psref); curlwp_bindx(bound); break; } case IPV6_LEAVE_GROUP: { /* * Drop a multicast group membership. * Group must be a valid IP6 multicast address. */ error = sockopt_get(sopt, &mreq, sizeof(mreq)); if (error != 0) break; if (IN6_IS_ADDR_V4MAPPED(&mreq.ipv6mr_multiaddr)) { error = ip_setmoptions(&in6p->in6p_v4moptions, sopt); break; } /* * If an interface address was specified, get a pointer * to its ifnet structure. */ if (mreq.ipv6mr_interface != 0) { if ((ifp = if_byindex(mreq.ipv6mr_interface)) == NULL) { error = ENXIO; /* XXX EINVAL? */ break; } } else ifp = NULL; /* Fill in the scope zone ID */ if (ifp) { if (in6_setscope(&mreq.ipv6mr_multiaddr, ifp, NULL)) { /* XXX: should not happen */ error = EADDRNOTAVAIL; break; } } else if (mreq.ipv6mr_interface != 0) { /* * XXX: This case would happens when the (positive) * index is in the valid range, but the corresponding * interface has been detached dynamically. The above * check probably avoids such case to happen here, but * we check it explicitly for safety. */ error = EADDRNOTAVAIL; break; } else { /* ipv6mr_interface == 0 */ struct sockaddr_in6 sa6_mc; /* * The API spec says as follows: * If the interface index is specified as 0, the * system may choose a multicast group membership to * drop by matching the multicast address only. * On the other hand, we cannot disambiguate the scope * zone unless an interface is provided. Thus, we * check if there's ambiguity with the default scope * zone as the last resort. */ sockaddr_in6_init(&sa6_mc, &mreq.ipv6mr_multiaddr, 0, 0, 0); error = sa6_embedscope(&sa6_mc, ip6_use_defzone); if (error != 0) break; mreq.ipv6mr_multiaddr = sa6_mc.sin6_addr; } /* * Find the membership in the membership list. */ LIST_FOREACH(imm, &im6o->im6o_memberships, i6mm_chain) { if ((ifp == NULL || imm->i6mm_maddr->in6m_ifp == ifp) && IN6_ARE_ADDR_EQUAL(&imm->i6mm_maddr->in6m_addr, &mreq.ipv6mr_multiaddr)) break; } if (imm == NULL) { /* Unable to resolve interface */ error = EADDRNOTAVAIL; break; } /* * Give up the multicast address record to which the * membership points. */ LIST_REMOVE(imm, i6mm_chain); in6_leavegroup(imm); /* in6m_ifp should not leave thanks to in6p_lock */ break; } default: error = EOPNOTSUPP; break; } /* * If all options have default values, no need to keep the mbuf. */ if (im6o->im6o_multicast_if_index == 0 && im6o->im6o_multicast_hlim == ip6_defmcasthlim && im6o->im6o_multicast_loop == IPV6_DEFAULT_MULTICAST_LOOP && LIST_EMPTY(&im6o->im6o_memberships)) { free(in6p->in6p_moptions, M_IPMOPTS); in6p->in6p_moptions = NULL; } return (error); } /* * Return the IP6 multicast options in response to user getsockopt(). */ static int ip6_getmoptions(struct sockopt *sopt, struct in6pcb *in6p) { u_int optval; int error; struct ip6_moptions *im6o = in6p->in6p_moptions; switch (sopt->sopt_name) { case IPV6_MULTICAST_IF: if (im6o == NULL || im6o->im6o_multicast_if_index == 0) optval = 0; else optval = im6o->im6o_multicast_if_index; error = sockopt_set(sopt, &optval, sizeof(optval)); break; case IPV6_MULTICAST_HOPS: if (im6o == NULL) optval = ip6_defmcasthlim; else optval = im6o->im6o_multicast_hlim; error = sockopt_set(sopt, &optval, sizeof(optval)); break; case IPV6_MULTICAST_LOOP: if (im6o == NULL) optval = IPV6_DEFAULT_MULTICAST_LOOP; else optval = im6o->im6o_multicast_loop; error = sockopt_set(sopt, &optval, sizeof(optval)); break; default: error = EOPNOTSUPP; } return (error); } /* * Discard the IP6 multicast options. */ void ip6_freemoptions(struct ip6_moptions *im6o) { struct in6_multi_mship *imm, *nimm; if (im6o == NULL) return; /* The owner of im6o (in6p) should be protected by solock */ LIST_FOREACH_SAFE(imm, &im6o->im6o_memberships, i6mm_chain, nimm) { LIST_REMOVE(imm, i6mm_chain); in6_leavegroup(imm); } free(im6o, M_IPMOPTS); } /* * Set IPv6 outgoing packet options based on advanced API. */ int ip6_setpktopts(struct mbuf *control, struct ip6_pktopts *opt, struct ip6_pktopts *stickyopt, kauth_cred_t cred, int uproto) { struct cmsghdr *cm = 0; if (control == NULL || opt == NULL) return (EINVAL); ip6_initpktopts(opt); if (stickyopt) { int error; /* * If stickyopt is provided, make a local copy of the options * for this particular packet, then override them by ancillary * objects. * XXX: copypktopts() does not copy the cached route to a next * hop (if any). This is not very good in terms of efficiency, * but we can allow this since this option should be rarely * used. */ if ((error = copypktopts(opt, stickyopt, M_NOWAIT)) != 0) return (error); } /* * XXX: Currently, we assume all the optional information is stored * in a single mbuf. */ if (control->m_next) return (EINVAL); /* XXX if cm->cmsg_len is not aligned, control->m_len can become <0 */ for (; control->m_len > 0; control->m_data += CMSG_ALIGN(cm->cmsg_len), control->m_len -= CMSG_ALIGN(cm->cmsg_len)) { int error; if (control->m_len < CMSG_LEN(0)) return (EINVAL); cm = mtod(control, struct cmsghdr *); if (cm->cmsg_len < CMSG_LEN(0) || cm->cmsg_len > control->m_len) return (EINVAL); if (cm->cmsg_level != IPPROTO_IPV6) continue; error = ip6_setpktopt(cm->cmsg_type, CMSG_DATA(cm), cm->cmsg_len - CMSG_LEN(0), opt, cred, 0, 1, uproto); if (error) return (error); } return (0); } /* * Set a particular packet option, as a sticky option or an ancillary data * item. "len" can be 0 only when it's a sticky option. * We have 4 cases of combination of "sticky" and "cmsg": * "sticky=0, cmsg=0": impossible * "sticky=0, cmsg=1": RFC2292 or RFC3542 ancillary data * "sticky=1, cmsg=0": RFC3542 socket option * "sticky=1, cmsg=1": RFC2292 socket option */ static int ip6_setpktopt(int optname, u_char *buf, int len, struct ip6_pktopts *opt, kauth_cred_t cred, int sticky, int cmsg, int uproto) { int minmtupolicy; int error; if (!sticky && !cmsg) { #ifdef DIAGNOSTIC printf("ip6_setpktopt: impossible case\n"); #endif return (EINVAL); } /* * IPV6_2292xxx is for backward compatibility to RFC2292, and should * not be specified in the context of RFC3542. Conversely, * RFC3542 types should not be specified in the context of RFC2292. */ if (!cmsg) { switch (optname) { case IPV6_2292PKTINFO: case IPV6_2292HOPLIMIT: case IPV6_2292NEXTHOP: case IPV6_2292HOPOPTS: case IPV6_2292DSTOPTS: case IPV6_2292RTHDR: case IPV6_2292PKTOPTIONS: return (ENOPROTOOPT); } } if (sticky && cmsg) { switch (optname) { case IPV6_PKTINFO: case IPV6_HOPLIMIT: case IPV6_NEXTHOP: case IPV6_HOPOPTS: case IPV6_DSTOPTS: case IPV6_RTHDRDSTOPTS: case IPV6_RTHDR: case IPV6_USE_MIN_MTU: case IPV6_DONTFRAG: case IPV6_OTCLASS: case IPV6_TCLASS: case IPV6_PREFER_TEMPADDR: /* XXX not an RFC3542 option */ return (ENOPROTOOPT); } } switch (optname) { #ifdef RFC2292 case IPV6_2292PKTINFO: #endif case IPV6_PKTINFO: { struct in6_pktinfo *pktinfo; if (len != sizeof(struct in6_pktinfo)) return (EINVAL); pktinfo = (struct in6_pktinfo *)buf; /* * An application can clear any sticky IPV6_PKTINFO option by * doing a "regular" setsockopt with ipi6_addr being * in6addr_any and ipi6_ifindex being zero. * [RFC 3542, Section 6] */ if (optname == IPV6_PKTINFO && opt->ip6po_pktinfo && pktinfo->ipi6_ifindex == 0 && IN6_IS_ADDR_UNSPECIFIED(&pktinfo->ipi6_addr)) { ip6_clearpktopts(opt, optname); break; } if (uproto == IPPROTO_TCP && optname == IPV6_PKTINFO && sticky && !IN6_IS_ADDR_UNSPECIFIED(&pktinfo->ipi6_addr)) { return (EINVAL); } /* Validate the interface index if specified. */ if (pktinfo->ipi6_ifindex) { struct ifnet *ifp; int s = pserialize_read_enter(); ifp = if_byindex(pktinfo->ipi6_ifindex); if (ifp == NULL) { pserialize_read_exit(s); return ENXIO; } pserialize_read_exit(s); } /* * We store the address anyway, and let in6_selectsrc() * validate the specified address. This is because ipi6_addr * may not have enough information about its scope zone, and * we may need additional information (such as outgoing * interface or the scope zone of a destination address) to * disambiguate the scope. * XXX: the delay of the validation may confuse the * application when it is used as a sticky option. */ if (opt->ip6po_pktinfo == NULL) { opt->ip6po_pktinfo = malloc(sizeof(*pktinfo), M_IP6OPT, M_NOWAIT); if (opt->ip6po_pktinfo == NULL) return (ENOBUFS); } memcpy(opt->ip6po_pktinfo, pktinfo, sizeof(*pktinfo)); break; } #ifdef RFC2292 case IPV6_2292HOPLIMIT: #endif case IPV6_HOPLIMIT: { int *hlimp; /* * RFC 3542 deprecated the usage of sticky IPV6_HOPLIMIT * to simplify the ordering among hoplimit options. */ if (optname == IPV6_HOPLIMIT && sticky) return (ENOPROTOOPT); if (len != sizeof(int)) return (EINVAL); hlimp = (int *)buf; if (*hlimp < -1 || *hlimp > 255) return (EINVAL); opt->ip6po_hlim = *hlimp; break; } case IPV6_OTCLASS: if (len != sizeof(u_int8_t)) return (EINVAL); opt->ip6po_tclass = *(u_int8_t *)buf; break; case IPV6_TCLASS: { int tclass; if (len != sizeof(int)) return (EINVAL); tclass = *(int *)buf; if (tclass < -1 || tclass > 255) return (EINVAL); opt->ip6po_tclass = tclass; break; } #ifdef RFC2292 case IPV6_2292NEXTHOP: #endif case IPV6_NEXTHOP: error = kauth_authorize_network(cred, KAUTH_NETWORK_IPV6, KAUTH_REQ_NETWORK_IPV6_HOPBYHOP, NULL, NULL, NULL); if (error) return (error); if (len == 0) { /* just remove the option */ ip6_clearpktopts(opt, IPV6_NEXTHOP); break; } /* check if cmsg_len is large enough for sa_len */ if (len < sizeof(struct sockaddr) || len < *buf) return (EINVAL); switch (((struct sockaddr *)buf)->sa_family) { case AF_INET6: { struct sockaddr_in6 *sa6 = (struct sockaddr_in6 *)buf; if (sa6->sin6_len != sizeof(struct sockaddr_in6)) return (EINVAL); if (IN6_IS_ADDR_UNSPECIFIED(&sa6->sin6_addr) || IN6_IS_ADDR_MULTICAST(&sa6->sin6_addr)) { return (EINVAL); } if ((error = sa6_embedscope(sa6, ip6_use_defzone)) != 0) { return (error); } break; } case AF_LINK: /* eventually be supported? */ default: return (EAFNOSUPPORT); } /* turn off the previous option, then set the new option. */ ip6_clearpktopts(opt, IPV6_NEXTHOP); opt->ip6po_nexthop = malloc(*buf, M_IP6OPT, M_NOWAIT); if (opt->ip6po_nexthop == NULL) return (ENOBUFS); memcpy(opt->ip6po_nexthop, buf, *buf); break; #ifdef RFC2292 case IPV6_2292HOPOPTS: #endif case IPV6_HOPOPTS: { struct ip6_hbh *hbh; int hbhlen; /* * XXX: We don't allow a non-privileged user to set ANY HbH * options, since per-option restriction has too much * overhead. */ error = kauth_authorize_network(cred, KAUTH_NETWORK_IPV6, KAUTH_REQ_NETWORK_IPV6_HOPBYHOP, NULL, NULL, NULL); if (error) return (error); if (len == 0) { ip6_clearpktopts(opt, IPV6_HOPOPTS); break; /* just remove the option */ } /* message length validation */ if (len < sizeof(struct ip6_hbh)) return (EINVAL); hbh = (struct ip6_hbh *)buf; hbhlen = (hbh->ip6h_len + 1) << 3; if (len != hbhlen) return (EINVAL); /* turn off the previous option, then set the new option. */ ip6_clearpktopts(opt, IPV6_HOPOPTS); opt->ip6po_hbh = malloc(hbhlen, M_IP6OPT, M_NOWAIT); if (opt->ip6po_hbh == NULL) return (ENOBUFS); memcpy(opt->ip6po_hbh, hbh, hbhlen); break; } #ifdef RFC2292 case IPV6_2292DSTOPTS: #endif case IPV6_DSTOPTS: case IPV6_RTHDRDSTOPTS: { struct ip6_dest *dest, **newdest = NULL; int destlen; /* XXX: see the comment for IPV6_HOPOPTS */ error = kauth_authorize_network(cred, KAUTH_NETWORK_IPV6, KAUTH_REQ_NETWORK_IPV6_HOPBYHOP, NULL, NULL, NULL); if (error) return (error); if (len == 0) { ip6_clearpktopts(opt, optname); break; /* just remove the option */ } /* message length validation */ if (len < sizeof(struct ip6_dest)) return (EINVAL); dest = (struct ip6_dest *)buf; destlen = (dest->ip6d_len + 1) << 3; if (len != destlen) return (EINVAL); /* * Determine the position that the destination options header * should be inserted; before or after the routing header. */ switch (optname) { case IPV6_2292DSTOPTS: /* * The old advanced API is ambiguous on this point. * Our approach is to determine the position based * according to the existence of a routing header. * Note, however, that this depends on the order of the * extension headers in the ancillary data; the 1st * part of the destination options header must appear * before the routing header in the ancillary data, * too. * RFC3542 solved the ambiguity by introducing * separate ancillary data or option types. */ if (opt->ip6po_rthdr == NULL) newdest = &opt->ip6po_dest1; else newdest = &opt->ip6po_dest2; break; case IPV6_RTHDRDSTOPTS: newdest = &opt->ip6po_dest1; break; case IPV6_DSTOPTS: newdest = &opt->ip6po_dest2; break; } /* turn off the previous option, then set the new option. */ ip6_clearpktopts(opt, optname); *newdest = malloc(destlen, M_IP6OPT, M_NOWAIT); if (*newdest == NULL) return (ENOBUFS); memcpy(*newdest, dest, destlen); break; } #ifdef RFC2292 case IPV6_2292RTHDR: #endif case IPV6_RTHDR: { struct ip6_rthdr *rth; int rthlen; if (len == 0) { ip6_clearpktopts(opt, IPV6_RTHDR); break; /* just remove the option */ } /* message length validation */ if (len < sizeof(struct ip6_rthdr)) return (EINVAL); rth = (struct ip6_rthdr *)buf; rthlen = (rth->ip6r_len + 1) << 3; if (len != rthlen) return (EINVAL); switch (rth->ip6r_type) { case IPV6_RTHDR_TYPE_0: /* Dropped, RFC5095. */ default: return (EINVAL); /* not supported */ } /* turn off the previous option */ ip6_clearpktopts(opt, IPV6_RTHDR); opt->ip6po_rthdr = malloc(rthlen, M_IP6OPT, M_NOWAIT); if (opt->ip6po_rthdr == NULL) return (ENOBUFS); memcpy(opt->ip6po_rthdr, rth, rthlen); break; } case IPV6_USE_MIN_MTU: if (len != sizeof(int)) return (EINVAL); minmtupolicy = *(int *)buf; if (minmtupolicy != IP6PO_MINMTU_MCASTONLY && minmtupolicy != IP6PO_MINMTU_DISABLE && minmtupolicy != IP6PO_MINMTU_ALL) { return (EINVAL); } opt->ip6po_minmtu = minmtupolicy; break; case IPV6_DONTFRAG: if (len != sizeof(int)) return (EINVAL); if (uproto == IPPROTO_TCP || *(int *)buf == 0) { /* * we ignore this option for TCP sockets. * (RFC3542 leaves this case unspecified.) */ opt->ip6po_flags &= ~IP6PO_DONTFRAG; } else opt->ip6po_flags |= IP6PO_DONTFRAG; break; case IPV6_PREFER_TEMPADDR: { int preftemp; if (len != sizeof(int)) return (EINVAL); preftemp = *(int *)buf; switch (preftemp) { case IP6PO_TEMPADDR_SYSTEM: case IP6PO_TEMPADDR_NOTPREFER: case IP6PO_TEMPADDR_PREFER: break; default: return (EINVAL); } opt->ip6po_prefer_tempaddr = preftemp; break; } default: return (ENOPROTOOPT); } /* end of switch */ return (0); } /* * Routine called from ip6_output() to loop back a copy of an IP6 multicast * packet to the input queue of a specified interface. Note that this * calls the output routine of the loopback "driver", but with an interface * pointer that might NOT be lo0ifp -- easier than replicating that code here. */ void ip6_mloopback(struct ifnet *ifp, struct mbuf *m, const struct sockaddr_in6 *dst) { struct mbuf *copym; struct ip6_hdr *ip6; copym = m_copypacket(m, M_DONTWAIT); if (copym == NULL) return; /* * Make sure to deep-copy IPv6 header portion in case the data * is in an mbuf cluster, so that we can safely override the IPv6 * header portion later. */ if ((copym->m_flags & M_EXT) != 0 || copym->m_len < sizeof(struct ip6_hdr)) { copym = m_pullup(copym, sizeof(struct ip6_hdr)); if (copym == NULL) return; } #ifdef DIAGNOSTIC if (copym->m_len < sizeof(*ip6)) { m_freem(copym); return; } #endif ip6 = mtod(copym, struct ip6_hdr *); /* * clear embedded scope identifiers if necessary. * in6_clearscope will touch the addresses only when necessary. */ in6_clearscope(&ip6->ip6_src); in6_clearscope(&ip6->ip6_dst); (void)looutput(ifp, copym, (const struct sockaddr *)dst, NULL); } /* * Chop IPv6 header off from the payload. */ static int ip6_splithdr(struct mbuf *m, struct ip6_exthdrs *exthdrs) { struct mbuf *mh; struct ip6_hdr *ip6; ip6 = mtod(m, struct ip6_hdr *); if (m->m_len > sizeof(*ip6)) { MGETHDR(mh, M_DONTWAIT, MT_HEADER); if (mh == NULL) { m_freem(m); return ENOBUFS; } m_move_pkthdr(mh, m); m_align(mh, sizeof(*ip6)); m->m_len -= sizeof(*ip6); m->m_data += sizeof(*ip6); mh->m_next = m; mh->m_len = sizeof(*ip6); memcpy(mtod(mh, void *), (void *)ip6, sizeof(*ip6)); m = mh; } exthdrs->ip6e_ip6 = m; return 0; } /* * Compute IPv6 extension header length. */ int ip6_optlen(struct in6pcb *in6p) { int len; if (!in6p->in6p_outputopts) return 0; len = 0; #define elen(x) \ (((struct ip6_ext *)(x)) ? (((struct ip6_ext *)(x))->ip6e_len + 1) << 3 : 0) len += elen(in6p->in6p_outputopts->ip6po_hbh); len += elen(in6p->in6p_outputopts->ip6po_dest1); len += elen(in6p->in6p_outputopts->ip6po_rthdr); len += elen(in6p->in6p_outputopts->ip6po_dest2); return len; #undef elen } /* * Ensure sending address is valid. * Returns 0 on success, -1 if an error should be sent back or 1 * if the packet could be dropped without error (protocol dependent). */ static int ip6_ifaddrvalid(const struct in6_addr *src, const struct in6_addr *dst) { struct sockaddr_in6 sin6; int s, error; struct ifaddr *ifa; struct in6_ifaddr *ia6; if (IN6_IS_ADDR_UNSPECIFIED(src)) return 0; memset(&sin6, 0, sizeof(sin6)); sin6.sin6_family = AF_INET6; sin6.sin6_len = sizeof(sin6); sin6.sin6_addr = *src; s = pserialize_read_enter(); ifa = ifa_ifwithaddr(sin6tosa(&sin6)); if ((ia6 = ifatoia6(ifa)) == NULL || ia6->ia6_flags & (IN6_IFF_ANYCAST | IN6_IFF_DUPLICATED)) error = -1; else if (ia6->ia6_flags & IN6_IFF_TENTATIVE) error = 1; else if (ia6->ia6_flags & IN6_IFF_DETACHED && (sin6.sin6_addr = *dst, ifa_ifwithaddr(sin6tosa(&sin6)) == NULL)) /* Allow internal traffic to DETACHED addresses */ error = 1; else error = 0; pserialize_read_exit(s); return error; } |
| 47 47 11 12 6 2 2 2 6 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 197 198 199 200 201 202 203 204 205 206 207 208 209 210 211 212 213 214 215 216 217 218 219 220 221 222 223 224 225 226 227 228 229 230 231 232 233 234 235 236 237 238 239 240 241 242 243 244 245 246 247 248 249 250 251 252 253 254 255 256 257 258 259 260 261 262 263 264 265 266 267 268 269 270 271 272 273 274 275 276 277 278 279 280 281 282 283 284 285 286 287 288 289 290 291 292 293 294 295 296 297 298 299 300 301 302 303 304 305 306 307 308 309 310 311 312 313 314 315 316 317 318 319 320 321 322 323 324 325 326 327 328 329 330 331 332 333 334 335 336 337 338 339 340 341 342 343 344 345 346 347 348 349 350 351 352 353 354 355 356 357 358 359 360 361 362 363 364 365 366 367 368 369 370 371 372 373 374 375 376 377 378 379 380 381 382 383 384 385 386 387 388 389 390 391 392 393 394 395 396 397 398 399 400 401 402 403 404 405 406 407 408 409 410 411 412 413 414 415 416 417 418 419 420 421 422 423 424 425 426 427 428 429 430 431 432 433 434 435 436 437 438 439 440 441 442 443 444 445 446 447 448 449 450 451 452 453 454 455 456 457 458 459 460 461 462 463 464 465 466 467 468 469 470 471 472 473 474 475 476 477 478 479 480 481 482 483 484 485 486 487 488 489 490 491 492 493 494 495 496 497 498 499 500 501 502 503 504 505 506 507 508 509 510 511 512 513 514 515 516 517 518 519 520 521 522 523 524 525 526 527 528 529 530 531 532 533 534 535 536 537 538 539 540 541 542 543 544 545 546 547 548 549 550 551 552 553 554 555 556 557 558 559 560 561 562 563 564 565 566 567 568 569 570 571 572 573 574 575 576 577 578 579 580 581 582 583 584 585 586 587 588 589 590 591 592 593 594 595 596 597 598 599 600 601 602 603 604 605 606 607 608 609 610 611 612 613 614 615 616 617 618 619 620 621 622 623 624 625 626 627 628 629 630 631 632 633 634 635 636 637 638 639 640 641 642 643 644 645 646 647 648 649 650 651 652 653 654 655 656 657 658 659 660 661 662 663 664 665 666 667 668 669 670 671 672 673 674 675 676 677 678 679 680 681 682 683 684 685 686 687 688 689 690 691 692 693 694 695 696 697 698 699 700 701 702 703 704 705 706 707 708 709 710 711 712 713 714 715 716 717 718 719 720 721 722 723 724 725 726 727 728 729 730 731 732 733 734 735 736 737 738 739 740 741 742 743 744 745 746 747 748 749 750 751 752 753 754 755 756 757 758 759 760 761 762 763 764 765 766 767 768 769 770 771 772 773 774 775 776 777 778 779 780 781 782 783 784 785 786 787 788 789 790 791 792 793 794 795 796 797 798 799 800 801 802 803 804 805 806 807 808 809 810 811 812 813 814 815 816 817 818 819 820 821 822 823 824 825 826 827 828 829 830 831 832 833 834 835 836 837 838 839 840 841 842 843 844 845 846 847 848 849 850 851 852 853 854 855 856 857 858 859 860 861 862 863 864 865 866 867 868 869 870 871 872 873 874 875 876 877 878 879 880 881 882 883 884 885 886 887 888 889 890 891 892 893 894 895 896 897 898 899 900 901 902 903 904 905 906 907 908 909 910 911 912 913 914 915 916 917 918 919 920 921 922 923 924 925 926 927 928 929 930 931 932 933 934 935 936 937 938 939 940 941 942 943 944 945 946 947 948 949 950 951 952 953 954 955 956 957 958 959 960 961 962 963 964 965 966 967 968 969 970 971 972 973 974 975 976 977 978 979 980 981 982 983 984 985 986 987 988 989 990 991 992 993 994 995 996 997 998 999 1000 1001 1002 1003 1004 1005 1006 1007 1008 1009 1010 1011 1012 1013 1014 1015 1016 1017 1018 1019 1020 1021 1022 1023 1024 1025 1026 1027 1028 1029 1030 1031 1032 1033 1034 1035 1036 1037 1038 1039 1040 1041 1042 1043 1044 1045 1046 1047 1048 1049 1050 1051 1052 1053 1054 1055 1056 1057 1058 1059 1060 1061 1062 1063 1064 1065 1066 1067 1068 1069 1070 1071 1072 1073 1074 1075 1076 1077 1078 1079 1080 1081 1082 1083 1084 1085 1086 1087 1088 1089 1090 1091 1092 1093 1094 1095 1096 1097 1098 1099 1100 1101 1102 1103 1104 1105 1106 1107 1108 1109 1110 1111 1112 1113 1114 1115 1116 1117 1118 1119 1120 1121 1122 1123 1124 1125 1126 1127 1128 1129 1130 1131 1132 1133 1134 1135 1136 1137 1138 1139 1140 1141 1142 1143 1144 1145 1146 1147 1148 1149 1150 1151 1152 1153 1154 1155 1156 1157 1158 1159 1160 1161 1162 1163 1164 1165 1166 | /* $NetBSD: kern_pmf.c,v 1.48 2022/03/28 12:38:59 riastradh Exp $ */ /*- * Copyright (c) 2007 Jared D. McNeill <jmcneill@invisible.ca> * All rights reserved. * * Redistribution and use in source and binary forms, with or without * modification, are permitted provided that the following conditions * are met: * 1. Redistributions of source code must retain the above copyright * notice, this list of conditions and the following disclaimer. * 2. Redistributions in binary form must reproduce the above copyright * notice, this list of conditions and the following disclaimer in the * documentation and/or other materials provided with the distribution. * * THIS SOFTWARE IS PROVIDED BY THE NETBSD FOUNDATION, INC. AND CONTRIBUTORS * ``AS IS'' AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED * TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR * PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE FOUNDATION OR CONTRIBUTORS * BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR * CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF * SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS * INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN * CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) * ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE * POSSIBILITY OF SUCH DAMAGE. */ #include <sys/cdefs.h> __KERNEL_RCSID(0, "$NetBSD: kern_pmf.c,v 1.48 2022/03/28 12:38:59 riastradh Exp $"); #include <sys/types.h> #include <sys/param.h> #include <sys/kmem.h> #include <sys/buf.h> #include <sys/callout.h> #include <sys/kernel.h> #include <sys/device.h> #include <sys/device_impl.h> #include <sys/pmf.h> #include <sys/queue.h> #include <sys/sched.h> #include <sys/workqueue.h> #include <prop/proplib.h> #include <sys/condvar.h> #include <sys/mutex.h> #include <sys/proc.h> #include <sys/reboot.h> /* for RB_NOSYNC */ #include <sys/sched.h> #include <sys/sysctl.h> #include <sys/vfs_syscalls.h> /* XXX ugly special case, but for now the only client */ #include "wsdisplay.h" #if NWSDISPLAY > 0 #include <dev/wscons/wsdisplayvar.h> #endif #define PMF_DEBUG #ifdef PMF_DEBUG int pmf_debug_event; int pmf_debug_suspend; int pmf_debug_suspensor; int pmf_debug_idle; int pmf_debug_transition; #define PMF_SUSPENSOR_PRINTF(x) if (pmf_debug_suspensor) printf x #define PMF_SUSPEND_PRINTF(x) if (pmf_debug_suspend) printf x #define PMF_EVENT_PRINTF(x) if (pmf_debug_event) printf x #define PMF_IDLE_PRINTF(x) if (pmf_debug_idle) printf x #define PMF_TRANSITION_PRINTF(x) if (pmf_debug_transition) printf x #define PMF_TRANSITION_PRINTF2(y,x) if (pmf_debug_transition>y) printf x #else #define PMF_SUSPENSOR_PRINTF(x) do { } while (0) #define PMF_SUSPEND_PRINTF(x) do { } while (0) #define PMF_EVENT_PRINTF(x) do { } while (0) #define PMF_IDLE_PRINTF(x) do { } while (0) #define PMF_TRANSITION_PRINTF(x) do { } while (0) #define PMF_TRANSITION_PRINTF2(y,x) do { } while (0) #endif static prop_dictionary_t pmf_platform = NULL; static struct workqueue *pmf_event_workqueue; static struct workqueue *pmf_suspend_workqueue; typedef struct pmf_event_handler { TAILQ_ENTRY(pmf_event_handler) pmf_link; pmf_generic_event_t pmf_event; void (*pmf_handler)(device_t); device_t pmf_device; bool pmf_global; } pmf_event_handler_t; static TAILQ_HEAD(, pmf_event_handler) pmf_all_events = TAILQ_HEAD_INITIALIZER(pmf_all_events); typedef struct pmf_event_workitem { struct work pew_work; pmf_generic_event_t pew_event; device_t pew_device; } pmf_event_workitem_t; typedef struct pmf_suspend_workitem { struct work psw_work; device_t psw_dev; pmf_qual_t psw_qual; } pmf_suspend_workitem_t; static struct pool pew_pl; static pmf_event_workitem_t *pmf_event_workitem_get(void); static void pmf_event_workitem_put(pmf_event_workitem_t *); bool pmf_device_resume_locked(device_t, const pmf_qual_t *); bool pmf_device_suspend_locked(device_t, const pmf_qual_t *); static bool device_pmf_any_suspensor(device_t, devact_level_t); static bool complete_suspension(device_t dev, const device_suspensor_t **susp, const pmf_qual_t *pqp) { int i; pmf_qual_t pq; const device_suspensor_t *ds; ds = pmf_qual_suspension(pqp); KASSERT(ds->ds_delegator != NULL); pq = *pqp; pq.pq_suspensor = ds->ds_delegator; for (i = 0; i < DEVICE_SUSPENSORS_MAX; i++) { if (susp[i] != ds) continue; if (!pmf_device_suspend(dev, &pq)) return false; } return true; } static void pmf_suspend_worker(struct work *wk, void *dummy) { pmf_suspend_workitem_t *psw; deviter_t di; device_t dev; psw = (void *)wk; KASSERT(wk == &psw->psw_work); KASSERT(psw != NULL); for (dev = deviter_first(&di, 0); dev != NULL; dev = deviter_next(&di)) { if (dev == psw->psw_dev && device_pmf_lock(dev)) break; } deviter_release(&di); if (dev == NULL) return; switch (pmf_qual_depth(&psw->psw_qual)) { case DEVACT_LEVEL_FULL: if (!complete_suspension(dev, dev->dv_class_suspensors, &psw->psw_qual)) break; /*FALLTHROUGH*/ case DEVACT_LEVEL_DRIVER: if (!complete_suspension(dev, dev->dv_driver_suspensors, &psw->psw_qual)) break; /*FALLTHROUGH*/ case DEVACT_LEVEL_BUS: if (!complete_suspension(dev, dev->dv_bus_suspensors, &psw->psw_qual)) break; } device_pmf_unlock(dev); kmem_free(psw, sizeof(*psw)); } static void pmf_event_worker(struct work *wk, void *dummy) { pmf_event_workitem_t *pew; pmf_event_handler_t *event; pew = (void *)wk; KASSERT(wk == &pew->pew_work); KASSERT(pew != NULL); TAILQ_FOREACH(event, &pmf_all_events, pmf_link) { if (event->pmf_event != pew->pew_event) continue; if (event->pmf_device == pew->pew_device || event->pmf_global) (*event->pmf_handler)(event->pmf_device); } pmf_event_workitem_put(pew); } static bool pmf_check_system_drivers(void) { device_t curdev; bool unsupported_devs; deviter_t di; unsupported_devs = false; for (curdev = deviter_first(&di, 0); curdev != NULL; curdev = deviter_next(&di)) { if (device_pmf_is_registered(curdev)) continue; if (!unsupported_devs) printf("Devices without power management support:"); printf(" %s", device_xname(curdev)); unsupported_devs = true; } deviter_release(&di); if (unsupported_devs) { printf("\n"); return false; } return true; } bool pmf_system_bus_resume(const pmf_qual_t *qual) { bool rv; device_t curdev; deviter_t di; aprint_debug("Powering devices:"); /* D0 handlers are run in order */ rv = true; for (curdev = deviter_first(&di, DEVITER_F_ROOT_FIRST); curdev != NULL; curdev = deviter_next(&di)) { if (!device_pmf_is_registered(curdev)) continue; if (device_is_active(curdev) || !device_is_enabled(curdev)) continue; aprint_debug(" %s", device_xname(curdev)); if (!device_pmf_bus_resume(curdev, qual)) { rv = false; aprint_debug("(failed)"); } } deviter_release(&di); aprint_debug("\n"); return rv; } bool pmf_system_resume(const pmf_qual_t *qual) { bool rv; device_t curdev, parent; deviter_t di; if (!pmf_check_system_drivers()) return false; aprint_debug("Resuming devices:"); /* D0 handlers are run in order */ rv = true; for (curdev = deviter_first(&di, DEVITER_F_ROOT_FIRST); curdev != NULL; curdev = deviter_next(&di)) { if (device_is_active(curdev) || !device_is_enabled(curdev)) continue; parent = device_parent(curdev); if (parent != NULL && !device_is_active(parent)) continue; aprint_debug(" %s", device_xname(curdev)); if (!pmf_device_resume(curdev, qual)) { rv = false; aprint_debug("(failed)"); } } deviter_release(&di); aprint_debug(".\n"); KERNEL_UNLOCK_ONE(0); #if NWSDISPLAY > 0 if (rv) wsdisplay_handlex(1); #endif return rv; } bool pmf_system_suspend(const pmf_qual_t *qual) { device_t curdev; deviter_t di; if (!pmf_check_system_drivers()) return false; #if NWSDISPLAY > 0 if (wsdisplay_handlex(0)) return false; #endif KERNEL_LOCK(1, NULL); /* * Flush buffers only if the shutdown didn't do so * already and if there was no panic. */ if (doing_shutdown == 0 && panicstr == NULL) { printf("Flushing disk caches: "); do_sys_sync(&lwp0); if (vfs_syncwait() != 0) printf("giving up\n"); else printf("done\n"); } aprint_debug("Suspending devices:"); for (curdev = deviter_first(&di, DEVITER_F_LEAVES_FIRST); curdev != NULL; curdev = deviter_next(&di)) { if (!device_is_active(curdev)) continue; aprint_debug(" %s", device_xname(curdev)); /* XXX joerg check return value and abort suspend */ if (!pmf_device_suspend(curdev, qual)) aprint_debug("(failed)"); } deviter_release(&di); aprint_debug(".\n"); return true; } static bool shutdown_all(int how) { static struct shutdown_state s; device_t curdev; bool progress = false; KERNEL_LOCK(1, NULL); for (curdev = shutdown_first(&s); curdev != NULL; curdev = shutdown_next(&s)) { aprint_debug(" shutting down %s, ", device_xname(curdev)); if (!device_pmf_is_registered(curdev)) aprint_debug("skipped."); #if 0 /* needed? */ else if (!device_pmf_class_shutdown(curdev, how)) aprint_debug("failed."); #endif else if (!device_pmf_driver_shutdown(curdev, how)) aprint_debug("failed."); else if (!device_pmf_bus_shutdown(curdev, how)) aprint_debug("failed."); else { progress = true; aprint_debug("success."); } } KERNEL_UNLOCK_ONE(NULL); return progress; } void pmf_system_shutdown(int how) { if (panicstr != NULL) return; aprint_debug("Shutting down devices:"); shutdown_all(how); } bool pmf_set_platform(const char *key, const char *value) { if (pmf_platform == NULL) pmf_platform = prop_dictionary_create(); if (pmf_platform == NULL) return false; return prop_dictionary_set_string(pmf_platform, key, value); } const char * pmf_get_platform(const char *key) { const char *value; if (pmf_platform == NULL) return NULL; if (!prop_dictionary_get_string(pmf_platform, key, &value)) return NULL; return value; } bool pmf_device_register1(device_t dev, bool (*suspend)(device_t, const pmf_qual_t *), bool (*resume)(device_t, const pmf_qual_t *), bool (*shutdown)(device_t, int)) { if (!device_pmf_driver_register(dev, suspend, resume, shutdown)) return false; if (!device_pmf_driver_child_register(dev)) { device_pmf_driver_deregister(dev); return false; } return true; } void pmf_device_deregister(device_t dev) { device_pmf_class_deregister(dev); device_pmf_bus_deregister(dev); device_pmf_driver_deregister(dev); } static const device_suspensor_t _device_suspensor_drvctl = { .ds_delegator = NULL , .ds_name = "drvctl" }; static const device_suspensor_t _device_suspensor_self = { .ds_delegator = NULL , .ds_name = "self" }; #if 0 static const device_suspensor_t _device_suspensor_self_delegate = { .ds_delegator = &_device_suspensor_self , .ds_name = "self delegate" }; #endif static const device_suspensor_t _device_suspensor_system = { .ds_delegator = NULL , .ds_name = "system" }; const device_suspensor_t * const device_suspensor_self = &_device_suspensor_self, #if 0 * const device_suspensor_self_delegate = &_device_suspensor_self_delegate, #endif * const device_suspensor_system = &_device_suspensor_system, * const device_suspensor_drvctl = &_device_suspensor_drvctl; static const pmf_qual_t _pmf_qual_system = { .pq_actlvl = DEVACT_LEVEL_FULL , .pq_suspensor = &_device_suspensor_system }; static const pmf_qual_t _pmf_qual_drvctl = { .pq_actlvl = DEVACT_LEVEL_FULL , .pq_suspensor = &_device_suspensor_drvctl }; static const pmf_qual_t _pmf_qual_self = { .pq_actlvl = DEVACT_LEVEL_DRIVER , .pq_suspensor = &_device_suspensor_self }; const pmf_qual_t * const PMF_Q_DRVCTL = &_pmf_qual_drvctl, * const PMF_Q_NONE = &_pmf_qual_system, * const PMF_Q_SELF = &_pmf_qual_self; static bool device_suspensor_delegates_to(const device_suspensor_t *ds, const device_suspensor_t *delegate) { const device_suspensor_t *iter; for (iter = delegate->ds_delegator; iter != NULL; iter = iter->ds_delegator) { if (ds == iter) return true; } return false; } static bool add_suspensor(device_t dev, const char *kind, const device_suspensor_t **susp, const device_suspensor_t *ds) { int i; for (i = 0; i < DEVICE_SUSPENSORS_MAX; i++) { if (susp[i] == NULL) continue; if (ds == susp[i]) { PMF_SUSPENSOR_PRINTF(( "%s: %s-suspended by %s (delegator %s) already\n", device_xname(dev), kind, susp[i]->ds_name, (susp[i]->ds_delegator != NULL) ? susp[i]->ds_delegator->ds_name : "<none>")); return true; } if (device_suspensor_delegates_to(ds, susp[i])) { PMF_SUSPENSOR_PRINTF(( "%s: %s assumes %s-suspension by %s " "(delegator %s)\n", device_xname(dev), ds->ds_name, kind, susp[i]->ds_name, (susp[i]->ds_delegator != NULL) ? susp[i]->ds_delegator->ds_name : "<none>")); susp[i] = ds; return true; } } for (i = 0; i < DEVICE_SUSPENSORS_MAX; i++) { if (susp[i] == NULL) { susp[i] = ds; PMF_SUSPENSOR_PRINTF(( "%s: newly %s-suspended by %s (delegator %s)\n", device_xname(dev), kind, susp[i]->ds_name, (susp[i]->ds_delegator != NULL) ? susp[i]->ds_delegator->ds_name : "<none>")); return true; } } return false; } static bool device_pmf_add_suspensor(device_t dev, const pmf_qual_t *pq) { const device_suspensor_t *ds; KASSERT(pq != NULL); ds = pmf_qual_suspension(pq); KASSERT(ds != NULL); if (!add_suspensor(dev, "class", dev->dv_class_suspensors, ds)) return false; if (!add_suspensor(dev, "driver", dev->dv_driver_suspensors, ds)) return false; if (!add_suspensor(dev, "bus", dev->dv_bus_suspensors, ds)) return false; return true; } #if 0 static bool device_pmf_has_suspension(device_t dev, const device_suspensor_t *ds) { int i; for (i = 0; i < DEVICE_SUSPENSORS_MAX; i++) { if (dev->dv_suspensions[i] == ds) return true; if (device_suspensor_delegates_to(dev->dv_suspensions[i], ds)) return true; } return false; } #endif static bool any_suspensor(device_t dev, const char *kind, const device_suspensor_t **susp) { int i; bool suspended = false; for (i = 0; i < DEVICE_SUSPENSORS_MAX; i++) { if (susp[i] != NULL) { PMF_SUSPENSOR_PRINTF(("%s: %s is suspended by %s " "(delegator %s)\n", device_xname(dev), kind, susp[i]->ds_name, (susp[i]->ds_delegator != NULL) ? susp[i]->ds_delegator->ds_name : "<none>")); suspended = true; } } return suspended; } static bool device_pmf_any_suspensor(device_t dev, devact_level_t depth) { switch (depth) { case DEVACT_LEVEL_FULL: if (any_suspensor(dev, "class", dev->dv_class_suspensors)) return true; /*FALLTHROUGH*/ case DEVACT_LEVEL_DRIVER: if (any_suspensor(dev, "driver", dev->dv_driver_suspensors)) return true; /*FALLTHROUGH*/ case DEVACT_LEVEL_BUS: if (any_suspensor(dev, "bus", dev->dv_bus_suspensors)) return true; } return false; } static bool remove_suspensor(device_t dev, const char *kind, const device_suspensor_t **susp, const device_suspensor_t *ds) { int i; for (i = 0; i < DEVICE_SUSPENSORS_MAX; i++) { if (susp[i] == NULL) continue; if (ds == susp[i] || device_suspensor_delegates_to(ds, susp[i])) { PMF_SUSPENSOR_PRINTF(("%s: %s suspension %s " "(delegator %s) removed by %s\n", device_xname(dev), kind, susp[i]->ds_name, (susp[i]->ds_delegator != NULL) ? susp[i]->ds_delegator->ds_name : "<none>", ds->ds_name)); susp[i] = NULL; return true; } } return false; } static bool device_pmf_remove_suspensor(device_t dev, const pmf_qual_t *pq) { const device_suspensor_t *ds; KASSERT(pq != NULL); ds = pmf_qual_suspension(pq); KASSERT(ds != NULL); if (!remove_suspensor(dev, "class", dev->dv_class_suspensors, ds)) return false; if (!remove_suspensor(dev, "driver", dev->dv_driver_suspensors, ds)) return false; if (!remove_suspensor(dev, "bus", dev->dv_bus_suspensors, ds)) return false; return true; } void pmf_self_suspensor_init(device_t dev, device_suspensor_t *ds, pmf_qual_t *pq) { ds->ds_delegator = device_suspensor_self; snprintf(ds->ds_name, sizeof(ds->ds_name), "%s-self", device_xname(dev)); pq->pq_actlvl = DEVACT_LEVEL_DRIVER; pq->pq_suspensor = ds; } bool pmf_device_suspend(device_t dev, const pmf_qual_t *qual) { bool rc; PMF_TRANSITION_PRINTF(("%s: suspend enter\n", device_xname(dev))); if (!device_pmf_is_registered(dev)) return false; if (!device_pmf_lock(dev)) return false; rc = pmf_device_suspend_locked(dev, qual); device_pmf_unlock(dev); PMF_TRANSITION_PRINTF(("%s: suspend exit\n", device_xname(dev))); return rc; } bool pmf_device_suspend_locked(device_t dev, const pmf_qual_t *qual) { if (!device_pmf_add_suspensor(dev, qual)) return false; PMF_TRANSITION_PRINTF2(1, ("%s: class suspend\n", device_xname(dev))); if (!device_pmf_class_suspend(dev, qual)) return false; PMF_TRANSITION_PRINTF2(1, ("%s: driver suspend\n", device_xname(dev))); if (!device_pmf_driver_suspend(dev, qual)) return false; PMF_TRANSITION_PRINTF2(1, ("%s: bus suspend\n", device_xname(dev))); if (!device_pmf_bus_suspend(dev, qual)) return false; return true; } bool pmf_device_resume(device_t dev, const pmf_qual_t *qual) { bool rc; PMF_TRANSITION_PRINTF(("%s: resume enter\n", device_xname(dev))); if (!device_pmf_is_registered(dev)) return false; if (!device_pmf_lock(dev)) return false; rc = pmf_device_resume_locked(dev, qual); device_pmf_unlock(dev); PMF_TRANSITION_PRINTF(("%s: resume exit\n", device_xname(dev))); return rc; } bool pmf_device_resume_locked(device_t dev, const pmf_qual_t *qual) { device_pmf_remove_suspensor(dev, qual); if (device_pmf_any_suspensor(dev, DEVACT_LEVEL_FULL)) return true; PMF_TRANSITION_PRINTF2(1, ("%s: bus resume\n", device_xname(dev))); if (!device_pmf_bus_resume(dev, qual)) return false; PMF_TRANSITION_PRINTF2(1, ("%s: driver resume\n", device_xname(dev))); if (!device_pmf_driver_resume(dev, qual)) return false; PMF_TRANSITION_PRINTF2(1, ("%s: class resume\n", device_xname(dev))); if (!device_pmf_class_resume(dev, qual)) return false; return true; } bool pmf_device_recursive_suspend(device_t dv, const pmf_qual_t *qual) { bool rv = true; device_t curdev; deviter_t di; pmf_qual_t pq; pmf_qual_recursive_copy(&pq, qual); for (curdev = deviter_first(&di, 0); curdev != NULL; curdev = deviter_next(&di)) { if (device_parent(curdev) != dv) continue; if (!pmf_device_recursive_suspend(curdev, &pq)) { rv = false; break; } } deviter_release(&di); return rv && pmf_device_suspend(dv, qual); } void pmf_qual_recursive_copy(pmf_qual_t *dst, const pmf_qual_t *src) { *dst = *src; dst->pq_actlvl = DEVACT_LEVEL_FULL; } bool pmf_device_recursive_resume(device_t dv, const pmf_qual_t *qual) { device_t parent; pmf_qual_t pq; if (device_is_active(dv)) return true; pmf_qual_recursive_copy(&pq, qual); parent = device_parent(dv); if (parent != NULL) { if (!pmf_device_recursive_resume(parent, &pq)) return false; } return pmf_device_resume(dv, qual); } bool pmf_device_descendants_release(device_t dv, const pmf_qual_t *qual) { bool rv = true; device_t curdev; deviter_t di; for (curdev = deviter_first(&di, 0); curdev != NULL; curdev = deviter_next(&di)) { if (device_parent(curdev) != dv) continue; device_pmf_remove_suspensor(curdev, qual); if (!pmf_device_descendants_release(curdev, qual)) { rv = false; break; } } deviter_release(&di); return rv; } bool pmf_device_descendants_resume(device_t dv, const pmf_qual_t *qual) { bool rv = true; device_t curdev; deviter_t di; KASSERT(pmf_qual_descend_ok(qual)); for (curdev = deviter_first(&di, 0); curdev != NULL; curdev = deviter_next(&di)) { if (device_parent(curdev) != dv) continue; if (!pmf_device_resume(curdev, qual) || !pmf_device_descendants_resume(curdev, qual)) { rv = false; break; } } deviter_release(&di); return rv; } bool pmf_device_subtree_release(device_t dv, const pmf_qual_t *qual) { pmf_qual_t pq; device_pmf_remove_suspensor(dv, qual); pmf_qual_recursive_copy(&pq, qual); return pmf_device_descendants_release(dv, &pq); } bool pmf_device_subtree_resume(device_t dv, const pmf_qual_t *qual) { pmf_qual_t pq; if (!pmf_device_subtree_release(dv, qual)) return false; if (!pmf_device_recursive_resume(dv, qual)) return false; pmf_qual_recursive_copy(&pq, qual); return pmf_device_descendants_resume(dv, &pq); } #include <net/if.h> static bool pmf_class_network_suspend(device_t dev, const pmf_qual_t *qual) { struct ifnet *ifp = device_pmf_class_private(dev); int s; s = splnet(); IFNET_LOCK(ifp); (*ifp->if_stop)(ifp, 0); IFNET_UNLOCK(ifp); splx(s); return true; } static bool pmf_class_network_resume(device_t dev, const pmf_qual_t *qual) { struct ifnet *ifp = device_pmf_class_private(dev); int s; bool restart = false; s = splnet(); IFNET_LOCK(ifp); if (ifp->if_flags & IFF_UP) { ifp->if_flags &= ~IFF_RUNNING; if ((*ifp->if_init)(ifp) != 0) aprint_normal_ifnet(ifp, "resume failed\n"); restart = true; } IFNET_UNLOCK(ifp); if (restart) if_start_lock(ifp); splx(s); return true; } void pmf_class_network_register(device_t dev, struct ifnet *ifp) { device_pmf_class_register(dev, ifp, pmf_class_network_suspend, pmf_class_network_resume, NULL); } bool pmf_event_inject(device_t dv, pmf_generic_event_t ev) { pmf_event_workitem_t *pew; pew = pmf_event_workitem_get(); if (pew == NULL) { PMF_EVENT_PRINTF(("%s: PMF event %d dropped (no memory)\n", dv ? device_xname(dv) : "<anonymous>", ev)); return false; } pew->pew_event = ev; pew->pew_device = dv; workqueue_enqueue(pmf_event_workqueue, &pew->pew_work, NULL); PMF_EVENT_PRINTF(("%s: PMF event %d injected\n", dv ? device_xname(dv) : "<anonymous>", ev)); return true; } bool pmf_event_register(device_t dv, pmf_generic_event_t ev, void (*handler)(device_t), bool global) { pmf_event_handler_t *event; event = kmem_alloc(sizeof(*event), KM_SLEEP); event->pmf_event = ev; event->pmf_handler = handler; event->pmf_device = dv; event->pmf_global = global; TAILQ_INSERT_TAIL(&pmf_all_events, event, pmf_link); return true; } void pmf_event_deregister(device_t dv, pmf_generic_event_t ev, void (*handler)(device_t), bool global) { pmf_event_handler_t *event; TAILQ_FOREACH(event, &pmf_all_events, pmf_link) { if (event->pmf_event != ev) continue; if (event->pmf_device != dv) continue; if (event->pmf_global != global) continue; if (event->pmf_handler != handler) continue; TAILQ_REMOVE(&pmf_all_events, event, pmf_link); kmem_free(event, sizeof(*event)); return; } } struct display_class_softc { TAILQ_ENTRY(display_class_softc) dc_link; device_t dc_dev; }; static TAILQ_HEAD(, display_class_softc) all_displays; static callout_t global_idle_counter; static int idle_timeout = 30; static void input_idle(void *dummy) { PMF_IDLE_PRINTF(("Input idle handler called\n")); pmf_event_inject(NULL, PMFE_DISPLAY_OFF); } static void input_activity_handler(device_t dv, devactive_t type) { if (!TAILQ_EMPTY(&all_displays)) callout_schedule(&global_idle_counter, idle_timeout * hz); } static void pmf_class_input_deregister(device_t dv) { device_active_deregister(dv, input_activity_handler); } bool pmf_class_input_register(device_t dv) { if (!device_active_register(dv, input_activity_handler)) return false; device_pmf_class_register(dv, NULL, NULL, NULL, pmf_class_input_deregister); return true; } static void pmf_class_display_deregister(device_t dv) { struct display_class_softc *sc = device_pmf_class_private(dv); int s; s = splsoftclock(); TAILQ_REMOVE(&all_displays, sc, dc_link); if (TAILQ_EMPTY(&all_displays)) callout_stop(&global_idle_counter); splx(s); kmem_free(sc, sizeof(*sc)); } bool pmf_class_display_register(device_t dv) { struct display_class_softc *sc; int s; sc = kmem_alloc(sizeof(*sc), KM_SLEEP); s = splsoftclock(); if (TAILQ_EMPTY(&all_displays)) callout_schedule(&global_idle_counter, idle_timeout * hz); TAILQ_INSERT_HEAD(&all_displays, sc, dc_link); splx(s); device_pmf_class_register(dv, sc, NULL, NULL, pmf_class_display_deregister); return true; } static void pmf_event_workitem_put(pmf_event_workitem_t *pew) { KASSERT(pew != NULL); pool_put(&pew_pl, pew); } static pmf_event_workitem_t * pmf_event_workitem_get(void) { return pool_get(&pew_pl, PR_NOWAIT); } SYSCTL_SETUP(sysctl_pmf_setup, "PMF subtree setup") { const struct sysctlnode *node = NULL; sysctl_createv(clog, 0, NULL, &node, CTLFLAG_PERMANENT, CTLTYPE_NODE, "pmf", SYSCTL_DESCR("pmf controls"), NULL, 0, NULL, 0, CTL_KERN, CTL_CREATE, CTL_EOL); #ifdef PMF_DEBUG sysctl_createv(clog, 0, &node, &node, CTLFLAG_PERMANENT, CTLTYPE_NODE, "debug", SYSCTL_DESCR("debug levels"), NULL, 0, NULL, 0, CTL_CREATE, CTL_EOL); sysctl_createv(clog, 0, &node, NULL, CTLFLAG_PERMANENT|CTLFLAG_READWRITE, CTLTYPE_INT, "event", SYSCTL_DESCR("event"), NULL, 0, &pmf_debug_event, sizeof(pmf_debug_event), CTL_CREATE, CTL_EOL); sysctl_createv(clog, 0, &node, NULL, CTLFLAG_PERMANENT|CTLFLAG_READWRITE, CTLTYPE_INT, "suspend", SYSCTL_DESCR("suspend"), NULL, 0, &pmf_debug_suspend, sizeof(pmf_debug_suspend), CTL_CREATE, CTL_EOL); sysctl_createv(clog, 0, &node, NULL, CTLFLAG_PERMANENT|CTLFLAG_READWRITE, CTLTYPE_INT, "suspensor", SYSCTL_DESCR("suspensor"), NULL, 0, &pmf_debug_suspensor, sizeof(pmf_debug_suspensor), CTL_CREATE, CTL_EOL); sysctl_createv(clog, 0, &node, NULL, CTLFLAG_PERMANENT|CTLFLAG_READWRITE, CTLTYPE_INT, "idle", SYSCTL_DESCR("idle"), NULL, 0, &pmf_debug_idle, sizeof(pmf_debug_idle), CTL_CREATE, CTL_EOL); sysctl_createv(clog, 0, &node, NULL, CTLFLAG_PERMANENT|CTLFLAG_READWRITE, CTLTYPE_INT, "transition", SYSCTL_DESCR("event"), NULL, 0, &pmf_debug_transition, sizeof(pmf_debug_transition), CTL_CREATE, CTL_EOL); #endif } void pmf_init(void) { int err; pool_init(&pew_pl, sizeof(pmf_event_workitem_t), 0, 0, 0, "pewpl", NULL, IPL_HIGH); pool_setlowat(&pew_pl, 1); pool_sethiwat(&pew_pl, 8); KASSERT(pmf_event_workqueue == NULL); err = workqueue_create(&pmf_event_workqueue, "pmfevent", pmf_event_worker, NULL, PRI_NONE, IPL_VM, 0); if (err) panic("couldn't create pmfevent workqueue"); KASSERT(pmf_suspend_workqueue == NULL); err = workqueue_create(&pmf_suspend_workqueue, "pmfsuspend", pmf_suspend_worker, NULL, PRI_NONE, IPL_VM, 0); if (err) panic("couldn't create pmfsuspend workqueue"); callout_init(&global_idle_counter, 0); callout_setfunc(&global_idle_counter, input_idle, NULL); } |
| 4 4 3 1 4 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 197 198 199 200 201 202 203 204 205 206 207 208 209 210 211 212 213 214 215 216 217 218 219 220 221 222 223 224 225 226 227 228 229 230 | /* $NetBSD: kern_module_vfs.c,v 1.18 2021/06/29 22:40:53 dholland Exp $ */ /*- * Copyright (c) 2008 The NetBSD Foundation, Inc. * All rights reserved. * * This code is derived from software developed for The NetBSD Foundation * by Andrew Doran. * * Redistribution and use in source and binary forms, with or without * modification, are permitted provided that the following conditions * are met: * 1. Redistributions of source code must retain the above copyright * notice, this list of conditions and the following disclaimer. * 2. Redistributions in binary form must reproduce the above copyright * notice, this list of conditions and the following disclaimer in the * documentation and/or other materials provided with the distribution. * * THIS SOFTWARE IS PROVIDED BY THE NETBSD FOUNDATION, INC. AND CONTRIBUTORS * ``AS IS'' AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED * TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR * PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE FOUNDATION OR CONTRIBUTORS * BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR * CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF * SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS * INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN * CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) * ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE * POSSIBILITY OF SUCH DAMAGE. */ /* * Kernel module file system interaction. */ #include <sys/cdefs.h> __KERNEL_RCSID(0, "$NetBSD: kern_module_vfs.c,v 1.18 2021/06/29 22:40:53 dholland Exp $"); #define _MODULE_INTERNAL #include <sys/param.h> #include <sys/fcntl.h> #include <sys/kmem.h> #include <sys/kobj.h> #include <sys/module.h> #include <sys/namei.h> #include <sys/pool.h> #include <sys/stat.h> #include <sys/vnode.h> #include <prop/proplib.h> static int module_load_plist_vfs(const char *, const bool, prop_dictionary_t *); void module_load_vfs_init(void) { module_load_vfs_vec = module_load_vfs; aprint_normal("kern.module.path=%s\n", module_base); } int module_load_vfs(const char *name, int flags, bool autoload, module_t *mod, prop_dictionary_t *filedictp) { char *path; bool nochroot; int error; prop_bool_t noload; prop_dictionary_t moduledict; nochroot = false; error = 0; path = NULL; moduledict = NULL; if (filedictp) *filedictp = NULL; path = PNBUF_GET(); if (!autoload) { if (strchr(name, '/') != NULL) { nochroot = false; snprintf(path, MAXPATHLEN, "%s", name); module_print("Loading module from %s", path); error = kobj_load_vfs(&mod->mod_kobj, path, nochroot); } else error = ENOENT; } if (autoload || (error == ENOENT)) { if (strchr(name, '/') == NULL) { nochroot = true; snprintf(path, MAXPATHLEN, "%s/%s/%s.kmod", module_base, name, name); module_print("Loading module from %s", path); error = kobj_load_vfs(&mod->mod_kobj, path, nochroot); } else error = ENOENT; } if (error != 0) { PNBUF_PUT(path); module_print("Cannot %sload kernel object `%s'" " error=%d", autoload ? "auto" : "", name, error); return error; } /* * Load and process <module>.plist if it exists. */ if ((!ISSET(flags, MODCTL_NO_PROP) && filedictp) || autoload) { error = module_load_plist_vfs(path, nochroot, &moduledict); if (error != 0) { module_print("plist load returned error %d for `%s'", error, path); if (error != ENOENT) goto fail; } else if (autoload) { noload = prop_dictionary_get(moduledict, "noautoload"); if (noload != NULL && prop_bool_true(noload)) { module_error("autoloading is disallowed for %s", path); prop_object_release(moduledict); error = EPERM; goto fail; } } if (error == 0) { /* can get here if error == ENOENT */ if (!ISSET(flags, MODCTL_NO_PROP) && filedictp) *filedictp = moduledict; else prop_object_release(moduledict); } } PNBUF_PUT(path); return 0; fail: kobj_unload(mod->mod_kobj); PNBUF_PUT(path); return error; } /* * module_load_plist_vfs: * * Load a plist located in the file system into memory. */ static int module_load_plist_vfs(const char *modpath, const bool nochroot, prop_dictionary_t *filedictp) { struct pathbuf *pb; struct vnode *vp; struct stat sb; void *base; char *proppath; const size_t plistsize = 8192; size_t resid; int error, pathlen; KASSERT(filedictp != NULL); base = NULL; proppath = PNBUF_GET(); strlcpy(proppath, modpath, MAXPATHLEN); pathlen = strlen(proppath); if ((pathlen >= 6) && (strcmp(&proppath[pathlen - 5], ".kmod") == 0)) { strcpy(&proppath[pathlen - 5], ".plist"); } else if (pathlen < MAXPATHLEN - 6) { strcat(proppath, ".plist"); } else { error = ENOENT; goto out1; } /* XXX this makes an unnecessary extra copy of the path */ pb = pathbuf_create(proppath); if (pb == NULL) { error = ENOMEM; goto out1; } module_print("Loading plist from %s", proppath); error = vn_open(NULL, pb, (nochroot ? NOCHROOT : 0), FREAD, 0, &vp, NULL, NULL); if (error != 0) { goto out2; } error = vn_stat(vp, &sb); if (error != 0) { goto out3; } if (sb.st_size >= (plistsize - 1)) { /* leave space for term \0 */ error = EFBIG; goto out3; } base = kmem_alloc(plistsize, KM_SLEEP); error = vn_rdwr(UIO_READ, vp, base, sb.st_size, 0, UIO_SYSSPACE, IO_NODELOCKED, curlwp->l_cred, &resid, curlwp); *((uint8_t *)base + sb.st_size) = '\0'; if (error == 0 && resid != 0) { error = EFBIG; } if (error != 0) { kmem_free(base, plistsize); base = NULL; goto out3; } *filedictp = prop_dictionary_internalize(base); if (*filedictp == NULL) { error = EINVAL; } kmem_free(base, plistsize); base = NULL; KASSERT(error == 0); out3: VOP_UNLOCK(vp); vn_close(vp, FREAD, kauth_cred_get()); out2: pathbuf_destroy(pb); out1: PNBUF_PUT(proppath); return error; } |
| 142 142 24 24 24 11 11 11 11 10 5 2 5 5 5 2 3 12 12 12 12 1 5 5 6 11 11 6 2 3 5 6 8 2 3 1 1 202 202 202 110 108 110 142 201 202 202 201 190 2 104 89 88 105 192 105 89 192 192 192 192 191 191 7 6 7 7 7 7 7 7 5 1 6 6 1 5 5 136 136 136 127 9 136 17 119 7 113 127 9 136 26 110 134 134 134 109 24 41 42 42 2 192 2 2861 2817 2818 2817 1370 1371 1369 555 556 555 33 32 9 9 9 4 5 4 2 7 4 1 4 32 32 32 32 281 279 281 276 276 276 275 275 276 6 2 78 78 77 78 4 9 9 9 9 9 4622 7 7 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 197 198 199 200 201 202 203 204 205 206 207 208 209 210 211 212 213 214 215 216 217 218 219 220 221 222 223 224 225 226 227 228 229 230 231 232 233 234 235 236 237 238 239 240 241 242 243 244 245 246 247 248 249 250 251 252 253 254 255 256 257 258 259 260 261 262 263 264 265 266 267 268 269 270 271 272 273 274 275 276 277 278 279 280 281 282 283 284 285 286 287 288 289 290 291 292 293 294 295 296 297 298 299 300 301 302 303 304 305 306 307 308 309 310 311 312 313 314 315 316 317 318 319 320 321 322 323 324 325 326 327 328 329 330 331 332 333 334 335 336 337 338 339 340 341 342 343 344 345 346 347 348 349 350 351 352 353 354 355 356 357 358 359 360 361 362 363 364 365 366 367 368 369 370 371 372 373 374 375 376 377 378 379 380 381 382 383 384 385 386 387 388 389 390 391 392 393 394 395 396 397 398 399 400 401 402 403 404 405 406 407 408 409 410 411 412 413 414 415 416 417 418 419 420 421 422 423 424 425 426 427 428 429 430 431 432 433 434 435 436 437 438 439 440 441 442 443 444 445 446 447 448 449 450 451 452 453 454 455 456 457 458 459 460 461 462 463 464 465 466 467 468 469 470 471 472 473 474 475 476 477 478 479 480 481 482 483 484 485 486 487 488 489 490 491 492 493 494 495 496 497 498 499 500 501 502 503 504 505 506 507 508 509 510 511 512 513 514 515 516 517 518 519 520 521 522 523 524 525 526 527 528 529 530 531 532 533 534 535 536 537 538 539 540 541 542 543 544 545 546 547 548 549 550 551 552 553 554 555 556 557 558 559 560 561 562 563 564 565 566 567 568 569 570 571 572 573 574 575 576 577 578 579 580 581 582 583 584 585 586 587 588 589 590 591 592 593 594 595 596 597 598 599 600 601 602 603 604 605 606 607 608 609 610 611 612 613 614 615 616 617 618 619 620 621 622 623 624 625 626 627 628 629 630 631 632 633 634 635 636 637 638 639 640 641 642 643 644 645 646 647 648 649 650 651 652 653 654 655 656 657 658 659 660 661 662 663 664 665 666 667 668 669 670 671 672 673 674 675 676 677 678 679 680 681 682 683 684 685 686 687 688 689 690 691 692 693 694 695 696 697 698 699 700 701 702 703 704 705 706 707 708 709 710 711 712 713 714 715 716 717 718 719 720 721 722 723 724 725 726 727 728 729 730 731 732 733 734 735 736 737 738 739 740 741 742 743 744 745 746 747 748 749 750 751 752 753 754 755 756 757 758 759 760 761 762 763 764 765 766 767 768 769 770 771 772 773 774 775 776 777 778 779 780 781 782 783 784 785 786 787 788 789 790 791 792 793 794 795 796 797 798 799 800 801 802 803 804 805 806 807 808 809 810 811 812 813 814 815 816 817 818 819 820 821 822 823 824 825 826 827 828 829 830 831 832 833 834 835 836 837 838 839 840 841 842 843 844 845 846 847 848 849 850 851 852 853 854 855 856 857 858 859 860 861 862 863 864 865 866 867 868 869 870 871 872 873 874 875 876 877 878 879 880 881 882 883 884 885 886 887 888 889 890 891 892 893 894 895 896 897 898 899 900 901 902 903 904 905 906 907 908 909 910 911 912 913 914 915 916 917 918 919 920 921 922 923 924 925 926 927 928 929 930 931 932 933 934 935 936 937 938 939 940 941 942 943 944 945 946 947 948 949 950 951 952 953 954 955 956 957 958 959 960 961 962 963 964 965 966 967 968 969 970 971 972 973 974 975 976 977 978 979 980 981 982 983 984 985 986 987 988 989 990 991 992 993 994 995 996 997 998 999 1000 1001 1002 1003 1004 1005 1006 1007 1008 1009 1010 1011 1012 1013 1014 1015 1016 1017 1018 1019 1020 1021 1022 1023 1024 1025 1026 1027 1028 1029 1030 1031 1032 1033 1034 1035 1036 1037 1038 1039 1040 1041 1042 1043 1044 1045 1046 1047 1048 1049 1050 1051 1052 1053 1054 1055 1056 1057 1058 1059 1060 1061 1062 1063 1064 1065 1066 1067 1068 1069 1070 1071 1072 1073 1074 1075 1076 1077 1078 1079 1080 1081 1082 1083 1084 1085 1086 1087 1088 1089 1090 1091 1092 1093 1094 1095 1096 1097 1098 1099 1100 1101 1102 1103 1104 1105 1106 1107 1108 1109 1110 1111 1112 1113 1114 1115 1116 1117 1118 1119 1120 1121 1122 1123 1124 1125 1126 1127 1128 1129 1130 1131 1132 1133 1134 1135 1136 1137 1138 1139 1140 1141 1142 1143 1144 1145 1146 1147 1148 1149 1150 1151 1152 1153 1154 1155 1156 1157 1158 1159 1160 1161 1162 1163 1164 1165 1166 1167 1168 1169 1170 1171 1172 1173 1174 1175 1176 1177 1178 1179 1180 1181 1182 1183 1184 1185 1186 1187 1188 1189 1190 1191 1192 1193 1194 1195 1196 1197 1198 1199 1200 1201 1202 1203 1204 1205 1206 1207 1208 1209 1210 1211 1212 1213 1214 1215 1216 1217 1218 1219 1220 1221 1222 1223 1224 1225 1226 1227 1228 1229 1230 1231 1232 1233 1234 1235 1236 1237 1238 1239 1240 1241 1242 1243 1244 1245 1246 1247 1248 1249 1250 1251 1252 1253 1254 1255 1256 1257 1258 1259 1260 1261 1262 1263 1264 1265 1266 1267 1268 1269 1270 1271 1272 1273 1274 1275 1276 1277 1278 1279 1280 1281 1282 1283 1284 1285 1286 1287 1288 1289 1290 1291 1292 1293 1294 1295 1296 1297 1298 1299 1300 1301 1302 1303 1304 1305 1306 1307 1308 1309 1310 1311 1312 1313 1314 1315 1316 1317 1318 1319 1320 1321 1322 1323 1324 1325 1326 1327 1328 1329 1330 1331 1332 1333 1334 1335 1336 1337 1338 1339 1340 1341 1342 1343 1344 1345 1346 1347 1348 1349 1350 1351 1352 1353 1354 1355 1356 1357 1358 1359 1360 1361 1362 1363 1364 1365 1366 1367 1368 1369 1370 1371 1372 1373 1374 1375 1376 1377 1378 1379 1380 1381 1382 1383 1384 1385 1386 1387 1388 1389 1390 1391 1392 1393 1394 1395 1396 1397 1398 1399 1400 1401 1402 1403 1404 1405 1406 1407 1408 1409 1410 1411 1412 1413 1414 1415 1416 1417 1418 1419 1420 1421 1422 1423 1424 1425 1426 1427 1428 1429 1430 1431 1432 1433 1434 1435 1436 1437 1438 1439 1440 1441 1442 1443 1444 1445 1446 1447 1448 1449 1450 1451 1452 1453 1454 1455 1456 1457 1458 1459 1460 1461 1462 1463 1464 1465 1466 1467 1468 1469 1470 1471 1472 1473 1474 1475 1476 1477 1478 1479 1480 1481 1482 1483 1484 1485 1486 1487 1488 1489 1490 1491 1492 1493 1494 1495 1496 1497 1498 1499 1500 1501 1502 1503 1504 1505 1506 1507 1508 1509 1510 1511 1512 1513 1514 1515 1516 1517 1518 1519 1520 1521 1522 1523 1524 1525 1526 1527 1528 1529 1530 1531 1532 1533 1534 1535 1536 1537 1538 1539 1540 1541 1542 1543 1544 1545 1546 1547 1548 1549 1550 1551 1552 1553 1554 1555 1556 1557 1558 1559 1560 1561 1562 1563 1564 1565 1566 1567 1568 1569 1570 1571 1572 1573 1574 1575 1576 1577 1578 1579 1580 1581 1582 1583 1584 1585 1586 1587 1588 1589 1590 1591 1592 1593 1594 1595 1596 1597 1598 1599 1600 1601 1602 1603 1604 1605 1606 1607 1608 1609 1610 1611 1612 1613 1614 1615 1616 1617 1618 1619 1620 1621 1622 1623 1624 1625 1626 1627 1628 1629 1630 1631 1632 1633 1634 1635 1636 1637 1638 1639 1640 1641 1642 1643 1644 1645 1646 1647 1648 1649 1650 1651 1652 1653 1654 1655 1656 1657 1658 1659 1660 1661 1662 1663 1664 1665 1666 1667 1668 1669 1670 1671 1672 1673 1674 1675 1676 1677 1678 1679 1680 1681 1682 1683 1684 1685 1686 1687 1688 1689 1690 1691 1692 1693 1694 1695 1696 1697 1698 1699 1700 1701 1702 1703 1704 1705 1706 1707 1708 1709 1710 1711 1712 1713 1714 1715 1716 1717 1718 1719 1720 1721 1722 1723 1724 1725 1726 1727 1728 1729 1730 1731 1732 1733 1734 1735 1736 1737 1738 1739 1740 1741 1742 1743 1744 1745 1746 1747 1748 1749 1750 1751 1752 1753 1754 1755 1756 1757 1758 1759 1760 1761 1762 1763 1764 1765 1766 1767 1768 1769 1770 1771 1772 1773 1774 1775 1776 1777 1778 1779 1780 1781 1782 1783 1784 1785 1786 1787 1788 1789 1790 1791 1792 1793 1794 1795 1796 1797 1798 1799 1800 1801 1802 1803 1804 1805 1806 1807 1808 1809 1810 1811 1812 1813 1814 1815 1816 1817 1818 1819 1820 1821 1822 1823 1824 1825 1826 1827 1828 1829 1830 1831 1832 1833 1834 1835 1836 1837 1838 1839 1840 1841 1842 1843 1844 1845 1846 1847 1848 1849 1850 1851 1852 1853 1854 1855 1856 1857 1858 1859 1860 1861 1862 1863 1864 1865 1866 1867 1868 1869 1870 1871 1872 1873 1874 1875 1876 1877 1878 1879 1880 1881 1882 1883 1884 1885 1886 1887 1888 1889 1890 1891 1892 1893 1894 1895 1896 1897 1898 1899 1900 1901 1902 1903 1904 1905 1906 1907 1908 1909 1910 1911 1912 1913 1914 1915 1916 1917 1918 1919 1920 1921 1922 1923 1924 1925 1926 1927 1928 1929 1930 1931 1932 1933 1934 1935 1936 1937 1938 1939 1940 1941 1942 1943 1944 1945 1946 1947 1948 1949 1950 1951 1952 1953 1954 1955 1956 1957 1958 1959 1960 1961 1962 1963 1964 1965 1966 1967 1968 1969 1970 1971 1972 1973 1974 1975 1976 1977 1978 1979 1980 1981 1982 1983 1984 1985 1986 1987 1988 1989 1990 1991 1992 1993 1994 1995 1996 1997 1998 1999 2000 2001 2002 2003 2004 2005 2006 2007 2008 2009 2010 2011 2012 2013 2014 2015 2016 2017 2018 2019 2020 2021 2022 2023 2024 2025 2026 2027 2028 2029 2030 2031 2032 2033 2034 2035 2036 2037 2038 2039 2040 2041 2042 2043 2044 2045 2046 2047 2048 2049 2050 2051 2052 2053 2054 2055 2056 2057 2058 2059 2060 2061 2062 2063 2064 2065 2066 2067 2068 2069 2070 2071 2072 2073 2074 2075 2076 2077 2078 2079 2080 2081 2082 2083 2084 2085 2086 2087 2088 2089 2090 2091 2092 2093 2094 2095 2096 2097 2098 2099 2100 2101 2102 2103 2104 2105 2106 2107 | /* $NetBSD: kern_lwp.c,v 1.251 2022/07/01 01:06:04 riastradh Exp $ */ /*- * Copyright (c) 2001, 2006, 2007, 2008, 2009, 2019, 2020 * The NetBSD Foundation, Inc. * All rights reserved. * * This code is derived from software contributed to The NetBSD Foundation * by Nathan J. Williams, and Andrew Doran. * * Redistribution and use in source and binary forms, with or without * modification, are permitted provided that the following conditions * are met: * 1. Redistributions of source code must retain the above copyright * notice, this list of conditions and the following disclaimer. * 2. Redistributions in binary form must reproduce the above copyright * notice, this list of conditions and the following disclaimer in the * documentation and/or other materials provided with the distribution. * * THIS SOFTWARE IS PROVIDED BY THE NETBSD FOUNDATION, INC. AND CONTRIBUTORS * ``AS IS'' AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED * TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR * PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE FOUNDATION OR CONTRIBUTORS * BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR * CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF * SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS * INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN * CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) * ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE * POSSIBILITY OF SUCH DAMAGE. */ /* * Overview * * Lightweight processes (LWPs) are the basic unit or thread of * execution within the kernel. The core state of an LWP is described * by "struct lwp", also known as lwp_t. * * Each LWP is contained within a process (described by "struct proc"), * Every process contains at least one LWP, but may contain more. The * process describes attributes shared among all of its LWPs such as a * private address space, global execution state (stopped, active, * zombie, ...), signal disposition and so on. On a multiprocessor * machine, multiple LWPs be executing concurrently in the kernel. * * Execution states * * At any given time, an LWP has overall state that is described by * lwp::l_stat. The states are broken into two sets below. The first * set is guaranteed to represent the absolute, current state of the * LWP: * * LSONPROC * * On processor: the LWP is executing on a CPU, either in the * kernel or in user space. * * LSRUN * * Runnable: the LWP is parked on a run queue, and may soon be * chosen to run by an idle processor, or by a processor that * has been asked to preempt a currently runnning but lower * priority LWP. * * LSIDL * * Idle: the LWP has been created but has not yet executed, or * it has ceased executing a unit of work and is waiting to be * started again. This state exists so that the LWP can occupy * a slot in the process & PID table, but without having to * worry about being touched; lookups of the LWP by ID will * fail while in this state. The LWP will become visible for * lookup once its state transitions further. Some special * kernel threads also (ab)use this state to indicate that they * are idle (soft interrupts and idle LWPs). * * LSSUSPENDED: * * Suspended: the LWP has had its execution suspended by * another LWP in the same process using the _lwp_suspend() * system call. User-level LWPs also enter the suspended * state when the system is shutting down. * * The second set represent a "statement of intent" on behalf of the * LWP. The LWP may in fact be executing on a processor, may be * sleeping or idle. It is expected to take the necessary action to * stop executing or become "running" again within a short timeframe. * The LP_RUNNING flag in lwp::l_pflag indicates that an LWP is running. * Importantly, it indicates that its state is tied to a CPU. * * LSZOMB: * * Dead or dying: the LWP has released most of its resources * and is about to switch away into oblivion, or has already * switched away. When it switches away, its few remaining * resources can be collected. * * LSSLEEP: * * Sleeping: the LWP has entered itself onto a sleep queue, and * has switched away or will switch away shortly to allow other * LWPs to run on the CPU. * * LSSTOP: * * Stopped: the LWP has been stopped as a result of a job * control signal, or as a result of the ptrace() interface. * * Stopped LWPs may run briefly within the kernel to handle * signals that they receive, but will not return to user space * until their process' state is changed away from stopped. * * Single LWPs within a process can not be set stopped * selectively: all actions that can stop or continue LWPs * occur at the process level. * * State transitions * * Note that the LSSTOP state may only be set when returning to * user space in userret(), or when sleeping interruptably. The * LSSUSPENDED state may only be set in userret(). Before setting * those states, we try to ensure that the LWPs will release all * locks that they hold, and at a minimum try to ensure that the * LWP can be set runnable again by a signal. * * LWPs may transition states in the following ways: * * RUN -------> ONPROC ONPROC -----> RUN * > SLEEP * > STOPPED * > SUSPENDED * > ZOMB * > IDL (special cases) * * STOPPED ---> RUN SUSPENDED --> RUN * > SLEEP * * SLEEP -----> ONPROC IDL --------> RUN * > RUN > SUSPENDED * > STOPPED > STOPPED * > ONPROC (special cases) * * Some state transitions are only possible with kernel threads (eg * ONPROC -> IDL) and happen under tightly controlled circumstances * free of unwanted side effects. * * Migration * * Migration of threads from one CPU to another could be performed * internally by the scheduler via sched_takecpu() or sched_catchlwp() * functions. The universal lwp_migrate() function should be used for * any other cases. Subsystems in the kernel must be aware that CPU * of LWP may change, while it is not locked. * * Locking * * The majority of fields in 'struct lwp' are covered by a single, * general spin lock pointed to by lwp::l_mutex. The locks covering * each field are documented in sys/lwp.h. * * State transitions must be made with the LWP's general lock held, * and may cause the LWP's lock pointer to change. Manipulation of * the general lock is not performed directly, but through calls to * lwp_lock(), lwp_unlock() and others. It should be noted that the * adaptive locks are not allowed to be released while the LWP's lock * is being held (unlike for other spin-locks). * * States and their associated locks: * * LSIDL, LSONPROC, LSZOMB, LSSUPENDED: * * Always covered by spc_lwplock, which protects LWPs not * associated with any other sync object. This is a per-CPU * lock and matches lwp::l_cpu. * * LSRUN: * * Always covered by spc_mutex, which protects the run queues. * This is a per-CPU lock and matches lwp::l_cpu. * * LSSLEEP: * * Covered by a lock associated with the sleep queue (sometimes * a turnstile sleep queue) that the LWP resides on. This can * be spc_lwplock for SOBJ_SLEEPQ_NULL (an "untracked" sleep). * * LSSTOP: * * If the LWP was previously sleeping (l_wchan != NULL), then * l_mutex references the sleep queue lock. If the LWP was * runnable or on the CPU when halted, or has been removed from * the sleep queue since halted, then the lock is spc_lwplock. * * The lock order is as follows: * * sleepq -> turnstile -> spc_lwplock -> spc_mutex * * Each process has a scheduler state lock (proc::p_lock), and a * number of counters on LWPs and their states: p_nzlwps, p_nrlwps, and * so on. When an LWP is to be entered into or removed from one of the * following states, p_lock must be held and the process wide counters * adjusted: * * LSIDL, LSZOMB, LSSTOP, LSSUSPENDED * * (But not always for kernel threads. There are some special cases * as mentioned above: soft interrupts, and the idle loops.) * * Note that an LWP is considered running or likely to run soon if in * one of the following states. This affects the value of p_nrlwps: * * LSRUN, LSONPROC, LSSLEEP * * p_lock does not need to be held when transitioning among these * three states, hence p_lock is rarely taken for state transitions. */ #include <sys/cdefs.h> __KERNEL_RCSID(0, "$NetBSD: kern_lwp.c,v 1.251 2022/07/01 01:06:04 riastradh Exp $"); #include "opt_ddb.h" #include "opt_lockdebug.h" #include "opt_dtrace.h" #define _LWP_API_PRIVATE #include <sys/param.h> #include <sys/systm.h> #include <sys/cpu.h> #include <sys/pool.h> #include <sys/proc.h> #include <sys/syscallargs.h> #include <sys/syscall_stats.h> #include <sys/kauth.h> #include <sys/sleepq.h> #include <sys/lockdebug.h> #include <sys/kmem.h> #include <sys/pset.h> #include <sys/intr.h> #include <sys/lwpctl.h> #include <sys/atomic.h> #include <sys/filedesc.h> #include <sys/fstrans.h> #include <sys/dtrace_bsd.h> #include <sys/sdt.h> #include <sys/ptrace.h> #include <sys/xcall.h> #include <sys/uidinfo.h> #include <sys/sysctl.h> #include <sys/psref.h> #include <sys/msan.h> #include <sys/kcov.h> #include <sys/cprng.h> #include <sys/futex.h> #include <uvm/uvm_extern.h> #include <uvm/uvm_object.h> static pool_cache_t lwp_cache __read_mostly; struct lwplist alllwp __cacheline_aligned; static int lwp_ctor(void *, void *, int); static void lwp_dtor(void *, void *); /* DTrace proc provider probes */ SDT_PROVIDER_DEFINE(proc); SDT_PROBE_DEFINE1(proc, kernel, , lwp__create, "struct lwp *"); SDT_PROBE_DEFINE1(proc, kernel, , lwp__start, "struct lwp *"); SDT_PROBE_DEFINE1(proc, kernel, , lwp__exit, "struct lwp *"); struct turnstile turnstile0 __cacheline_aligned; struct lwp lwp0 __aligned(MIN_LWP_ALIGNMENT) = { #ifdef LWP0_CPU_INFO .l_cpu = LWP0_CPU_INFO, #endif #ifdef LWP0_MD_INITIALIZER .l_md = LWP0_MD_INITIALIZER, #endif .l_proc = &proc0, .l_lid = 0, /* we own proc0's slot in the pid table */ .l_flag = LW_SYSTEM, .l_stat = LSONPROC, .l_ts = &turnstile0, .l_syncobj = &sched_syncobj, .l_refcnt = 0, .l_priority = PRI_USER + NPRI_USER - 1, .l_inheritedprio = -1, .l_class = SCHED_OTHER, .l_psid = PS_NONE, .l_pi_lenders = SLIST_HEAD_INITIALIZER(&lwp0.l_pi_lenders), .l_name = __UNCONST("swapper"), .l_fd = &filedesc0, }; static int lwp_maxlwp(void) { /* Assume 1 LWP per 1MiB. */ uint64_t lwps_per = ctob(physmem) / (1024 * 1024); return MAX(MIN(MAXMAXLWP, lwps_per), MAXLWP); } static int sysctl_kern_maxlwp(SYSCTLFN_PROTO); /* * sysctl helper routine for kern.maxlwp. Ensures that the new * values are not too low or too high. */ static int sysctl_kern_maxlwp(SYSCTLFN_ARGS) { int error, nmaxlwp; struct sysctlnode node; nmaxlwp = maxlwp; node = *rnode; node.sysctl_data = &nmaxlwp; error = sysctl_lookup(SYSCTLFN_CALL(&node)); if (error || newp == NULL) return error; if (nmaxlwp < 0 || nmaxlwp >= MAXMAXLWP) return EINVAL; if (nmaxlwp > lwp_maxlwp()) return EINVAL; maxlwp = nmaxlwp; return 0; } static void sysctl_kern_lwp_setup(void) { sysctl_createv(NULL, 0, NULL, NULL, CTLFLAG_PERMANENT|CTLFLAG_READWRITE, CTLTYPE_INT, "maxlwp", SYSCTL_DESCR("Maximum number of simultaneous threads"), sysctl_kern_maxlwp, 0, NULL, 0, CTL_KERN, CTL_CREATE, CTL_EOL); } void lwpinit(void) { LIST_INIT(&alllwp); lwpinit_specificdata(); /* * Provide a barrier to ensure that all mutex_oncpu() and rw_oncpu() * calls will exit before memory of LWPs is returned to the pool, where * KVA of LWP structure might be freed and re-used for other purposes. * Kernel preemption is disabled around mutex_oncpu() and rw_oncpu() * callers, therefore a regular passive serialization barrier will * do the job. */ lwp_cache = pool_cache_init(sizeof(lwp_t), MIN_LWP_ALIGNMENT, 0, PR_PSERIALIZE, "lwppl", NULL, IPL_NONE, lwp_ctor, lwp_dtor, NULL); maxlwp = lwp_maxlwp(); sysctl_kern_lwp_setup(); } void lwp0_init(void) { struct lwp *l = &lwp0; KASSERT((void *)uvm_lwp_getuarea(l) != NULL); LIST_INSERT_HEAD(&alllwp, l, l_list); callout_init(&l->l_timeout_ch, CALLOUT_MPSAFE); callout_setfunc(&l->l_timeout_ch, sleepq_timeout, l); cv_init(&l->l_sigcv, "sigwait"); cv_init(&l->l_waitcv, "vfork"); kauth_cred_hold(proc0.p_cred); l->l_cred = proc0.p_cred; kdtrace_thread_ctor(NULL, l); lwp_initspecific(l); SYSCALL_TIME_LWP_INIT(l); } /* * Initialize the non-zeroed portion of an lwp_t. */ static int lwp_ctor(void *arg, void *obj, int flags) { lwp_t *l = obj; l->l_stat = LSIDL; l->l_cpu = curcpu(); l->l_mutex = l->l_cpu->ci_schedstate.spc_lwplock; l->l_ts = pool_get(&turnstile_pool, flags); if (l->l_ts == NULL) { return ENOMEM; } else { turnstile_ctor(l->l_ts); return 0; } } static void lwp_dtor(void *arg, void *obj) { lwp_t *l = obj; /* * The value of l->l_cpu must still be valid at this point. */ KASSERT(l->l_cpu != NULL); /* * We can't return turnstile0 to the pool (it didn't come from it), * so if it comes up just drop it quietly and move on. */ if (l->l_ts != &turnstile0) pool_put(&turnstile_pool, l->l_ts); } /* * Set an LWP suspended. * * Must be called with p_lock held, and the LWP locked. Will unlock the * LWP before return. */ int lwp_suspend(struct lwp *curl, struct lwp *t) { int error; KASSERT(mutex_owned(t->l_proc->p_lock)); KASSERT(lwp_locked(t, NULL)); KASSERT(curl != t || curl->l_stat == LSONPROC); /* * If the current LWP has been told to exit, we must not suspend anyone * else or deadlock could occur. We won't return to userspace. */ if ((curl->l_flag & (LW_WEXIT | LW_WCORE)) != 0) { lwp_unlock(t); return (EDEADLK); } if ((t->l_flag & LW_DBGSUSPEND) != 0) { lwp_unlock(t); return 0; } error = 0; switch (t->l_stat) { case LSRUN: case LSONPROC: t->l_flag |= LW_WSUSPEND; lwp_need_userret(t); lwp_unlock(t); break; case LSSLEEP: t->l_flag |= LW_WSUSPEND; /* * Kick the LWP and try to get it to the kernel boundary * so that it will release any locks that it holds. * setrunnable() will release the lock. */ if ((t->l_flag & LW_SINTR) != 0) setrunnable(t); else lwp_unlock(t); break; case LSSUSPENDED: lwp_unlock(t); break; case LSSTOP: t->l_flag |= LW_WSUSPEND; setrunnable(t); break; case LSIDL: case LSZOMB: error = EINTR; /* It's what Solaris does..... */ lwp_unlock(t); break; } return (error); } /* * Restart a suspended LWP. * * Must be called with p_lock held, and the LWP locked. Will unlock the * LWP before return. */ void lwp_continue(struct lwp *l) { KASSERT(mutex_owned(l->l_proc->p_lock)); KASSERT(lwp_locked(l, NULL)); /* If rebooting or not suspended, then just bail out. */ if ((l->l_flag & LW_WREBOOT) != 0) { lwp_unlock(l); return; } l->l_flag &= ~LW_WSUSPEND; if (l->l_stat != LSSUSPENDED || (l->l_flag & LW_DBGSUSPEND) != 0) { lwp_unlock(l); return; } /* setrunnable() will release the lock. */ setrunnable(l); } /* * Restart a stopped LWP. * * Must be called with p_lock held, and the LWP NOT locked. Will unlock the * LWP before return. */ void lwp_unstop(struct lwp *l) { struct proc *p = l->l_proc; KASSERT(mutex_owned(&proc_lock)); KASSERT(mutex_owned(p->p_lock)); lwp_lock(l); KASSERT((l->l_flag & LW_DBGSUSPEND) == 0); /* If not stopped, then just bail out. */ if (l->l_stat != LSSTOP) { lwp_unlock(l); return; } p->p_stat = SACTIVE; p->p_sflag &= ~PS_STOPPING; if (!p->p_waited) p->p_pptr->p_nstopchild--; if (l->l_wchan == NULL) { /* setrunnable() will release the lock. */ setrunnable(l); } else if (p->p_xsig && (l->l_flag & LW_SINTR) != 0) { /* setrunnable() so we can receive the signal */ setrunnable(l); } else { l->l_stat = LSSLEEP; p->p_nrlwps++; lwp_unlock(l); } } /* * Wait for an LWP within the current process to exit. If 'lid' is * non-zero, we are waiting for a specific LWP. * * Must be called with p->p_lock held. */ int lwp_wait(struct lwp *l, lwpid_t lid, lwpid_t *departed, bool exiting) { const lwpid_t curlid = l->l_lid; proc_t *p = l->l_proc; lwp_t *l2, *next; int error; KASSERT(mutex_owned(p->p_lock)); p->p_nlwpwait++; l->l_waitingfor = lid; for (;;) { int nfound; /* * Avoid a race between exit1() and sigexit(): if the * process is dumping core, then we need to bail out: call * into lwp_userret() where we will be suspended until the * deed is done. */ if ((p->p_sflag & PS_WCORE) != 0) { mutex_exit(p->p_lock); lwp_userret(l); KASSERT(false); } /* * First off, drain any detached LWP that is waiting to be * reaped. */ while ((l2 = p->p_zomblwp) != NULL) { p->p_zomblwp = NULL; lwp_free(l2, false, false);/* releases proc mutex */ mutex_enter(p->p_lock); } /* * Now look for an LWP to collect. If the whole process is * exiting, count detached LWPs as eligible to be collected, * but don't drain them here. */ nfound = 0; error = 0; /* * If given a specific LID, go via pid_table and make sure * it's not detached. */ if (lid != 0) { l2 = proc_find_lwp(p, lid); if (l2 == NULL) { error = ESRCH; break; } KASSERT(l2->l_lid == lid); if ((l2->l_prflag & LPR_DETACHED) != 0) { error = EINVAL; break; } } else { l2 = LIST_FIRST(&p->p_lwps); } for (; l2 != NULL; l2 = next) { next = (lid != 0 ? NULL : LIST_NEXT(l2, l_sibling)); /* * If a specific wait and the target is waiting on * us, then avoid deadlock. This also traps LWPs * that try to wait on themselves. * * Note that this does not handle more complicated * cycles, like: t1 -> t2 -> t3 -> t1. The process * can still be killed so it is not a major problem. */ if (l2->l_lid == lid && l2->l_waitingfor == curlid) { error = EDEADLK; break; } if (l2 == l) continue; if ((l2->l_prflag & LPR_DETACHED) != 0) { nfound += exiting; continue; } if (lid != 0) { /* * Mark this LWP as the first waiter, if there * is no other. */ if (l2->l_waiter == 0) l2->l_waiter = curlid; } else if (l2->l_waiter != 0) { /* * It already has a waiter - so don't * collect it. If the waiter doesn't * grab it we'll get another chance * later. */ nfound++; continue; } nfound++; /* No need to lock the LWP in order to see LSZOMB. */ if (l2->l_stat != LSZOMB) continue; /* * We're no longer waiting. Reset the "first waiter" * pointer on the target, in case it was us. */ l->l_waitingfor = 0; l2->l_waiter = 0; p->p_nlwpwait--; if (departed) *departed = l2->l_lid; sched_lwp_collect(l2); /* lwp_free() releases the proc lock. */ lwp_free(l2, false, false); mutex_enter(p->p_lock); return 0; } if (error != 0) break; if (nfound == 0) { error = ESRCH; break; } /* * Note: since the lock will be dropped, need to restart on * wakeup to run all LWPs again, e.g. there may be new LWPs. */ if (exiting) { KASSERT(p->p_nlwps > 1); error = cv_timedwait(&p->p_lwpcv, p->p_lock, 1); break; } /* * Break out if all LWPs are in _lwp_wait(). There are * other ways to hang the process with _lwp_wait(), but the * sleep is interruptable so little point checking for them. */ if (p->p_nlwpwait == p->p_nlwps) { error = EDEADLK; break; } /* * Sit around and wait for something to happen. We'll be * awoken if any of the conditions examined change: if an * LWP exits, is collected, or is detached. */ if ((error = cv_wait_sig(&p->p_lwpcv, p->p_lock)) != 0) break; } /* * We didn't find any LWPs to collect, we may have received a * signal, or some other condition has caused us to bail out. * * If waiting on a specific LWP, clear the waiters marker: some * other LWP may want it. Then, kick all the remaining waiters * so that they can re-check for zombies and for deadlock. */ if (lid != 0) { l2 = proc_find_lwp(p, lid); KASSERT(l2 == NULL || l2->l_lid == lid); if (l2 != NULL && l2->l_waiter == curlid) l2->l_waiter = 0; } p->p_nlwpwait--; l->l_waitingfor = 0; cv_broadcast(&p->p_lwpcv); return error; } /* * Create a new LWP within process 'p2', using LWP 'l1' as a template. * The new LWP is created in state LSIDL and must be set running, * suspended, or stopped by the caller. */ int lwp_create(lwp_t *l1, proc_t *p2, vaddr_t uaddr, int flags, void *stack, size_t stacksize, void (*func)(void *), void *arg, lwp_t **rnewlwpp, int sclass, const sigset_t *sigmask, const stack_t *sigstk) { struct lwp *l2; KASSERT(l1 == curlwp || l1->l_proc == &proc0); /* * Enforce limits, excluding the first lwp and kthreads. We must * use the process credentials here when adjusting the limit, as * they are what's tied to the accounting entity. However for * authorizing the action, we'll use the LWP's credentials. */ mutex_enter(p2->p_lock); if (p2->p_nlwps != 0 && p2 != &proc0) { uid_t uid = kauth_cred_getuid(p2->p_cred); int count = chglwpcnt(uid, 1); if (__predict_false(count > p2->p_rlimit[RLIMIT_NTHR].rlim_cur)) { if (kauth_authorize_process(l1->l_cred, KAUTH_PROCESS_RLIMIT, p2, KAUTH_ARG(KAUTH_REQ_PROCESS_RLIMIT_BYPASS), &p2->p_rlimit[RLIMIT_NTHR], KAUTH_ARG(RLIMIT_NTHR)) != 0) { (void)chglwpcnt(uid, -1); mutex_exit(p2->p_lock); return EAGAIN; } } } /* * First off, reap any detached LWP waiting to be collected. * We can re-use its LWP structure and turnstile. */ if ((l2 = p2->p_zomblwp) != NULL) { p2->p_zomblwp = NULL; lwp_free(l2, true, false); /* p2 now unlocked by lwp_free() */ KASSERT(l2->l_ts != NULL); KASSERT(l2->l_inheritedprio == -1); KASSERT(SLIST_EMPTY(&l2->l_pi_lenders)); memset(&l2->l_startzero, 0, sizeof(*l2) - offsetof(lwp_t, l_startzero)); } else { mutex_exit(p2->p_lock); l2 = pool_cache_get(lwp_cache, PR_WAITOK); memset(&l2->l_startzero, 0, sizeof(*l2) - offsetof(lwp_t, l_startzero)); SLIST_INIT(&l2->l_pi_lenders); } /* * Because of lockless lookup via pid_table, the LWP can be locked * and inspected briefly even after it's freed, so a few fields are * kept stable. */ KASSERT(l2->l_stat == LSIDL); KASSERT(l2->l_cpu != NULL); KASSERT(l2->l_ts != NULL); KASSERT(l2->l_mutex == l2->l_cpu->ci_schedstate.spc_lwplock); l2->l_proc = p2; l2->l_refcnt = 0; l2->l_class = sclass; /* * Allocate a process ID for this LWP. We need to do this now * while we can still unwind if it fails. Because we're marked * as LSIDL, no lookups by the ID will succeed. * * N.B. this will always succeed for the first LWP in a process, * because proc_alloc_lwpid() will usurp the slot. Also note * that l2->l_proc MUST be valid so that lookups of the proc * will succeed, even if the LWP itself is not visible. */ if (__predict_false(proc_alloc_lwpid(p2, l2) == -1)) { pool_cache_put(lwp_cache, l2); return EAGAIN; } /* * If vfork(), we want the LWP to run fast and on the same CPU * as its parent, so that it can reuse the VM context and cache * footprint on the local CPU. */ l2->l_kpriority = ((flags & LWP_VFORK) ? true : false); l2->l_kpribase = PRI_KERNEL; l2->l_priority = l1->l_priority; l2->l_inheritedprio = -1; l2->l_protectprio = -1; l2->l_auxprio = -1; l2->l_flag = 0; l2->l_pflag = LP_MPSAFE; TAILQ_INIT(&l2->l_ld_locks); l2->l_psrefs = 0; kmsan_lwp_alloc(l2); /* * For vfork, borrow parent's lwpctl context if it exists. * This also causes us to return via lwp_userret. */ if (flags & LWP_VFORK && l1->l_lwpctl) { l2->l_lwpctl = l1->l_lwpctl; l2->l_flag |= LW_LWPCTL; } /* * If not the first LWP in the process, grab a reference to the * descriptor table. */ l2->l_fd = p2->p_fd; if (p2->p_nlwps != 0) { KASSERT(l1->l_proc == p2); fd_hold(l2); } else { KASSERT(l1->l_proc != p2); } if (p2->p_flag & PK_SYSTEM) { /* Mark it as a system LWP. */ l2->l_flag |= LW_SYSTEM; } kdtrace_thread_ctor(NULL, l2); lwp_initspecific(l2); sched_lwp_fork(l1, l2); lwp_update_creds(l2); callout_init(&l2->l_timeout_ch, CALLOUT_MPSAFE); callout_setfunc(&l2->l_timeout_ch, sleepq_timeout, l2); cv_init(&l2->l_sigcv, "sigwait"); cv_init(&l2->l_waitcv, "vfork"); l2->l_syncobj = &sched_syncobj; PSREF_DEBUG_INIT_LWP(l2); if (rnewlwpp != NULL) *rnewlwpp = l2; /* * PCU state needs to be saved before calling uvm_lwp_fork() so that * the MD cpu_lwp_fork() can copy the saved state to the new LWP. */ pcu_save_all(l1); #if PCU_UNIT_COUNT > 0 l2->l_pcu_valid = l1->l_pcu_valid; #endif uvm_lwp_setuarea(l2, uaddr); uvm_lwp_fork(l1, l2, stack, stacksize, func, (arg != NULL) ? arg : l2); mutex_enter(p2->p_lock); if ((flags & LWP_DETACHED) != 0) { l2->l_prflag = LPR_DETACHED; p2->p_ndlwps++; } else l2->l_prflag = 0; if (l1->l_proc == p2) { /* * These flags are set while p_lock is held. Copy with * p_lock held too, so the LWP doesn't sneak into the * process without them being set. */ l2->l_flag |= (l1->l_flag & (LW_WEXIT | LW_WREBOOT | LW_WCORE)); } else { /* fork(): pending core/exit doesn't apply to child. */ l2->l_flag |= (l1->l_flag & LW_WREBOOT); } l2->l_sigstk = *sigstk; l2->l_sigmask = *sigmask; TAILQ_INIT(&l2->l_sigpend.sp_info); sigemptyset(&l2->l_sigpend.sp_set); LIST_INSERT_HEAD(&p2->p_lwps, l2, l_sibling); p2->p_nlwps++; p2->p_nrlwps++; KASSERT(l2->l_affinity == NULL); /* Inherit the affinity mask. */ if (l1->l_affinity) { /* * Note that we hold the state lock while inheriting * the affinity to avoid race with sched_setaffinity(). */ lwp_lock(l1); if (l1->l_affinity) { kcpuset_use(l1->l_affinity); l2->l_affinity = l1->l_affinity; } lwp_unlock(l1); } /* This marks the end of the "must be atomic" section. */ mutex_exit(p2->p_lock); SDT_PROBE(proc, kernel, , lwp__create, l2, 0, 0, 0, 0); mutex_enter(&proc_lock); LIST_INSERT_HEAD(&alllwp, l2, l_list); /* Inherit a processor-set */ l2->l_psid = l1->l_psid; mutex_exit(&proc_lock); SYSCALL_TIME_LWP_INIT(l2); if (p2->p_emul->e_lwp_fork) (*p2->p_emul->e_lwp_fork)(l1, l2); return (0); } /* * Set a new LWP running. If the process is stopping, then the LWP is * created stopped. */ void lwp_start(lwp_t *l, int flags) { proc_t *p = l->l_proc; mutex_enter(p->p_lock); lwp_lock(l); KASSERT(l->l_stat == LSIDL); if ((flags & LWP_SUSPENDED) != 0) { /* It'll suspend itself in lwp_userret(). */ l->l_flag |= LW_WSUSPEND; } if (p->p_stat == SSTOP || (p->p_sflag & PS_STOPPING) != 0) { KASSERT(l->l_wchan == NULL); l->l_stat = LSSTOP; p->p_nrlwps--; lwp_unlock(l); } else { setrunnable(l); /* LWP now unlocked */ } mutex_exit(p->p_lock); } /* * Called by MD code when a new LWP begins execution. Must be called * with the previous LWP locked (so at splsched), or if there is no * previous LWP, at splsched. */ void lwp_startup(struct lwp *prev, struct lwp *new_lwp) { kmutex_t *lock; KASSERTMSG(new_lwp == curlwp, "l %p curlwp %p prevlwp %p", new_lwp, curlwp, prev); KASSERT(kpreempt_disabled()); KASSERT(prev != NULL); KASSERT((prev->l_pflag & LP_RUNNING) != 0); KASSERT(curcpu()->ci_mtx_count == -2); /* * Immediately mark the previous LWP as no longer running and * unlock (to keep lock wait times short as possible). If a * zombie, don't touch after clearing LP_RUNNING as it could be * reaped by another CPU. Use atomic_store_release to ensure * this -- matches atomic_load_acquire in lwp_free. */ lock = prev->l_mutex; if (__predict_false(prev->l_stat == LSZOMB)) { atomic_store_release(&prev->l_pflag, prev->l_pflag & ~LP_RUNNING); } else { prev->l_pflag &= ~LP_RUNNING; } mutex_spin_exit(lock); /* Correct spin mutex count after mi_switch(). */ curcpu()->ci_mtx_count = 0; /* Install new VM context. */ if (__predict_true(new_lwp->l_proc->p_vmspace)) { pmap_activate(new_lwp); } /* We remain at IPL_SCHED from mi_switch() - reset it. */ spl0(); LOCKDEBUG_BARRIER(NULL, 0); SDT_PROBE(proc, kernel, , lwp__start, new_lwp, 0, 0, 0, 0); /* For kthreads, acquire kernel lock if not MPSAFE. */ if (__predict_false((new_lwp->l_pflag & LP_MPSAFE) == 0)) { KERNEL_LOCK(1, new_lwp); } } /* * Exit an LWP. * * *** WARNING *** This can be called with (l != curlwp) in error paths. */ void lwp_exit(struct lwp *l) { struct proc *p = l->l_proc; struct lwp *l2; bool current; current = (l == curlwp); KASSERT(current || (l->l_stat == LSIDL && l->l_target_cpu == NULL)); KASSERT(p == curproc); SDT_PROBE(proc, kernel, , lwp__exit, l, 0, 0, 0, 0); /* Verify that we hold no locks; for DIAGNOSTIC check kernel_lock. */ LOCKDEBUG_BARRIER(NULL, 0); KASSERTMSG(curcpu()->ci_biglock_count == 0, "kernel_lock leaked"); /* * If we are the last live LWP in a process, we need to exit the * entire process. We do so with an exit status of zero, because * it's a "controlled" exit, and because that's what Solaris does. * * We are not quite a zombie yet, but for accounting purposes we * must increment the count of zombies here. * * Note: the last LWP's specificdata will be deleted here. */ mutex_enter(p->p_lock); if (p->p_nlwps - p->p_nzlwps == 1) { KASSERT(current == true); KASSERT(p != &proc0); exit1(l, 0, 0); /* NOTREACHED */ } p->p_nzlwps++; /* * Perform any required thread cleanup. Do this early so * anyone wanting to look us up with lwp_getref_lwpid() will * fail to find us before we become a zombie. * * N.B. this will unlock p->p_lock on our behalf. */ lwp_thread_cleanup(l); if (p->p_emul->e_lwp_exit) (*p->p_emul->e_lwp_exit)(l); /* Drop filedesc reference. */ fd_free(); /* Release fstrans private data. */ fstrans_lwp_dtor(l); /* Delete the specificdata while it's still safe to sleep. */ lwp_finispecific(l); /* * Release our cached credentials. */ kauth_cred_free(l->l_cred); callout_destroy(&l->l_timeout_ch); /* * If traced, report LWP exit event to the debugger. * * Remove the LWP from the global list. * Free its LID from the PID namespace if needed. */ mutex_enter(&proc_lock); if ((p->p_slflag & (PSL_TRACED|PSL_TRACELWP_EXIT)) == (PSL_TRACED|PSL_TRACELWP_EXIT)) { mutex_enter(p->p_lock); if (ISSET(p->p_sflag, PS_WEXIT)) { mutex_exit(p->p_lock); /* * We are exiting, bail out without informing parent * about a terminating LWP as it would deadlock. */ } else { eventswitch(TRAP_LWP, PTRACE_LWP_EXIT, l->l_lid); mutex_enter(&proc_lock); } } LIST_REMOVE(l, l_list); mutex_exit(&proc_lock); /* * Get rid of all references to the LWP that others (e.g. procfs) * may have, and mark the LWP as a zombie. If the LWP is detached, * mark it waiting for collection in the proc structure. Note that * before we can do that, we need to free any other dead, deatched * LWP waiting to meet its maker. * * All conditions need to be observed upon under the same hold of * p_lock, because if the lock is dropped any of them can change. */ mutex_enter(p->p_lock); for (;;) { if (lwp_drainrefs(l)) continue; if ((l->l_prflag & LPR_DETACHED) != 0) { if ((l2 = p->p_zomblwp) != NULL) { p->p_zomblwp = NULL; lwp_free(l2, false, false); /* proc now unlocked */ mutex_enter(p->p_lock); continue; } p->p_zomblwp = l; } break; } /* * If we find a pending signal for the process and we have been * asked to check for signals, then we lose: arrange to have * all other LWPs in the process check for signals. */ if ((l->l_flag & LW_PENDSIG) != 0 && firstsig(&p->p_sigpend.sp_set) != 0) { LIST_FOREACH(l2, &p->p_lwps, l_sibling) { lwp_lock(l2); signotify(l2); lwp_unlock(l2); } } /* * Release any PCU resources before becoming a zombie. */ pcu_discard_all(l); lwp_lock(l); l->l_stat = LSZOMB; if (l->l_name != NULL) { strcpy(l->l_name, "(zombie)"); } lwp_unlock(l); p->p_nrlwps--; cv_broadcast(&p->p_lwpcv); if (l->l_lwpctl != NULL) l->l_lwpctl->lc_curcpu = LWPCTL_CPU_EXITED; mutex_exit(p->p_lock); /* * We can no longer block. At this point, lwp_free() may already * be gunning for us. On a multi-CPU system, we may be off p_lwps. * * Free MD LWP resources. */ cpu_lwp_free(l, 0); if (current) { /* Switch away into oblivion. */ lwp_lock(l); spc_lock(l->l_cpu); mi_switch(l); panic("lwp_exit"); } } /* * Free a dead LWP's remaining resources. * * XXXLWP limits. */ void lwp_free(struct lwp *l, bool recycle, bool last) { struct proc *p = l->l_proc; struct rusage *ru; ksiginfoq_t kq; KASSERT(l != curlwp); KASSERT(last || mutex_owned(p->p_lock)); /* * We use the process credentials instead of the lwp credentials here * because the lwp credentials maybe cached (just after a setuid call) * and we don't want pay for syncing, since the lwp is going away * anyway */ if (p != &proc0 && p->p_nlwps != 1) (void)chglwpcnt(kauth_cred_getuid(p->p_cred), -1); /* * In the unlikely event that the LWP is still on the CPU, * then spin until it has switched away. * * atomic_load_acquire matches atomic_store_release in * lwp_startup and mi_switch. */ while (__predict_false((atomic_load_acquire(&l->l_pflag) & LP_RUNNING) != 0)) { SPINLOCK_BACKOFF_HOOK; } /* * Now that the LWP's known off the CPU, reset its state back to * LSIDL, which defeats anything that might have gotten a hold on * the LWP via pid_table before the ID was freed. It's important * to do this with both the LWP locked and p_lock held. * * Also reset the CPU and lock pointer back to curcpu(), since the * LWP will in all likelyhood be cached with the current CPU in * lwp_cache when we free it and later allocated from there again * (avoid incidental lock contention). */ lwp_lock(l); l->l_stat = LSIDL; l->l_cpu = curcpu(); lwp_unlock_to(l, l->l_cpu->ci_schedstate.spc_lwplock); /* * If this was not the last LWP in the process, then adjust counters * and unlock. This is done differently for the last LWP in exit1(). */ if (!last) { /* * Add the LWP's run time to the process' base value. * This needs to co-incide with coming off p_lwps. */ bintime_add(&p->p_rtime, &l->l_rtime); p->p_pctcpu += l->l_pctcpu; ru = &p->p_stats->p_ru; ruadd(ru, &l->l_ru); ru->ru_nvcsw += (l->l_ncsw - l->l_nivcsw); ru->ru_nivcsw += l->l_nivcsw; LIST_REMOVE(l, l_sibling); p->p_nlwps--; p->p_nzlwps--; if ((l->l_prflag & LPR_DETACHED) != 0) p->p_ndlwps--; /* * Have any LWPs sleeping in lwp_wait() recheck for * deadlock. */ cv_broadcast(&p->p_lwpcv); mutex_exit(p->p_lock); /* Free the LWP ID. */ mutex_enter(&proc_lock); proc_free_lwpid(p, l->l_lid); mutex_exit(&proc_lock); } /* * Destroy the LWP's remaining signal information. */ ksiginfo_queue_init(&kq); sigclear(&l->l_sigpend, NULL, &kq); ksiginfo_queue_drain(&kq); cv_destroy(&l->l_sigcv); cv_destroy(&l->l_waitcv); /* * Free lwpctl structure and affinity. */ if (l->l_lwpctl) { lwp_ctl_free(l); } if (l->l_affinity) { kcpuset_unuse(l->l_affinity, NULL); l->l_affinity = NULL; } /* * Free remaining data structures and the LWP itself unless the * caller wants to recycle. */ if (l->l_name != NULL) kmem_free(l->l_name, MAXCOMLEN); kmsan_lwp_free(l); kcov_lwp_free(l); cpu_lwp_free2(l); uvm_lwp_exit(l); KASSERT(SLIST_EMPTY(&l->l_pi_lenders)); KASSERT(l->l_inheritedprio == -1); KASSERT(l->l_blcnt == 0); kdtrace_thread_dtor(NULL, l); if (!recycle) pool_cache_put(lwp_cache, l); } /* * Migrate the LWP to the another CPU. Unlocks the LWP. */ void lwp_migrate(lwp_t *l, struct cpu_info *tci) { struct schedstate_percpu *tspc; int lstat = l->l_stat; KASSERT(lwp_locked(l, NULL)); KASSERT(tci != NULL); /* If LWP is still on the CPU, it must be handled like LSONPROC */ if ((l->l_pflag & LP_RUNNING) != 0) { lstat = LSONPROC; } /* * The destination CPU could be changed while previous migration * was not finished. */ if (l->l_target_cpu != NULL) { l->l_target_cpu = tci; lwp_unlock(l); return; } /* Nothing to do if trying to migrate to the same CPU */ if (l->l_cpu == tci) { lwp_unlock(l); return; } KASSERT(l->l_target_cpu == NULL); tspc = &tci->ci_schedstate; switch (lstat) { case LSRUN: l->l_target_cpu = tci; break; case LSSLEEP: l->l_cpu = tci; break; case LSIDL: case LSSTOP: case LSSUSPENDED: l->l_cpu = tci; if (l->l_wchan == NULL) { lwp_unlock_to(l, tspc->spc_lwplock); return; } break; case LSONPROC: l->l_target_cpu = tci; spc_lock(l->l_cpu); sched_resched_cpu(l->l_cpu, PRI_USER_RT, true); /* spc now unlocked */ break; } lwp_unlock(l); } #define lwp_find_exclude(l) \ ((l)->l_stat == LSIDL || (l)->l_stat == LSZOMB) /* * Find the LWP in the process. Arguments may be zero, in such case, * the calling process and first LWP in the list will be used. * On success - returns proc locked. * * => pid == 0 -> look in curproc. * => pid == -1 -> match any proc. * => otherwise look up the proc. * * => lid == 0 -> first LWP in the proc * => otherwise specific LWP */ struct lwp * lwp_find2(pid_t pid, lwpid_t lid) { proc_t *p; lwp_t *l; /* First LWP of specified proc. */ if (lid == 0) { switch (pid) { case -1: /* No lookup keys. */ return NULL; case 0: p = curproc; mutex_enter(p->p_lock); break; default: mutex_enter(&proc_lock); p = proc_find(pid); if (__predict_false(p == NULL)) { mutex_exit(&proc_lock); return NULL; } mutex_enter(p->p_lock); mutex_exit(&proc_lock); break; } LIST_FOREACH(l, &p->p_lwps, l_sibling) { if (__predict_true(!lwp_find_exclude(l))) break; } goto out; } l = proc_find_lwp_acquire_proc(lid, &p); if (l == NULL) return NULL; KASSERT(p != NULL); KASSERT(mutex_owned(p->p_lock)); if (__predict_false(lwp_find_exclude(l))) { l = NULL; goto out; } /* Apply proc filter, if applicable. */ switch (pid) { case -1: /* Match anything. */ break; case 0: if (p != curproc) l = NULL; break; default: if (p->p_pid != pid) l = NULL; break; } out: if (__predict_false(l == NULL)) { mutex_exit(p->p_lock); } return l; } /* * Look up a live LWP within the specified process. * * Must be called with p->p_lock held (as it looks at the radix tree, * and also wants to exclude idle and zombie LWPs). */ struct lwp * lwp_find(struct proc *p, lwpid_t id) { struct lwp *l; KASSERT(mutex_owned(p->p_lock)); l = proc_find_lwp(p, id); KASSERT(l == NULL || l->l_lid == id); /* * No need to lock - all of these conditions will * be visible with the process level mutex held. */ if (__predict_false(l != NULL && lwp_find_exclude(l))) l = NULL; return l; } /* * Update an LWP's cached credentials to mirror the process' master copy. * * This happens early in the syscall path, on user trap, and on LWP * creation. A long-running LWP can also voluntarily choose to update * its credentials by calling this routine. This may be called from * LWP_CACHE_CREDS(), which checks l->l_prflag & LPR_CRMOD beforehand. */ void lwp_update_creds(struct lwp *l) { kauth_cred_t oc; struct proc *p; p = l->l_proc; oc = l->l_cred; mutex_enter(p->p_lock); kauth_cred_hold(p->p_cred); l->l_cred = p->p_cred; l->l_prflag &= ~LPR_CRMOD; mutex_exit(p->p_lock); if (oc != NULL) kauth_cred_free(oc); } /* * Verify that an LWP is locked, and optionally verify that the lock matches * one we specify. */ int lwp_locked(struct lwp *l, kmutex_t *mtx) { kmutex_t *cur = l->l_mutex; return mutex_owned(cur) && (mtx == cur || mtx == NULL); } /* * Lend a new mutex to an LWP. The old mutex must be held. */ kmutex_t * lwp_setlock(struct lwp *l, kmutex_t *mtx) { kmutex_t *oldmtx = l->l_mutex; KASSERT(mutex_owned(oldmtx)); atomic_store_release(&l->l_mutex, mtx); return oldmtx; } /* * Lend a new mutex to an LWP, and release the old mutex. The old mutex * must be held. */ void lwp_unlock_to(struct lwp *l, kmutex_t *mtx) { kmutex_t *old; KASSERT(lwp_locked(l, NULL)); old = l->l_mutex; atomic_store_release(&l->l_mutex, mtx); mutex_spin_exit(old); } int lwp_trylock(struct lwp *l) { kmutex_t *old; for (;;) { if (!mutex_tryenter(old = atomic_load_consume(&l->l_mutex))) return 0; if (__predict_true(atomic_load_relaxed(&l->l_mutex) == old)) return 1; mutex_spin_exit(old); } } void lwp_unsleep(lwp_t *l, bool unlock) { KASSERT(mutex_owned(l->l_mutex)); (*l->l_syncobj->sobj_unsleep)(l, unlock); } /* * Handle exceptions for mi_userret(). Called if a member of LW_USERRET is * set. */ void lwp_userret(struct lwp *l) { struct proc *p; int sig; KASSERT(l == curlwp); KASSERT(l->l_stat == LSONPROC); p = l->l_proc; /* * It is safe to do this read unlocked on a MP system.. */ while ((l->l_flag & LW_USERRET) != 0) { /* * Process pending signals first, unless the process * is dumping core or exiting, where we will instead * enter the LW_WSUSPEND case below. */ if ((l->l_flag & (LW_PENDSIG | LW_WCORE | LW_WEXIT)) == LW_PENDSIG) { mutex_enter(p->p_lock); while ((sig = issignal(l)) != 0) postsig(sig); mutex_exit(p->p_lock); } /* * Core-dump or suspend pending. * * In case of core dump, suspend ourselves, so that the kernel * stack and therefore the userland registers saved in the * trapframe are around for coredump() to write them out. * We also need to save any PCU resources that we have so that * they accessible for coredump(). We issue a wakeup on * p->p_lwpcv so that sigexit() will write the core file out * once all other LWPs are suspended. */ if ((l->l_flag & LW_WSUSPEND) != 0) { pcu_save_all(l); mutex_enter(p->p_lock); p->p_nrlwps--; cv_broadcast(&p->p_lwpcv); lwp_lock(l); l->l_stat = LSSUSPENDED; lwp_unlock(l); mutex_exit(p->p_lock); lwp_lock(l); spc_lock(l->l_cpu); mi_switch(l); } /* Process is exiting. */ if ((l->l_flag & LW_WEXIT) != 0) { lwp_exit(l); KASSERT(0); /* NOTREACHED */ } /* update lwpctl processor (for vfork child_return) */ if (l->l_flag & LW_LWPCTL) { lwp_lock(l); KASSERT(kpreempt_disabled()); l->l_lwpctl->lc_curcpu = (int)cpu_index(l->l_cpu); l->l_lwpctl->lc_pctr++; l->l_flag &= ~LW_LWPCTL; lwp_unlock(l); } } } /* * Force an LWP to enter the kernel, to take a trip through lwp_userret(). */ void lwp_need_userret(struct lwp *l) { KASSERT(!cpu_intr_p()); KASSERT(lwp_locked(l, NULL)); /* * If the LWP is in any state other than LSONPROC, we know that it * is executing in-kernel and will hit userret() on the way out. * * If the LWP is curlwp, then we know we'll be back out to userspace * soon (can't be called from a hardware interrupt here). * * Otherwise, we can't be sure what the LWP is doing, so first make * sure the update to l_flag will be globally visible, and then * force the LWP to take a trip through trap() where it will do * userret(). */ if (l->l_stat == LSONPROC && l != curlwp) { membar_producer(); cpu_signotify(l); } } /* * Add one reference to an LWP. This will prevent the LWP from * exiting, thus keep the lwp structure and PCB around to inspect. */ void lwp_addref(struct lwp *l) { KASSERT(mutex_owned(l->l_proc->p_lock)); KASSERT(l->l_stat != LSZOMB); l->l_refcnt++; } /* * Remove one reference to an LWP. If this is the last reference, * then we must finalize the LWP's death. */ void lwp_delref(struct lwp *l) { struct proc *p = l->l_proc; mutex_enter(p->p_lock); lwp_delref2(l); mutex_exit(p->p_lock); } /* * Remove one reference to an LWP. If this is the last reference, * then we must finalize the LWP's death. The proc mutex is held * on entry. */ void lwp_delref2(struct lwp *l) { struct proc *p = l->l_proc; KASSERT(mutex_owned(p->p_lock)); KASSERT(l->l_stat != LSZOMB); KASSERT(l->l_refcnt > 0); if (--l->l_refcnt == 0) cv_broadcast(&p->p_lwpcv); } /* * Drain all references to the current LWP. Returns true if * we blocked. */ bool lwp_drainrefs(struct lwp *l) { struct proc *p = l->l_proc; bool rv = false; KASSERT(mutex_owned(p->p_lock)); l->l_prflag |= LPR_DRAINING; while (l->l_refcnt > 0) { rv = true; cv_wait(&p->p_lwpcv, p->p_lock); } return rv; } /* * Return true if the specified LWP is 'alive'. Only p->p_lock need * be held. */ bool lwp_alive(lwp_t *l) { KASSERT(mutex_owned(l->l_proc->p_lock)); switch (l->l_stat) { case LSSLEEP: case LSRUN: case LSONPROC: case LSSTOP: case LSSUSPENDED: return true; default: return false; } } /* * Return first live LWP in the process. */ lwp_t * lwp_find_first(proc_t *p) { lwp_t *l; KASSERT(mutex_owned(p->p_lock)); LIST_FOREACH(l, &p->p_lwps, l_sibling) { if (lwp_alive(l)) { return l; } } return NULL; } /* * Allocate a new lwpctl structure for a user LWP. */ int lwp_ctl_alloc(vaddr_t *uaddr) { lcproc_t *lp; u_int bit, i, offset; struct uvm_object *uao; int error; lcpage_t *lcp; proc_t *p; lwp_t *l; l = curlwp; p = l->l_proc; /* don't allow a vforked process to create lwp ctls */ if (p->p_lflag & PL_PPWAIT) return EBUSY; if (l->l_lcpage != NULL) { lcp = l->l_lcpage; *uaddr = lcp->lcp_uaddr + (vaddr_t)l->l_lwpctl - lcp->lcp_kaddr; return 0; } /* First time around, allocate header structure for the process. */ if ((lp = p->p_lwpctl) == NULL) { lp = kmem_alloc(sizeof(*lp), KM_SLEEP); mutex_init(&lp->lp_lock, MUTEX_DEFAULT, IPL_NONE); lp->lp_uao = NULL; TAILQ_INIT(&lp->lp_pages); mutex_enter(p->p_lock); if (p->p_lwpctl == NULL) { p->p_lwpctl = lp; mutex_exit(p->p_lock); } else { mutex_exit(p->p_lock); mutex_destroy(&lp->lp_lock); kmem_free(lp, sizeof(*lp)); lp = p->p_lwpctl; } } /* * Set up an anonymous memory region to hold the shared pages. * Map them into the process' address space. The user vmspace * gets the first reference on the UAO. */ mutex_enter(&lp->lp_lock); if (lp->lp_uao == NULL) { lp->lp_uao = uao_create(LWPCTL_UAREA_SZ, 0); lp->lp_cur = 0; lp->lp_max = LWPCTL_UAREA_SZ; lp->lp_uva = p->p_emul->e_vm_default_addr(p, (vaddr_t)p->p_vmspace->vm_daddr, LWPCTL_UAREA_SZ, p->p_vmspace->vm_map.flags & VM_MAP_TOPDOWN); error = uvm_map(&p->p_vmspace->vm_map, &lp->lp_uva, LWPCTL_UAREA_SZ, lp->lp_uao, 0, 0, UVM_MAPFLAG(UVM_PROT_RW, UVM_PROT_RW, UVM_INH_NONE, UVM_ADV_NORMAL, 0)); if (error != 0) { uao_detach(lp->lp_uao); lp->lp_uao = NULL; mutex_exit(&lp->lp_lock); return error; } } /* Get a free block and allocate for this LWP. */ TAILQ_FOREACH(lcp, &lp->lp_pages, lcp_chain) { if (lcp->lcp_nfree != 0) break; } if (lcp == NULL) { /* Nothing available - try to set up a free page. */ if (lp->lp_cur == lp->lp_max) { mutex_exit(&lp->lp_lock); return ENOMEM; } lcp = kmem_alloc(LWPCTL_LCPAGE_SZ, KM_SLEEP); /* * Wire the next page down in kernel space. Since this * is a new mapping, we must add a reference. */ uao = lp->lp_uao; (*uao->pgops->pgo_reference)(uao); lcp->lcp_kaddr = vm_map_min(kernel_map); error = uvm_map(kernel_map, &lcp->lcp_kaddr, PAGE_SIZE, uao, lp->lp_cur, PAGE_SIZE, UVM_MAPFLAG(UVM_PROT_RW, UVM_PROT_RW, UVM_INH_NONE, UVM_ADV_RANDOM, 0)); if (error != 0) { mutex_exit(&lp->lp_lock); kmem_free(lcp, LWPCTL_LCPAGE_SZ); (*uao->pgops->pgo_detach)(uao); return error; } error = uvm_map_pageable(kernel_map, lcp->lcp_kaddr, lcp->lcp_kaddr + PAGE_SIZE, FALSE, 0); if (error != 0) { mutex_exit(&lp->lp_lock); uvm_unmap(kernel_map, lcp->lcp_kaddr, lcp->lcp_kaddr + PAGE_SIZE); kmem_free(lcp, LWPCTL_LCPAGE_SZ); return error; } /* Prepare the page descriptor and link into the list. */ lcp->lcp_uaddr = lp->lp_uva + lp->lp_cur; lp->lp_cur += PAGE_SIZE; lcp->lcp_nfree = LWPCTL_PER_PAGE; lcp->lcp_rotor = 0; memset(lcp->lcp_bitmap, 0xff, LWPCTL_BITMAP_SZ); TAILQ_INSERT_HEAD(&lp->lp_pages, lcp, lcp_chain); } for (i = lcp->lcp_rotor; lcp->lcp_bitmap[i] == 0;) { if (++i >= LWPCTL_BITMAP_ENTRIES) i = 0; } bit = ffs(lcp->lcp_bitmap[i]) - 1; lcp->lcp_bitmap[i] ^= (1U << bit); lcp->lcp_rotor = i; lcp->lcp_nfree--; l->l_lcpage = lcp; offset = (i << 5) + bit; l->l_lwpctl = (lwpctl_t *)lcp->lcp_kaddr + offset; *uaddr = lcp->lcp_uaddr + offset * sizeof(lwpctl_t); mutex_exit(&lp->lp_lock); KPREEMPT_DISABLE(l); l->l_lwpctl->lc_curcpu = (int)cpu_index(curcpu()); KPREEMPT_ENABLE(l); return 0; } /* * Free an lwpctl structure back to the per-process list. */ void lwp_ctl_free(lwp_t *l) { struct proc *p = l->l_proc; lcproc_t *lp; lcpage_t *lcp; u_int map, offset; /* don't free a lwp context we borrowed for vfork */ if (p->p_lflag & PL_PPWAIT) { l->l_lwpctl = NULL; return; } lp = p->p_lwpctl; KASSERT(lp != NULL); lcp = l->l_lcpage; offset = (u_int)((lwpctl_t *)l->l_lwpctl - (lwpctl_t *)lcp->lcp_kaddr); KASSERT(offset < LWPCTL_PER_PAGE); mutex_enter(&lp->lp_lock); lcp->lcp_nfree++; map = offset >> 5; lcp->lcp_bitmap[map] |= (1U << (offset & 31)); if (lcp->lcp_bitmap[lcp->lcp_rotor] == 0) lcp->lcp_rotor = map; if (TAILQ_FIRST(&lp->lp_pages)->lcp_nfree == 0) { TAILQ_REMOVE(&lp->lp_pages, lcp, lcp_chain); TAILQ_INSERT_HEAD(&lp->lp_pages, lcp, lcp_chain); } mutex_exit(&lp->lp_lock); } /* * Process is exiting; tear down lwpctl state. This can only be safely * called by the last LWP in the process. */ void lwp_ctl_exit(void) { lcpage_t *lcp, *next; lcproc_t *lp; proc_t *p; lwp_t *l; l = curlwp; l->l_lwpctl = NULL; l->l_lcpage = NULL; p = l->l_proc; lp = p->p_lwpctl; KASSERT(lp != NULL); KASSERT(p->p_nlwps == 1); for (lcp = TAILQ_FIRST(&lp->lp_pages); lcp != NULL; lcp = next) { next = TAILQ_NEXT(lcp, lcp_chain); uvm_unmap(kernel_map, lcp->lcp_kaddr, lcp->lcp_kaddr + PAGE_SIZE); kmem_free(lcp, LWPCTL_LCPAGE_SZ); } if (lp->lp_uao != NULL) { uvm_unmap(&p->p_vmspace->vm_map, lp->lp_uva, lp->lp_uva + LWPCTL_UAREA_SZ); } mutex_destroy(&lp->lp_lock); kmem_free(lp, sizeof(*lp)); p->p_lwpctl = NULL; } /* * Return the current LWP's "preemption counter". Used to detect * preemption across operations that can tolerate preemption without * crashing, but which may generate incorrect results if preempted. */ uint64_t lwp_pctr(void) { return curlwp->l_ncsw; } /* * Set an LWP's private data pointer. */ int lwp_setprivate(struct lwp *l, void *ptr) { int error = 0; l->l_private = ptr; #ifdef __HAVE_CPU_LWP_SETPRIVATE error = cpu_lwp_setprivate(l, ptr); #endif return error; } /* * Perform any thread-related cleanup on LWP exit. * N.B. l->l_proc->p_lock must be HELD on entry but will * be released before returning! */ void lwp_thread_cleanup(struct lwp *l) { KASSERT(mutex_owned(l->l_proc->p_lock)); mutex_exit(l->l_proc->p_lock); /* * If the LWP has robust futexes, release them all * now. */ if (__predict_false(l->l_robust_head != 0)) { futex_release_all_lwp(l); } } #if defined(DDB) #include <machine/pcb.h> void lwp_whatis(uintptr_t addr, void (*pr)(const char *, ...)) { lwp_t *l; LIST_FOREACH(l, &alllwp, l_list) { uintptr_t stack = (uintptr_t)KSTACK_LOWEST_ADDR(l); if (addr < stack || stack + KSTACK_SIZE <= addr) { continue; } (*pr)("%p is %p+%zu, LWP %p's stack\n", (void *)addr, (void *)stack, (size_t)(addr - stack), l); } } #endif /* defined(DDB) */ |
| 766 755 52 189 15 189 189 188 189 189 18 75 122 189 7 185 1701 1699 1691 1667 1399 1653 1400 1400 967 966 135 959 668 932 887 61 17 1 108 1676 1675 251 1660 908 257 1660 1660 1 338 598 816 18 1387 1388 458 709 1385 65 64 20 45 45 6 64 64 63 59 63 63 11 1 55 63 837 838 64 64 824 824 823 74 816 1 532 701 579 158 427 45 45 45 134 134 134 133 134 134 135 135 257 774 60 60 60 154 154 122 161 161 160 154 153 154 23 24 21 823 823 822 1 8 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 197 198 199 200 201 202 203 204 205 206 207 208 209 210 211 212 213 214 215 216 217 218 219 220 221 222 223 224 225 226 227 228 229 230 231 232 233 234 235 236 237 238 239 240 241 242 243 244 245 246 247 248 249 250 251 252 253 254 255 256 257 258 259 260 261 262 263 264 265 266 267 268 269 270 271 272 273 274 275 276 277 278 279 280 281 282 283 284 285 286 287 288 289 290 291 292 293 294 295 296 297 298 299 300 301 302 303 304 305 306 307 308 309 310 311 312 313 314 315 316 317 318 319 320 321 322 323 324 325 326 327 328 329 330 331 332 333 334 335 336 337 338 339 340 341 342 343 344 345 346 347 348 349 350 351 352 353 354 355 356 357 358 359 360 361 362 363 364 365 366 367 368 369 370 371 372 373 374 375 376 377 378 379 380 381 382 383 384 385 386 387 388 389 390 391 392 393 394 395 396 397 398 399 400 401 402 403 404 405 406 407 408 409 410 411 412 413 414 415 416 417 418 419 420 421 422 423 424 425 426 427 428 429 430 431 432 433 434 435 436 437 438 439 440 441 442 443 444 445 446 447 448 449 450 451 452 453 454 455 456 457 458 459 460 461 462 463 464 465 466 467 468 469 470 471 472 473 474 475 476 477 478 479 480 481 482 483 484 485 486 487 488 489 490 491 492 493 494 495 496 497 498 499 500 501 502 503 504 505 506 507 508 509 510 511 512 513 514 515 516 517 518 519 520 521 522 523 524 525 526 527 528 529 530 531 532 533 534 535 536 537 538 539 540 541 542 543 544 545 546 547 548 549 550 551 552 553 554 555 556 557 558 559 560 561 562 563 564 565 566 567 568 569 570 571 572 573 574 575 576 577 578 579 580 581 582 583 584 585 586 587 588 589 590 591 592 593 594 595 596 597 598 599 600 601 602 603 604 605 606 607 608 609 610 611 612 613 614 615 616 617 618 619 620 621 622 623 624 625 626 627 628 629 630 631 632 633 634 635 636 637 638 639 640 641 642 643 644 645 646 647 648 649 650 651 652 653 654 655 656 657 658 659 660 661 662 663 664 665 666 667 668 669 670 671 672 673 674 675 676 677 678 679 680 681 682 683 684 685 686 687 688 689 690 691 692 693 694 695 696 697 698 699 700 701 702 703 704 705 706 707 708 709 710 711 712 713 714 715 716 717 718 719 720 721 722 723 724 725 726 727 728 729 730 731 732 733 734 735 736 737 738 739 740 741 742 743 744 745 746 747 748 749 750 751 752 753 754 755 756 757 758 759 760 761 762 763 764 765 766 767 768 769 770 771 772 773 774 775 776 777 778 779 780 781 782 783 784 785 786 787 788 789 790 791 792 793 794 795 796 797 798 799 800 801 802 803 804 805 806 807 808 809 810 811 812 813 814 815 816 817 818 819 820 821 822 823 824 825 826 827 828 829 830 831 832 833 834 835 836 837 838 839 840 841 842 843 844 845 846 847 848 849 850 851 852 853 854 855 856 857 858 859 860 861 862 863 864 865 866 867 868 869 870 871 872 873 874 875 876 877 878 879 880 881 882 883 884 885 886 887 888 889 890 891 892 893 894 895 896 897 898 899 900 901 902 903 904 905 906 907 908 909 910 911 912 913 914 915 916 917 918 919 920 921 922 923 924 925 926 927 928 929 930 931 932 933 934 935 936 937 938 939 940 941 942 943 944 945 946 947 948 949 950 951 952 953 954 955 956 957 958 959 960 961 962 963 964 965 966 967 968 969 970 971 972 973 974 975 976 977 978 979 980 981 982 983 984 985 986 987 988 989 990 991 992 993 994 995 996 997 998 999 1000 1001 1002 1003 1004 1005 1006 1007 1008 1009 1010 1011 1012 1013 1014 1015 1016 1017 1018 1019 1020 1021 1022 1023 1024 1025 1026 1027 1028 1029 1030 1031 1032 1033 1034 1035 1036 1037 1038 1039 1040 1041 1042 1043 1044 1045 1046 1047 1048 1049 1050 1051 1052 1053 1054 1055 1056 1057 1058 1059 1060 1061 1062 1063 1064 1065 1066 1067 1068 1069 1070 1071 1072 1073 1074 1075 1076 1077 1078 1079 1080 1081 1082 1083 1084 1085 1086 1087 1088 1089 1090 1091 1092 1093 1094 1095 1096 1097 1098 1099 1100 1101 1102 1103 1104 1105 1106 1107 1108 1109 1110 1111 1112 1113 1114 1115 1116 1117 1118 1119 1120 1121 1122 1123 1124 1125 1126 1127 1128 1129 1130 1131 1132 1133 1134 1135 1136 1137 1138 1139 1140 1141 1142 1143 1144 1145 1146 1147 1148 1149 1150 1151 1152 1153 1154 1155 1156 1157 1158 1159 1160 1161 1162 1163 1164 1165 1166 1167 1168 1169 1170 1171 1172 1173 1174 1175 1176 1177 1178 1179 1180 1181 1182 1183 1184 1185 1186 1187 1188 1189 1190 1191 1192 1193 1194 1195 1196 1197 1198 1199 1200 1201 1202 1203 1204 1205 1206 1207 1208 1209 1210 1211 1212 1213 1214 1215 1216 1217 1218 1219 1220 1221 1222 1223 1224 1225 1226 1227 1228 1229 1230 1231 1232 1233 1234 1235 1236 1237 1238 1239 1240 1241 1242 1243 1244 1245 1246 1247 1248 1249 1250 1251 1252 1253 1254 1255 1256 1257 1258 1259 1260 1261 1262 1263 1264 1265 1266 1267 1268 1269 1270 1271 1272 1273 1274 1275 1276 1277 1278 1279 1280 1281 1282 1283 1284 1285 1286 1287 1288 1289 1290 1291 1292 1293 1294 1295 1296 1297 1298 1299 1300 1301 1302 1303 1304 1305 1306 1307 1308 1309 1310 1311 1312 1313 1314 1315 1316 1317 1318 1319 1320 1321 1322 1323 1324 1325 1326 1327 1328 1329 1330 1331 1332 1333 1334 1335 1336 1337 1338 1339 1340 1341 1342 1343 1344 1345 1346 1347 1348 1349 1350 1351 1352 1353 1354 1355 1356 1357 1358 1359 1360 1361 1362 1363 1364 1365 1366 1367 1368 1369 1370 1371 1372 1373 1374 1375 1376 1377 1378 1379 1380 1381 1382 1383 1384 1385 1386 1387 1388 1389 1390 1391 1392 1393 1394 1395 1396 1397 1398 1399 1400 1401 1402 1403 1404 1405 1406 1407 1408 1409 1410 1411 1412 1413 1414 1415 1416 1417 1418 1419 1420 1421 1422 1423 1424 1425 1426 1427 1428 1429 1430 1431 1432 1433 1434 1435 1436 1437 1438 1439 1440 1441 1442 1443 1444 1445 1446 1447 1448 1449 1450 1451 1452 1453 1454 1455 1456 1457 1458 1459 1460 1461 1462 1463 1464 1465 1466 1467 1468 1469 1470 1471 1472 1473 1474 1475 1476 1477 1478 1479 1480 1481 1482 1483 1484 1485 1486 1487 1488 1489 1490 1491 1492 1493 1494 1495 1496 1497 1498 1499 1500 1501 1502 1503 1504 1505 1506 1507 1508 1509 1510 1511 1512 1513 1514 1515 1516 1517 1518 1519 1520 1521 1522 1523 1524 1525 1526 1527 1528 1529 1530 1531 1532 1533 1534 1535 1536 1537 1538 1539 1540 1541 1542 1543 1544 1545 | /* $NetBSD: vfs_cache.c,v 1.152 2021/11/01 21:28:03 andvar Exp $ */ /*- * Copyright (c) 2008, 2019, 2020 The NetBSD Foundation, Inc. * All rights reserved. * * This code is derived from software contributed to The NetBSD Foundation * by Andrew Doran. * * Redistribution and use in source and binary forms, with or without * modification, are permitted provided that the following conditions * are met: * 1. Redistributions of source code must retain the above copyright * notice, this list of conditions and the following disclaimer. * 2. Redistributions in binary form must reproduce the above copyright * notice, this list of conditions and the following disclaimer in the * documentation and/or other materials provided with the distribution. * * THIS SOFTWARE IS PROVIDED BY THE NETBSD FOUNDATION, INC. AND CONTRIBUTORS * ``AS IS'' AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED * TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR * PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE FOUNDATION OR CONTRIBUTORS * BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR * CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF * SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS * INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN * CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) * ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE * POSSIBILITY OF SUCH DAMAGE. */ /* * Copyright (c) 1989, 1993 * The Regents of the University of California. All rights reserved. * * Redistribution and use in source and binary forms, with or without * modification, are permitted provided that the following conditions * are met: * 1. Redistributions of source code must retain the above copyright * notice, this list of conditions and the following disclaimer. * 2. Redistributions in binary form must reproduce the above copyright * notice, this list of conditions and the following disclaimer in the * documentation and/or other materials provided with the distribution. * 3. Neither the name of the University nor the names of its contributors * may be used to endorse or promote products derived from this software * without specific prior written permission. * * THIS SOFTWARE IS PROVIDED BY THE REGENTS AND CONTRIBUTORS ``AS IS'' AND * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE * ARE DISCLAIMED. IN NO EVENT SHALL THE REGENTS OR CONTRIBUTORS BE LIABLE * FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL * DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS * OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) * HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT * LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY * OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF * SUCH DAMAGE. * * @(#)vfs_cache.c 8.3 (Berkeley) 8/22/94 */ /* * Name caching: * * Names found by directory scans are retained in a cache for future * reference. It is managed LRU, so frequently used names will hang * around. The cache is indexed by hash value obtained from the name. * * The name cache is the brainchild of Robert Elz and was introduced in * 4.3BSD. See "Using gprof to Tune the 4.2BSD Kernel", Marshall Kirk * McKusick, May 21 1984. * * Data structures: * * Most Unix namecaches very sensibly use a global hash table to index * names. The global hash table works well, but can cause concurrency * headaches for the kernel hacker. In the NetBSD 10.0 implementation * we are not sensible, and use a per-directory data structure to index * names, but the cache otherwise functions the same. * * The index is a red-black tree. There are no special concurrency * requirements placed on it, because it's per-directory and protected * by the namecache's per-directory locks. It should therefore not be * difficult to experiment with other types of index. * * Each cached name is stored in a struct namecache, along with a * pointer to the associated vnode (nc_vp). Names longer than a * maximum length of NCHNAMLEN are allocated with kmem_alloc(); they * occur infrequently, and names shorter than this are stored directly * in struct namecache. If it is a "negative" entry, (i.e. for a name * that is known NOT to exist) the vnode pointer will be NULL. * * For a directory with 3 cached names for 3 distinct vnodes, the * various vnodes and namecache structs would be connected like this * (the root is at the bottom of the diagram): * * ... * ^ * |- vi_nc_tree * | * +----o----+ +---------+ +---------+ * | VDIR | | VCHR | | VREG | * | vnode o-----+ | vnode o-----+ | vnode o------+ * +---------+ | +---------+ | +---------+ | * ^ | ^ | ^ | * |- nc_vp |- vi_nc_list |- nc_vp |- vi_nc_list |- nc_vp | * | | | | | | * +----o----+ | +----o----+ | +----o----+ | * +---onamecache|<----+ +---onamecache|<----+ +---onamecache|<-----+ * | +---------+ | +---------+ | +---------+ * | ^ | ^ | ^ * | | | | | | * | | +----------------------+ | | * |-nc_dvp | +-------------------------------------------------+ * | |/- vi_nc_tree | | * | | |- nc_dvp |- nc_dvp * | +----o----+ | | * +-->| VDIR |<----------+ | * | vnode |<------------------------------------+ * +---------+ * * START HERE * * Replacement: * * As the cache becomes full, old and unused entries are purged as new * entries are added. The synchronization overhead in maintaining a * strict ordering would be prohibitive, so the VM system's "clock" or * "second chance" page replacement algorithm is aped here. New * entries go to the tail of the active list. After they age out and * reach the head of the list, they are moved to the tail of the * inactive list. Any use of the deactivated cache entry reactivates * it, saving it from impending doom; if not reactivated, the entry * eventually reaches the head of the inactive list and is purged. * * Concurrency: * * From a performance perspective, cache_lookup(nameiop == LOOKUP) is * what really matters; insertion of new entries with cache_enter() is * comparatively infrequent, and overshadowed by the cost of expensive * file system metadata operations (which may involve disk I/O). We * therefore want to make everything simplest in the lookup path. * * struct namecache is mostly stable except for list and tree related * entries, changes to which don't affect the cached name or vnode. * For changes to name+vnode, entries are purged in preference to * modifying them. * * Read access to namecache entries is made via tree, list, or LRU * list. A lock corresponding to the direction of access should be * held. See definition of "struct namecache" in src/sys/namei.src, * and the definition of "struct vnode" for the particulars. * * Per-CPU statistics, and LRU list totals are read unlocked, since * an approximate value is OK. We maintain 32-bit sized per-CPU * counters and 64-bit global counters under the theory that 32-bit * sized counters are less likely to be hosed by nonatomic increment * (on 32-bit platforms). * * The lock order is: * * 1) vi->vi_nc_lock (tree or parent -> child direction, * used during forward lookup) * * 2) vi->vi_nc_listlock (list or child -> parent direction, * used during reverse lookup) * * 3) cache_lru_lock (LRU list direction, used during reclaim) * * 4) vp->v_interlock (what the cache entry points to) */ #include <sys/cdefs.h> __KERNEL_RCSID(0, "$NetBSD: vfs_cache.c,v 1.152 2021/11/01 21:28:03 andvar Exp $"); #define __NAMECACHE_PRIVATE #ifdef _KERNEL_OPT #include "opt_ddb.h" #include "opt_dtrace.h" #endif #include <sys/param.h> #include <sys/types.h> #include <sys/atomic.h> #include <sys/callout.h> #include <sys/cpu.h> #include <sys/errno.h> #include <sys/evcnt.h> #include <sys/hash.h> #include <sys/kernel.h> #include <sys/mount.h> #include <sys/mutex.h> #include <sys/namei.h> #include <sys/param.h> #include <sys/pool.h> #include <sys/sdt.h> #include <sys/sysctl.h> #include <sys/systm.h> #include <sys/time.h> #include <sys/vnode_impl.h> #include <miscfs/genfs/genfs.h> static void cache_activate(struct namecache *); static void cache_update_stats(void *); static int cache_compare_nodes(void *, const void *, const void *); static void cache_deactivate(void); static void cache_reclaim(void); static int cache_stat_sysctl(SYSCTLFN_ARGS); /* * Global pool cache. */ static pool_cache_t cache_pool __read_mostly; /* * LRU replacement. */ enum cache_lru_id { LRU_ACTIVE, LRU_INACTIVE, LRU_COUNT }; static struct { TAILQ_HEAD(, namecache) list[LRU_COUNT]; u_int count[LRU_COUNT]; } cache_lru __cacheline_aligned; static kmutex_t cache_lru_lock __cacheline_aligned; /* * Cache effectiveness statistics. nchstats holds system-wide total. */ struct nchstats nchstats; struct nchstats_percpu _NAMEI_CACHE_STATS(uint32_t); struct nchcpu { struct nchstats_percpu cur; struct nchstats_percpu last; }; static callout_t cache_stat_callout; static kmutex_t cache_stat_lock __cacheline_aligned; #define COUNT(f) do { \ lwp_t *l = curlwp; \ KPREEMPT_DISABLE(l); \ struct nchcpu *nchcpu = curcpu()->ci_data.cpu_nch; \ nchcpu->cur.f++; \ KPREEMPT_ENABLE(l); \ } while (/* CONSTCOND */ 0); #define UPDATE(nchcpu, f) do { \ uint32_t cur = atomic_load_relaxed(&nchcpu->cur.f); \ nchstats.f += (uint32_t)(cur - nchcpu->last.f); \ nchcpu->last.f = cur; \ } while (/* CONSTCOND */ 0) /* * Tunables. cache_maxlen replaces the historical doingcache: * set it zero to disable caching for debugging purposes. */ int cache_lru_maxdeact __read_mostly = 2; /* max # to deactivate */ int cache_lru_maxscan __read_mostly = 64; /* max # to scan/reclaim */ int cache_maxlen __read_mostly = USHRT_MAX; /* max name length to cache */ int cache_stat_interval __read_mostly = 300; /* in seconds */ /* * sysctl stuff. */ static struct sysctllog *cache_sysctllog; /* * This is a dummy name that cannot usually occur anywhere in the cache nor * file system. It's used when caching the root vnode of mounted file * systems. The name is attached to the directory that the file system is * mounted on. */ static const char cache_mp_name[] = ""; static const int cache_mp_nlen = sizeof(cache_mp_name) - 1; /* * Red-black tree stuff. */ static const rb_tree_ops_t cache_rbtree_ops = { .rbto_compare_nodes = cache_compare_nodes, .rbto_compare_key = cache_compare_nodes, .rbto_node_offset = offsetof(struct namecache, nc_tree), .rbto_context = NULL }; /* * dtrace probes. */ SDT_PROVIDER_DEFINE(vfs); SDT_PROBE_DEFINE1(vfs, namecache, invalidate, done, "struct vnode *"); SDT_PROBE_DEFINE1(vfs, namecache, purge, parents, "struct vnode *"); SDT_PROBE_DEFINE1(vfs, namecache, purge, children, "struct vnode *"); SDT_PROBE_DEFINE2(vfs, namecache, purge, name, "char *", "size_t"); SDT_PROBE_DEFINE1(vfs, namecache, purge, vfs, "struct mount *"); SDT_PROBE_DEFINE3(vfs, namecache, lookup, hit, "struct vnode *", "char *", "size_t"); SDT_PROBE_DEFINE3(vfs, namecache, lookup, miss, "struct vnode *", "char *", "size_t"); SDT_PROBE_DEFINE3(vfs, namecache, lookup, toolong, "struct vnode *", "char *", "size_t"); SDT_PROBE_DEFINE2(vfs, namecache, revlookup, success, "struct vnode *", "struct vnode *"); SDT_PROBE_DEFINE2(vfs, namecache, revlookup, fail, "struct vnode *", "int"); SDT_PROBE_DEFINE2(vfs, namecache, prune, done, "int", "int"); SDT_PROBE_DEFINE3(vfs, namecache, enter, toolong, "struct vnode *", "char *", "size_t"); SDT_PROBE_DEFINE3(vfs, namecache, enter, done, "struct vnode *", "char *", "size_t"); /* * rbtree: compare two nodes. */ static int cache_compare_nodes(void *context, const void *n1, const void *n2) { const struct namecache *nc1 = n1; const struct namecache *nc2 = n2; if (nc1->nc_key < nc2->nc_key) { return -1; } if (nc1->nc_key > nc2->nc_key) { return 1; } KASSERT(nc1->nc_nlen == nc2->nc_nlen); return memcmp(nc1->nc_name, nc2->nc_name, nc1->nc_nlen); } /* * Compute a key value for the given name. The name length is encoded in * the key value to try and improve uniqueness, and so that length doesn't * need to be compared separately for string comparisons. */ static inline uint64_t cache_key(const char *name, size_t nlen) { uint64_t key; KASSERT(nlen <= USHRT_MAX); key = hash32_buf(name, nlen, HASH32_STR_INIT); return (key << 32) | nlen; } /* * Remove an entry from the cache. vi_nc_lock must be held, and if dir2node * is true, then we're locking in the conventional direction and the list * lock will be acquired when removing the entry from the vnode list. */ static void cache_remove(struct namecache *ncp, const bool dir2node) { struct vnode *vp, *dvp = ncp->nc_dvp; vnode_impl_t *dvi = VNODE_TO_VIMPL(dvp); KASSERT(rw_write_held(&dvi->vi_nc_lock)); KASSERT(cache_key(ncp->nc_name, ncp->nc_nlen) == ncp->nc_key); KASSERT(rb_tree_find_node(&dvi->vi_nc_tree, ncp) == ncp); SDT_PROBE(vfs, namecache, invalidate, done, ncp, 0, 0, 0, 0); /* * Remove from the vnode's list. This excludes cache_revlookup(), * and then it's safe to remove from the LRU lists. */ if ((vp = ncp->nc_vp) != NULL) { vnode_impl_t *vi = VNODE_TO_VIMPL(vp); if (__predict_true(dir2node)) { rw_enter(&vi->vi_nc_listlock, RW_WRITER); TAILQ_REMOVE(&vi->vi_nc_list, ncp, nc_list); rw_exit(&vi->vi_nc_listlock); } else { TAILQ_REMOVE(&vi->vi_nc_list, ncp, nc_list); } } /* Remove from the directory's rbtree. */ rb_tree_remove_node(&dvi->vi_nc_tree, ncp); /* Remove from the LRU lists. */ mutex_enter(&cache_lru_lock); TAILQ_REMOVE(&cache_lru.list[ncp->nc_lrulist], ncp, nc_lru); cache_lru.count[ncp->nc_lrulist]--; mutex_exit(&cache_lru_lock); /* Finally, free it. */ if (ncp->nc_nlen > NCHNAMLEN) { size_t sz = offsetof(struct namecache, nc_name[ncp->nc_nlen]); kmem_free(ncp, sz); } else { pool_cache_put(cache_pool, ncp); } } /* * Find a single cache entry and return it. vi_nc_lock must be held. */ static struct namecache * __noinline cache_lookup_entry(struct vnode *dvp, const char *name, size_t namelen, uint64_t key) { vnode_impl_t *dvi = VNODE_TO_VIMPL(dvp); struct rb_node *node = dvi->vi_nc_tree.rbt_root; struct namecache *ncp; int lrulist, diff; KASSERT(rw_lock_held(&dvi->vi_nc_lock)); /* * Search the RB tree for the key. This is an inlined lookup * tailored for exactly what's needed here (64-bit key and so on) * that is quite a bit faster than using rb_tree_find_node(). * * For a matching key memcmp() needs to be called once to confirm * that the correct name has been found. Very rarely there will be * a key value collision and the search will continue. */ for (;;) { if (__predict_false(RB_SENTINEL_P(node))) { return NULL; } ncp = (struct namecache *)node; KASSERT((void *)&ncp->nc_tree == (void *)ncp); KASSERT(ncp->nc_dvp == dvp); if (ncp->nc_key == key) { KASSERT(ncp->nc_nlen == namelen); diff = memcmp(ncp->nc_name, name, namelen); if (__predict_true(diff == 0)) { break; } node = node->rb_nodes[diff < 0]; } else { node = node->rb_nodes[ncp->nc_key < key]; } } /* * If the entry is on the wrong LRU list, requeue it. This is an * unlocked check, but it will rarely be wrong and even then there * will be no harm caused. */ lrulist = atomic_load_relaxed(&ncp->nc_lrulist); if (__predict_false(lrulist != LRU_ACTIVE)) { cache_activate(ncp); } return ncp; } /* * Look for a the name in the cache. We don't do this * if the segment name is long, simply so the cache can avoid * holding long names (which would either waste space, or * add greatly to the complexity). * * Lookup is called with DVP pointing to the directory to search, * and CNP providing the name of the entry being sought: cn_nameptr * is the name, cn_namelen is its length, and cn_flags is the flags * word from the namei operation. * * DVP must be locked. * * There are three possible non-error return states: * 1. Nothing was found in the cache. Nothing is known about * the requested name. * 2. A negative entry was found in the cache, meaning that the * requested name definitely does not exist. * 3. A positive entry was found in the cache, meaning that the * requested name does exist and that we are providing the * vnode. * In these cases the results are: * 1. 0 returned; VN is set to NULL. * 2. 1 returned; VN is set to NULL. * 3. 1 returned; VN is set to the vnode found. * * The additional result argument ISWHT is set to zero, unless a * negative entry is found that was entered as a whiteout, in which * case ISWHT is set to one. * * The ISWHT_RET argument pointer may be null. In this case an * assertion is made that the whiteout flag is not set. File systems * that do not support whiteouts can/should do this. * * Filesystems that do support whiteouts should add ISWHITEOUT to * cnp->cn_flags if ISWHT comes back nonzero. * * When a vnode is returned, it is locked, as per the vnode lookup * locking protocol. * * There is no way for this function to fail, in the sense of * generating an error that requires aborting the namei operation. * * (Prior to October 2012, this function returned an integer status, * and a vnode, and mucked with the flags word in CNP for whiteouts. * The integer status was -1 for "nothing found", ENOENT for "a * negative entry found", 0 for "a positive entry found", and possibly * other errors, and the value of VN might or might not have been set * depending on what error occurred.) */ bool cache_lookup(struct vnode *dvp, const char *name, size_t namelen, uint32_t nameiop, uint32_t cnflags, int *iswht_ret, struct vnode **vn_ret) { vnode_impl_t *dvi = VNODE_TO_VIMPL(dvp); struct namecache *ncp; struct vnode *vp; uint64_t key; int error; bool hit; krw_t op; KASSERT(namelen != cache_mp_nlen || name == cache_mp_name); /* Establish default result values */ if (iswht_ret != NULL) { *iswht_ret = 0; } *vn_ret = NULL; if (__predict_false(namelen > cache_maxlen)) { SDT_PROBE(vfs, namecache, lookup, toolong, dvp, name, namelen, 0, 0); COUNT(ncs_long); return false; } /* Compute the key up front - don't need the lock. */ key = cache_key(name, namelen); /* Could the entry be purged below? */ if ((cnflags & ISLASTCN) != 0 && ((cnflags & MAKEENTRY) == 0 || nameiop == CREATE)) { op = RW_WRITER; } else { op = RW_READER; } /* Now look for the name. */ rw_enter(&dvi->vi_nc_lock, op); ncp = cache_lookup_entry(dvp, name, namelen, key); if (__predict_false(ncp == NULL)) { rw_exit(&dvi->vi_nc_lock); COUNT(ncs_miss); SDT_PROBE(vfs, namecache, lookup, miss, dvp, name, namelen, 0, 0); return false; } if (__predict_false((cnflags & MAKEENTRY) == 0)) { /* * Last component and we are renaming or deleting, * the cache entry is invalid, or otherwise don't * want cache entry to exist. */ KASSERT((cnflags & ISLASTCN) != 0); cache_remove(ncp, true); rw_exit(&dvi->vi_nc_lock); COUNT(ncs_badhits); return false; } if (ncp->nc_vp == NULL) { if (iswht_ret != NULL) { /* * Restore the ISWHITEOUT flag saved earlier. */ *iswht_ret = ncp->nc_whiteout; } else { KASSERT(!ncp->nc_whiteout); } if (nameiop == CREATE && (cnflags & ISLASTCN) != 0) { /* * Last component and we are preparing to create * the named object, so flush the negative cache * entry. */ COUNT(ncs_badhits); cache_remove(ncp, true); hit = false; } else { COUNT(ncs_neghits); SDT_PROBE(vfs, namecache, lookup, hit, dvp, name, namelen, 0, 0); /* found neg entry; vn is already null from above */ hit = true; } rw_exit(&dvi->vi_nc_lock); return hit; } vp = ncp->nc_vp; error = vcache_tryvget(vp); rw_exit(&dvi->vi_nc_lock); if (error) { KASSERT(error == EBUSY); /* * This vnode is being cleaned out. * XXX badhits? */ COUNT(ncs_falsehits); return false; } COUNT(ncs_goodhits); SDT_PROBE(vfs, namecache, lookup, hit, dvp, name, namelen, 0, 0); /* found it */ *vn_ret = vp; return true; } /* * Version of the above without the nameiop argument, for NFS. */ bool cache_lookup_raw(struct vnode *dvp, const char *name, size_t namelen, uint32_t cnflags, int *iswht_ret, struct vnode **vn_ret) { return cache_lookup(dvp, name, namelen, LOOKUP, cnflags | MAKEENTRY, iswht_ret, vn_ret); } /* * Used by namei() to walk down a path, component by component by looking up * names in the cache. The node locks are chained along the way: a parent's * lock is not dropped until the child's is acquired. */ bool cache_lookup_linked(struct vnode *dvp, const char *name, size_t namelen, struct vnode **vn_ret, krwlock_t **plock, kauth_cred_t cred) { vnode_impl_t *dvi = VNODE_TO_VIMPL(dvp); struct namecache *ncp; krwlock_t *oldlock, *newlock; uint64_t key; int error; KASSERT(namelen != cache_mp_nlen || name == cache_mp_name); /* If disabled, or file system doesn't support this, bail out. */ if (__predict_false((dvp->v_mount->mnt_iflag & IMNT_NCLOOKUP) == 0)) { return false; } if (__predict_false(namelen > cache_maxlen)) { COUNT(ncs_long); return false; } /* Compute the key up front - don't need the lock. */ key = cache_key(name, namelen); /* * Acquire the directory lock. Once we have that, we can drop the * previous one (if any). * * The two lock holds mean that the directory can't go away while * here: the directory must be purged with cache_purge() before * being freed, and both parent & child's vi_nc_lock must be taken * before that point is passed. * * However if there's no previous lock, like at the root of the * chain, then "dvp" must be referenced to prevent dvp going away * before we get its lock. * * Note that the two locks can be the same if looking up a dot, for * example: /usr/bin/. If looking up the parent (..) we can't wait * on the lock as child -> parent is the wrong direction. */ if (*plock != &dvi->vi_nc_lock) { oldlock = *plock; newlock = &dvi->vi_nc_lock; if (!rw_tryenter(&dvi->vi_nc_lock, RW_READER)) { return false; } } else { oldlock = NULL; newlock = NULL; if (*plock == NULL) { KASSERT(vrefcnt(dvp) > 0); } } /* * First up check if the user is allowed to look up files in this * directory. */ if (cred != FSCRED) { if (dvi->vi_nc_mode == VNOVAL) { if (newlock != NULL) { rw_exit(newlock); } return false; } KASSERT(dvi->vi_nc_uid != VNOVAL && dvi->vi_nc_gid != VNOVAL); error = kauth_authorize_vnode(cred, KAUTH_ACCESS_ACTION(VEXEC, dvp->v_type, dvi->vi_nc_mode & ALLPERMS), dvp, NULL, genfs_can_access(dvp, cred, dvi->vi_nc_uid, dvi->vi_nc_gid, dvi->vi_nc_mode & ALLPERMS, NULL, VEXEC)); if (error != 0) { if (newlock != NULL) { rw_exit(newlock); } COUNT(ncs_denied); return false; } } /* * Now look for a matching cache entry. */ ncp = cache_lookup_entry(dvp, name, namelen, key); if (__predict_false(ncp == NULL)) { if (newlock != NULL) { rw_exit(newlock); } COUNT(ncs_miss); SDT_PROBE(vfs, namecache, lookup, miss, dvp, name, namelen, 0, 0); return false; } if (ncp->nc_vp == NULL) { /* found negative entry; vn is already null from above */ KASSERT(namelen != cache_mp_nlen && name != cache_mp_name); COUNT(ncs_neghits); } else { COUNT(ncs_goodhits); /* XXX can be "badhits" */ } SDT_PROBE(vfs, namecache, lookup, hit, dvp, name, namelen, 0, 0); /* * Return with the directory lock still held. It will either be * returned to us with another call to cache_lookup_linked() when * looking up the next component, or the caller will release it * manually when finished. */ if (oldlock) { rw_exit(oldlock); } if (newlock) { *plock = newlock; } *vn_ret = ncp->nc_vp; return true; } /* * Scan cache looking for name of directory entry pointing at vp. * Will not search for "." or "..". * * If the lookup succeeds the vnode is referenced and stored in dvpp. * * If bufp is non-NULL, also place the name in the buffer which starts * at bufp, immediately before *bpp, and move bpp backwards to point * at the start of it. (Yes, this is a little baroque, but it's done * this way to cater to the whims of getcwd). * * Returns 0 on success, -1 on cache miss, positive errno on failure. */ int cache_revlookup(struct vnode *vp, struct vnode **dvpp, char **bpp, char *bufp, bool checkaccess, accmode_t accmode) { vnode_impl_t *vi = VNODE_TO_VIMPL(vp); struct namecache *ncp; struct vnode *dvp; int error, nlen, lrulist; char *bp; KASSERT(vp != NULL); if (cache_maxlen == 0) goto out; rw_enter(&vi->vi_nc_listlock, RW_READER); if (checkaccess) { /* * Check if the user is allowed to see. NOTE: this is * checking for access on the "wrong" directory. getcwd() * wants to see that there is access on every component * along the way, not that there is access to any individual * component. Don't use this to check you can look in vp. * * I don't like it, I didn't come up with it, don't blame me! */ if (vi->vi_nc_mode == VNOVAL) { rw_exit(&vi->vi_nc_listlock); return -1; } KASSERT(vi->vi_nc_uid != VNOVAL && vi->vi_nc_gid != VNOVAL); error = kauth_authorize_vnode(kauth_cred_get(), KAUTH_ACCESS_ACTION(VEXEC, vp->v_type, vi->vi_nc_mode & ALLPERMS), vp, NULL, genfs_can_access(vp, curlwp->l_cred, vi->vi_nc_uid, vi->vi_nc_gid, vi->vi_nc_mode & ALLPERMS, NULL, accmode)); if (error != 0) { rw_exit(&vi->vi_nc_listlock); COUNT(ncs_denied); return EACCES; } } TAILQ_FOREACH(ncp, &vi->vi_nc_list, nc_list) { KASSERT(ncp->nc_vp == vp); KASSERT(ncp->nc_dvp != NULL); nlen = ncp->nc_nlen; /* * Ignore mountpoint entries. */ if (ncp->nc_nlen == cache_mp_nlen) { continue; } /* * The queue is partially sorted. Once we hit dots, nothing * else remains but dots and dotdots, so bail out. */ if (ncp->nc_name[0] == '.') { if (nlen == 1 || (nlen == 2 && ncp->nc_name[1] == '.')) { break; } } /* * Record a hit on the entry. This is an unlocked read but * even if wrong it doesn't matter too much. */ lrulist = atomic_load_relaxed(&ncp->nc_lrulist); if (lrulist != LRU_ACTIVE) { cache_activate(ncp); } if (bufp) { bp = *bpp; bp -= nlen; if (bp <= bufp) { *dvpp = NULL; rw_exit(&vi->vi_nc_listlock); SDT_PROBE(vfs, namecache, revlookup, fail, vp, ERANGE, 0, 0, 0); return (ERANGE); } memcpy(bp, ncp->nc_name, nlen); *bpp = bp; } dvp = ncp->nc_dvp; error = vcache_tryvget(dvp); rw_exit(&vi->vi_nc_listlock); if (error) { KASSERT(error == EBUSY); if (bufp) (*bpp) += nlen; *dvpp = NULL; SDT_PROBE(vfs, namecache, revlookup, fail, vp, error, 0, 0, 0); return -1; } *dvpp = dvp; SDT_PROBE(vfs, namecache, revlookup, success, vp, dvp, 0, 0, 0); COUNT(ncs_revhits); return (0); } rw_exit(&vi->vi_nc_listlock); COUNT(ncs_revmiss); out: *dvpp = NULL; return (-1); } /* * Add an entry to the cache. */ void cache_enter(struct vnode *dvp, struct vnode *vp, const char *name, size_t namelen, uint32_t cnflags) { vnode_impl_t *dvi = VNODE_TO_VIMPL(dvp); struct namecache *ncp, *oncp; int total; KASSERT(namelen != cache_mp_nlen || name == cache_mp_name); /* First, check whether we can/should add a cache entry. */ if ((cnflags & MAKEENTRY) == 0 || __predict_false(namelen > cache_maxlen)) { SDT_PROBE(vfs, namecache, enter, toolong, vp, name, namelen, 0, 0); return; } SDT_PROBE(vfs, namecache, enter, done, vp, name, namelen, 0, 0); /* * Reclaim some entries if over budget. This is an unlocked check, * but it doesn't matter. Just need to catch up with things * eventually: it doesn't matter if we go over temporarily. */ total = atomic_load_relaxed(&cache_lru.count[LRU_ACTIVE]); total += atomic_load_relaxed(&cache_lru.count[LRU_INACTIVE]); if (__predict_false(total > desiredvnodes)) { cache_reclaim(); } /* Now allocate a fresh entry. */ if (__predict_true(namelen <= NCHNAMLEN)) { ncp = pool_cache_get(cache_pool, PR_WAITOK); } else { size_t sz = offsetof(struct namecache, nc_name[namelen]); ncp = kmem_alloc(sz, KM_SLEEP); } /* * Fill in cache info. For negative hits, save the ISWHITEOUT flag * so we can restore it later when the cache entry is used again. */ ncp->nc_vp = vp; ncp->nc_dvp = dvp; ncp->nc_key = cache_key(name, namelen); ncp->nc_nlen = namelen; ncp->nc_whiteout = ((cnflags & ISWHITEOUT) != 0); memcpy(ncp->nc_name, name, namelen); /* * Insert to the directory. Concurrent lookups may race for a cache * entry. If there's a entry there already, purge it. */ rw_enter(&dvi->vi_nc_lock, RW_WRITER); oncp = rb_tree_insert_node(&dvi->vi_nc_tree, ncp); if (oncp != ncp) { KASSERT(oncp->nc_key == ncp->nc_key); KASSERT(oncp->nc_nlen == ncp->nc_nlen); KASSERT(memcmp(oncp->nc_name, name, namelen) == 0); cache_remove(oncp, true); oncp = rb_tree_insert_node(&dvi->vi_nc_tree, ncp); KASSERT(oncp == ncp); } /* * With the directory lock still held, insert to the tail of the * ACTIVE LRU list (new) and take the opportunity to incrementally * balance the lists. */ mutex_enter(&cache_lru_lock); ncp->nc_lrulist = LRU_ACTIVE; cache_lru.count[LRU_ACTIVE]++; TAILQ_INSERT_TAIL(&cache_lru.list[LRU_ACTIVE], ncp, nc_lru); cache_deactivate(); mutex_exit(&cache_lru_lock); /* * Finally, insert to the vnode and unlock. With everything set up * it's safe to let cache_revlookup() see the entry. Partially sort * the per-vnode list: dots go to back so cache_revlookup() doesn't * have to consider them. */ if (vp != NULL) { vnode_impl_t *vi = VNODE_TO_VIMPL(vp); rw_enter(&vi->vi_nc_listlock, RW_WRITER); if ((namelen == 1 && name[0] == '.') || (namelen == 2 && name[0] == '.' && name[1] == '.')) { TAILQ_INSERT_TAIL(&vi->vi_nc_list, ncp, nc_list); } else { TAILQ_INSERT_HEAD(&vi->vi_nc_list, ncp, nc_list); } rw_exit(&vi->vi_nc_listlock); } rw_exit(&dvi->vi_nc_lock); } /* * Set identity info in cache for a vnode. We only care about directories * so ignore other updates. The cached info may be marked invalid if the * inode has an ACL. */ void cache_enter_id(struct vnode *vp, mode_t mode, uid_t uid, gid_t gid, bool valid) { vnode_impl_t *vi = VNODE_TO_VIMPL(vp); if (vp->v_type == VDIR) { /* Grab both locks, for forward & reverse lookup. */ rw_enter(&vi->vi_nc_lock, RW_WRITER); rw_enter(&vi->vi_nc_listlock, RW_WRITER); if (valid) { vi->vi_nc_mode = mode; vi->vi_nc_uid = uid; vi->vi_nc_gid = gid; } else { vi->vi_nc_mode = VNOVAL; vi->vi_nc_uid = VNOVAL; vi->vi_nc_gid = VNOVAL; } rw_exit(&vi->vi_nc_listlock); rw_exit(&vi->vi_nc_lock); } } /* * Return true if we have identity for the given vnode, and use as an * opportunity to confirm that everything squares up. * * Because of shared code, some file systems could provide partial * information, missing some updates, so check the mount flag too. */ bool cache_have_id(struct vnode *vp) { if (vp->v_type == VDIR && (vp->v_mount->mnt_iflag & IMNT_NCLOOKUP) != 0 && atomic_load_relaxed(&VNODE_TO_VIMPL(vp)->vi_nc_mode) != VNOVAL) { return true; } else { return false; } } /* * Enter a mount point. cvp is the covered vnode, and rvp is the root of * the mounted file system. */ void cache_enter_mount(struct vnode *cvp, struct vnode *rvp) { KASSERT(vrefcnt(cvp) > 0); KASSERT(vrefcnt(rvp) > 0); KASSERT(cvp->v_type == VDIR); KASSERT((rvp->v_vflag & VV_ROOT) != 0); if (rvp->v_type == VDIR) { cache_enter(cvp, rvp, cache_mp_name, cache_mp_nlen, MAKEENTRY); } } /* * Look up a cached mount point. Used in the strongly locked path. */ bool cache_lookup_mount(struct vnode *dvp, struct vnode **vn_ret) { bool ret; ret = cache_lookup(dvp, cache_mp_name, cache_mp_nlen, LOOKUP, MAKEENTRY, NULL, vn_ret); KASSERT((*vn_ret != NULL) == ret); return ret; } /* * Try to cross a mount point. For use with cache_lookup_linked(). */ bool cache_cross_mount(struct vnode **dvp, krwlock_t **plock) { return cache_lookup_linked(*dvp, cache_mp_name, cache_mp_nlen, dvp, plock, FSCRED); } /* * Name cache initialization, from vfs_init() when the system is booting. */ void nchinit(void) { cache_pool = pool_cache_init(sizeof(struct namecache), coherency_unit, 0, 0, "namecache", NULL, IPL_NONE, NULL, NULL, NULL); KASSERT(cache_pool != NULL); mutex_init(&cache_lru_lock, MUTEX_DEFAULT, IPL_NONE); TAILQ_INIT(&cache_lru.list[LRU_ACTIVE]); TAILQ_INIT(&cache_lru.list[LRU_INACTIVE]); mutex_init(&cache_stat_lock, MUTEX_DEFAULT, IPL_NONE); callout_init(&cache_stat_callout, CALLOUT_MPSAFE); callout_setfunc(&cache_stat_callout, cache_update_stats, NULL); callout_schedule(&cache_stat_callout, cache_stat_interval * hz); KASSERT(cache_sysctllog == NULL); sysctl_createv(&cache_sysctllog, 0, NULL, NULL, CTLFLAG_PERMANENT, CTLTYPE_STRUCT, "namecache_stats", SYSCTL_DESCR("namecache statistics"), cache_stat_sysctl, 0, NULL, 0, CTL_VFS, CTL_CREATE, CTL_EOL); } /* * Called once for each CPU in the system as attached. */ void cache_cpu_init(struct cpu_info *ci) { void *p; size_t sz; sz = roundup2(sizeof(struct nchcpu), coherency_unit) + coherency_unit; p = kmem_zalloc(sz, KM_SLEEP); ci->ci_data.cpu_nch = (void *)roundup2((uintptr_t)p, coherency_unit); } /* * A vnode is being allocated: set up cache structures. */ void cache_vnode_init(struct vnode *vp) { vnode_impl_t *vi = VNODE_TO_VIMPL(vp); rw_init(&vi->vi_nc_lock); rw_init(&vi->vi_nc_listlock); rb_tree_init(&vi->vi_nc_tree, &cache_rbtree_ops); TAILQ_INIT(&vi->vi_nc_list); vi->vi_nc_mode = VNOVAL; vi->vi_nc_uid = VNOVAL; vi->vi_nc_gid = VNOVAL; } /* * A vnode is being freed: finish cache structures. */ void cache_vnode_fini(struct vnode *vp) { vnode_impl_t *vi = VNODE_TO_VIMPL(vp); KASSERT(RB_TREE_MIN(&vi->vi_nc_tree) == NULL); KASSERT(TAILQ_EMPTY(&vi->vi_nc_list)); rw_destroy(&vi->vi_nc_lock); rw_destroy(&vi->vi_nc_listlock); } /* * Helper for cache_purge1(): purge cache entries for the given vnode from * all directories that the vnode is cached in. */ static void cache_purge_parents(struct vnode *vp) { vnode_impl_t *dvi, *vi = VNODE_TO_VIMPL(vp); struct vnode *dvp, *blocked; struct namecache *ncp; SDT_PROBE(vfs, namecache, purge, parents, vp, 0, 0, 0, 0); blocked = NULL; rw_enter(&vi->vi_nc_listlock, RW_WRITER); while ((ncp = TAILQ_FIRST(&vi->vi_nc_list)) != NULL) { /* * Locking in the wrong direction. Try for a hold on the * directory node's lock, and if we get it then all good, * nuke the entry and move on to the next. */ dvp = ncp->nc_dvp; dvi = VNODE_TO_VIMPL(dvp); if (rw_tryenter(&dvi->vi_nc_lock, RW_WRITER)) { cache_remove(ncp, false); rw_exit(&dvi->vi_nc_lock); blocked = NULL; continue; } /* * We can't wait on the directory node's lock with our list * lock held or the system could deadlock. * * Take a hold on the directory vnode to prevent it from * being freed (taking the vnode & lock with it). Then * wait for the lock to become available with no other locks * held, and retry. * * If this happens twice in a row, give the other side a * breather; we can do nothing until it lets go. */ vhold(dvp); rw_exit(&vi->vi_nc_listlock); rw_enter(&dvi->vi_nc_lock, RW_WRITER); /* Do nothing. */ rw_exit(&dvi->vi_nc_lock); holdrele(dvp); if (blocked == dvp) { kpause("ncpurge", false, 1, NULL); } rw_enter(&vi->vi_nc_listlock, RW_WRITER); blocked = dvp; } rw_exit(&vi->vi_nc_listlock); } /* * Helper for cache_purge1(): purge all cache entries hanging off the given * directory vnode. */ static void cache_purge_children(struct vnode *dvp) { vnode_impl_t *dvi = VNODE_TO_VIMPL(dvp); struct namecache *ncp; SDT_PROBE(vfs, namecache, purge, children, dvp, 0, 0, 0, 0); rw_enter(&dvi->vi_nc_lock, RW_WRITER); while ((ncp = RB_TREE_MIN(&dvi->vi_nc_tree)) != NULL) { cache_remove(ncp, true); } rw_exit(&dvi->vi_nc_lock); } /* * Helper for cache_purge1(): purge cache entry from the given vnode, * finding it by name. */ static void cache_purge_name(struct vnode *dvp, const char *name, size_t namelen) { vnode_impl_t *dvi = VNODE_TO_VIMPL(dvp); struct namecache *ncp; uint64_t key; SDT_PROBE(vfs, namecache, purge, name, name, namelen, 0, 0, 0); key = cache_key(name, namelen); rw_enter(&dvi->vi_nc_lock, RW_WRITER); ncp = cache_lookup_entry(dvp, name, namelen, key); if (ncp) { cache_remove(ncp, true); } rw_exit(&dvi->vi_nc_lock); } /* * Cache flush, a particular vnode; called when a vnode is renamed to * hide entries that would now be invalid. */ void cache_purge1(struct vnode *vp, const char *name, size_t namelen, int flags) { if (flags & PURGE_PARENTS) { cache_purge_parents(vp); } if (flags & PURGE_CHILDREN) { cache_purge_children(vp); } if (name != NULL) { cache_purge_name(vp, name, namelen); } } /* * vnode filter for cache_purgevfs(). */ static bool cache_vdir_filter(void *cookie, vnode_t *vp) { return vp->v_type == VDIR; } /* * Cache flush, a whole filesystem; called when filesys is umounted to * remove entries that would now be invalid. */ void cache_purgevfs(struct mount *mp) { struct vnode_iterator *iter; vnode_t *dvp; vfs_vnode_iterator_init(mp, &iter); for (;;) { dvp = vfs_vnode_iterator_next(iter, cache_vdir_filter, NULL); if (dvp == NULL) { break; } cache_purge_children(dvp); vrele(dvp); } vfs_vnode_iterator_destroy(iter); } /* * Re-queue an entry onto the tail of the active LRU list, after it has * scored a hit. */ static void cache_activate(struct namecache *ncp) { mutex_enter(&cache_lru_lock); TAILQ_REMOVE(&cache_lru.list[ncp->nc_lrulist], ncp, nc_lru); TAILQ_INSERT_TAIL(&cache_lru.list[LRU_ACTIVE], ncp, nc_lru); cache_lru.count[ncp->nc_lrulist]--; cache_lru.count[LRU_ACTIVE]++; ncp->nc_lrulist = LRU_ACTIVE; mutex_exit(&cache_lru_lock); } /* * Try to balance the LRU lists. Pick some victim entries, and re-queue * them from the head of the active list to the tail of the inactive list. */ static void cache_deactivate(void) { struct namecache *ncp; int total, i; KASSERT(mutex_owned(&cache_lru_lock)); /* If we're nowhere near budget yet, don't bother. */ total = cache_lru.count[LRU_ACTIVE] + cache_lru.count[LRU_INACTIVE]; if (total < (desiredvnodes >> 1)) { return; } /* * Aim for a 1:1 ratio of active to inactive. This is to allow each * potential victim a reasonable amount of time to cycle through the * inactive list in order to score a hit and be reactivated, while * trying not to cause reactivations too frequently. */ if (cache_lru.count[LRU_ACTIVE] < cache_lru.count[LRU_INACTIVE]) { return; } /* Move only a few at a time; will catch up eventually. */ for (i = 0; i < cache_lru_maxdeact; i++) { ncp = TAILQ_FIRST(&cache_lru.list[LRU_ACTIVE]); if (ncp == NULL) { break; } KASSERT(ncp->nc_lrulist == LRU_ACTIVE); ncp->nc_lrulist = LRU_INACTIVE; TAILQ_REMOVE(&cache_lru.list[LRU_ACTIVE], ncp, nc_lru); TAILQ_INSERT_TAIL(&cache_lru.list[LRU_INACTIVE], ncp, nc_lru); cache_lru.count[LRU_ACTIVE]--; cache_lru.count[LRU_INACTIVE]++; } } /* * Free some entries from the cache, when we have gone over budget. * * We don't want to cause too much work for any individual caller, and it * doesn't matter if we temporarily go over budget. This is also "just a * cache" so it's not a big deal if we screw up and throw out something we * shouldn't. So we take a relaxed attitude to this process to reduce its * impact. */ static void cache_reclaim(void) { struct namecache *ncp; vnode_impl_t *dvi; int toscan; /* * Scan up to a preset maximum number of entries, but no more than * 0.8% of the total at once (to allow for very small systems). * * On bigger systems, do a larger chunk of work to reduce the number * of times that cache_lru_lock is held for any length of time. */ mutex_enter(&cache_lru_lock); toscan = MIN(cache_lru_maxscan, desiredvnodes >> 7); toscan = MAX(toscan, 1); SDT_PROBE(vfs, namecache, prune, done, cache_lru.count[LRU_ACTIVE] + cache_lru.count[LRU_INACTIVE], toscan, 0, 0, 0); while (toscan-- != 0) { /* First try to balance the lists. */ cache_deactivate(); /* Now look for a victim on head of inactive list (old). */ ncp = TAILQ_FIRST(&cache_lru.list[LRU_INACTIVE]); if (ncp == NULL) { break; } dvi = VNODE_TO_VIMPL(ncp->nc_dvp); KASSERT(ncp->nc_lrulist == LRU_INACTIVE); KASSERT(dvi != NULL); /* * Locking in the wrong direction. If we can't get the * lock, the directory is actively busy, and it could also * cause problems for the next guy in here, so send the * entry to the back of the list. */ if (!rw_tryenter(&dvi->vi_nc_lock, RW_WRITER)) { TAILQ_REMOVE(&cache_lru.list[LRU_INACTIVE], ncp, nc_lru); TAILQ_INSERT_TAIL(&cache_lru.list[LRU_INACTIVE], ncp, nc_lru); continue; } /* * Now have the victim entry locked. Drop the LRU list * lock, purge the entry, and start over. The hold on * vi_nc_lock will prevent the vnode from vanishing until * finished (cache_purge() will be called on dvp before it * disappears, and that will wait on vi_nc_lock). */ mutex_exit(&cache_lru_lock); cache_remove(ncp, true); rw_exit(&dvi->vi_nc_lock); mutex_enter(&cache_lru_lock); } mutex_exit(&cache_lru_lock); } /* * For file system code: count a lookup that required a full re-scan of * directory metadata. */ void namecache_count_pass2(void) { COUNT(ncs_pass2); } /* * For file system code: count a lookup that scored a hit in the directory * metadata near the location of the last lookup. */ void namecache_count_2passes(void) { COUNT(ncs_2passes); } /* * Sum the stats from all CPUs into nchstats. This need |