1// SPDX-License-Identifier: GPL-2.0-or-later 2/* 3 * Budget Fair Queueing (BFQ) I/O scheduler. 4 * 5 * Based on ideas and code from CFQ: 6 * Copyright (C) 2003 Jens Axboe <axboe@kernel.dk> 7 * 8 * Copyright (C) 2008 Fabio Checconi <fabio@gandalf.sssup.it> 9 * Paolo Valente <paolo.valente@unimore.it> 10 * 11 * Copyright (C) 2010 Paolo Valente <paolo.valente@unimore.it> 12 * Arianna Avanzini <avanzini@google.com> 13 * 14 * Copyright (C) 2017 Paolo Valente <paolo.valente@linaro.org> 15 * 16 * BFQ is a proportional-share I/O scheduler, with some extra 17 * low-latency capabilities. BFQ also supports full hierarchical 18 * scheduling through cgroups. Next paragraphs provide an introduction 19 * on BFQ inner workings. Details on BFQ benefits, usage and 20 * limitations can be found in Documentation/block/bfq-iosched.rst. 21 * 22 * BFQ is a proportional-share storage-I/O scheduling algorithm based 23 * on the slice-by-slice service scheme of CFQ. But BFQ assigns 24 * budgets, measured in number of sectors, to processes instead of 25 * time slices. The device is not granted to the in-service process 26 * for a given time slice, but until it has exhausted its assigned 27 * budget. This change from the time to the service domain enables BFQ 28 * to distribute the device throughput among processes as desired, 29 * without any distortion due to throughput fluctuations, or to device 30 * internal queueing. BFQ uses an ad hoc internal scheduler, called 31 * B-WF2Q+, to schedule processes according to their budgets. More 32 * precisely, BFQ schedules queues associated with processes. Each 33 * process/queue is assigned a user-configurable weight, and B-WF2Q+ 34 * guarantees that each queue receives a fraction of the throughput 35 * proportional to its weight. Thanks to the accurate policy of 36 * B-WF2Q+, BFQ can afford to assign high budgets to I/O-bound 37 * processes issuing sequential requests (to boost the throughput), 38 * and yet guarantee a low latency to interactive and soft real-time 39 * applications. 40 * 41 * In particular, to provide these low-latency guarantees, BFQ 42 * explicitly privileges the I/O of two classes of time-sensitive 43 * applications: interactive and soft real-time. In more detail, BFQ 44 * behaves this way if the low_latency parameter is set (default 45 * configuration). This feature enables BFQ to provide applications in 46 * these classes with a very low latency. 47 * 48 * To implement this feature, BFQ constantly tries to detect whether 49 * the I/O requests in a bfq_queue come from an interactive or a soft 50 * real-time application. For brevity, in these cases, the queue is 51 * said to be interactive or soft real-time. In both cases, BFQ 52 * privileges the service of the queue, over that of non-interactive 53 * and non-soft-real-time queues. This privileging is performed, 54 * mainly, by raising the weight of the queue. So, for brevity, we 55 * call just weight-raising periods the time periods during which a 56 * queue is privileged, because deemed interactive or soft real-time. 57 * 58 * The detection of soft real-time queues/applications is described in 59 * detail in the comments on the function 60 * bfq_bfqq_softrt_next_start. On the other hand, the detection of an 61 * interactive queue works as follows: a queue is deemed interactive 62 * if it is constantly non empty only for a limited time interval, 63 * after which it does become empty. The queue may be deemed 64 * interactive again (for a limited time), if it restarts being 65 * constantly non empty, provided that this happens only after the 66 * queue has remained empty for a given minimum idle time. 67 * 68 * By default, BFQ computes automatically the above maximum time 69 * interval, i.e., the time interval after which a constantly 70 * non-empty queue stops being deemed interactive. Since a queue is 71 * weight-raised while it is deemed interactive, this maximum time 72 * interval happens to coincide with the (maximum) duration of the 73 * weight-raising for interactive queues. 74 * 75 * Finally, BFQ also features additional heuristics for 76 * preserving both a low latency and a high throughput on NCQ-capable, 77 * rotational or flash-based devices, and to get the job done quickly 78 * for applications consisting in many I/O-bound processes. 79 * 80 * NOTE: if the main or only goal, with a given device, is to achieve 81 * the maximum-possible throughput at all times, then do switch off 82 * all low-latency heuristics for that device, by setting low_latency 83 * to 0. 84 * 85 * BFQ is described in [1], where also a reference to the initial, 86 * more theoretical paper on BFQ can be found. The interested reader 87 * can find in the latter paper full details on the main algorithm, as 88 * well as formulas of the guarantees and formal proofs of all the 89 * properties. With respect to the version of BFQ presented in these 90 * papers, this implementation adds a few more heuristics, such as the 91 * ones that guarantee a low latency to interactive and soft real-time 92 * applications, and a hierarchical extension based on H-WF2Q+. 93 * 94 * B-WF2Q+ is based on WF2Q+, which is described in [2], together with 95 * H-WF2Q+, while the augmented tree used here to implement B-WF2Q+ 96 * with O(log N) complexity derives from the one introduced with EEVDF 97 * in [3]. 98 * 99 * [1] P. Valente, A. Avanzini, "Evolution of the BFQ Storage I/O 100 * Scheduler", Proceedings of the First Workshop on Mobile System 101 * Technologies (MST-2015), May 2015. 102 * http://algogroup.unimore.it/people/paolo/disk_sched/mst-2015.pdf 103 * 104 * [2] Jon C.R. Bennett and H. Zhang, "Hierarchical Packet Fair Queueing 105 * Algorithms", IEEE/ACM Transactions on Networking, 5(5):675-689, 106 * Oct 1997. 107 * 108 * http://www.cs.cmu.edu/~hzhang/papers/TON-97-Oct.ps.gz 109 * 110 * [3] I. Stoica and H. Abdel-Wahab, "Earliest Eligible Virtual Deadline 111 * First: A Flexible and Accurate Mechanism for Proportional Share 112 * Resource Allocation", technical report. 113 * 114 * http://www.cs.berkeley.edu/~istoica/papers/eevdf-tr-95.pdf 115 */ 116#include <linux/module.h> 117#include <linux/slab.h> 118#include <linux/blkdev.h> 119#include <linux/cgroup.h> 120#include <linux/elevator.h> 121#include <linux/ktime.h> 122#include <linux/rbtree.h> 123#include <linux/ioprio.h> 124#include <linux/sbitmap.h> 125#include <linux/delay.h> 126#include <linux/backing-dev.h> 127 128#include <trace/events/block.h> 129 130#include "blk.h" 131#include "blk-mq.h" 132#include "blk-mq-tag.h" 133#include "blk-mq-sched.h" 134#include "bfq-iosched.h" 135#include "blk-wbt.h" 136 137#define BFQ_BFQQ_FNS(name) \ 138void bfq_mark_bfqq_##name(struct bfq_queue *bfqq) \ 139{ \ 140 __set_bit(BFQQF_##name, &(bfqq)->flags); \ 141} \ 142void bfq_clear_bfqq_##name(struct bfq_queue *bfqq) \ 143{ \ 144 __clear_bit(BFQQF_##name, &(bfqq)->flags); \ 145} \ 146int bfq_bfqq_##name(const struct bfq_queue *bfqq) \ 147{ \ 148 return test_bit(BFQQF_##name, &(bfqq)->flags); \ 149} 150 151BFQ_BFQQ_FNS(just_created); 152BFQ_BFQQ_FNS(busy); 153BFQ_BFQQ_FNS(wait_request); 154BFQ_BFQQ_FNS(non_blocking_wait_rq); 155BFQ_BFQQ_FNS(fifo_expire); 156BFQ_BFQQ_FNS(has_short_ttime); 157BFQ_BFQQ_FNS(sync); 158BFQ_BFQQ_FNS(IO_bound); 159BFQ_BFQQ_FNS(in_large_burst); 160BFQ_BFQQ_FNS(coop); 161BFQ_BFQQ_FNS(split_coop); 162BFQ_BFQQ_FNS(softrt_update); 163#undef BFQ_BFQQ_FNS \ 164 165/* Expiration time of async (0) and sync (1) requests, in ns. */ 166static const u64 bfq_fifo_expire[2] = { NSEC_PER_SEC / 4, NSEC_PER_SEC / 8 }; 167 168/* Maximum backwards seek (magic number lifted from CFQ), in KiB. */ 169static const int bfq_back_max = 16 * 1024; 170 171/* Penalty of a backwards seek, in number of sectors. */ 172static const int bfq_back_penalty = 2; 173 174/* Idling period duration, in ns. */ 175static u64 bfq_slice_idle = NSEC_PER_SEC / 125; 176 177/* Minimum number of assigned budgets for which stats are safe to compute. */ 178static const int bfq_stats_min_budgets = 194; 179 180/* Default maximum budget values, in sectors and number of requests. */ 181static const int bfq_default_max_budget = 16 * 1024; 182 183/* 184 * When a sync request is dispatched, the queue that contains that 185 * request, and all the ancestor entities of that queue, are charged 186 * with the number of sectors of the request. In contrast, if the 187 * request is async, then the queue and its ancestor entities are 188 * charged with the number of sectors of the request, multiplied by 189 * the factor below. This throttles the bandwidth for async I/O, 190 * w.r.t. to sync I/O, and it is done to counter the tendency of async 191 * writes to steal I/O throughput to reads. 192 * 193 * The current value of this parameter is the result of a tuning with 194 * several hardware and software configurations. We tried to find the 195 * lowest value for which writes do not cause noticeable problems to 196 * reads. In fact, the lower this parameter, the stabler I/O control, 197 * in the following respect. The lower this parameter is, the less 198 * the bandwidth enjoyed by a group decreases 199 * - when the group does writes, w.r.t. to when it does reads; 200 * - when other groups do reads, w.r.t. to when they do writes. 201 */ 202static const int bfq_async_charge_factor = 3; 203 204/* Default timeout values, in jiffies, approximating CFQ defaults. */ 205const int bfq_timeout = HZ / 8; 206 207/* 208 * Time limit for merging (see comments in bfq_setup_cooperator). Set 209 * to the slowest value that, in our tests, proved to be effective in 210 * removing false positives, while not causing true positives to miss 211 * queue merging. 212 * 213 * As can be deduced from the low time limit below, queue merging, if 214 * successful, happens at the very beginning of the I/O of the involved 215 * cooperating processes, as a consequence of the arrival of the very 216 * first requests from each cooperator. After that, there is very 217 * little chance to find cooperators. 218 */ 219static const unsigned long bfq_merge_time_limit = HZ/10; 220 221static struct kmem_cache *bfq_pool; 222 223/* Below this threshold (in ns), we consider thinktime immediate. */ 224#define BFQ_MIN_TT (2 * NSEC_PER_MSEC) 225 226/* hw_tag detection: parallel requests threshold and min samples needed. */ 227#define BFQ_HW_QUEUE_THRESHOLD 3 228#define BFQ_HW_QUEUE_SAMPLES 32 229 230#define BFQQ_SEEK_THR (sector_t)(8 * 100) 231#define BFQQ_SECT_THR_NONROT (sector_t)(2 * 32) 232#define BFQ_RQ_SEEKY(bfqd, last_pos, rq) \ 233 (get_sdist(last_pos, rq) > \ 234 BFQQ_SEEK_THR && \ 235 (!blk_queue_nonrot(bfqd->queue) || \ 236 blk_rq_sectors(rq) < BFQQ_SECT_THR_NONROT)) 237#define BFQQ_CLOSE_THR (sector_t)(8 * 1024) 238#define BFQQ_SEEKY(bfqq) (hweight32(bfqq->seek_history) > 19) 239/* 240 * Sync random I/O is likely to be confused with soft real-time I/O, 241 * because it is characterized by limited throughput and apparently 242 * isochronous arrival pattern. To avoid false positives, queues 243 * containing only random (seeky) I/O are prevented from being tagged 244 * as soft real-time. 245 */ 246#define BFQQ_TOTALLY_SEEKY(bfqq) (bfqq->seek_history == -1) 247 248/* Min number of samples required to perform peak-rate update */ 249#define BFQ_RATE_MIN_SAMPLES 32 250/* Min observation time interval required to perform a peak-rate update (ns) */ 251#define BFQ_RATE_MIN_INTERVAL (300*NSEC_PER_MSEC) 252/* Target observation time interval for a peak-rate update (ns) */ 253#define BFQ_RATE_REF_INTERVAL NSEC_PER_SEC 254 255/* 256 * Shift used for peak-rate fixed precision calculations. 257 * With 258 * - the current shift: 16 positions 259 * - the current type used to store rate: u32 260 * - the current unit of measure for rate: [sectors/usec], or, more precisely, 261 * [(sectors/usec) / 2^BFQ_RATE_SHIFT] to take into account the shift, 262 * the range of rates that can be stored is 263 * [1 / 2^BFQ_RATE_SHIFT, 2^(32 - BFQ_RATE_SHIFT)] sectors/usec = 264 * [1 / 2^16, 2^16] sectors/usec = [15e-6, 65536] sectors/usec = 265 * [15, 65G] sectors/sec 266 * Which, assuming a sector size of 512B, corresponds to a range of 267 * [7.5K, 33T] B/sec 268 */ 269#define BFQ_RATE_SHIFT 16 270 271/* 272 * When configured for computing the duration of the weight-raising 273 * for interactive queues automatically (see the comments at the 274 * beginning of this file), BFQ does it using the following formula: 275 * duration = (ref_rate / r) * ref_wr_duration, 276 * where r is the peak rate of the device, and ref_rate and 277 * ref_wr_duration are two reference parameters. In particular, 278 * ref_rate is the peak rate of the reference storage device (see 279 * below), and ref_wr_duration is about the maximum time needed, with 280 * BFQ and while reading two files in parallel, to load typical large 281 * applications on the reference device (see the comments on 282 * max_service_from_wr below, for more details on how ref_wr_duration 283 * is obtained). In practice, the slower/faster the device at hand 284 * is, the more/less it takes to load applications with respect to the 285 * reference device. Accordingly, the longer/shorter BFQ grants 286 * weight raising to interactive applications. 287 * 288 * BFQ uses two different reference pairs (ref_rate, ref_wr_duration), 289 * depending on whether the device is rotational or non-rotational. 290 * 291 * In the following definitions, ref_rate[0] and ref_wr_duration[0] 292 * are the reference values for a rotational device, whereas 293 * ref_rate[1] and ref_wr_duration[1] are the reference values for a 294 * non-rotational device. The reference rates are not the actual peak 295 * rates of the devices used as a reference, but slightly lower 296 * values. The reason for using slightly lower values is that the 297 * peak-rate estimator tends to yield slightly lower values than the 298 * actual peak rate (it can yield the actual peak rate only if there 299 * is only one process doing I/O, and the process does sequential 300 * I/O). 301 * 302 * The reference peak rates are measured in sectors/usec, left-shifted 303 * by BFQ_RATE_SHIFT. 304 */ 305static int ref_rate[2] = {14000, 33000}; 306/* 307 * To improve readability, a conversion function is used to initialize 308 * the following array, which entails that the array can be 309 * initialized only in a function. 310 */ 311static int ref_wr_duration[2]; 312 313/* 314 * BFQ uses the above-detailed, time-based weight-raising mechanism to 315 * privilege interactive tasks. This mechanism is vulnerable to the 316 * following false positives: I/O-bound applications that will go on 317 * doing I/O for much longer than the duration of weight 318 * raising. These applications have basically no benefit from being 319 * weight-raised at the beginning of their I/O. On the opposite end, 320 * while being weight-raised, these applications 321 * a) unjustly steal throughput to applications that may actually need 322 * low latency; 323 * b) make BFQ uselessly perform device idling; device idling results 324 * in loss of device throughput with most flash-based storage, and may 325 * increase latencies when used purposelessly. 326 * 327 * BFQ tries to reduce these problems, by adopting the following 328 * countermeasure. To introduce this countermeasure, we need first to 329 * finish explaining how the duration of weight-raising for 330 * interactive tasks is computed. 331 * 332 * For a bfq_queue deemed as interactive, the duration of weight 333 * raising is dynamically adjusted, as a function of the estimated 334 * peak rate of the device, so as to be equal to the time needed to 335 * execute the 'largest' interactive task we benchmarked so far. By 336 * largest task, we mean the task for which each involved process has 337 * to do more I/O than for any of the other tasks we benchmarked. This 338 * reference interactive task is the start-up of LibreOffice Writer, 339 * and in this task each process/bfq_queue needs to have at most ~110K 340 * sectors transferred. 341 * 342 * This last piece of information enables BFQ to reduce the actual 343 * duration of weight-raising for at least one class of I/O-bound 344 * applications: those doing sequential or quasi-sequential I/O. An 345 * example is file copy. In fact, once started, the main I/O-bound 346 * processes of these applications usually consume the above 110K 347 * sectors in much less time than the processes of an application that 348 * is starting, because these I/O-bound processes will greedily devote 349 * almost all their CPU cycles only to their target, 350 * throughput-friendly I/O operations. This is even more true if BFQ 351 * happens to be underestimating the device peak rate, and thus 352 * overestimating the duration of weight raising. But, according to 353 * our measurements, once transferred 110K sectors, these processes 354 * have no right to be weight-raised any longer. 355 * 356 * Basing on the last consideration, BFQ ends weight-raising for a 357 * bfq_queue if the latter happens to have received an amount of 358 * service at least equal to the following constant. The constant is 359 * set to slightly more than 110K, to have a minimum safety margin. 360 * 361 * This early ending of weight-raising reduces the amount of time 362 * during which interactive false positives cause the two problems 363 * described at the beginning of these comments. 364 */ 365static const unsigned long max_service_from_wr = 120000; 366 367#define RQ_BIC(rq) icq_to_bic((rq)->elv.priv[0]) 368#define RQ_BFQQ(rq) ((rq)->elv.priv[1]) 369 370struct bfq_queue *bic_to_bfqq(struct bfq_io_cq *bic, bool is_sync) 371{ 372 return bic->bfqq[is_sync]; 373} 374 375static void bfq_put_stable_ref(struct bfq_queue *bfqq); 376 377void bic_set_bfqq(struct bfq_io_cq *bic, struct bfq_queue *bfqq, bool is_sync) 378{ 379 /* 380 * If bfqq != NULL, then a non-stable queue merge between 381 * bic->bfqq and bfqq is happening here. This causes troubles 382 * in the following case: bic->bfqq has also been scheduled 383 * for a possible stable merge with bic->stable_merge_bfqq, 384 * and bic->stable_merge_bfqq == bfqq happens to 385 * hold. Troubles occur because bfqq may then undergo a split, 386 * thereby becoming eligible for a stable merge. Yet, if 387 * bic->stable_merge_bfqq points exactly to bfqq, then bfqq 388 * would be stably merged with itself. To avoid this anomaly, 389 * we cancel the stable merge if 390 * bic->stable_merge_bfqq == bfqq. 391 */ 392 bic->bfqq[is_sync] = bfqq; 393 394 if (bfqq && bic->stable_merge_bfqq == bfqq) { 395 /* 396 * Actually, these same instructions are executed also 397 * in bfq_setup_cooperator, in case of abort or actual 398 * execution of a stable merge. We could avoid 399 * repeating these instructions there too, but if we 400 * did so, we would nest even more complexity in this 401 * function. 402 */ 403 bfq_put_stable_ref(bic->stable_merge_bfqq); 404 405 bic->stable_merge_bfqq = NULL; 406 } 407} 408 409struct bfq_data *bic_to_bfqd(struct bfq_io_cq *bic) 410{ 411 return bic->icq.q->elevator->elevator_data; 412} 413 414/** 415 * icq_to_bic - convert iocontext queue structure to bfq_io_cq. 416 * @icq: the iocontext queue. 417 */ 418static struct bfq_io_cq *icq_to_bic(struct io_cq *icq) 419{ 420 /* bic->icq is the first member, %NULL will convert to %NULL */ 421 return container_of(icq, struct bfq_io_cq, icq); 422} 423 424/** 425 * bfq_bic_lookup - search into @ioc a bic associated to @bfqd. 426 * @bfqd: the lookup key. 427 * @ioc: the io_context of the process doing I/O. 428 * @q: the request queue. 429 */ 430static struct bfq_io_cq *bfq_bic_lookup(struct bfq_data *bfqd, 431 struct io_context *ioc, 432 struct request_queue *q) 433{ 434 if (ioc) { 435 unsigned long flags; 436 struct bfq_io_cq *icq; 437 438 spin_lock_irqsave(&q->queue_lock, flags); 439 icq = icq_to_bic(ioc_lookup_icq(ioc, q)); 440 spin_unlock_irqrestore(&q->queue_lock, flags); 441 442 return icq; 443 } 444 445 return NULL; 446} 447 448/* 449 * Scheduler run of queue, if there are requests pending and no one in the 450 * driver that will restart queueing. 451 */ 452void bfq_schedule_dispatch(struct bfq_data *bfqd) 453{ 454 if (bfqd->queued != 0) { 455 bfq_log(bfqd, "schedule dispatch"); 456 blk_mq_run_hw_queues(bfqd->queue, true); 457 } 458} 459 460#define bfq_class_idle(bfqq) ((bfqq)->ioprio_class == IOPRIO_CLASS_IDLE) 461 462#define bfq_sample_valid(samples) ((samples) > 80) 463 464/* 465 * Lifted from AS - choose which of rq1 and rq2 that is best served now. 466 * We choose the request that is closer to the head right now. Distance 467 * behind the head is penalized and only allowed to a certain extent. 468 */ 469static struct request *bfq_choose_req(struct bfq_data *bfqd, 470 struct request *rq1, 471 struct request *rq2, 472 sector_t last) 473{ 474 sector_t s1, s2, d1 = 0, d2 = 0; 475 unsigned long back_max; 476#define BFQ_RQ1_WRAP 0x01 /* request 1 wraps */ 477#define BFQ_RQ2_WRAP 0x02 /* request 2 wraps */ 478 unsigned int wrap = 0; /* bit mask: requests behind the disk head? */ 479 480 if (!rq1 || rq1 == rq2) 481 return rq2; 482 if (!rq2) 483 return rq1; 484 485 if (rq_is_sync(rq1) && !rq_is_sync(rq2)) 486 return rq1; 487 else if (rq_is_sync(rq2) && !rq_is_sync(rq1)) 488 return rq2; 489 if ((rq1->cmd_flags & REQ_META) && !(rq2->cmd_flags & REQ_META)) 490 return rq1; 491 else if ((rq2->cmd_flags & REQ_META) && !(rq1->cmd_flags & REQ_META)) 492 return rq2; 493 494 s1 = blk_rq_pos(rq1); 495 s2 = blk_rq_pos(rq2); 496 497 /* 498 * By definition, 1KiB is 2 sectors. 499 */ 500 back_max = bfqd->bfq_back_max * 2; 501 502 /* 503 * Strict one way elevator _except_ in the case where we allow 504 * short backward seeks which are biased as twice the cost of a 505 * similar forward seek. 506 */ 507 if (s1 >= last) 508 d1 = s1 - last; 509 else if (s1 + back_max >= last) 510 d1 = (last - s1) * bfqd->bfq_back_penalty; 511 else 512 wrap |= BFQ_RQ1_WRAP; 513 514 if (s2 >= last) 515 d2 = s2 - last; 516 else if (s2 + back_max >= last) 517 d2 = (last - s2) * bfqd->bfq_back_penalty; 518 else 519 wrap |= BFQ_RQ2_WRAP; 520 521 /* Found required data */ 522 523 /* 524 * By doing switch() on the bit mask "wrap" we avoid having to 525 * check two variables for all permutations: --> faster! 526 */ 527 switch (wrap) { 528 case 0: /* common case for CFQ: rq1 and rq2 not wrapped */ 529 if (d1 < d2) 530 return rq1; 531 else if (d2 < d1) 532 return rq2; 533 534 if (s1 >= s2) 535 return rq1; 536 else 537 return rq2; 538 539 case BFQ_RQ2_WRAP: 540 return rq1; 541 case BFQ_RQ1_WRAP: 542 return rq2; 543 case BFQ_RQ1_WRAP|BFQ_RQ2_WRAP: /* both rqs wrapped */ 544 default: 545 /* 546 * Since both rqs are wrapped, 547 * start with the one that's further behind head 548 * (--> only *one* back seek required), 549 * since back seek takes more time than forward. 550 */ 551 if (s1 <= s2) 552 return rq1; 553 else 554 return rq2; 555 } 556} 557 558/* 559 * Async I/O can easily starve sync I/O (both sync reads and sync 560 * writes), by consuming all tags. Similarly, storms of sync writes, 561 * such as those that sync(2) may trigger, can starve sync reads. 562 * Limit depths of async I/O and sync writes so as to counter both 563 * problems. 564 */ 565static void bfq_limit_depth(unsigned int op, struct blk_mq_alloc_data *data) 566{ 567 struct bfq_data *bfqd = data->q->elevator->elevator_data; 568 569 if (op_is_sync(op) && !op_is_write(op)) 570 return; 571 572 data->shallow_depth = 573 bfqd->word_depths[!!bfqd->wr_busy_queues][op_is_sync(op)]; 574 575 bfq_log(bfqd, "[%s] wr_busy %d sync %d depth %u", 576 __func__, bfqd->wr_busy_queues, op_is_sync(op), 577 data->shallow_depth); 578} 579 580static struct bfq_queue * 581bfq_rq_pos_tree_lookup(struct bfq_data *bfqd, struct rb_root *root, 582 sector_t sector, struct rb_node **ret_parent, 583 struct rb_node ***rb_link) 584{ 585 struct rb_node **p, *parent; 586 struct bfq_queue *bfqq = NULL; 587 588 parent = NULL; 589 p = &root->rb_node; 590 while (*p) { 591 struct rb_node **n; 592 593 parent = *p; 594 bfqq = rb_entry(parent, struct bfq_queue, pos_node); 595 596 /* 597 * Sort strictly based on sector. Smallest to the left, 598 * largest to the right. 599 */ 600 if (sector > blk_rq_pos(bfqq->next_rq)) 601 n = &(*p)->rb_right; 602 else if (sector < blk_rq_pos(bfqq->next_rq)) 603 n = &(*p)->rb_left; 604 else 605 break; 606 p = n; 607 bfqq = NULL; 608 } 609 610 *ret_parent = parent; 611 if (rb_link) 612 *rb_link = p; 613 614 bfq_log(bfqd, "rq_pos_tree_lookup %llu: returning %d", 615 (unsigned long long)sector, 616 bfqq ? bfqq->pid : 0); 617 618 return bfqq; 619} 620 621static bool bfq_too_late_for_merging(struct bfq_queue *bfqq) 622{ 623 return bfqq->service_from_backlogged > 0 && 624 time_is_before_jiffies(bfqq->first_IO_time + 625 bfq_merge_time_limit); 626} 627 628/* 629 * The following function is not marked as __cold because it is 630 * actually cold, but for the same performance goal described in the 631 * comments on the likely() at the beginning of 632 * bfq_setup_cooperator(). Unexpectedly, to reach an even lower 633 * execution time for the case where this function is not invoked, we 634 * had to add an unlikely() in each involved if(). 635 */ 636void __cold 637bfq_pos_tree_add_move(struct bfq_data *bfqd, struct bfq_queue *bfqq) 638{ 639 struct rb_node **p, *parent; 640 struct bfq_queue *__bfqq; 641 642 if (bfqq->pos_root) { 643 rb_erase(&bfqq->pos_node, bfqq->pos_root); 644 bfqq->pos_root = NULL; 645 } 646 647 /* oom_bfqq does not participate in queue merging */ 648 if (bfqq == &bfqd->oom_bfqq) 649 return; 650 651 /* 652 * bfqq cannot be merged any longer (see comments in 653 * bfq_setup_cooperator): no point in adding bfqq into the 654 * position tree. 655 */ 656 if (bfq_too_late_for_merging(bfqq)) 657 return; 658 659 if (bfq_class_idle(bfqq)) 660 return; 661 if (!bfqq->next_rq) 662 return; 663 664 bfqq->pos_root = &bfq_bfqq_to_bfqg(bfqq)->rq_pos_tree; 665 __bfqq = bfq_rq_pos_tree_lookup(bfqd, bfqq->pos_root, 666 blk_rq_pos(bfqq->next_rq), &parent, &p); 667 if (!__bfqq) { 668 rb_link_node(&bfqq->pos_node, parent, p); 669 rb_insert_color(&bfqq->pos_node, bfqq->pos_root); 670 } else 671 bfqq->pos_root = NULL; 672} 673 674/* 675 * The following function returns false either if every active queue 676 * must receive the same share of the throughput (symmetric scenario), 677 * or, as a special case, if bfqq must receive a share of the 678 * throughput lower than or equal to the share that every other active 679 * queue must receive. If bfqq does sync I/O, then these are the only 680 * two cases where bfqq happens to be guaranteed its share of the 681 * throughput even if I/O dispatching is not plugged when bfqq remains 682 * temporarily empty (for more details, see the comments in the 683 * function bfq_better_to_idle()). For this reason, the return value 684 * of this function is used to check whether I/O-dispatch plugging can 685 * be avoided. 686 * 687 * The above first case (symmetric scenario) occurs when: 688 * 1) all active queues have the same weight, 689 * 2) all active queues belong to the same I/O-priority class, 690 * 3) all active groups at the same level in the groups tree have the same 691 * weight, 692 * 4) all active groups at the same level in the groups tree have the same 693 * number of children. 694 * 695 * Unfortunately, keeping the necessary state for evaluating exactly 696 * the last two symmetry sub-conditions above would be quite complex 697 * and time consuming. Therefore this function evaluates, instead, 698 * only the following stronger three sub-conditions, for which it is 699 * much easier to maintain the needed state: 700 * 1) all active queues have the same weight, 701 * 2) all active queues belong to the same I/O-priority class, 702 * 3) there are no active groups. 703 * In particular, the last condition is always true if hierarchical 704 * support or the cgroups interface are not enabled, thus no state 705 * needs to be maintained in this case. 706 */ 707static bool bfq_asymmetric_scenario(struct bfq_data *bfqd, 708 struct bfq_queue *bfqq) 709{ 710 bool smallest_weight = bfqq && 711 bfqq->weight_counter && 712 bfqq->weight_counter == 713 container_of( 714 rb_first_cached(&bfqd->queue_weights_tree), 715 struct bfq_weight_counter, 716 weights_node); 717 718 /* 719 * For queue weights to differ, queue_weights_tree must contain 720 * at least two nodes. 721 */ 722 bool varied_queue_weights = !smallest_weight && 723 !RB_EMPTY_ROOT(&bfqd->queue_weights_tree.rb_root) && 724 (bfqd->queue_weights_tree.rb_root.rb_node->rb_left || 725 bfqd->queue_weights_tree.rb_root.rb_node->rb_right); 726 727 bool multiple_classes_busy = 728 (bfqd->busy_queues[0] && bfqd->busy_queues[1]) || 729 (bfqd->busy_queues[0] && bfqd->busy_queues[2]) || 730 (bfqd->busy_queues[1] && bfqd->busy_queues[2]); 731 732 return varied_queue_weights || multiple_classes_busy 733#ifdef CONFIG_BFQ_GROUP_IOSCHED 734 || bfqd->num_groups_with_pending_reqs > 0 735#endif 736 ; 737} 738 739/* 740 * If the weight-counter tree passed as input contains no counter for 741 * the weight of the input queue, then add that counter; otherwise just 742 * increment the existing counter. 743 * 744 * Note that weight-counter trees contain few nodes in mostly symmetric 745 * scenarios. For example, if all queues have the same weight, then the 746 * weight-counter tree for the queues may contain at most one node. 747 * This holds even if low_latency is on, because weight-raised queues 748 * are not inserted in the tree. 749 * In most scenarios, the rate at which nodes are created/destroyed 750 * should be low too. 751 */ 752void bfq_weights_tree_add(struct bfq_data *bfqd, struct bfq_queue *bfqq, 753 struct rb_root_cached *root) 754{ 755 struct bfq_entity *entity = &bfqq->entity; 756 struct rb_node **new = &(root->rb_root.rb_node), *parent = NULL; 757 bool leftmost = true; 758 759 /* 760 * Do not insert if the queue is already associated with a 761 * counter, which happens if: 762 * 1) a request arrival has caused the queue to become both 763 * non-weight-raised, and hence change its weight, and 764 * backlogged; in this respect, each of the two events 765 * causes an invocation of this function, 766 * 2) this is the invocation of this function caused by the 767 * second event. This second invocation is actually useless, 768 * and we handle this fact by exiting immediately. More 769 * efficient or clearer solutions might possibly be adopted. 770 */ 771 if (bfqq->weight_counter) 772 return; 773 774 while (*new) { 775 struct bfq_weight_counter *__counter = container_of(*new, 776 struct bfq_weight_counter, 777 weights_node); 778 parent = *new; 779 780 if (entity->weight == __counter->weight) { 781 bfqq->weight_counter = __counter; 782 goto inc_counter; 783 } 784 if (entity->weight < __counter->weight) 785 new = &((*new)->rb_left); 786 else { 787 new = &((*new)->rb_right); 788 leftmost = false; 789 } 790 } 791 792 bfqq->weight_counter = kzalloc(sizeof(struct bfq_weight_counter), 793 GFP_ATOMIC); 794 795 /* 796 * In the unlucky event of an allocation failure, we just 797 * exit. This will cause the weight of queue to not be 798 * considered in bfq_asymmetric_scenario, which, in its turn, 799 * causes the scenario to be deemed wrongly symmetric in case 800 * bfqq's weight would have been the only weight making the 801 * scenario asymmetric. On the bright side, no unbalance will 802 * however occur when bfqq becomes inactive again (the 803 * invocation of this function is triggered by an activation 804 * of queue). In fact, bfq_weights_tree_remove does nothing 805 * if !bfqq->weight_counter. 806 */ 807 if (unlikely(!bfqq->weight_counter)) 808 return; 809 810 bfqq->weight_counter->weight = entity->weight; 811 rb_link_node(&bfqq->weight_counter->weights_node, parent, new); 812 rb_insert_color_cached(&bfqq->weight_counter->weights_node, root, 813 leftmost); 814 815inc_counter: 816 bfqq->weight_counter->num_active++; 817 bfqq->ref++; 818} 819 820/* 821 * Decrement the weight counter associated with the queue, and, if the 822 * counter reaches 0, remove the counter from the tree. 823 * See the comments to the function bfq_weights_tree_add() for considerations 824 * about overhead. 825 */ 826void __bfq_weights_tree_remove(struct bfq_data *bfqd, 827 struct bfq_queue *bfqq, 828 struct rb_root_cached *root) 829{ 830 if (!bfqq->weight_counter) 831 return; 832 833 bfqq->weight_counter->num_active--; 834 if (bfqq->weight_counter->num_active > 0) 835 goto reset_entity_pointer; 836 837 rb_erase_cached(&bfqq->weight_counter->weights_node, root); 838 kfree(bfqq->weight_counter); 839 840reset_entity_pointer: 841 bfqq->weight_counter = NULL; 842 bfq_put_queue(bfqq); 843} 844 845/* 846 * Invoke __bfq_weights_tree_remove on bfqq and decrement the number 847 * of active groups for each queue's inactive parent entity. 848 */ 849void bfq_weights_tree_remove(struct bfq_data *bfqd, 850 struct bfq_queue *bfqq) 851{ 852 struct bfq_entity *entity = bfqq->entity.parent; 853 854 for_each_entity(entity) { 855 struct bfq_sched_data *sd = entity->my_sched_data; 856 857 if (sd->next_in_service || sd->in_service_entity) { 858 /* 859 * entity is still active, because either 860 * next_in_service or in_service_entity is not 861 * NULL (see the comments on the definition of 862 * next_in_service for details on why 863 * in_service_entity must be checked too). 864 * 865 * As a consequence, its parent entities are 866 * active as well, and thus this loop must 867 * stop here. 868 */ 869 break; 870 } 871 872 /* 873 * The decrement of num_groups_with_pending_reqs is 874 * not performed immediately upon the deactivation of 875 * entity, but it is delayed to when it also happens 876 * that the first leaf descendant bfqq of entity gets 877 * all its pending requests completed. The following 878 * instructions perform this delayed decrement, if 879 * needed. See the comments on 880 * num_groups_with_pending_reqs for details. 881 */ 882 if (entity->in_groups_with_pending_reqs) { 883 entity->in_groups_with_pending_reqs = false; 884 bfqd->num_groups_with_pending_reqs--; 885 } 886 } 887 888 /* 889 * Next function is invoked last, because it causes bfqq to be 890 * freed if the following holds: bfqq is not in service and 891 * has no dispatched request. DO NOT use bfqq after the next 892 * function invocation. 893 */ 894 __bfq_weights_tree_remove(bfqd, bfqq, 895 &bfqd->queue_weights_tree); 896} 897 898/* 899 * Return expired entry, or NULL to just start from scratch in rbtree. 900 */ 901static struct request *bfq_check_fifo(struct bfq_queue *bfqq, 902 struct request *last) 903{ 904 struct request *rq; 905 906 if (bfq_bfqq_fifo_expire(bfqq)) 907 return NULL; 908 909 bfq_mark_bfqq_fifo_expire(bfqq); 910 911 rq = rq_entry_fifo(bfqq->fifo.next); 912 913 if (rq == last || ktime_get_ns() < rq->fifo_time) 914 return NULL; 915 916 bfq_log_bfqq(bfqq->bfqd, bfqq, "check_fifo: returned %p", rq); 917 return rq; 918} 919 920static struct request *bfq_find_next_rq(struct bfq_data *bfqd, 921 struct bfq_queue *bfqq, 922 struct request *last) 923{ 924 struct rb_node *rbnext = rb_next(&last->rb_node); 925 struct rb_node *rbprev = rb_prev(&last->rb_node); 926 struct request *next, *prev = NULL; 927 928 /* Follow expired path, else get first next available. */ 929 next = bfq_check_fifo(bfqq, last); 930 if (next) 931 return next; 932 933 if (rbprev) 934 prev = rb_entry_rq(rbprev); 935 936 if (rbnext) 937 next = rb_entry_rq(rbnext); 938 else { 939 rbnext = rb_first(&bfqq->sort_list); 940 if (rbnext && rbnext != &last->rb_node) 941 next = rb_entry_rq(rbnext); 942 } 943 944 return bfq_choose_req(bfqd, next, prev, blk_rq_pos(last)); 945} 946 947/* see the definition of bfq_async_charge_factor for details */ 948static unsigned long bfq_serv_to_charge(struct request *rq, 949 struct bfq_queue *bfqq) 950{ 951 if (bfq_bfqq_sync(bfqq) || bfqq->wr_coeff > 1 || 952 bfq_asymmetric_scenario(bfqq->bfqd, bfqq)) 953 return blk_rq_sectors(rq); 954 955 return blk_rq_sectors(rq) * bfq_async_charge_factor; 956} 957 958/** 959 * bfq_updated_next_req - update the queue after a new next_rq selection. 960 * @bfqd: the device data the queue belongs to. 961 * @bfqq: the queue to update. 962 * 963 * If the first request of a queue changes we make sure that the queue 964 * has enough budget to serve at least its first request (if the 965 * request has grown). We do this because if the queue has not enough 966 * budget for its first request, it has to go through two dispatch 967 * rounds to actually get it dispatched. 968 */ 969static void bfq_updated_next_req(struct bfq_data *bfqd, 970 struct bfq_queue *bfqq) 971{ 972 struct bfq_entity *entity = &bfqq->entity; 973 struct request *next_rq = bfqq->next_rq; 974 unsigned long new_budget; 975 976 if (!next_rq) 977 return; 978 979 if (bfqq == bfqd->in_service_queue) 980 /* 981 * In order not to break guarantees, budgets cannot be 982 * changed after an entity has been selected. 983 */ 984 return; 985 986 new_budget = max_t(unsigned long, 987 max_t(unsigned long, bfqq->max_budget, 988 bfq_serv_to_charge(next_rq, bfqq)), 989 entity->service); 990 if (entity->budget != new_budget) { 991 entity->budget = new_budget; 992 bfq_log_bfqq(bfqd, bfqq, "updated next rq: new budget %lu", 993 new_budget); 994 bfq_requeue_bfqq(bfqd, bfqq, false); 995 } 996} 997 998static unsigned int bfq_wr_duration(struct bfq_data *bfqd) 999{ 1000 u64 dur;
1001 1002 if (bfqd->bfq_wr_max_time > 0) 1003 return bfqd->bfq_wr_max_time; 1004 1005 dur = bfqd->rate_dur_prod; 1006 do_div(dur, bfqd->peak_rate); 1007 1008 /* 1009 * Limit duration between 3 and 25 seconds. The upper limit 1010 * has been conservatively set after the following worst case: 1011 * on a QEMU/KVM virtual machine 1012 * - running in a slow PC 1013 * - with a virtual disk stacked on a slow low-end 5400rpm HDD 1014 * - serving a heavy I/O workload, such as the sequential reading 1015 * of several files 1016 * mplayer took 23 seconds to start, if constantly weight-raised. 1017 * 1018 * As for higher values than that accommodating the above bad 1019 * scenario, tests show that higher values would often yield 1020 * the opposite of the desired result, i.e., would worsen 1021 * responsiveness by allowing non-interactive applications to 1022 * preserve weight raising for too long. 1023 * 1024 * On the other end, lower values than 3 seconds make it 1025 * difficult for most interactive tasks to complete their jobs 1026 * before weight-raising finishes. 1027 */ 1028 return clamp_val(dur, msecs_to_jiffies(3000), msecs_to_jiffies(25000)); 1029} 1030 1031/* switch back from soft real-time to interactive weight raising */ 1032static void switch_back_to_interactive_wr(struct bfq_queue *bfqq, 1033 struct bfq_data *bfqd) 1034{ 1035 bfqq->wr_coeff = bfqd->bfq_wr_coeff; 1036 bfqq->wr_cur_max_time = bfq_wr_duration(bfqd); 1037 bfqq->last_wr_start_finish = bfqq->wr_start_at_switch_to_srt; 1038} 1039 1040static void 1041bfq_bfqq_resume_state(struct bfq_queue *bfqq, struct bfq_data *bfqd, 1042 struct bfq_io_cq *bic, bool bfq_already_existing) 1043{ 1044 unsigned int old_wr_coeff = 1; 1045 bool busy = bfq_already_existing && bfq_bfqq_busy(bfqq); 1046 1047 if (bic->saved_has_short_ttime) 1048 bfq_mark_bfqq_has_short_ttime(bfqq); 1049 else 1050 bfq_clear_bfqq_has_short_ttime(bfqq); 1051 1052 if (bic->saved_IO_bound) 1053 bfq_mark_bfqq_IO_bound(bfqq); 1054 else 1055 bfq_clear_bfqq_IO_bound(bfqq); 1056 1057 bfqq->last_serv_time_ns = bic->saved_last_serv_time_ns; 1058 bfqq->inject_limit = bic->saved_inject_limit; 1059 bfqq->decrease_time_jif = bic->saved_decrease_time_jif; 1060 1061 bfqq->entity.new_weight = bic->saved_weight; 1062 bfqq->ttime = bic->saved_ttime; 1063 bfqq->io_start_time = bic->saved_io_start_time; 1064 bfqq->tot_idle_time = bic->saved_tot_idle_time; 1065 /* 1066 * Restore weight coefficient only if low_latency is on 1067 */ 1068 if (bfqd->low_latency) { 1069 old_wr_coeff = bfqq->wr_coeff; 1070 bfqq->wr_coeff = bic->saved_wr_coeff; 1071 } 1072 bfqq->service_from_wr = bic->saved_service_from_wr; 1073 bfqq->wr_start_at_switch_to_srt = bic->saved_wr_start_at_switch_to_srt; 1074 bfqq->last_wr_start_finish = bic->saved_last_wr_start_finish; 1075 bfqq->wr_cur_max_time = bic->saved_wr_cur_max_time; 1076 1077 if (bfqq->wr_coeff > 1 && (bfq_bfqq_in_large_burst(bfqq) || 1078 time_is_before_jiffies(bfqq->last_wr_start_finish + 1079 bfqq->wr_cur_max_time))) { 1080 if (bfqq->wr_cur_max_time == bfqd->bfq_wr_rt_max_time && 1081 !bfq_bfqq_in_large_burst(bfqq) && 1082 time_is_after_eq_jiffies(bfqq->wr_start_at_switch_to_srt + 1083 bfq_wr_duration(bfqd))) { 1084 switch_back_to_interactive_wr(bfqq, bfqd); 1085 } else { 1086 bfqq->wr_coeff = 1; 1087 bfq_log_bfqq(bfqq->bfqd, bfqq, 1088 "resume state: switching off wr"); 1089 } 1090 } 1091 1092 /* make sure weight will be updated, however we got here */ 1093 bfqq->entity.prio_changed = 1; 1094 1095 if (likely(!busy)) 1096 return; 1097 1098 if (old_wr_coeff == 1 && bfqq->wr_coeff > 1) 1099 bfqd->wr_busy_queues++; 1100 else if (old_wr_coeff > 1 && bfqq->wr_coeff == 1) 1101 bfqd->wr_busy_queues--; 1102} 1103 1104static int bfqq_process_refs(struct bfq_queue *bfqq) 1105{ 1106 return bfqq->ref - bfqq->allocated - bfqq->entity.on_st_or_in_serv - 1107 (bfqq->weight_counter != NULL) - bfqq->stable_ref; 1108} 1109 1110/* Empty burst list and add just bfqq (see comments on bfq_handle_burst) */ 1111static void bfq_reset_burst_list(struct bfq_data *bfqd, struct bfq_queue *bfqq) 1112{ 1113 struct bfq_queue *item; 1114 struct hlist_node *n; 1115 1116 hlist_for_each_entry_safe(item, n, &bfqd->burst_list, burst_list_node) 1117 hlist_del_init(&item->burst_list_node); 1118 1119 /* 1120 * Start the creation of a new burst list only if there is no 1121 * active queue. See comments on the conditional invocation of 1122 * bfq_handle_burst(). 1123 */ 1124 if (bfq_tot_busy_queues(bfqd) == 0) { 1125 hlist_add_head(&bfqq->burst_list_node, &bfqd->burst_list); 1126 bfqd->burst_size = 1; 1127 } else 1128 bfqd->burst_size = 0; 1129 1130 bfqd->burst_parent_entity = bfqq->entity.parent; 1131} 1132 1133/* Add bfqq to the list of queues in current burst (see bfq_handle_burst) */ 1134static void bfq_add_to_burst(struct bfq_data *bfqd, struct bfq_queue *bfqq) 1135{ 1136 /* Increment burst size to take into account also bfqq */ 1137 bfqd->burst_size++; 1138 1139 if (bfqd->burst_size == bfqd->bfq_large_burst_thresh) { 1140 struct bfq_queue *pos, *bfqq_item; 1141 struct hlist_node *n; 1142 1143 /* 1144 * Enough queues have been activated shortly after each 1145 * other to consider this burst as large. 1146 */ 1147 bfqd->large_burst = true; 1148 1149 /* 1150 * We can now mark all queues in the burst list as 1151 * belonging to a large burst. 1152 */ 1153 hlist_for_each_entry(bfqq_item, &bfqd->burst_list, 1154 burst_list_node) 1155 bfq_mark_bfqq_in_large_burst(bfqq_item); 1156 bfq_mark_bfqq_in_large_burst(bfqq); 1157 1158 /* 1159 * From now on, and until the current burst finishes, any 1160 * new queue being activated shortly after the last queue 1161 * was inserted in the burst can be immediately marked as 1162 * belonging to a large burst. So the burst list is not 1163 * needed any more. Remove it. 1164 */ 1165 hlist_for_each_entry_safe(pos, n, &bfqd->burst_list, 1166 burst_list_node) 1167 hlist_del_init(&pos->burst_list_node); 1168 } else /* 1169 * Burst not yet large: add bfqq to the burst list. Do 1170 * not increment the ref counter for bfqq, because bfqq 1171 * is removed from the burst list before freeing bfqq 1172 * in put_queue. 1173 */ 1174 hlist_add_head(&bfqq->burst_list_node, &bfqd->burst_list); 1175} 1176 1177/* 1178 * If many queues belonging to the same group happen to be created 1179 * shortly after each other, then the processes associated with these 1180 * queues have typically a common goal. In particular, bursts of queue 1181 * creations are usually caused by services or applications that spawn 1182 * many parallel threads/processes. Examples are systemd during boot, 1183 * or git grep. To help these processes get their job done as soon as 1184 * possible, it is usually better to not grant either weight-raising 1185 * or device idling to their queues, unless these queues must be 1186 * protected from the I/O flowing through other active queues. 1187 * 1188 * In this comment we describe, firstly, the reasons why this fact 1189 * holds, and, secondly, the next function, which implements the main 1190 * steps needed to properly mark these queues so that they can then be 1191 * treated in a different way. 1192 * 1193 * The above services or applications benefit mostly from a high 1194 * throughput: the quicker the requests of the activated queues are 1195 * cumulatively served, the sooner the target job of these queues gets 1196 * completed. As a consequence, weight-raising any of these queues, 1197 * which also implies idling the device for it, is almost always 1198 * counterproductive, unless there are other active queues to isolate 1199 * these new queues from. If there no other active queues, then 1200 * weight-raising these new queues just lowers throughput in most 1201 * cases. 1202 * 1203 * On the other hand, a burst of queue creations may be caused also by 1204 * the start of an application that does not consist of a lot of 1205 * parallel I/O-bound threads. In fact, with a complex application, 1206 * several short processes may need to be executed to start-up the 1207 * application. In this respect, to start an application as quickly as 1208 * possible, the best thing to do is in any case to privilege the I/O 1209 * related to the application with respect to all other 1210 * I/O. Therefore, the best strategy to start as quickly as possible 1211 * an application that causes a burst of queue creations is to 1212 * weight-raise all the queues created during the burst. This is the 1213 * exact opposite of the best strategy for the other type of bursts. 1214 * 1215 * In the end, to take the best action for each of the two cases, the 1216 * two types of bursts need to be distinguished. Fortunately, this 1217 * seems relatively easy, by looking at the sizes of the bursts. In 1218 * particular, we found a threshold such that only bursts with a 1219 * larger size than that threshold are apparently caused by 1220 * services or commands such as systemd or git grep. For brevity, 1221 * hereafter we call just 'large' these bursts. BFQ *does not* 1222 * weight-raise queues whose creation occurs in a large burst. In 1223 * addition, for each of these queues BFQ performs or does not perform 1224 * idling depending on which choice boosts the throughput more. The 1225 * exact choice depends on the device and request pattern at 1226 * hand. 1227 * 1228 * Unfortunately, false positives may occur while an interactive task 1229 * is starting (e.g., an application is being started). The 1230 * consequence is that the queues associated with the task do not 1231 * enjoy weight raising as expected. Fortunately these false positives 1232 * are very rare. They typically occur if some service happens to 1233 * start doing I/O exactly when the interactive task starts. 1234 * 1235 * Turning back to the next function, it is invoked only if there are 1236 * no active queues (apart from active queues that would belong to the 1237 * same, possible burst bfqq would belong to), and it implements all 1238 * the steps needed to detect the occurrence of a large burst and to 1239 * properly mark all the queues belonging to it (so that they can then 1240 * be treated in a different way). This goal is achieved by 1241 * maintaining a "burst list" that holds, temporarily, the queues that 1242 * belong to the burst in progress. The list is then used to mark 1243 * these queues as belonging to a large burst if the burst does become 1244 * large. The main steps are the following. 1245 * 1246 * . when the very first queue is created, the queue is inserted into the 1247 * list (as it could be the first queue in a possible burst) 1248 * 1249 * . if the current burst has not yet become large, and a queue Q that does 1250 * not yet belong to the burst is activated shortly after the last time 1251 * at which a new queue entered the burst list, then the function appends 1252 * Q to the burst list 1253 * 1254 * . if, as a consequence of the previous step, the burst size reaches 1255 * the large-burst threshold, then 1256 * 1257 * . all the queues in the burst list are marked as belonging to a 1258 * large burst 1259 * 1260 * . the burst list is deleted; in fact, the burst list already served 1261 * its purpose (keeping temporarily track of the queues in a burst, 1262 * so as to be able to mark them as belonging to a large burst in the 1263 * previous sub-step), and now is not needed any more 1264 * 1265 * . the device enters a large-burst mode 1266 * 1267 * . if a queue Q that does not belong to the burst is created while 1268 * the device is in large-burst mode and shortly after the last time 1269 * at which a queue either entered the burst list or was marked as 1270 * belonging to the current large burst, then Q is immediately marked 1271 * as belonging to a large burst. 1272 * 1273 * . if a queue Q that does not belong to the burst is created a while 1274 * later, i.e., not shortly after, than the last time at which a queue 1275 * either entered the burst list or was marked as belonging to the 1276 * current large burst, then the current burst is deemed as finished and: 1277 * 1278 * . the large-burst mode is reset if set 1279 * 1280 * . the burst list is emptied 1281 * 1282 * . Q is inserted in the burst list, as Q may be the first queue 1283 * in a possible new burst (then the burst list contains just Q 1284 * after this step). 1285 */ 1286static void bfq_handle_burst(struct bfq_data *bfqd, struct bfq_queue *bfqq) 1287{ 1288 /* 1289 * If bfqq is already in the burst list or is part of a large 1290 * burst, or finally has just been split, then there is 1291 * nothing else to do. 1292 */ 1293 if (!hlist_unhashed(&bfqq->burst_list_node) || 1294 bfq_bfqq_in_large_burst(bfqq) || 1295 time_is_after_eq_jiffies(bfqq->split_time + 1296 msecs_to_jiffies(10))) 1297 return; 1298 1299 /* 1300 * If bfqq's creation happens late enough, or bfqq belongs to 1301 * a different group than the burst group, then the current 1302 * burst is finished, and related data structures must be 1303 * reset. 1304 * 1305 * In this respect, consider the special case where bfqq is 1306 * the very first queue created after BFQ is selected for this 1307 * device. In this case, last_ins_in_burst and 1308 * burst_parent_entity are not yet significant when we get 1309 * here. But it is easy to verify that, whether or not the 1310 * following condition is true, bfqq will end up being 1311 * inserted into the burst list. In particular the list will 1312 * happen to contain only bfqq. And this is exactly what has 1313 * to happen, as bfqq may be the first queue of the first 1314 * burst. 1315 */ 1316 if (time_is_before_jiffies(bfqd->last_ins_in_burst + 1317 bfqd->bfq_burst_interval) || 1318 bfqq->entity.parent != bfqd->burst_parent_entity) { 1319 bfqd->large_burst = false; 1320 bfq_reset_burst_list(bfqd, bfqq); 1321 goto end; 1322 } 1323 1324 /* 1325 * If we get here, then bfqq is being activated shortly after the 1326 * last queue. So, if the current burst is also large, we can mark 1327 * bfqq as belonging to this large burst immediately. 1328 */ 1329 if (bfqd->large_burst) { 1330 bfq_mark_bfqq_in_large_burst(bfqq); 1331 goto end; 1332 } 1333 1334 /* 1335 * If we get here, then a large-burst state has not yet been 1336 * reached, but bfqq is being activated shortly after the last 1337 * queue. Then we add bfqq to the burst. 1338 */ 1339 bfq_add_to_burst(bfqd, bfqq); 1340end: 1341 /* 1342 * At this point, bfqq either has been added to the current 1343 * burst or has caused the current burst to terminate and a 1344 * possible new burst to start. In particular, in the second 1345 * case, bfqq has become the first queue in the possible new 1346 * burst. In both cases last_ins_in_burst needs to be moved 1347 * forward. 1348 */ 1349 bfqd->last_ins_in_burst = jiffies; 1350} 1351 1352static int bfq_bfqq_budget_left(struct bfq_queue *bfqq) 1353{ 1354 struct bfq_entity *entity = &bfqq->entity; 1355 1356 return entity->budget - entity->service; 1357} 1358 1359/* 1360 * If enough samples have been computed, return the current max budget 1361 * stored in bfqd, which is dynamically updated according to the 1362 * estimated disk peak rate; otherwise return the default max budget 1363 */ 1364static int bfq_max_budget(struct bfq_data *bfqd) 1365{ 1366 if (bfqd->budgets_assigned < bfq_stats_min_budgets) 1367 return bfq_default_max_budget; 1368 else 1369 return bfqd->bfq_max_budget; 1370} 1371 1372/* 1373 * Return min budget, which is a fraction of the current or default 1374 * max budget (trying with 1/32) 1375 */ 1376static int bfq_min_budget(struct bfq_data *bfqd) 1377{ 1378 if (bfqd->budgets_assigned < bfq_stats_min_budgets) 1379 return bfq_default_max_budget / 32; 1380 else 1381 return bfqd->bfq_max_budget / 32; 1382} 1383 1384/* 1385 * The next function, invoked after the input queue bfqq switches from 1386 * idle to busy, updates the budget of bfqq. The function also tells 1387 * whether the in-service queue should be expired, by returning 1388 * true. The purpose of expiring the in-service queue is to give bfqq 1389 * the chance to possibly preempt the in-service queue, and the reason 1390 * for preempting the in-service queue is to achieve one of the two 1391 * goals below. 1392 * 1393 * 1. Guarantee to bfqq its reserved bandwidth even if bfqq has 1394 * expired because it has remained idle. In particular, bfqq may have 1395 * expired for one of the following two reasons: 1396 * 1397 * - BFQQE_NO_MORE_REQUESTS bfqq did not enjoy any device idling 1398 * and did not make it to issue a new request before its last 1399 * request was served; 1400 * 1401 * - BFQQE_TOO_IDLE bfqq did enjoy device idling, but did not issue 1402 * a new request before the expiration of the idling-time. 1403 * 1404 * Even if bfqq has expired for one of the above reasons, the process 1405 * associated with the queue may be however issuing requests greedily, 1406 * and thus be sensitive to the bandwidth it receives (bfqq may have 1407 * remained idle for other reasons: CPU high load, bfqq not enjoying 1408 * idling, I/O throttling somewhere in the path from the process to 1409 * the I/O scheduler, ...). But if, after every expiration for one of 1410 * the above two reasons, bfqq has to wait for the service of at least 1411 * one full budget of another queue before being served again, then 1412 * bfqq is likely to get a much lower bandwidth or resource time than 1413 * its reserved ones. To address this issue, two countermeasures need 1414 * to be taken. 1415 * 1416 * First, the budget and the timestamps of bfqq need to be updated in 1417 * a special way on bfqq reactivation: they need to be updated as if 1418 * bfqq did not remain idle and did not expire. In fact, if they are 1419 * computed as if bfqq expired and remained idle until reactivation, 1420 * then the process associated with bfqq is treated as if, instead of 1421 * being greedy, it stopped issuing requests when bfqq remained idle, 1422 * and restarts issuing requests only on this reactivation. In other 1423 * words, the scheduler does not help the process recover the "service 1424 * hole" between bfqq expiration and reactivation. As a consequence, 1425 * the process receives a lower bandwidth than its reserved one. In 1426 * contrast, to recover this hole, the budget must be updated as if 1427 * bfqq was not expired at all before this reactivation, i.e., it must 1428 * be set to the value of the remaining budget when bfqq was 1429 * expired. Along the same line, timestamps need to be assigned the 1430 * value they had the last time bfqq was selected for service, i.e., 1431 * before last expiration. Thus timestamps need to be back-shifted 1432 * with respect to their normal computation (see [1] for more details 1433 * on this tricky aspect). 1434 * 1435 * Secondly, to allow the process to recover the hole, the in-service 1436 * queue must be expired too, to give bfqq the chance to preempt it 1437 * immediately. In fact, if bfqq has to wait for a full budget of the 1438 * in-service queue to be completed, then it may become impossible to 1439 * let the process recover the hole, even if the back-shifted 1440 * timestamps of bfqq are lower than those of the in-service queue. If 1441 * this happens for most or all of the holes, then the process may not 1442 * receive its reserved bandwidth. In this respect, it is worth noting 1443 * that, being the service of outstanding requests unpreemptible, a 1444 * little fraction of the holes may however be unrecoverable, thereby 1445 * causing a little loss of bandwidth. 1446 * 1447 * The last important point is detecting whether bfqq does need this 1448 * bandwidth recovery. In this respect, the next function deems the 1449 * process associated with bfqq greedy, and thus allows it to recover 1450 * the hole, if: 1) the process is waiting for the arrival of a new 1451 * request (which implies that bfqq expired for one of the above two 1452 * reasons), and 2) such a request has arrived soon. The first 1453 * condition is controlled through the flag non_blocking_wait_rq, 1454 * while the second through the flag arrived_in_time. If both 1455 * conditions hold, then the function computes the budget in the 1456 * above-described special way, and signals that the in-service queue 1457 * should be expired. Timestamp back-shifting is done later in 1458 * __bfq_activate_entity. 1459 * 1460 * 2. Reduce latency. Even if timestamps are not backshifted to let 1461 * the process associated with bfqq recover a service hole, bfqq may 1462 * however happen to have, after being (re)activated, a lower finish 1463 * timestamp than the in-service queue. That is, the next budget of 1464 * bfqq may have to be completed before the one of the in-service 1465 * queue. If this is the case, then preempting the in-service queue 1466 * allows this goal to be achieved, apart from the unpreemptible, 1467 * outstanding requests mentioned above. 1468 * 1469 * Unfortunately, regardless of which of the above two goals one wants 1470 * to achieve, service trees need first to be updated to know whether 1471 * the in-service queue must be preempted. To have service trees 1472 * correctly updated, the in-service queue must be expired and 1473 * rescheduled, and bfqq must be scheduled too. This is one of the 1474 * most costly operations (in future versions, the scheduling 1475 * mechanism may be re-designed in such a way to make it possible to 1476 * know whether preemption is needed without needing to update service 1477 * trees). In addition, queue preemptions almost always cause random 1478 * I/O, which may in turn cause loss of throughput. Finally, there may 1479 * even be no in-service queue when the next function is invoked (so, 1480 * no queue to compare timestamps with). Because of these facts, the 1481 * next function adopts the following simple scheme to avoid costly 1482 * operations, too frequent preemptions and too many dependencies on 1483 * the state of the scheduler: it requests the expiration of the 1484 * in-service queue (unconditionally) only for queues that need to 1485 * recover a hole. Then it delegates to other parts of the code the 1486 * responsibility of handling the above case 2. 1487 */ 1488static bool bfq_bfqq_update_budg_for_activation(struct bfq_data *bfqd, 1489 struct bfq_queue *bfqq, 1490 bool arrived_in_time) 1491{ 1492 struct bfq_entity *entity = &bfqq->entity; 1493 1494 /* 1495 * In the next compound condition, we check also whether there 1496 * is some budget left, because otherwise there is no point in 1497 * trying to go on serving bfqq with this same budget: bfqq 1498 * would be expired immediately after being selected for 1499 * service. This would only cause useless overhead. 1500 */ 1501 if (bfq_bfqq_non_blocking_wait_rq(bfqq) && arrived_in_time && 1502 bfq_bfqq_budget_left(bfqq) > 0) { 1503 /* 1504 * We do not clear the flag non_blocking_wait_rq here, as 1505 * the latter is used in bfq_activate_bfqq to signal 1506 * that timestamps need to be back-shifted (and is 1507 * cleared right after). 1508 */ 1509 1510 /* 1511 * In next assignment we rely on that either 1512 * entity->service or entity->budget are not updated 1513 * on expiration if bfqq is empty (see 1514 * __bfq_bfqq_recalc_budget). Thus both quantities 1515 * remain unchanged after such an expiration, and the 1516 * following statement therefore assigns to 1517 * entity->budget the remaining budget on such an 1518 * expiration. 1519 */ 1520 entity->budget = min_t(unsigned long, 1521 bfq_bfqq_budget_left(bfqq), 1522 bfqq->max_budget); 1523 1524 /* 1525 * At this point, we have used entity->service to get 1526 * the budget left (needed for updating 1527 * entity->budget). Thus we finally can, and have to, 1528 * reset entity->service. The latter must be reset 1529 * because bfqq would otherwise be charged again for 1530 * the service it has received during its previous 1531 * service slot(s). 1532 */ 1533 entity->service = 0; 1534 1535 return true; 1536 } 1537 1538 /* 1539 * We can finally complete expiration, by setting service to 0. 1540 */ 1541 entity->service = 0; 1542 entity->budget = max_t(unsigned long, bfqq->max_budget, 1543 bfq_serv_to_charge(bfqq->next_rq, bfqq)); 1544 bfq_clear_bfqq_non_blocking_wait_rq(bfqq); 1545 return false; 1546} 1547 1548/* 1549 * Return the farthest past time instant according to jiffies 1550 * macros. 1551 */ 1552static unsigned long bfq_smallest_from_now(void) 1553{ 1554 return jiffies - MAX_JIFFY_OFFSET; 1555} 1556 1557static void bfq_update_bfqq_wr_on_rq_arrival(struct bfq_data *bfqd, 1558 struct bfq_queue *bfqq, 1559 unsigned int old_wr_coeff, 1560 bool wr_or_deserves_wr, 1561 bool interactive, 1562 bool in_burst, 1563 bool soft_rt) 1564{ 1565 if (old_wr_coeff == 1 && wr_or_deserves_wr) { 1566 /* start a weight-raising period */ 1567 if (interactive) { 1568 bfqq->service_from_wr = 0; 1569 bfqq->wr_coeff = bfqd->bfq_wr_coeff; 1570 bfqq->wr_cur_max_time = bfq_wr_duration(bfqd); 1571 } else { 1572 /* 1573 * No interactive weight raising in progress 1574 * here: assign minus infinity to 1575 * wr_start_at_switch_to_srt, to make sure 1576 * that, at the end of the soft-real-time 1577 * weight raising periods that is starting 1578 * now, no interactive weight-raising period 1579 * may be wrongly considered as still in 1580 * progress (and thus actually started by 1581 * mistake). 1582 */ 1583 bfqq->wr_start_at_switch_to_srt = 1584 bfq_smallest_from_now(); 1585 bfqq->wr_coeff = bfqd->bfq_wr_coeff * 1586 BFQ_SOFTRT_WEIGHT_FACTOR; 1587 bfqq->wr_cur_max_time = 1588 bfqd->bfq_wr_rt_max_time; 1589 } 1590 1591 /* 1592 * If needed, further reduce budget to make sure it is 1593 * close to bfqq's backlog, so as to reduce the 1594 * scheduling-error component due to a too large 1595 * budget. Do not care about throughput consequences, 1596 * but only about latency. Finally, do not assign a 1597 * too small budget either, to avoid increasing 1598 * latency by causing too frequent expirations. 1599 */ 1600 bfqq->entity.budget = min_t(unsigned long, 1601 bfqq->entity.budget, 1602 2 * bfq_min_budget(bfqd)); 1603 } else if (old_wr_coeff > 1) { 1604 if (interactive) { /* update wr coeff and duration */ 1605 bfqq->wr_coeff = bfqd->bfq_wr_coeff; 1606 bfqq->wr_cur_max_time = bfq_wr_duration(bfqd); 1607 } else if (in_burst) 1608 bfqq->wr_coeff = 1; 1609 else if (soft_rt) { 1610 /* 1611 * The application is now or still meeting the 1612 * requirements for being deemed soft rt. We 1613 * can then correctly and safely (re)charge 1614 * the weight-raising duration for the 1615 * application with the weight-raising 1616 * duration for soft rt applications. 1617 * 1618 * In particular, doing this recharge now, i.e., 1619 * before the weight-raising period for the 1620 * application finishes, reduces the probability 1621 * of the following negative scenario: 1622 * 1) the weight of a soft rt application is 1623 * raised at startup (as for any newly 1624 * created application), 1625 * 2) since the application is not interactive, 1626 * at a certain time weight-raising is 1627 * stopped for the application, 1628 * 3) at that time the application happens to 1629 * still have pending requests, and hence 1630 * is destined to not have a chance to be 1631 * deemed soft rt before these requests are 1632 * completed (see the comments to the 1633 * function bfq_bfqq_softrt_next_start() 1634 * for details on soft rt detection), 1635 * 4) these pending requests experience a high 1636 * latency because the application is not 1637 * weight-raised while they are pending. 1638 */ 1639 if (bfqq->wr_cur_max_time != 1640 bfqd->bfq_wr_rt_max_time) { 1641 bfqq->wr_start_at_switch_to_srt = 1642 bfqq->last_wr_start_finish; 1643 1644 bfqq->wr_cur_max_time = 1645 bfqd->bfq_wr_rt_max_time; 1646 bfqq->wr_coeff = bfqd->bfq_wr_coeff * 1647 BFQ_SOFTRT_WEIGHT_FACTOR; 1648 } 1649 bfqq->last_wr_start_finish = jiffies; 1650 } 1651 } 1652} 1653 1654static bool bfq_bfqq_idle_for_long_time(struct bfq_data *bfqd, 1655 struct bfq_queue *bfqq) 1656{ 1657 return bfqq->dispatched == 0 && 1658 time_is_before_jiffies( 1659 bfqq->budget_timeout + 1660 bfqd->bfq_wr_min_idle_time); 1661} 1662 1663 1664/* 1665 * Return true if bfqq is in a higher priority class, or has a higher 1666 * weight than the in-service queue. 1667 */ 1668static bool bfq_bfqq_higher_class_or_weight(struct bfq_queue *bfqq, 1669 struct bfq_queue *in_serv_bfqq) 1670{ 1671 int bfqq_weight, in_serv_weight; 1672 1673 if (bfqq->ioprio_class < in_serv_bfqq->ioprio_class) 1674 return true; 1675 1676 if (in_serv_bfqq->entity.parent == bfqq->entity.parent) { 1677 bfqq_weight = bfqq->entity.weight; 1678 in_serv_weight = in_serv_bfqq->entity.weight; 1679 } else { 1680 if (bfqq->entity.parent) 1681 bfqq_weight = bfqq->entity.parent->weight; 1682 else 1683 bfqq_weight = bfqq->entity.weight; 1684 if (in_serv_bfqq->entity.parent) 1685 in_serv_weight = in_serv_bfqq->entity.parent->weight; 1686 else 1687 in_serv_weight = in_serv_bfqq->entity.weight; 1688 } 1689 1690 return bfqq_weight > in_serv_weight; 1691} 1692 1693static bool bfq_better_to_idle(struct bfq_queue *bfqq); 1694 1695static void bfq_bfqq_handle_idle_busy_switch(struct bfq_data *bfqd, 1696 struct bfq_queue *bfqq, 1697 int old_wr_coeff, 1698 struct request *rq, 1699 bool *interactive) 1700{ 1701 bool soft_rt, in_burst, wr_or_deserves_wr, 1702 bfqq_wants_to_preempt, 1703 idle_for_long_time = bfq_bfqq_idle_for_long_time(bfqd, bfqq), 1704 /* 1705 * See the comments on 1706 * bfq_bfqq_update_budg_for_activation for 1707 * details on the usage of the next variable. 1708 */ 1709 arrived_in_time = ktime_get_ns() <= 1710 bfqq->ttime.last_end_request + 1711 bfqd->bfq_slice_idle * 3; 1712 1713 1714 /* 1715 * bfqq deserves to be weight-raised if: 1716 * - it is sync, 1717 * - it does not belong to a large burst, 1718 * - it has been idle for enough time or is soft real-time, 1719 * - is linked to a bfq_io_cq (it is not shared in any sense), 1720 * - has a default weight (otherwise we assume the user wanted 1721 * to control its weight explicitly) 1722 */ 1723 in_burst = bfq_bfqq_in_large_burst(bfqq); 1724 soft_rt = bfqd->bfq_wr_max_softrt_rate > 0 && 1725 !BFQQ_TOTALLY_SEEKY(bfqq) && 1726 !in_burst && 1727 time_is_before_jiffies(bfqq->soft_rt_next_start) && 1728 bfqq->dispatched == 0 && 1729 bfqq->entity.new_weight == 40; 1730 *interactive = !in_burst && idle_for_long_time && 1731 bfqq->entity.new_weight == 40; 1732 wr_or_deserves_wr = bfqd->low_latency && 1733 (bfqq->wr_coeff > 1 || 1734 (bfq_bfqq_sync(bfqq) && 1735 bfqq->bic && (*interactive || soft_rt))); 1736 1737 /* 1738 * Using the last flag, update budget and check whether bfqq 1739 * may want to preempt the in-service queue. 1740 */ 1741 bfqq_wants_to_preempt = 1742 bfq_bfqq_update_budg_for_activation(bfqd, bfqq, 1743 arrived_in_time); 1744 1745 /* 1746 * If bfqq happened to be activated in a burst, but has been 1747 * idle for much more than an interactive queue, then we 1748 * assume that, in the overall I/O initiated in the burst, the 1749 * I/O associated with bfqq is finished. So bfqq does not need 1750 * to be treated as a queue belonging to a burst 1751 * anymore. Accordingly, we reset bfqq's in_large_burst flag 1752 * if set, and remove bfqq from the burst list if it's 1753 * there. We do not decrement burst_size, because the fact 1754 * that bfqq does not need to belong to the burst list any 1755 * more does not invalidate the fact that bfqq was created in 1756 * a burst. 1757 */ 1758 if (likely(!bfq_bfqq_just_created(bfqq)) && 1759 idle_for_long_time && 1760 time_is_before_jiffies( 1761 bfqq->budget_timeout + 1762 msecs_to_jiffies(10000))) { 1763 hlist_del_init(&bfqq->burst_list_node); 1764 bfq_clear_bfqq_in_large_burst(bfqq); 1765 } 1766 1767 bfq_clear_bfqq_just_created(bfqq); 1768 1769 if (bfqd->low_latency) { 1770 if (unlikely(time_is_after_jiffies(bfqq->split_time))) 1771 /* wraparound */ 1772 bfqq->split_time = 1773 jiffies - bfqd->bfq_wr_min_idle_time - 1; 1774 1775 if (time_is_before_jiffies(bfqq->split_time + 1776 bfqd->bfq_wr_min_idle_time)) { 1777 bfq_update_bfqq_wr_on_rq_arrival(bfqd, bfqq, 1778 old_wr_coeff, 1779 wr_or_deserves_wr, 1780 *interactive, 1781 in_burst, 1782 soft_rt); 1783 1784 if (old_wr_coeff != bfqq->wr_coeff) 1785 bfqq->entity.prio_changed = 1; 1786 } 1787 } 1788 1789 bfqq->last_idle_bklogged = jiffies; 1790 bfqq->service_from_backlogged = 0; 1791 bfq_clear_bfqq_softrt_update(bfqq); 1792 1793 bfq_add_bfqq_busy(bfqd, bfqq); 1794 1795 /* 1796 * Expire in-service queue if preemption may be needed for 1797 * guarantees or throughput. As for guarantees, we care 1798 * explicitly about two cases. The first is that bfqq has to 1799 * recover a service hole, as explained in the comments on 1800 * bfq_bfqq_update_budg_for_activation(), i.e., that 1801 * bfqq_wants_to_preempt is true. However, if bfqq does not 1802 * carry time-critical I/O, then bfqq's bandwidth is less 1803 * important than that of queues that carry time-critical I/O. 1804 * So, as a further constraint, we consider this case only if 1805 * bfqq is at least as weight-raised, i.e., at least as time 1806 * critical, as the in-service queue. 1807 * 1808 * The second case is that bfqq is in a higher priority class, 1809 * or has a higher weight than the in-service queue. If this 1810 * condition does not hold, we don't care because, even if 1811 * bfqq does not start to be served immediately, the resulting 1812 * delay for bfqq's I/O is however lower or much lower than 1813 * the ideal completion time to be guaranteed to bfqq's I/O. 1814 * 1815 * In both cases, preemption is needed only if, according to 1816 * the timestamps of both bfqq and of the in-service queue, 1817 * bfqq actually is the next queue to serve. So, to reduce 1818 * useless preemptions, the return value of 1819 * next_queue_may_preempt() is considered in the next compound 1820 * condition too. Yet next_queue_may_preempt() just checks a 1821 * simple, necessary condition for bfqq to be the next queue 1822 * to serve. In fact, to evaluate a sufficient condition, the 1823 * timestamps of the in-service queue would need to be 1824 * updated, and this operation is quite costly (see the 1825 * comments on bfq_bfqq_update_budg_for_activation()). 1826 * 1827 * As for throughput, we ask bfq_better_to_idle() whether we 1828 * still need to plug I/O dispatching. If bfq_better_to_idle() 1829 * says no, then plugging is not needed any longer, either to 1830 * boost throughput or to perserve service guarantees. Then 1831 * the best option is to stop plugging I/O, as not doing so 1832 * would certainly lower throughput. We may end up in this 1833 * case if: (1) upon a dispatch attempt, we detected that it 1834 * was better to plug I/O dispatch, and to wait for a new 1835 * request to arrive for the currently in-service queue, but 1836 * (2) this switch of bfqq to busy changes the scenario. 1837 */ 1838 if (bfqd->in_service_queue && 1839 ((bfqq_wants_to_preempt && 1840 bfqq->wr_coeff >= bfqd->in_service_queue->wr_coeff) || 1841 bfq_bfqq_higher_class_or_weight(bfqq, bfqd->in_service_queue) || 1842 !bfq_better_to_idle(bfqd->in_service_queue)) && 1843 next_queue_may_preempt(bfqd)) 1844 bfq_bfqq_expire(bfqd, bfqd->in_service_queue, 1845 false, BFQQE_PREEMPTED); 1846} 1847 1848static void bfq_reset_inject_limit(struct bfq_data *bfqd, 1849 struct bfq_queue *bfqq) 1850{ 1851 /* invalidate baseline total service time */ 1852 bfqq->last_serv_time_ns = 0; 1853 1854 /* 1855 * Reset pointer in case we are waiting for 1856 * some request completion. 1857 */ 1858 bfqd->waited_rq = NULL; 1859 1860 /* 1861 * If bfqq has a short think time, then start by setting the 1862 * inject limit to 0 prudentially, because the service time of 1863 * an injected I/O request may be higher than the think time 1864 * of bfqq, and therefore, if one request was injected when 1865 * bfqq remains empty, this injected request might delay the 1866 * service of the next I/O request for bfqq significantly. In 1867 * case bfqq can actually tolerate some injection, then the 1868 * adaptive update will however raise the limit soon. This 1869 * lucky circumstance holds exactly because bfqq has a short 1870 * think time, and thus, after remaining empty, is likely to 1871 * get new I/O enqueued---and then completed---before being 1872 * expired. This is the very pattern that gives the 1873 * limit-update algorithm the chance to measure the effect of 1874 * injection on request service times, and then to update the 1875 * limit accordingly. 1876 * 1877 * However, in the following special case, the inject limit is 1878 * left to 1 even if the think time is short: bfqq's I/O is 1879 * synchronized with that of some other queue, i.e., bfqq may 1880 * receive new I/O only after the I/O of the other queue is 1881 * completed. Keeping the inject limit to 1 allows the 1882 * blocking I/O to be served while bfqq is in service. And 1883 * this is very convenient both for bfqq and for overall 1884 * throughput, as explained in detail in the comments in 1885 * bfq_update_has_short_ttime(). 1886 * 1887 * On the opposite end, if bfqq has a long think time, then 1888 * start directly by 1, because: 1889 * a) on the bright side, keeping at most one request in 1890 * service in the drive is unlikely to cause any harm to the 1891 * latency of bfqq's requests, as the service time of a single 1892 * request is likely to be lower than the think time of bfqq; 1893 * b) on the downside, after becoming empty, bfqq is likely to 1894 * expire before getting its next request. With this request 1895 * arrival pattern, it is very hard to sample total service 1896 * times and update the inject limit accordingly (see comments 1897 * on bfq_update_inject_limit()). So the limit is likely to be 1898 * never, or at least seldom, updated. As a consequence, by 1899 * setting the limit to 1, we avoid that no injection ever 1900 * occurs with bfqq. On the downside, this proactive step 1901 * further reduces chances to actually compute the baseline 1902 * total service time. Thus it reduces chances to execute the 1903 * limit-update algorithm and possibly raise the limit to more 1904 * than 1. 1905 */ 1906 if (bfq_bfqq_has_short_ttime(bfqq)) 1907 bfqq->inject_limit = 0; 1908 else 1909 bfqq->inject_limit = 1; 1910 1911 bfqq->decrease_time_jif = jiffies; 1912} 1913 1914static void bfq_update_io_intensity(struct bfq_queue *bfqq, u64 now_ns) 1915{ 1916 u64 tot_io_time = now_ns - bfqq->io_start_time; 1917 1918 if (RB_EMPTY_ROOT(&bfqq->sort_list) && bfqq->dispatched == 0) 1919 bfqq->tot_idle_time += 1920 now_ns - bfqq->ttime.last_end_request; 1921 1922 if (unlikely(bfq_bfqq_just_created(bfqq))) 1923 return; 1924 1925 /* 1926 * Must be busy for at least about 80% of the time to be 1927 * considered I/O bound. 1928 */ 1929 if (bfqq->tot_idle_time * 5 > tot_io_time) 1930 bfq_clear_bfqq_IO_bound(bfqq); 1931 else 1932 bfq_mark_bfqq_IO_bound(bfqq); 1933 1934 /* 1935 * Keep an observation window of at most 200 ms in the past 1936 * from now. 1937 */ 1938 if (tot_io_time > 200 * NSEC_PER_MSEC) { 1939 bfqq->io_start_time = now_ns - (tot_io_time>>1); 1940 bfqq->tot_idle_time >>= 1; 1941 } 1942} 1943 1944/* 1945 * Detect whether bfqq's I/O seems synchronized with that of some 1946 * other queue, i.e., whether bfqq, after remaining empty, happens to 1947 * receive new I/O only right after some I/O request of the other 1948 * queue has been completed. We call waker queue the other queue, and 1949 * we assume, for simplicity, that bfqq may have at most one waker 1950 * queue. 1951 * 1952 * A remarkable throughput boost can be reached by unconditionally 1953 * injecting the I/O of the waker queue, every time a new 1954 * bfq_dispatch_request happens to be invoked while I/O is being 1955 * plugged for bfqq. In addition to boosting throughput, this 1956 * unblocks bfqq's I/O, thereby improving bandwidth and latency for 1957 * bfqq. Note that these same results may be achieved with the general 1958 * injection mechanism, but less effectively. For details on this 1959 * aspect, see the comments on the choice of the queue for injection 1960 * in bfq_select_queue(). 1961 * 1962 * Turning back to the detection of a waker queue, a queue Q is deemed 1963 * as a waker queue for bfqq if, for three consecutive times, bfqq 1964 * happens to become non empty right after a request of Q has been 1965 * completed. In particular, on the first time, Q is tentatively set 1966 * as a candidate waker queue, while on the third consecutive time 1967 * that Q is detected, the field waker_bfqq is set to Q, to confirm 1968 * that Q is a waker queue for bfqq. These detection steps are 1969 * performed only if bfqq has a long think time, so as to make it more 1970 * likely that bfqq's I/O is actually being blocked by a 1971 * synchronization. This last filter, plus the above three-times 1972 * requirement, make false positives less likely. 1973 * 1974 * NOTE 1975 * 1976 * The sooner a waker queue is detected, the sooner throughput can be 1977 * boosted by injecting I/O from the waker queue. Fortunately, 1978 * detection is likely to be actually fast, for the following 1979 * reasons. While blocked by synchronization, bfqq has a long think 1980 * time. This implies that bfqq's inject limit is at least equal to 1 1981 * (see the comments in bfq_update_inject_limit()). So, thanks to 1982 * injection, the waker queue is likely to be served during the very 1983 * first I/O-plugging time interval for bfqq. This triggers the first 1984 * step of the detection mechanism. Thanks again to injection, the 1985 * candidate waker queue is then likely to be confirmed no later than 1986 * during the next I/O-plugging interval for bfqq. 1987 * 1988 * ISSUE 1989 * 1990 * On queue merging all waker information is lost. 1991 */ 1992static void bfq_check_waker(struct bfq_data *bfqd, struct bfq_queue *bfqq, 1993 u64 now_ns) 1994{ 1995 if (!bfqd->last_completed_rq_bfqq || 1996 bfqd->last_completed_rq_bfqq == bfqq || 1997 bfq_bfqq_has_short_ttime(bfqq) || 1998 now_ns - bfqd->last_completion >= 4 * NSEC_PER_MSEC || 1999 bfqd->last_completed_rq_bfqq == bfqq->waker_bfqq) 2000 return;
2001 2002 if (bfqd->last_completed_rq_bfqq != 2003 bfqq->tentative_waker_bfqq) { 2004 /* 2005 * First synchronization detected with a 2006 * candidate waker queue, or with a different 2007 * candidate waker queue from the current one. 2008 */ 2009 bfqq->tentative_waker_bfqq = 2010 bfqd->last_completed_rq_bfqq; 2011 bfqq->num_waker_detections = 1; 2012 } else /* Same tentative waker queue detected again */ 2013 bfqq->num_waker_detections++; 2014 2015 if (bfqq->num_waker_detections == 3) { 2016 bfqq->waker_bfqq = bfqd->last_completed_rq_bfqq; 2017 bfqq->tentative_waker_bfqq = NULL; 2018 2019 /* 2020 * If the waker queue disappears, then 2021 * bfqq->waker_bfqq must be reset. To 2022 * this goal, we maintain in each 2023 * waker queue a list, woken_list, of 2024 * all the queues that reference the 2025 * waker queue through their 2026 * waker_bfqq pointer. When the waker 2027 * queue exits, the waker_bfqq pointer 2028 * of all the queues in the woken_list 2029 * is reset. 2030 * 2031 * In addition, if bfqq is already in 2032 * the woken_list of a waker queue, 2033 * then, before being inserted into 2034 * the woken_list of a new waker 2035 * queue, bfqq must be removed from 2036 * the woken_list of the old waker 2037 * queue. 2038 */ 2039 if (!hlist_unhashed(&bfqq->woken_list_node)) 2040 hlist_del_init(&bfqq->woken_list_node); 2041 hlist_add_head(&bfqq->woken_list_node, 2042 &bfqd->last_completed_rq_bfqq->woken_list); 2043 } 2044} 2045 2046static void bfq_add_request(struct request *rq) 2047{ 2048 struct bfq_queue *bfqq = RQ_BFQQ(rq); 2049 struct bfq_data *bfqd = bfqq->bfqd; 2050 struct request *next_rq, *prev; 2051 unsigned int old_wr_coeff = bfqq->wr_coeff; 2052 bool interactive = false; 2053 u64 now_ns = ktime_get_ns(); 2054 2055 bfq_log_bfqq(bfqd, bfqq, "add_request %d", rq_is_sync(rq)); 2056 bfqq->queued[rq_is_sync(rq)]++; 2057 bfqd->queued++; 2058 2059 if (RB_EMPTY_ROOT(&bfqq->sort_list) && bfq_bfqq_sync(bfqq)) { 2060 bfq_check_waker(bfqd, bfqq, now_ns); 2061 2062 /* 2063 * Periodically reset inject limit, to make sure that 2064 * the latter eventually drops in case workload 2065 * changes, see step (3) in the comments on 2066 * bfq_update_inject_limit(). 2067 */ 2068 if (time_is_before_eq_jiffies(bfqq->decrease_time_jif + 2069 msecs_to_jiffies(1000))) 2070 bfq_reset_inject_limit(bfqd, bfqq); 2071 2072 /* 2073 * The following conditions must hold to setup a new 2074 * sampling of total service time, and then a new 2075 * update of the inject limit: 2076 * - bfqq is in service, because the total service 2077 * time is evaluated only for the I/O requests of 2078 * the queues in service; 2079 * - this is the right occasion to compute or to 2080 * lower the baseline total service time, because 2081 * there are actually no requests in the drive, 2082 * or 2083 * the baseline total service time is available, and 2084 * this is the right occasion to compute the other 2085 * quantity needed to update the inject limit, i.e., 2086 * the total service time caused by the amount of 2087 * injection allowed by the current value of the 2088 * limit. It is the right occasion because injection 2089 * has actually been performed during the service 2090 * hole, and there are still in-flight requests, 2091 * which are very likely to be exactly the injected 2092 * requests, or part of them; 2093 * - the minimum interval for sampling the total 2094 * service time and updating the inject limit has 2095 * elapsed. 2096 */ 2097 if (bfqq == bfqd->in_service_queue && 2098 (bfqd->rq_in_driver == 0 || 2099 (bfqq->last_serv_time_ns > 0 && 2100 bfqd->rqs_injected && bfqd->rq_in_driver > 0)) && 2101 time_is_before_eq_jiffies(bfqq->decrease_time_jif + 2102 msecs_to_jiffies(10))) { 2103 bfqd->last_empty_occupied_ns = ktime_get_ns(); 2104 /* 2105 * Start the state machine for measuring the 2106 * total service time of rq: setting 2107 * wait_dispatch will cause bfqd->waited_rq to 2108 * be set when rq will be dispatched. 2109 */ 2110 bfqd->wait_dispatch = true; 2111 /* 2112 * If there is no I/O in service in the drive, 2113 * then possible injection occurred before the 2114 * arrival of rq will not affect the total 2115 * service time of rq. So the injection limit 2116 * must not be updated as a function of such 2117 * total service time, unless new injection 2118 * occurs before rq is completed. To have the 2119 * injection limit updated only in the latter 2120 * case, reset rqs_injected here (rqs_injected 2121 * will be set in case injection is performed 2122 * on bfqq before rq is completed). 2123 */ 2124 if (bfqd->rq_in_driver == 0) 2125 bfqd->rqs_injected = false; 2126 } 2127 } 2128 2129 if (bfq_bfqq_sync(bfqq)) 2130 bfq_update_io_intensity(bfqq, now_ns); 2131 2132 elv_rb_add(&bfqq->sort_list, rq); 2133 2134 /* 2135 * Check if this request is a better next-serve candidate. 2136 */ 2137 prev = bfqq->next_rq; 2138 next_rq = bfq_choose_req(bfqd, bfqq->next_rq, rq, bfqd->last_position); 2139 bfqq->next_rq = next_rq; 2140 2141 /* 2142 * Adjust priority tree position, if next_rq changes. 2143 * See comments on bfq_pos_tree_add_move() for the unlikely(). 2144 */ 2145 if (unlikely(!bfqd->nonrot_with_queueing && prev != bfqq->next_rq)) 2146 bfq_pos_tree_add_move(bfqd, bfqq); 2147 2148 if (!bfq_bfqq_busy(bfqq)) /* switching to busy ... */ 2149 bfq_bfqq_handle_idle_busy_switch(bfqd, bfqq, old_wr_coeff, 2150 rq, &interactive); 2151 else { 2152 if (bfqd->low_latency && old_wr_coeff == 1 && !rq_is_sync(rq) && 2153 time_is_before_jiffies( 2154 bfqq->last_wr_start_finish + 2155 bfqd->bfq_wr_min_inter_arr_async)) { 2156 bfqq->wr_coeff = bfqd->bfq_wr_coeff; 2157 bfqq->wr_cur_max_time = bfq_wr_duration(bfqd); 2158 2159 bfqd->wr_busy_queues++; 2160 bfqq->entity.prio_changed = 1; 2161 } 2162 if (prev != bfqq->next_rq) 2163 bfq_updated_next_req(bfqd, bfqq); 2164 } 2165 2166 /* 2167 * Assign jiffies to last_wr_start_finish in the following 2168 * cases: 2169 * 2170 * . if bfqq is not going to be weight-raised, because, for 2171 * non weight-raised queues, last_wr_start_finish stores the 2172 * arrival time of the last request; as of now, this piece 2173 * of information is used only for deciding whether to 2174 * weight-raise async queues 2175 * 2176 * . if bfqq is not weight-raised, because, if bfqq is now 2177 * switching to weight-raised, then last_wr_start_finish 2178 * stores the time when weight-raising starts 2179 * 2180 * . if bfqq is interactive, because, regardless of whether 2181 * bfqq is currently weight-raised, the weight-raising 2182 * period must start or restart (this case is considered 2183 * separately because it is not detected by the above 2184 * conditions, if bfqq is already weight-raised) 2185 * 2186 * last_wr_start_finish has to be updated also if bfqq is soft 2187 * real-time, because the weight-raising period is constantly 2188 * restarted on idle-to-busy transitions for these queues, but 2189 * this is already done in bfq_bfqq_handle_idle_busy_switch if 2190 * needed. 2191 */ 2192 if (bfqd->low_latency && 2193 (old_wr_coeff == 1 || bfqq->wr_coeff == 1 || interactive)) 2194 bfqq->last_wr_start_finish = jiffies; 2195} 2196 2197static struct request *bfq_find_rq_fmerge(struct bfq_data *bfqd, 2198 struct bio *bio, 2199 struct request_queue *q) 2200{ 2201 struct bfq_queue *bfqq = bfqd->bio_bfqq; 2202 2203 2204 if (bfqq) 2205 return elv_rb_find(&bfqq->sort_list, bio_end_sector(bio)); 2206 2207 return NULL; 2208} 2209 2210static sector_t get_sdist(sector_t last_pos, struct request *rq) 2211{ 2212 if (last_pos) 2213 return abs(blk_rq_pos(rq) - last_pos); 2214 2215 return 0; 2216} 2217 2218#if 0 /* Still not clear if we can do without next two functions */ 2219static void bfq_activate_request(struct request_queue *q, struct request *rq) 2220{ 2221 struct bfq_data *bfqd = q->elevator->elevator_data; 2222 2223 bfqd->rq_in_driver++; 2224} 2225 2226static void bfq_deactivate_request(struct request_queue *q, struct request *rq) 2227{ 2228 struct bfq_data *bfqd = q->elevator->elevator_data; 2229 2230 bfqd->rq_in_driver--; 2231} 2232#endif 2233 2234static void bfq_remove_request(struct request_queue *q, 2235 struct request *rq) 2236{ 2237 struct bfq_queue *bfqq = RQ_BFQQ(rq); 2238 struct bfq_data *bfqd = bfqq->bfqd; 2239 const int sync = rq_is_sync(rq); 2240 2241 if (bfqq->next_rq == rq) { 2242 bfqq->next_rq = bfq_find_next_rq(bfqd, bfqq, rq); 2243 bfq_updated_next_req(bfqd, bfqq); 2244 } 2245 2246 if (rq->queuelist.prev != &rq->queuelist) 2247 list_del_init(&rq->queuelist); 2248 bfqq->queued[sync]--; 2249 bfqd->queued--; 2250 elv_rb_del(&bfqq->sort_list, rq); 2251 2252 elv_rqhash_del(q, rq); 2253 if (q->last_merge == rq) 2254 q->last_merge = NULL; 2255 2256 if (RB_EMPTY_ROOT(&bfqq->sort_list)) { 2257 bfqq->next_rq = NULL; 2258 2259 if (bfq_bfqq_busy(bfqq) && bfqq != bfqd->in_service_queue) { 2260 bfq_del_bfqq_busy(bfqd, bfqq, false); 2261 /* 2262 * bfqq emptied. In normal operation, when 2263 * bfqq is empty, bfqq->entity.service and 2264 * bfqq->entity.budget must contain, 2265 * respectively, the service received and the 2266 * budget used last time bfqq emptied. These 2267 * facts do not hold in this case, as at least 2268 * this last removal occurred while bfqq is 2269 * not in service. To avoid inconsistencies, 2270 * reset both bfqq->entity.service and 2271 * bfqq->entity.budget, if bfqq has still a 2272 * process that may issue I/O requests to it. 2273 */ 2274 bfqq->entity.budget = bfqq->entity.service = 0; 2275 } 2276 2277 /* 2278 * Remove queue from request-position tree as it is empty. 2279 */ 2280 if (bfqq->pos_root) { 2281 rb_erase(&bfqq->pos_node, bfqq->pos_root); 2282 bfqq->pos_root = NULL; 2283 } 2284 } else { 2285 /* see comments on bfq_pos_tree_add_move() for the unlikely() */ 2286 if (unlikely(!bfqd->nonrot_with_queueing)) 2287 bfq_pos_tree_add_move(bfqd, bfqq); 2288 } 2289 2290 if (rq->cmd_flags & REQ_META) 2291 bfqq->meta_pending--; 2292 2293} 2294 2295static bool bfq_bio_merge(struct request_queue *q, struct bio *bio, 2296 unsigned int nr_segs) 2297{ 2298 struct bfq_data *bfqd = q->elevator->elevator_data; 2299 struct request *free = NULL; 2300 /* 2301 * bfq_bic_lookup grabs the queue_lock: invoke it now and 2302 * store its return value for later use, to avoid nesting 2303 * queue_lock inside the bfqd->lock. We assume that the bic 2304 * returned by bfq_bic_lookup does not go away before 2305 * bfqd->lock is taken. 2306 */ 2307 struct bfq_io_cq *bic = bfq_bic_lookup(bfqd, current->io_context, q); 2308 bool ret; 2309 2310 spin_lock_irq(&bfqd->lock); 2311 2312 if (bic) 2313 bfqd->bio_bfqq = bic_to_bfqq(bic, op_is_sync(bio->bi_opf)); 2314 else 2315 bfqd->bio_bfqq = NULL; 2316 bfqd->bio_bic = bic; 2317 2318 ret = blk_mq_sched_try_merge(q, bio, nr_segs, &free); 2319 2320 if (free) 2321 blk_mq_free_request(free); 2322 spin_unlock_irq(&bfqd->lock); 2323 2324 return ret; 2325} 2326 2327static int bfq_request_merge(struct request_queue *q, struct request **req, 2328 struct bio *bio) 2329{ 2330 struct bfq_data *bfqd = q->elevator->elevator_data; 2331 struct request *__rq; 2332 2333 __rq = bfq_find_rq_fmerge(bfqd, bio, q); 2334 if (__rq && elv_bio_merge_ok(__rq, bio)) { 2335 *req = __rq; 2336 return ELEVATOR_FRONT_MERGE; 2337 } 2338 2339 return ELEVATOR_NO_MERGE; 2340} 2341 2342static struct bfq_queue *bfq_init_rq(struct request *rq); 2343 2344static void bfq_request_merged(struct request_queue *q, struct request *req, 2345 enum elv_merge type) 2346{ 2347 if (type == ELEVATOR_FRONT_MERGE && 2348 rb_prev(&req->rb_node) && 2349 blk_rq_pos(req) < 2350 blk_rq_pos(container_of(rb_prev(&req->rb_node), 2351 struct request, rb_node))) { 2352 struct bfq_queue *bfqq = bfq_init_rq(req); 2353 struct bfq_data *bfqd; 2354 struct request *prev, *next_rq; 2355 2356 if (!bfqq) 2357 return; 2358 2359 bfqd = bfqq->bfqd; 2360 2361 /* Reposition request in its sort_list */ 2362 elv_rb_del(&bfqq->sort_list, req); 2363 elv_rb_add(&bfqq->sort_list, req); 2364 2365 /* Choose next request to be served for bfqq */ 2366 prev = bfqq->next_rq; 2367 next_rq = bfq_choose_req(bfqd, bfqq->next_rq, req, 2368 bfqd->last_position); 2369 bfqq->next_rq = next_rq; 2370 /* 2371 * If next_rq changes, update both the queue's budget to 2372 * fit the new request and the queue's position in its 2373 * rq_pos_tree. 2374 */ 2375 if (prev != bfqq->next_rq) { 2376 bfq_updated_next_req(bfqd, bfqq); 2377 /* 2378 * See comments on bfq_pos_tree_add_move() for 2379 * the unlikely(). 2380 */ 2381 if (unlikely(!bfqd->nonrot_with_queueing)) 2382 bfq_pos_tree_add_move(bfqd, bfqq); 2383 } 2384 } 2385} 2386 2387/* 2388 * This function is called to notify the scheduler that the requests 2389 * rq and 'next' have been merged, with 'next' going away. BFQ 2390 * exploits this hook to address the following issue: if 'next' has a 2391 * fifo_time lower that rq, then the fifo_time of rq must be set to 2392 * the value of 'next', to not forget the greater age of 'next'. 2393 * 2394 * NOTE: in this function we assume that rq is in a bfq_queue, basing 2395 * on that rq is picked from the hash table q->elevator->hash, which, 2396 * in its turn, is filled only with I/O requests present in 2397 * bfq_queues, while BFQ is in use for the request queue q. In fact, 2398 * the function that fills this hash table (elv_rqhash_add) is called 2399 * only by bfq_insert_request. 2400 */ 2401static void bfq_requests_merged(struct request_queue *q, struct request *rq, 2402 struct request *next) 2403{ 2404 struct bfq_queue *bfqq = bfq_init_rq(rq), 2405 *next_bfqq = bfq_init_rq(next); 2406 2407 if (!bfqq) 2408 return; 2409 2410 /* 2411 * If next and rq belong to the same bfq_queue and next is older 2412 * than rq, then reposition rq in the fifo (by substituting next 2413 * with rq). Otherwise, if next and rq belong to different 2414 * bfq_queues, never reposition rq: in fact, we would have to 2415 * reposition it with respect to next's position in its own fifo, 2416 * which would most certainly be too expensive with respect to 2417 * the benefits. 2418 */ 2419 if (bfqq == next_bfqq && 2420 !list_empty(&rq->queuelist) && !list_empty(&next->queuelist) && 2421 next->fifo_time < rq->fifo_time) { 2422 list_del_init(&rq->queuelist); 2423 list_replace_init(&next->queuelist, &rq->queuelist); 2424 rq->fifo_time = next->fifo_time; 2425 } 2426 2427 if (bfqq->next_rq == next) 2428 bfqq->next_rq = rq; 2429 2430 bfqg_stats_update_io_merged(bfqq_group(bfqq), next->cmd_flags); 2431} 2432 2433/* Must be called with bfqq != NULL */ 2434static void bfq_bfqq_end_wr(struct bfq_queue *bfqq) 2435{ 2436 /* 2437 * If bfqq has been enjoying interactive weight-raising, then 2438 * reset soft_rt_next_start. We do it for the following 2439 * reason. bfqq may have been conveying the I/O needed to load 2440 * a soft real-time application. Such an application actually 2441 * exhibits a soft real-time I/O pattern after it finishes 2442 * loading, and finally starts doing its job. But, if bfqq has 2443 * been receiving a lot of bandwidth so far (likely to happen 2444 * on a fast device), then soft_rt_next_start now contains a 2445 * high value that. So, without this reset, bfqq would be 2446 * prevented from being possibly considered as soft_rt for a 2447 * very long time. 2448 */ 2449 2450 if (bfqq->wr_cur_max_time != 2451 bfqq->bfqd->bfq_wr_rt_max_time) 2452 bfqq->soft_rt_next_start = jiffies; 2453 2454 if (bfq_bfqq_busy(bfqq)) 2455 bfqq->bfqd->wr_busy_queues--; 2456 bfqq->wr_coeff = 1; 2457 bfqq->wr_cur_max_time = 0; 2458 bfqq->last_wr_start_finish = jiffies; 2459 /* 2460 * Trigger a weight change on the next invocation of 2461 * __bfq_entity_update_weight_prio. 2462 */ 2463 bfqq->entity.prio_changed = 1; 2464} 2465 2466void bfq_end_wr_async_queues(struct bfq_data *bfqd, 2467 struct bfq_group *bfqg) 2468{ 2469 int i, j; 2470 2471 for (i = 0; i < 2; i++) 2472 for (j = 0; j < IOPRIO_BE_NR; j++) 2473 if (bfqg->async_bfqq[i][j]) 2474 bfq_bfqq_end_wr(bfqg->async_bfqq[i][j]); 2475 if (bfqg->async_idle_bfqq) 2476 bfq_bfqq_end_wr(bfqg->async_idle_bfqq); 2477} 2478 2479static void bfq_end_wr(struct bfq_data *bfqd) 2480{ 2481 struct bfq_queue *bfqq; 2482 2483 spin_lock_irq(&bfqd->lock); 2484 2485 list_for_each_entry(bfqq, &bfqd->active_list, bfqq_list) 2486 bfq_bfqq_end_wr(bfqq); 2487 list_for_each_entry(bfqq, &bfqd->idle_list, bfqq_list) 2488 bfq_bfqq_end_wr(bfqq); 2489 bfq_end_wr_async(bfqd); 2490 2491 spin_unlock_irq(&bfqd->lock); 2492} 2493 2494static sector_t bfq_io_struct_pos(void *io_struct, bool request) 2495{ 2496 if (request) 2497 return blk_rq_pos(io_struct); 2498 else 2499 return ((struct bio *)io_struct)->bi_iter.bi_sector; 2500} 2501 2502static int bfq_rq_close_to_sector(void *io_struct, bool request, 2503 sector_t sector) 2504{ 2505 return abs(bfq_io_struct_pos(io_struct, request) - sector) <= 2506 BFQQ_CLOSE_THR; 2507} 2508 2509static struct bfq_queue *bfqq_find_close(struct bfq_data *bfqd, 2510 struct bfq_queue *bfqq, 2511 sector_t sector) 2512{ 2513 struct rb_root *root = &bfq_bfqq_to_bfqg(bfqq)->rq_pos_tree; 2514 struct rb_node *parent, *node; 2515 struct bfq_queue *__bfqq; 2516 2517 if (RB_EMPTY_ROOT(root)) 2518 return NULL; 2519 2520 /* 2521 * First, if we find a request starting at the end of the last 2522 * request, choose it. 2523 */ 2524 __bfqq = bfq_rq_pos_tree_lookup(bfqd, root, sector, &parent, NULL); 2525 if (__bfqq) 2526 return __bfqq; 2527 2528 /* 2529 * If the exact sector wasn't found, the parent of the NULL leaf 2530 * will contain the closest sector (rq_pos_tree sorted by 2531 * next_request position). 2532 */ 2533 __bfqq = rb_entry(parent, struct bfq_queue, pos_node); 2534 if (bfq_rq_close_to_sector(__bfqq->next_rq, true, sector)) 2535 return __bfqq; 2536 2537 if (blk_rq_pos(__bfqq->next_rq) < sector) 2538 node = rb_next(&__bfqq->pos_node); 2539 else 2540 node = rb_prev(&__bfqq->pos_node); 2541 if (!node) 2542 return NULL; 2543 2544 __bfqq = rb_entry(node, struct bfq_queue, pos_node); 2545 if (bfq_rq_close_to_sector(__bfqq->next_rq, true, sector)) 2546 return __bfqq; 2547 2548 return NULL; 2549} 2550 2551static struct bfq_queue *bfq_find_close_cooperator(struct bfq_data *bfqd, 2552 struct bfq_queue *cur_bfqq, 2553 sector_t sector) 2554{ 2555 struct bfq_queue *bfqq; 2556 2557 /* 2558 * We shall notice if some of the queues are cooperating, 2559 * e.g., working closely on the same area of the device. In 2560 * that case, we can group them together and: 1) don't waste 2561 * time idling, and 2) serve the union of their requests in 2562 * the best possible order for throughput. 2563 */ 2564 bfqq = bfqq_find_close(bfqd, cur_bfqq, sector); 2565 if (!bfqq || bfqq == cur_bfqq) 2566 return NULL; 2567 2568 return bfqq; 2569} 2570 2571static struct bfq_queue * 2572bfq_setup_merge(struct bfq_queue *bfqq, struct bfq_queue *new_bfqq) 2573{ 2574 int process_refs, new_process_refs; 2575 struct bfq_queue *__bfqq; 2576 2577 /* 2578 * If there are no process references on the new_bfqq, then it is 2579 * unsafe to follow the ->new_bfqq chain as other bfqq's in the chain 2580 * may have dropped their last reference (not just their last process 2581 * reference). 2582 */ 2583 if (!bfqq_process_refs(new_bfqq)) 2584 return NULL; 2585 2586 /* Avoid a circular list and skip interim queue merges. */ 2587 while ((__bfqq = new_bfqq->new_bfqq)) { 2588 if (__bfqq == bfqq) 2589 return NULL; 2590 new_bfqq = __bfqq; 2591 } 2592 2593 process_refs = bfqq_process_refs(bfqq); 2594 new_process_refs = bfqq_process_refs(new_bfqq); 2595 /* 2596 * If the process for the bfqq has gone away, there is no 2597 * sense in merging the queues. 2598 */ 2599 if (process_refs == 0 || new_process_refs == 0) 2600 return NULL; 2601 2602 bfq_log_bfqq(bfqq->bfqd, bfqq, "scheduling merge with queue %d", 2603 new_bfqq->pid); 2604 2605 /* 2606 * Merging is just a redirection: the requests of the process 2607 * owning one of the two queues are redirected to the other queue. 2608 * The latter queue, in its turn, is set as shared if this is the 2609 * first time that the requests of some process are redirected to 2610 * it. 2611 * 2612 * We redirect bfqq to new_bfqq and not the opposite, because 2613 * we are in the context of the process owning bfqq, thus we 2614 * have the io_cq of this process. So we can immediately 2615 * configure this io_cq to redirect the requests of the 2616 * process to new_bfqq. In contrast, the io_cq of new_bfqq is 2617 * not available any more (new_bfqq->bic == NULL). 2618 * 2619 * Anyway, even in case new_bfqq coincides with the in-service 2620 * queue, redirecting requests the in-service queue is the 2621 * best option, as we feed the in-service queue with new 2622 * requests close to the last request served and, by doing so, 2623 * are likely to increase the throughput. 2624 */ 2625 bfqq->new_bfqq = new_bfqq; 2626 new_bfqq->ref += process_refs; 2627 return new_bfqq; 2628} 2629 2630static bool bfq_may_be_close_cooperator(struct bfq_queue *bfqq, 2631 struct bfq_queue *new_bfqq) 2632{ 2633 if (bfq_too_late_for_merging(new_bfqq)) 2634 return false; 2635 2636 if (bfq_class_idle(bfqq) || bfq_class_idle(new_bfqq) || 2637 (bfqq->ioprio_class != new_bfqq->ioprio_class)) 2638 return false; 2639 2640 /* 2641 * If either of the queues has already been detected as seeky, 2642 * then merging it with the other queue is unlikely to lead to 2643 * sequential I/O. 2644 */ 2645 if (BFQQ_SEEKY(bfqq) || BFQQ_SEEKY(new_bfqq)) 2646 return false; 2647 2648 /* 2649 * Interleaved I/O is known to be done by (some) applications 2650 * only for reads, so it does not make sense to merge async 2651 * queues. 2652 */ 2653 if (!bfq_bfqq_sync(bfqq) || !bfq_bfqq_sync(new_bfqq)) 2654 return false; 2655 2656 return true; 2657} 2658 2659static bool idling_boosts_thr_without_issues(struct bfq_data *bfqd, 2660 struct bfq_queue *bfqq); 2661 2662/* 2663 * Attempt to schedule a merge of bfqq with the currently in-service 2664 * queue or with a close queue among the scheduled queues. Return 2665 * NULL if no merge was scheduled, a pointer to the shared bfq_queue 2666 * structure otherwise. 2667 * 2668 * The OOM queue is not allowed to participate to cooperation: in fact, since 2669 * the requests temporarily redirected to the OOM queue could be redirected 2670 * again to dedicated queues at any time, the state needed to correctly 2671 * handle merging with the OOM queue would be quite complex and expensive 2672 * to maintain. Besides, in such a critical condition as an out of memory, 2673 * the benefits of queue merging may be little relevant, or even negligible. 2674 * 2675 * WARNING: queue merging may impair fairness among non-weight raised 2676 * queues, for at least two reasons: 1) the original weight of a 2677 * merged queue may change during the merged state, 2) even being the 2678 * weight the same, a merged queue may be bloated with many more 2679 * requests than the ones produced by its originally-associated 2680 * process. 2681 */ 2682static struct bfq_queue * 2683bfq_setup_cooperator(struct bfq_data *bfqd, struct bfq_queue *bfqq, 2684 void *io_struct, bool request, struct bfq_io_cq *bic) 2685{ 2686 struct bfq_queue *in_service_bfqq, *new_bfqq; 2687 2688 /* 2689 * Check delayed stable merge for rotational or non-queueing 2690 * devs. For this branch to be executed, bfqq must not be 2691 * currently merged with some other queue (i.e., bfqq->bic 2692 * must be non null). If we considered also merged queues, 2693 * then we should also check whether bfqq has already been 2694 * merged with bic->stable_merge_bfqq. But this would be 2695 * costly and complicated. 2696 */ 2697 if (unlikely(!bfqd->nonrot_with_queueing)) { 2698 /* 2699 * Make sure also that bfqq is sync, because 2700 * bic->stable_merge_bfqq may point to some queue (for 2701 * stable merging) also if bic is associated with a 2702 * sync queue, but this bfqq is async 2703 */ 2704 if (bfq_bfqq_sync(bfqq) && bic->stable_merge_bfqq && 2705 !bfq_bfqq_just_created(bfqq) && 2706 time_is_before_jiffies(bfqq->split_time + 2707 msecs_to_jiffies(200))) { 2708 struct bfq_queue *stable_merge_bfqq = 2709 bic->stable_merge_bfqq; 2710 int proc_ref = min(bfqq_process_refs(bfqq), 2711 bfqq_process_refs(stable_merge_bfqq)); 2712 2713 /* deschedule stable merge, because done or aborted here */ 2714 bfq_put_stable_ref(stable_merge_bfqq); 2715 2716 bic->stable_merge_bfqq = NULL; 2717 2718 if (!idling_boosts_thr_without_issues(bfqd, bfqq) && 2719 proc_ref > 0) { 2720 /* next function will take at least one ref */ 2721 struct bfq_queue *new_bfqq = 2722 bfq_setup_merge(bfqq, stable_merge_bfqq); 2723 2724 bic->stably_merged = true; 2725 if (new_bfqq && new_bfqq->bic) 2726 new_bfqq->bic->stably_merged = true; 2727 return new_bfqq; 2728 } else 2729 return NULL; 2730 } 2731 } 2732 2733 /* 2734 * Do not perform queue merging if the device is non 2735 * rotational and performs internal queueing. In fact, such a 2736 * device reaches a high speed through internal parallelism 2737 * and pipelining. This means that, to reach a high 2738 * throughput, it must have many requests enqueued at the same 2739 * time. But, in this configuration, the internal scheduling 2740 * algorithm of the device does exactly the job of queue 2741 * merging: it reorders requests so as to obtain as much as 2742 * possible a sequential I/O pattern. As a consequence, with 2743 * the workload generated by processes doing interleaved I/O, 2744 * the throughput reached by the device is likely to be the 2745 * same, with and without queue merging. 2746 * 2747 * Disabling merging also provides a remarkable benefit in 2748 * terms of throughput. Merging tends to make many workloads 2749 * artificially more uneven, because of shared queues 2750 * remaining non empty for incomparably more time than 2751 * non-merged queues. This may accentuate workload 2752 * asymmetries. For example, if one of the queues in a set of 2753 * merged queues has a higher weight than a normal queue, then 2754 * the shared queue may inherit such a high weight and, by 2755 * staying almost always active, may force BFQ to perform I/O 2756 * plugging most of the time. This evidently makes it harder 2757 * for BFQ to let the device reach a high throughput. 2758 * 2759 * Finally, the likely() macro below is not used because one 2760 * of the two branches is more likely than the other, but to 2761 * have the code path after the following if() executed as 2762 * fast as possible for the case of a non rotational device 2763 * with queueing. We want it because this is the fastest kind 2764 * of device. On the opposite end, the likely() may lengthen 2765 * the execution time of BFQ for the case of slower devices 2766 * (rotational or at least without queueing). But in this case 2767 * the execution time of BFQ matters very little, if not at 2768 * all. 2769 */ 2770 if (likely(bfqd->nonrot_with_queueing)) 2771 return NULL; 2772 2773 /* 2774 * Prevent bfqq from being merged if it has been created too 2775 * long ago. The idea is that true cooperating processes, and 2776 * thus their associated bfq_queues, are supposed to be 2777 * created shortly after each other. This is the case, e.g., 2778 * for KVM/QEMU and dump I/O threads. Basing on this 2779 * assumption, the following filtering greatly reduces the 2780 * probability that two non-cooperating processes, which just 2781 * happen to do close I/O for some short time interval, have 2782 * their queues merged by mistake. 2783 */ 2784 if (bfq_too_late_for_merging(bfqq)) 2785 return NULL; 2786 2787 if (bfqq->new_bfqq) 2788 return bfqq->new_bfqq; 2789 2790 if (!io_struct || unlikely(bfqq == &bfqd->oom_bfqq)) 2791 return NULL; 2792 2793 /* If there is only one backlogged queue, don't search. */ 2794 if (bfq_tot_busy_queues(bfqd) == 1) 2795 return NULL; 2796 2797 in_service_bfqq = bfqd->in_service_queue; 2798 2799 if (in_service_bfqq && in_service_bfqq != bfqq && 2800 likely(in_service_bfqq != &bfqd->oom_bfqq) && 2801 bfq_rq_close_to_sector(io_struct, request, 2802 bfqd->in_serv_last_pos) && 2803 bfqq->entity.parent == in_service_bfqq->entity.parent && 2804 bfq_may_be_close_cooperator(bfqq, in_service_bfqq)) { 2805 new_bfqq = bfq_setup_merge(bfqq, in_service_bfqq); 2806 if (new_bfqq) 2807 return new_bfqq; 2808 } 2809 /* 2810 * Check whether there is a cooperator among currently scheduled 2811 * queues. The only thing we need is that the bio/request is not 2812 * NULL, as we need it to establish whether a cooperator exists. 2813 */ 2814 new_bfqq = bfq_find_close_cooperator(bfqd, bfqq, 2815 bfq_io_struct_pos(io_struct, request)); 2816 2817 if (new_bfqq && likely(new_bfqq != &bfqd->oom_bfqq) && 2818 bfq_may_be_close_cooperator(bfqq, new_bfqq)) 2819 return bfq_setup_merge(bfqq, new_bfqq); 2820 2821 return NULL; 2822} 2823 2824static void bfq_bfqq_save_state(struct bfq_queue *bfqq) 2825{ 2826 struct bfq_io_cq *bic = bfqq->bic; 2827 2828 /* 2829 * If !bfqq->bic, the queue is already shared or its requests 2830 * have already been redirected to a shared queue; both idle window 2831 * and weight raising state have already been saved. Do nothing. 2832 */ 2833 if (!bic) 2834 return; 2835 2836 bic->saved_last_serv_time_ns = bfqq->last_serv_time_ns; 2837 bic->saved_inject_limit = bfqq->inject_limit; 2838 bic->saved_decrease_time_jif = bfqq->decrease_time_jif; 2839 2840 bic->saved_weight = bfqq->entity.orig_weight; 2841 bic->saved_ttime = bfqq->ttime; 2842 bic->saved_has_short_ttime = bfq_bfqq_has_short_ttime(bfqq); 2843 bic->saved_IO_bound = bfq_bfqq_IO_bound(bfqq); 2844 bic->saved_io_start_time = bfqq->io_start_time; 2845 bic->saved_tot_idle_time = bfqq->tot_idle_time; 2846 bic->saved_in_large_burst = bfq_bfqq_in_large_burst(bfqq); 2847 bic->was_in_burst_list = !hlist_unhashed(&bfqq->burst_list_node); 2848 if (unlikely(bfq_bfqq_just_created(bfqq) && 2849 !bfq_bfqq_in_large_burst(bfqq) && 2850 bfqq->bfqd->low_latency)) { 2851 /* 2852 * bfqq being merged right after being created: bfqq 2853 * would have deserved interactive weight raising, but 2854 * did not make it to be set in a weight-raised state, 2855 * because of this early merge. Store directly the 2856 * weight-raising state that would have been assigned 2857 * to bfqq, so that to avoid that bfqq unjustly fails 2858 * to enjoy weight raising if split soon. 2859 */ 2860 bic->saved_wr_coeff = bfqq->bfqd->bfq_wr_coeff; 2861 bic->saved_wr_start_at_switch_to_srt = bfq_smallest_from_now(); 2862 bic->saved_wr_cur_max_time = bfq_wr_duration(bfqq->bfqd); 2863 bic->saved_last_wr_start_finish = jiffies; 2864 } else { 2865 bic->saved_wr_coeff = bfqq->wr_coeff; 2866 bic->saved_wr_start_at_switch_to_srt = 2867 bfqq->wr_start_at_switch_to_srt; 2868 bic->saved_service_from_wr = bfqq->service_from_wr; 2869 bic->saved_last_wr_start_finish = bfqq->last_wr_start_finish; 2870 bic->saved_wr_cur_max_time = bfqq->wr_cur_max_time; 2871 } 2872} 2873 2874 2875static void 2876bfq_reassign_last_bfqq(struct bfq_queue *cur_bfqq, struct bfq_queue *new_bfqq) 2877{ 2878 if (cur_bfqq->entity.parent && 2879 cur_bfqq->entity.parent->last_bfqq_created == cur_bfqq) 2880 cur_bfqq->entity.parent->last_bfqq_created = new_bfqq; 2881 else if (cur_bfqq->bfqd && cur_bfqq->bfqd->last_bfqq_created == cur_bfqq) 2882 cur_bfqq->bfqd->last_bfqq_created = new_bfqq; 2883} 2884 2885void bfq_release_process_ref(struct bfq_data *bfqd, struct bfq_queue *bfqq) 2886{ 2887 /* 2888 * To prevent bfqq's service guarantees from being violated, 2889 * bfqq may be left busy, i.e., queued for service, even if 2890 * empty (see comments in __bfq_bfqq_expire() for 2891 * details). But, if no process will send requests to bfqq any 2892 * longer, then there is no point in keeping bfqq queued for 2893 * service. In addition, keeping bfqq queued for service, but 2894 * with no process ref any longer, may have caused bfqq to be 2895 * freed when dequeued from service. But this is assumed to 2896 * never happen. 2897 */ 2898 if (bfq_bfqq_busy(bfqq) && RB_EMPTY_ROOT(&bfqq->sort_list) && 2899 bfqq != bfqd->in_service_queue) 2900 bfq_del_bfqq_busy(bfqd, bfqq, false); 2901 2902 bfq_reassign_last_bfqq(bfqq, NULL); 2903 2904 bfq_put_queue(bfqq); 2905} 2906 2907static void 2908bfq_merge_bfqqs(struct bfq_data *bfqd, struct bfq_io_cq *bic, 2909 struct bfq_queue *bfqq, struct bfq_queue *new_bfqq) 2910{ 2911 bfq_log_bfqq(bfqd, bfqq, "merging with queue %lu", 2912 (unsigned long)new_bfqq->pid); 2913 /* Save weight raising and idle window of the merged queues */ 2914 bfq_bfqq_save_state(bfqq); 2915 bfq_bfqq_save_state(new_bfqq); 2916 if (bfq_bfqq_IO_bound(bfqq)) 2917 bfq_mark_bfqq_IO_bound(new_bfqq); 2918 bfq_clear_bfqq_IO_bound(bfqq); 2919 2920 /* 2921 * The processes associated with bfqq are cooperators of the 2922 * processes associated with new_bfqq. So, if bfqq has a 2923 * waker, then assume that all these processes will be happy 2924 * to let bfqq's waker freely inject I/O when they have no 2925 * I/O. 2926 */ 2927 if (bfqq->waker_bfqq && !new_bfqq->waker_bfqq && 2928 bfqq->waker_bfqq != new_bfqq) { 2929 new_bfqq->waker_bfqq = bfqq->waker_bfqq; 2930 new_bfqq->tentative_waker_bfqq = NULL; 2931 2932 /* 2933 * If the waker queue disappears, then 2934 * new_bfqq->waker_bfqq must be reset. So insert 2935 * new_bfqq into the woken_list of the waker. See 2936 * bfq_check_waker for details. 2937 */ 2938 hlist_add_head(&new_bfqq->woken_list_node, 2939 &new_bfqq->waker_bfqq->woken_list); 2940 2941 } 2942 2943 /* 2944 * If bfqq is weight-raised, then let new_bfqq inherit 2945 * weight-raising. To reduce false positives, neglect the case 2946 * where bfqq has just been created, but has not yet made it 2947 * to be weight-raised (which may happen because EQM may merge 2948 * bfqq even before bfq_add_request is executed for the first 2949 * time for bfqq). Handling this case would however be very 2950 * easy, thanks to the flag just_created. 2951 */ 2952 if (new_bfqq->wr_coeff == 1 && bfqq->wr_coeff > 1) { 2953 new_bfqq->wr_coeff = bfqq->wr_coeff; 2954 new_bfqq->wr_cur_max_time = bfqq->wr_cur_max_time; 2955 new_bfqq->last_wr_start_finish = bfqq->last_wr_start_finish; 2956 new_bfqq->wr_start_at_switch_to_srt = 2957 bfqq->wr_start_at_switch_to_srt; 2958 if (bfq_bfqq_busy(new_bfqq)) 2959 bfqd->wr_busy_queues++; 2960 new_bfqq->entity.prio_changed = 1; 2961 } 2962 2963 if (bfqq->wr_coeff > 1) { /* bfqq has given its wr to new_bfqq */ 2964 bfqq->wr_coeff = 1; 2965 bfqq->entity.prio_changed = 1; 2966 if (bfq_bfqq_busy(bfqq)) 2967 bfqd->wr_busy_queues--; 2968 } 2969 2970 bfq_log_bfqq(bfqd, new_bfqq, "merge_bfqqs: wr_busy %d", 2971 bfqd->wr_busy_queues); 2972 2973 /* 2974 * Merge queues (that is, let bic redirect its requests to new_bfqq) 2975 */ 2976 bic_set_bfqq(bic, new_bfqq, 1); 2977 bfq_mark_bfqq_coop(new_bfqq); 2978 /* 2979 * new_bfqq now belongs to at least two bics (it is a shared queue): 2980 * set new_bfqq->bic to NULL. bfqq either: 2981 * - does not belong to any bic any more, and hence bfqq->bic must 2982 * be set to NULL, or 2983 * - is a queue whose owning bics have already been redirected to a 2984 * different queue, hence the queue is destined to not belong to 2985 * any bic soon and bfqq->bic is already NULL (therefore the next 2986 * assignment causes no harm). 2987 */ 2988 new_bfqq->bic = NULL; 2989 /* 2990 * If the queue is shared, the pid is the pid of one of the associated 2991 * processes. Which pid depends on the exact sequence of merge events 2992 * the queue underwent. So printing such a pid is useless and confusing 2993 * because it reports a random pid between those of the associated 2994 * processes. 2995 * We mark such a queue with a pid -1, and then print SHARED instead of 2996 * a pid in logging messages. 2997 */ 2998 new_bfqq->pid = -1; 2999 bfqq->bic = NULL; 3000
3001 bfq_reassign_last_bfqq(bfqq, new_bfqq); 3002 3003 bfq_release_process_ref(bfqd, bfqq); 3004} 3005 3006static bool bfq_allow_bio_merge(struct request_queue *q, struct request *rq, 3007 struct bio *bio) 3008{ 3009 struct bfq_data *bfqd = q->elevator->elevator_data; 3010 bool is_sync = op_is_sync(bio->bi_opf); 3011 struct bfq_queue *bfqq = bfqd->bio_bfqq, *new_bfqq; 3012 3013 /* 3014 * Disallow merge of a sync bio into an async request. 3015 */ 3016 if (is_sync && !rq_is_sync(rq)) 3017 return false; 3018 3019 /* 3020 * Lookup the bfqq that this bio will be queued with. Allow 3021 * merge only if rq is queued there. 3022 */ 3023 if (!bfqq) 3024 return false; 3025 3026 /* 3027 * We take advantage of this function to perform an early merge 3028 * of the queues of possible cooperating processes. 3029 */ 3030 new_bfqq = bfq_setup_cooperator(bfqd, bfqq, bio, false, bfqd->bio_bic); 3031 if (new_bfqq) { 3032 /* 3033 * bic still points to bfqq, then it has not yet been 3034 * redirected to some other bfq_queue, and a queue 3035 * merge between bfqq and new_bfqq can be safely 3036 * fulfilled, i.e., bic can be redirected to new_bfqq 3037 * and bfqq can be put. 3038 */ 3039 bfq_merge_bfqqs(bfqd, bfqd->bio_bic, bfqq, 3040 new_bfqq); 3041 /* 3042 * If we get here, bio will be queued into new_queue, 3043 * so use new_bfqq to decide whether bio and rq can be 3044 * merged. 3045 */ 3046 bfqq = new_bfqq; 3047 3048 /* 3049 * Change also bqfd->bio_bfqq, as 3050 * bfqd->bio_bic now points to new_bfqq, and 3051 * this function may be invoked again (and then may 3052 * use again bqfd->bio_bfqq). 3053 */ 3054 bfqd->bio_bfqq = bfqq; 3055 } 3056 3057 return bfqq == RQ_BFQQ(rq); 3058} 3059 3060/* 3061 * Set the maximum time for the in-service queue to consume its 3062 * budget. This prevents seeky processes from lowering the throughput. 3063 * In practice, a time-slice service scheme is used with seeky 3064 * processes. 3065 */ 3066static void bfq_set_budget_timeout(struct bfq_data *bfqd, 3067 struct bfq_queue *bfqq) 3068{ 3069 unsigned int timeout_coeff; 3070 3071 if (bfqq->wr_cur_max_time == bfqd->bfq_wr_rt_max_time) 3072 timeout_coeff = 1; 3073 else 3074 timeout_coeff = bfqq->entity.weight / bfqq->entity.orig_weight; 3075 3076 bfqd->last_budget_start = ktime_get(); 3077 3078 bfqq->budget_timeout = jiffies + 3079 bfqd->bfq_timeout * timeout_coeff; 3080} 3081 3082static void __bfq_set_in_service_queue(struct bfq_data *bfqd, 3083 struct bfq_queue *bfqq) 3084{ 3085 if (bfqq) { 3086 bfq_clear_bfqq_fifo_expire(bfqq); 3087 3088 bfqd->budgets_assigned = (bfqd->budgets_assigned * 7 + 256) / 8; 3089 3090 if (time_is_before_jiffies(bfqq->last_wr_start_finish) && 3091 bfqq->wr_coeff > 1 && 3092 bfqq->wr_cur_max_time == bfqd->bfq_wr_rt_max_time && 3093 time_is_before_jiffies(bfqq->budget_timeout)) { 3094 /* 3095 * For soft real-time queues, move the start 3096 * of the weight-raising period forward by the 3097 * time the queue has not received any 3098 * service. Otherwise, a relatively long 3099 * service delay is likely to cause the 3100 * weight-raising period of the queue to end, 3101 * because of the short duration of the 3102 * weight-raising period of a soft real-time 3103 * queue. It is worth noting that this move 3104 * is not so dangerous for the other queues, 3105 * because soft real-time queues are not 3106 * greedy. 3107 * 3108 * To not add a further variable, we use the 3109 * overloaded field budget_timeout to 3110 * determine for how long the queue has not 3111 * received service, i.e., how much time has 3112 * elapsed since the queue expired. However, 3113 * this is a little imprecise, because 3114 * budget_timeout is set to jiffies if bfqq 3115 * not only expires, but also remains with no 3116 * request. 3117 */ 3118 if (time_after(bfqq->budget_timeout, 3119 bfqq->last_wr_start_finish)) 3120 bfqq->last_wr_start_finish += 3121 jiffies - bfqq->budget_timeout; 3122 else 3123 bfqq->last_wr_start_finish = jiffies; 3124 } 3125 3126 bfq_set_budget_timeout(bfqd, bfqq); 3127 bfq_log_bfqq(bfqd, bfqq, 3128 "set_in_service_queue, cur-budget = %d", 3129 bfqq->entity.budget); 3130 } 3131 3132 bfqd->in_service_queue = bfqq; 3133 bfqd->in_serv_last_pos = 0; 3134} 3135 3136/* 3137 * Get and set a new queue for service. 3138 */ 3139static struct bfq_queue *bfq_set_in_service_queue(struct bfq_data *bfqd) 3140{ 3141 struct bfq_queue *bfqq = bfq_get_next_queue(bfqd); 3142 3143 __bfq_set_in_service_queue(bfqd, bfqq); 3144 return bfqq; 3145} 3146 3147static void bfq_arm_slice_timer(struct bfq_data *bfqd) 3148{ 3149 struct bfq_queue *bfqq = bfqd->in_service_queue; 3150 u32 sl; 3151 3152 bfq_mark_bfqq_wait_request(bfqq); 3153 3154 /* 3155 * We don't want to idle for seeks, but we do want to allow 3156 * fair distribution of slice time for a process doing back-to-back 3157 * seeks. So allow a little bit of time for him to submit a new rq. 3158 */ 3159 sl = bfqd->bfq_slice_idle; 3160 /* 3161 * Unless the queue is being weight-raised or the scenario is 3162 * asymmetric, grant only minimum idle time if the queue 3163 * is seeky. A long idling is preserved for a weight-raised 3164 * queue, or, more in general, in an asymmetric scenario, 3165 * because a long idling is needed for guaranteeing to a queue 3166 * its reserved share of the throughput (in particular, it is 3167 * needed if the queue has a higher weight than some other 3168 * queue). 3169 */ 3170 if (BFQQ_SEEKY(bfqq) && bfqq->wr_coeff == 1 && 3171 !bfq_asymmetric_scenario(bfqd, bfqq)) 3172 sl = min_t(u64, sl, BFQ_MIN_TT); 3173 else if (bfqq->wr_coeff > 1) 3174 sl = max_t(u32, sl, 20ULL * NSEC_PER_MSEC); 3175 3176 bfqd->last_idling_start = ktime_get(); 3177 bfqd->last_idling_start_jiffies = jiffies; 3178 3179 hrtimer_start(&bfqd->idle_slice_timer, ns_to_ktime(sl), 3180 HRTIMER_MODE_REL); 3181 bfqg_stats_set_start_idle_time(bfqq_group(bfqq)); 3182} 3183 3184/* 3185 * In autotuning mode, max_budget is dynamically recomputed as the 3186 * amount of sectors transferred in timeout at the estimated peak 3187 * rate. This enables BFQ to utilize a full timeslice with a full 3188 * budget, even if the in-service queue is served at peak rate. And 3189 * this maximises throughput with sequential workloads. 3190 */ 3191static unsigned long bfq_calc_max_budget(struct bfq_data *bfqd) 3192{ 3193 return (u64)bfqd->peak_rate * USEC_PER_MSEC * 3194 jiffies_to_msecs(bfqd->bfq_timeout)>>BFQ_RATE_SHIFT; 3195} 3196 3197/* 3198 * Update parameters related to throughput and responsiveness, as a 3199 * function of the estimated peak rate. See comments on 3200 * bfq_calc_max_budget(), and on the ref_wr_duration array. 3201 */ 3202static void update_thr_responsiveness_params(struct bfq_data *bfqd) 3203{ 3204 if (bfqd->bfq_user_max_budget == 0) { 3205 bfqd->bfq_max_budget = 3206 bfq_calc_max_budget(bfqd); 3207 bfq_log(bfqd, "new max_budget = %d", bfqd->bfq_max_budget); 3208 } 3209} 3210 3211static void bfq_reset_rate_computation(struct bfq_data *bfqd, 3212 struct request *rq) 3213{ 3214 if (rq != NULL) { /* new rq dispatch now, reset accordingly */ 3215 bfqd->last_dispatch = bfqd->first_dispatch = ktime_get_ns(); 3216 bfqd->peak_rate_samples = 1; 3217 bfqd->sequential_samples = 0; 3218 bfqd->tot_sectors_dispatched = bfqd->last_rq_max_size = 3219 blk_rq_sectors(rq); 3220 } else /* no new rq dispatched, just reset the number of samples */ 3221 bfqd->peak_rate_samples = 0; /* full re-init on next disp. */ 3222 3223 bfq_log(bfqd, 3224 "reset_rate_computation at end, sample %u/%u tot_sects %llu", 3225 bfqd->peak_rate_samples, bfqd->sequential_samples, 3226 bfqd->tot_sectors_dispatched); 3227} 3228 3229static void bfq_update_rate_reset(struct bfq_data *bfqd, struct request *rq) 3230{ 3231 u32 rate, weight, divisor; 3232 3233 /* 3234 * For the convergence property to hold (see comments on 3235 * bfq_update_peak_rate()) and for the assessment to be 3236 * reliable, a minimum number of samples must be present, and 3237 * a minimum amount of time must have elapsed. If not so, do 3238 * not compute new rate. Just reset parameters, to get ready 3239 * for a new evaluation attempt. 3240 */ 3241 if (bfqd->peak_rate_samples < BFQ_RATE_MIN_SAMPLES || 3242 bfqd->delta_from_first < BFQ_RATE_MIN_INTERVAL) 3243 goto reset_computation; 3244 3245 /* 3246 * If a new request completion has occurred after last 3247 * dispatch, then, to approximate the rate at which requests 3248 * have been served by the device, it is more precise to 3249 * extend the observation interval to the last completion. 3250 */ 3251 bfqd->delta_from_first = 3252 max_t(u64, bfqd->delta_from_first, 3253 bfqd->last_completion - bfqd->first_dispatch); 3254 3255 /* 3256 * Rate computed in sects/usec, and not sects/nsec, for 3257 * precision issues. 3258 */ 3259 rate = div64_ul(bfqd->tot_sectors_dispatched<<BFQ_RATE_SHIFT, 3260 div_u64(bfqd->delta_from_first, NSEC_PER_USEC)); 3261 3262 /* 3263 * Peak rate not updated if: 3264 * - the percentage of sequential dispatches is below 3/4 of the 3265 * total, and rate is below the current estimated peak rate 3266 * - rate is unreasonably high (> 20M sectors/sec) 3267 */ 3268 if ((bfqd->sequential_samples < (3 * bfqd->peak_rate_samples)>>2 && 3269 rate <= bfqd->peak_rate) || 3270 rate > 20<<BFQ_RATE_SHIFT) 3271 goto reset_computation; 3272 3273 /* 3274 * We have to update the peak rate, at last! To this purpose, 3275 * we use a low-pass filter. We compute the smoothing constant 3276 * of the filter as a function of the 'weight' of the new 3277 * measured rate. 3278 * 3279 * As can be seen in next formulas, we define this weight as a 3280 * quantity proportional to how sequential the workload is, 3281 * and to how long the observation time interval is. 3282 * 3283 * The weight runs from 0 to 8. The maximum value of the 3284 * weight, 8, yields the minimum value for the smoothing 3285 * constant. At this minimum value for the smoothing constant, 3286 * the measured rate contributes for half of the next value of 3287 * the estimated peak rate. 3288 * 3289 * So, the first step is to compute the weight as a function 3290 * of how sequential the workload is. Note that the weight 3291 * cannot reach 9, because bfqd->sequential_samples cannot 3292 * become equal to bfqd->peak_rate_samples, which, in its 3293 * turn, holds true because bfqd->sequential_samples is not 3294 * incremented for the first sample. 3295 */ 3296 weight = (9 * bfqd->sequential_samples) / bfqd->peak_rate_samples; 3297 3298 /* 3299 * Second step: further refine the weight as a function of the 3300 * duration of the observation interval. 3301 */ 3302 weight = min_t(u32, 8, 3303 div_u64(weight * bfqd->delta_from_first, 3304 BFQ_RATE_REF_INTERVAL)); 3305 3306 /* 3307 * Divisor ranging from 10, for minimum weight, to 2, for 3308 * maximum weight. 3309 */ 3310 divisor = 10 - weight; 3311 3312 /* 3313 * Finally, update peak rate: 3314 * 3315 * peak_rate = peak_rate * (divisor-1) / divisor + rate / divisor 3316 */ 3317 bfqd->peak_rate *= divisor-1; 3318 bfqd->peak_rate /= divisor; 3319 rate /= divisor; /* smoothing constant alpha = 1/divisor */ 3320 3321 bfqd->peak_rate += rate; 3322 3323 /* 3324 * For a very slow device, bfqd->peak_rate can reach 0 (see 3325 * the minimum representable values reported in the comments 3326 * on BFQ_RATE_SHIFT). Push to 1 if this happens, to avoid 3327 * divisions by zero where bfqd->peak_rate is used as a 3328 * divisor. 3329 */ 3330 bfqd->peak_rate = max_t(u32, 1, bfqd->peak_rate); 3331 3332 update_thr_responsiveness_params(bfqd); 3333 3334reset_computation: 3335 bfq_reset_rate_computation(bfqd, rq); 3336} 3337 3338/* 3339 * Update the read/write peak rate (the main quantity used for 3340 * auto-tuning, see update_thr_responsiveness_params()). 3341 * 3342 * It is not trivial to estimate the peak rate (correctly): because of 3343 * the presence of sw and hw queues between the scheduler and the 3344 * device components that finally serve I/O requests, it is hard to 3345 * say exactly when a given dispatched request is served inside the 3346 * device, and for how long. As a consequence, it is hard to know 3347 * precisely at what rate a given set of requests is actually served 3348 * by the device. 3349 * 3350 * On the opposite end, the dispatch time of any request is trivially 3351 * available, and, from this piece of information, the "dispatch rate" 3352 * of requests can be immediately computed. So, the idea in the next 3353 * function is to use what is known, namely request dispatch times 3354 * (plus, when useful, request completion times), to estimate what is 3355 * unknown, namely in-device request service rate. 3356 * 3357 * The main issue is that, because of the above facts, the rate at 3358 * which a certain set of requests is dispatched over a certain time 3359 * interval can vary greatly with respect to the rate at which the 3360 * same requests are then served. But, since the size of any 3361 * intermediate queue is limited, and the service scheme is lossless 3362 * (no request is silently dropped), the following obvious convergence 3363 * property holds: the number of requests dispatched MUST become 3364 * closer and closer to the number of requests completed as the 3365 * observation interval grows. This is the key property used in 3366 * the next function to estimate the peak service rate as a function 3367 * of the observed dispatch rate. The function assumes to be invoked 3368 * on every request dispatch. 3369 */ 3370static void bfq_update_peak_rate(struct bfq_data *bfqd, struct request *rq) 3371{ 3372 u64 now_ns = ktime_get_ns(); 3373 3374 if (bfqd->peak_rate_samples == 0) { /* first dispatch */ 3375 bfq_log(bfqd, "update_peak_rate: goto reset, samples %d", 3376 bfqd->peak_rate_samples); 3377 bfq_reset_rate_computation(bfqd, rq); 3378 goto update_last_values; /* will add one sample */ 3379 } 3380 3381 /* 3382 * Device idle for very long: the observation interval lasting 3383 * up to this dispatch cannot be a valid observation interval 3384 * for computing a new peak rate (similarly to the late- 3385 * completion event in bfq_completed_request()). Go to 3386 * update_rate_and_reset to have the following three steps 3387 * taken: 3388 * - close the observation interval at the last (previous) 3389 * request dispatch or completion 3390 * - compute rate, if possible, for that observation interval 3391 * - start a new observation interval with this dispatch 3392 */ 3393 if (now_ns - bfqd->last_dispatch > 100*NSEC_PER_MSEC && 3394 bfqd->rq_in_driver == 0) 3395 goto update_rate_and_reset; 3396 3397 /* Update sampling information */ 3398 bfqd->peak_rate_samples++; 3399 3400 if ((bfqd->rq_in_driver > 0 || 3401 now_ns - bfqd->last_completion < BFQ_MIN_TT) 3402 && !BFQ_RQ_SEEKY(bfqd, bfqd->last_position, rq)) 3403 bfqd->sequential_samples++; 3404 3405 bfqd->tot_sectors_dispatched += blk_rq_sectors(rq); 3406 3407 /* Reset max observed rq size every 32 dispatches */ 3408 if (likely(bfqd->peak_rate_samples % 32)) 3409 bfqd->last_rq_max_size = 3410 max_t(u32, blk_rq_sectors(rq), bfqd->last_rq_max_size); 3411 else 3412 bfqd->last_rq_max_size = blk_rq_sectors(rq); 3413 3414 bfqd->delta_from_first = now_ns - bfqd->first_dispatch; 3415 3416 /* Target observation interval not yet reached, go on sampling */ 3417 if (bfqd->delta_from_first < BFQ_RATE_REF_INTERVAL) 3418 goto update_last_values; 3419 3420update_rate_and_reset: 3421 bfq_update_rate_reset(bfqd, rq); 3422update_last_values: 3423 bfqd->last_position = blk_rq_pos(rq) + blk_rq_sectors(rq); 3424 if (RQ_BFQQ(rq) == bfqd->in_service_queue) 3425 bfqd->in_serv_last_pos = bfqd->last_position; 3426 bfqd->last_dispatch = now_ns; 3427} 3428 3429/* 3430 * Remove request from internal lists. 3431 */ 3432static void bfq_dispatch_remove(struct request_queue *q, struct request *rq) 3433{ 3434 struct bfq_queue *bfqq = RQ_BFQQ(rq); 3435 3436 /* 3437 * For consistency, the next instruction should have been 3438 * executed after removing the request from the queue and 3439 * dispatching it. We execute instead this instruction before 3440 * bfq_remove_request() (and hence introduce a temporary 3441 * inconsistency), for efficiency. In fact, should this 3442 * dispatch occur for a non in-service bfqq, this anticipated 3443 * increment prevents two counters related to bfqq->dispatched 3444 * from risking to be, first, uselessly decremented, and then 3445 * incremented again when the (new) value of bfqq->dispatched 3446 * happens to be taken into account. 3447 */ 3448 bfqq->dispatched++; 3449 bfq_update_peak_rate(q->elevator->elevator_data, rq); 3450 3451 bfq_remove_request(q, rq); 3452} 3453 3454/* 3455 * There is a case where idling does not have to be performed for 3456 * throughput concerns, but to preserve the throughput share of 3457 * the process associated with bfqq. 3458 * 3459 * To introduce this case, we can note that allowing the drive 3460 * to enqueue more than one request at a time, and hence 3461 * delegating de facto final scheduling decisions to the 3462 * drive's internal scheduler, entails loss of control on the 3463 * actual request service order. In particular, the critical 3464 * situation is when requests from different processes happen 3465 * to be present, at the same time, in the internal queue(s) 3466 * of the drive. In such a situation, the drive, by deciding 3467 * the service order of the internally-queued requests, does 3468 * determine also the actual throughput distribution among 3469 * these processes. But the drive typically has no notion or 3470 * concern about per-process throughput distribution, and 3471 * makes its decisions only on a per-request basis. Therefore, 3472 * the service distribution enforced by the drive's internal 3473 * scheduler is likely to coincide with the desired throughput 3474 * distribution only in a completely symmetric, or favorably 3475 * skewed scenario where: 3476 * (i-a) each of these processes must get the same throughput as 3477 * the others, 3478 * (i-b) in case (i-a) does not hold, it holds that the process 3479 * associated with bfqq must receive a lower or equal 3480 * throughput than any of the other processes; 3481 * (ii) the I/O of each process has the same properties, in 3482 * terms of locality (sequential or random), direction 3483 * (reads or writes), request sizes, greediness 3484 * (from I/O-bound to sporadic), and so on; 3485 3486 * In fact, in such a scenario, the drive tends to treat the requests 3487 * of each process in about the same way as the requests of the 3488 * others, and thus to provide each of these processes with about the 3489 * same throughput. This is exactly the desired throughput 3490 * distribution if (i-a) holds, or, if (i-b) holds instead, this is an 3491 * even more convenient distribution for (the process associated with) 3492 * bfqq. 3493 * 3494 * In contrast, in any asymmetric or unfavorable scenario, device 3495 * idling (I/O-dispatch plugging) is certainly needed to guarantee 3496 * that bfqq receives its assigned fraction of the device throughput 3497 * (see [1] for details). 3498 * 3499 * The problem is that idling may significantly reduce throughput with 3500 * certain combinations of types of I/O and devices. An important 3501 * example is sync random I/O on flash storage with command 3502 * queueing. So, unless bfqq falls in cases where idling also boosts 3503 * throughput, it is important to check conditions (i-a), i(-b) and 3504 * (ii) accurately, so as to avoid idling when not strictly needed for 3505 * service guarantees. 3506 * 3507 * Unfortunately, it is extremely difficult to thoroughly check 3508 * condition (ii). And, in case there are active groups, it becomes 3509 * very difficult to check conditions (i-a) and (i-b) too. In fact, 3510 * if there are active groups, then, for conditions (i-a) or (i-b) to 3511 * become false 'indirectly', it is enough that an active group 3512 * contains more active processes or sub-groups than some other active 3513 * group. More precisely, for conditions (i-a) or (i-b) to become 3514 * false because of such a group, it is not even necessary that the 3515 * group is (still) active: it is sufficient that, even if the group 3516 * has become inactive, some of its descendant processes still have 3517 * some request already dispatched but still waiting for 3518 * completion. In fact, requests have still to be guaranteed their 3519 * share of the throughput even after being dispatched. In this 3520 * respect, it is easy to show that, if a group frequently becomes 3521 * inactive while still having in-flight requests, and if, when this 3522 * happens, the group is not considered in the calculation of whether 3523 * the scenario is asymmetric, then the group may fail to be 3524 * guaranteed its fair share of the throughput (basically because 3525 * idling may not be performed for the descendant processes of the 3526 * group, but it had to be). We address this issue with the following 3527 * bi-modal behavior, implemented in the function 3528 * bfq_asymmetric_scenario(). 3529 * 3530 * If there are groups with requests waiting for completion 3531 * (as commented above, some of these groups may even be 3532 * already inactive), then the scenario is tagged as 3533 * asymmetric, conservatively, without checking any of the 3534 * conditions (i-a), (i-b) or (ii). So the device is idled for bfqq. 3535 * This behavior matches also the fact that groups are created 3536 * exactly if controlling I/O is a primary concern (to 3537 * preserve bandwidth and latency guarantees). 3538 * 3539 * On the opposite end, if there are no groups with requests waiting 3540 * for completion, then only conditions (i-a) and (i-b) are actually 3541 * controlled, i.e., provided that conditions (i-a) or (i-b) holds, 3542 * idling is not performed, regardless of whether condition (ii) 3543 * holds. In other words, only if conditions (i-a) and (i-b) do not 3544 * hold, then idling is allowed, and the device tends to be prevented 3545 * from queueing many requests, possibly of several processes. Since 3546 * there are no groups with requests waiting for completion, then, to 3547 * control conditions (i-a) and (i-b) it is enough to check just 3548 * whether all the queues with requests waiting for completion also 3549 * have the same weight. 3550 * 3551 * Not checking condition (ii) evidently exposes bfqq to the 3552 * risk of getting less throughput than its fair share. 3553 * However, for queues with the same weight, a further 3554 * mechanism, preemption, mitigates or even eliminates this 3555 * problem. And it does so without consequences on overall 3556 * throughput. This mechanism and its benefits are explained 3557 * in the next three paragraphs. 3558 * 3559 * Even if a queue, say Q, is expired when it remains idle, Q 3560 * can still preempt the new in-service queue if the next 3561 * request of Q arrives soon (see the comments on 3562 * bfq_bfqq_update_budg_for_activation). If all queues and 3563 * groups have the same weight, this form of preemption, 3564 * combined with the hole-recovery heuristic described in the 3565 * comments on function bfq_bfqq_update_budg_for_activation, 3566 * are enough to preserve a correct bandwidth distribution in 3567 * the mid term, even without idling. In fact, even if not 3568 * idling allows the internal queues of the device to contain 3569 * many requests, and thus to reorder requests, we can rather 3570 * safely assume that the internal scheduler still preserves a 3571 * minimum of mid-term fairness. 3572 * 3573 * More precisely, this preemption-based, idleless approach 3574 * provides fairness in terms of IOPS, and not sectors per 3575 * second. This can be seen with a simple example. Suppose 3576 * that there are two queues with the same weight, but that 3577 * the first queue receives requests of 8 sectors, while the 3578 * second queue receives requests of 1024 sectors. In 3579 * addition, suppose that each of the two queues contains at 3580 * most one request at a time, which implies that each queue 3581 * always remains idle after it is served. Finally, after 3582 * remaining idle, each queue receives very quickly a new 3583 * request. It follows that the two queues are served 3584 * alternatively, preempting each other if needed. This 3585 * implies that, although both queues have the same weight, 3586 * the queue with large requests receives a service that is 3587 * 1024/8 times as high as the service received by the other 3588 * queue. 3589 * 3590 * The motivation for using preemption instead of idling (for 3591 * queues with the same weight) is that, by not idling, 3592 * service guarantees are preserved (completely or at least in 3593 * part) without minimally sacrificing throughput. And, if 3594 * there is no active group, then the primary expectation for 3595 * this device is probably a high throughput. 3596 * 3597 * We are now left only with explaining the two sub-conditions in the 3598 * additional compound condition that is checked below for deciding 3599 * whether the scenario is asymmetric. To explain the first 3600 * sub-condition, we need to add that the function 3601 * bfq_asymmetric_scenario checks the weights of only 3602 * non-weight-raised queues, for efficiency reasons (see comments on 3603 * bfq_weights_tree_add()). Then the fact that bfqq is weight-raised 3604 * is checked explicitly here. More precisely, the compound condition 3605 * below takes into account also the fact that, even if bfqq is being 3606 * weight-raised, the scenario is still symmetric if all queues with 3607 * requests waiting for completion happen to be 3608 * weight-raised. Actually, we should be even more precise here, and 3609 * differentiate between interactive weight raising and soft real-time 3610 * weight raising. 3611 * 3612 * The second sub-condition checked in the compound condition is 3613 * whether there is a fair amount of already in-flight I/O not 3614 * belonging to bfqq. If so, I/O dispatching is to be plugged, for the 3615 * following reason. The drive may decide to serve in-flight 3616 * non-bfqq's I/O requests before bfqq's ones, thereby delaying the 3617 * arrival of new I/O requests for bfqq (recall that bfqq is sync). If 3618 * I/O-dispatching is not plugged, then, while bfqq remains empty, a 3619 * basically uncontrolled amount of I/O from other queues may be 3620 * dispatched too, possibly causing the service of bfqq's I/O to be 3621 * delayed even longer in the drive. This problem gets more and more 3622 * serious as the speed and the queue depth of the drive grow, 3623 * because, as these two quantities grow, the probability to find no 3624 * queue busy but many requests in flight grows too. By contrast, 3625 * plugging I/O dispatching minimizes the delay induced by already 3626 * in-flight I/O, and enables bfqq to recover the bandwidth it may 3627 * lose because of this delay. 3628 * 3629 * As a side note, it is worth considering that the above 3630 * device-idling countermeasures may however fail in the following 3631 * unlucky scenario: if I/O-dispatch plugging is (correctly) disabled 3632 * in a time period during which all symmetry sub-conditions hold, and 3633 * therefore the device is allowed to enqueue many requests, but at 3634 * some later point in time some sub-condition stops to hold, then it 3635 * may become impossible to make requests be served in the desired 3636 * order until all the requests already queued in the device have been 3637 * served. The last sub-condition commented above somewhat mitigates 3638 * this problem for weight-raised queues. 3639 * 3640 * However, as an additional mitigation for this problem, we preserve 3641 * plugging for a special symmetric case that may suddenly turn into 3642 * asymmetric: the case where only bfqq is busy. In this case, not 3643 * expiring bfqq does not cause any harm to any other queues in terms 3644 * of service guarantees. In contrast, it avoids the following unlucky 3645 * sequence of events: (1) bfqq is expired, (2) a new queue with a 3646 * lower weight than bfqq becomes busy (or more queues), (3) the new 3647 * queue is served until a new request arrives for bfqq, (4) when bfqq 3648 * is finally served, there are so many requests of the new queue in 3649 * the drive that the pending requests for bfqq take a lot of time to 3650 * be served. In particular, event (2) may case even already 3651 * dispatched requests of bfqq to be delayed, inside the drive. So, to 3652 * avoid this series of events, the scenario is preventively declared 3653 * as asymmetric also if bfqq is the only busy queues 3654 */ 3655static bool idling_needed_for_service_guarantees(struct bfq_data *bfqd, 3656 struct bfq_queue *bfqq) 3657{ 3658 int tot_busy_queues = bfq_tot_busy_queues(bfqd); 3659 3660 /* No point in idling for bfqq if it won't get requests any longer */ 3661 if (unlikely(!bfqq_process_refs(bfqq))) 3662 return false; 3663 3664 return (bfqq->wr_coeff > 1 && 3665 (bfqd->wr_busy_queues < 3666 tot_busy_queues || 3667 bfqd->rq_in_driver >= 3668 bfqq->dispatched + 4)) || 3669 bfq_asymmetric_scenario(bfqd, bfqq) || 3670 tot_busy_queues == 1; 3671} 3672 3673static bool __bfq_bfqq_expire(struct bfq_data *bfqd, struct bfq_queue *bfqq, 3674 enum bfqq_expiration reason) 3675{ 3676 /* 3677 * If this bfqq is shared between multiple processes, check 3678 * to make sure that those processes are still issuing I/Os 3679 * within the mean seek distance. If not, it may be time to 3680 * break the queues apart again. 3681 */ 3682 if (bfq_bfqq_coop(bfqq) && BFQQ_SEEKY(bfqq)) 3683 bfq_mark_bfqq_split_coop(bfqq); 3684 3685 /* 3686 * Consider queues with a higher finish virtual time than 3687 * bfqq. If idling_needed_for_service_guarantees(bfqq) returns 3688 * true, then bfqq's bandwidth would be violated if an 3689 * uncontrolled amount of I/O from these queues were 3690 * dispatched while bfqq is waiting for its new I/O to 3691 * arrive. This is exactly what may happen if this is a forced 3692 * expiration caused by a preemption attempt, and if bfqq is 3693 * not re-scheduled. To prevent this from happening, re-queue 3694 * bfqq if it needs I/O-dispatch plugging, even if it is 3695 * empty. By doing so, bfqq is granted to be served before the 3696 * above queues (provided that bfqq is of course eligible). 3697 */ 3698 if (RB_EMPTY_ROOT(&bfqq->sort_list) && 3699 !(reason == BFQQE_PREEMPTED && 3700 idling_needed_for_service_guarantees(bfqd, bfqq))) { 3701 if (bfqq->dispatched == 0) 3702 /* 3703 * Overloading budget_timeout field to store 3704 * the time at which the queue remains with no 3705 * backlog and no outstanding request; used by 3706 * the weight-raising mechanism. 3707 */ 3708 bfqq->budget_timeout = jiffies; 3709 3710 bfq_del_bfqq_busy(bfqd, bfqq, true); 3711 } else { 3712 bfq_requeue_bfqq(bfqd, bfqq, true); 3713 /* 3714 * Resort priority tree of potential close cooperators. 3715 * See comments on bfq_pos_tree_add_move() for the unlikely(). 3716 */ 3717 if (unlikely(!bfqd->nonrot_with_queueing && 3718 !RB_EMPTY_ROOT(&bfqq->sort_list))) 3719 bfq_pos_tree_add_move(bfqd, bfqq); 3720 } 3721 3722 /* 3723 * All in-service entities must have been properly deactivated 3724 * or requeued before executing the next function, which 3725 * resets all in-service entities as no more in service. This 3726 * may cause bfqq to be freed. If this happens, the next 3727 * function returns true. 3728 */ 3729 return __bfq_bfqd_reset_in_service(bfqd); 3730} 3731 3732/** 3733 * __bfq_bfqq_recalc_budget - try to adapt the budget to the @bfqq behavior. 3734 * @bfqd: device data. 3735 * @bfqq: queue to update. 3736 * @reason: reason for expiration. 3737 * 3738 * Handle the feedback on @bfqq budget at queue expiration. 3739 * See the body for detailed comments. 3740 */ 3741static void __bfq_bfqq_recalc_budget(struct bfq_data *bfqd, 3742 struct bfq_queue *bfqq, 3743 enum bfqq_expiration reason) 3744{ 3745 struct request *next_rq; 3746 int budget, min_budget; 3747 3748 min_budget = bfq_min_budget(bfqd); 3749 3750 if (bfqq->wr_coeff == 1) 3751 budget = bfqq->max_budget; 3752 else /* 3753 * Use a constant, low budget for weight-raised queues, 3754 * to help achieve a low latency. Keep it slightly higher 3755 * than the minimum possible budget, to cause a little 3756 * bit fewer expirations. 3757 */ 3758 budget = 2 * min_budget; 3759 3760 bfq_log_bfqq(bfqd, bfqq, "recalc_budg: last budg %d, budg left %d", 3761 bfqq->entity.budget, bfq_bfqq_budget_left(bfqq)); 3762 bfq_log_bfqq(bfqd, bfqq, "recalc_budg: last max_budg %d, min budg %d", 3763 budget, bfq_min_budget(bfqd)); 3764 bfq_log_bfqq(bfqd, bfqq, "recalc_budg: sync %d, seeky %d", 3765 bfq_bfqq_sync(bfqq), BFQQ_SEEKY(bfqd->in_service_queue)); 3766 3767 if (bfq_bfqq_sync(bfqq) && bfqq->wr_coeff == 1) { 3768 switch (reason) { 3769 /* 3770 * Caveat: in all the following cases we trade latency 3771 * for throughput. 3772 */ 3773 case BFQQE_TOO_IDLE: 3774 /* 3775 * This is the only case where we may reduce 3776 * the budget: if there is no request of the 3777 * process still waiting for completion, then 3778 * we assume (tentatively) that the timer has 3779 * expired because the batch of requests of 3780 * the process could have been served with a 3781 * smaller budget. Hence, betting that 3782 * process will behave in the same way when it 3783 * becomes backlogged again, we reduce its 3784 * next budget. As long as we guess right, 3785 * this budget cut reduces the latency 3786 * experienced by the process. 3787 * 3788 * However, if there are still outstanding 3789 * requests, then the process may have not yet 3790 * issued its next request just because it is 3791 * still waiting for the completion of some of 3792 * the still outstanding ones. So in this 3793 * subcase we do not reduce its budget, on the 3794 * contrary we increase it to possibly boost 3795 * the throughput, as discussed in the 3796 * comments to the BUDGET_TIMEOUT case. 3797 */ 3798 if (bfqq->dispatched > 0) /* still outstanding reqs */ 3799 budget = min(budget * 2, bfqd->bfq_max_budget); 3800 else { 3801 if (budget > 5 * min_budget) 3802 budget -= 4 * min_budget; 3803 else 3804 budget = min_budget; 3805 } 3806 break; 3807 case BFQQE_BUDGET_TIMEOUT: 3808 /* 3809 * We double the budget here because it gives 3810 * the chance to boost the throughput if this 3811 * is not a seeky process (and has bumped into 3812 * this timeout because of, e.g., ZBR). 3813 */ 3814 budget = min(budget * 2, bfqd->bfq_max_budget); 3815 break; 3816 case BFQQE_BUDGET_EXHAUSTED: 3817 /* 3818 * The process still has backlog, and did not 3819 * let either the budget timeout or the disk 3820 * idling timeout expire. Hence it is not 3821 * seeky, has a short thinktime and may be 3822 * happy with a higher budget too. So 3823 * definitely increase the budget of this good 3824 * candidate to boost the disk throughput. 3825 */ 3826 budget = min(budget * 4, bfqd->bfq_max_budget); 3827 break; 3828 case BFQQE_NO_MORE_REQUESTS: 3829 /* 3830 * For queues that expire for this reason, it 3831 * is particularly important to keep the 3832 * budget close to the actual service they 3833 * need. Doing so reduces the timestamp 3834 * misalignment problem described in the 3835 * comments in the body of 3836 * __bfq_activate_entity. In fact, suppose 3837 * that a queue systematically expires for 3838 * BFQQE_NO_MORE_REQUESTS and presents a 3839 * new request in time to enjoy timestamp 3840 * back-shifting. The larger the budget of the 3841 * queue is with respect to the service the 3842 * queue actually requests in each service 3843 * slot, the more times the queue can be 3844 * reactivated with the same virtual finish 3845 * time. It follows that, even if this finish 3846 * time is pushed to the system virtual time 3847 * to reduce the consequent timestamp 3848 * misalignment, the queue unjustly enjoys for 3849 * many re-activations a lower finish time 3850 * than all newly activated queues. 3851 * 3852 * The service needed by bfqq is measured 3853 * quite precisely by bfqq->entity.service. 3854 * Since bfqq does not enjoy device idling, 3855 * bfqq->entity.service is equal to the number 3856 * of sectors that the process associated with 3857 * bfqq requested to read/write before waiting 3858 * for request completions, or blocking for 3859 * other reasons. 3860 */ 3861 budget = max_t(int, bfqq->entity.service, min_budget); 3862 break; 3863 default: 3864 return; 3865 } 3866 } else if (!bfq_bfqq_sync(bfqq)) { 3867 /* 3868 * Async queues get always the maximum possible 3869 * budget, as for them we do not care about latency 3870 * (in addition, their ability to dispatch is limited 3871 * by the charging factor). 3872 */ 3873 budget = bfqd->bfq_max_budget; 3874 } 3875 3876 bfqq->max_budget = budget; 3877 3878 if (bfqd->budgets_assigned >= bfq_stats_min_budgets && 3879 !bfqd->bfq_user_max_budget) 3880 bfqq->max_budget = min(bfqq->max_budget, bfqd->bfq_max_budget); 3881 3882 /* 3883 * If there is still backlog, then assign a new budget, making 3884 * sure that it is large enough for the next request. Since 3885 * the finish time of bfqq must be kept in sync with the 3886 * budget, be sure to call __bfq_bfqq_expire() *after* this 3887 * update. 3888 * 3889 * If there is no backlog, then no need to update the budget; 3890 * it will be updated on the arrival of a new request. 3891 */ 3892 next_rq = bfqq->next_rq; 3893 if (next_rq) 3894 bfqq->entity.budget = max_t(unsigned long, bfqq->max_budget, 3895 bfq_serv_to_charge(next_rq, bfqq)); 3896 3897 bfq_log_bfqq(bfqd, bfqq, "head sect: %u, new budget %d", 3898 next_rq ? blk_rq_sectors(next_rq) : 0, 3899 bfqq->entity.budget); 3900} 3901 3902/* 3903 * Return true if the process associated with bfqq is "slow". The slow 3904 * flag is used, in addition to the budget timeout, to reduce the 3905 * amount of service provided to seeky processes, and thus reduce 3906 * their chances to lower the throughput. More details in the comments 3907 * on the function bfq_bfqq_expire(). 3908 * 3909 * An important observation is in order: as discussed in the comments 3910 * on the function bfq_update_peak_rate(), with devices with internal 3911 * queues, it is hard if ever possible to know when and for how long 3912 * an I/O request is processed by the device (apart from the trivial 3913 * I/O pattern where a new request is dispatched only after the 3914 * previous one has been completed). This makes it hard to evaluate 3915 * the real rate at which the I/O requests of each bfq_queue are 3916 * served. In fact, for an I/O scheduler like BFQ, serving a 3917 * bfq_queue means just dispatching its requests during its service 3918 * slot (i.e., until the budget of the queue is exhausted, or the 3919 * queue remains idle, or, finally, a timeout fires). But, during the 3920 * service slot of a bfq_queue, around 100 ms at most, the device may 3921 * be even still processing requests of bfq_queues served in previous 3922 * service slots. On the opposite end, the requests of the in-service 3923 * bfq_queue may be completed after the service slot of the queue 3924 * finishes. 3925 * 3926 * Anyway, unless more sophisticated solutions are used 3927 * (where possible), the sum of the sizes of the requests dispatched 3928 * during the service slot of a bfq_queue is probably the only 3929 * approximation available for the service received by the bfq_queue 3930 * during its service slot. And this sum is the quantity used in this 3931 * function to evaluate the I/O speed of a process. 3932 */ 3933static bool bfq_bfqq_is_slow(struct bfq_data *bfqd, struct bfq_queue *bfqq, 3934 bool compensate, enum bfqq_expiration reason, 3935 unsigned long *delta_ms) 3936{ 3937 ktime_t delta_ktime; 3938 u32 delta_usecs; 3939 bool slow = BFQQ_SEEKY(bfqq); /* if delta too short, use seekyness */ 3940 3941 if (!bfq_bfqq_sync(bfqq)) 3942 return false; 3943 3944 if (compensate) 3945 delta_ktime = bfqd->last_idling_start; 3946 else 3947 delta_ktime = ktime_get(); 3948 delta_ktime = ktime_sub(delta_ktime, bfqd->last_budget_start); 3949 delta_usecs = ktime_to_us(delta_ktime); 3950 3951 /* don't use too short time intervals */ 3952 if (delta_usecs < 1000) { 3953 if (blk_queue_nonrot(bfqd->queue)) 3954 /* 3955 * give same worst-case guarantees as idling 3956 * for seeky 3957 */ 3958 *delta_ms = BFQ_MIN_TT / NSEC_PER_MSEC; 3959 else /* charge at least one seek */ 3960 *delta_ms = bfq_slice_idle / NSEC_PER_MSEC; 3961 3962 return slow; 3963 } 3964 3965 *delta_ms = delta_usecs / USEC_PER_MSEC; 3966 3967 /* 3968 * Use only long (> 20ms) intervals to filter out excessive 3969 * spikes in service rate estimation. 3970 */ 3971 if (delta_usecs > 20000) { 3972 /* 3973 * Caveat for rotational devices: processes doing I/O 3974 * in the slower disk zones tend to be slow(er) even 3975 * if not seeky. In this respect, the estimated peak 3976 * rate is likely to be an average over the disk 3977 * surface. Accordingly, to not be too harsh with 3978 * unlucky processes, a process is deemed slow only if 3979 * its rate has been lower than half of the estimated 3980 * peak rate. 3981 */ 3982 slow = bfqq->entity.service < bfqd->bfq_max_budget / 2; 3983 } 3984 3985 bfq_log_bfqq(bfqd, bfqq, "bfq_bfqq_is_slow: slow %d", slow); 3986 3987 return slow; 3988} 3989 3990/* 3991 * To be deemed as soft real-time, an application must meet two 3992 * requirements. First, the application must not require an average 3993 * bandwidth higher than the approximate bandwidth required to playback or 3994 * record a compressed high-definition video. 3995 * The next function is invoked on the completion of the last request of a 3996 * batch, to compute the next-start time instant, soft_rt_next_start, such 3997 * that, if the next request of the application does not arrive before 3998 * soft_rt_next_start, then the above requirement on the bandwidth is met. 3999 * 4000 * The second requirement is that the request pattern of the application is
4001 * isochronous, i.e., that, after issuing a request or a batch of requests, 4002 * the application stops issuing new requests until all its pending requests 4003 * have been completed. After that, the application may issue a new batch, 4004 * and so on. 4005 * For this reason the next function is invoked to compute 4006 * soft_rt_next_start only for applications that meet this requirement, 4007 * whereas soft_rt_next_start is set to infinity for applications that do 4008 * not. 4009 * 4010 * Unfortunately, even a greedy (i.e., I/O-bound) application may 4011 * happen to meet, occasionally or systematically, both the above 4012 * bandwidth and isochrony requirements. This may happen at least in 4013 * the following circumstances. First, if the CPU load is high. The 4014 * application may stop issuing requests while the CPUs are busy 4015 * serving other processes, then restart, then stop again for a while, 4016 * and so on. The other circumstances are related to the storage 4017 * device: the storage device is highly loaded or reaches a low-enough 4018 * throughput with the I/O of the application (e.g., because the I/O 4019 * is random and/or the device is slow). In all these cases, the 4020 * I/O of the application may be simply slowed down enough to meet 4021 * the bandwidth and isochrony requirements. To reduce the probability 4022 * that greedy applications are deemed as soft real-time in these 4023 * corner cases, a further rule is used in the computation of 4024 * soft_rt_next_start: the return value of this function is forced to 4025 * be higher than the maximum between the following two quantities. 4026 * 4027 * (a) Current time plus: (1) the maximum time for which the arrival 4028 * of a request is waited for when a sync queue becomes idle, 4029 * namely bfqd->bfq_slice_idle, and (2) a few extra jiffies. We 4030 * postpone for a moment the reason for adding a few extra 4031 * jiffies; we get back to it after next item (b). Lower-bounding 4032 * the return value of this function with the current time plus 4033 * bfqd->bfq_slice_idle tends to filter out greedy applications, 4034 * because the latter issue their next request as soon as possible 4035 * after the last one has been completed. In contrast, a soft 4036 * real-time application spends some time processing data, after a 4037 * batch of its requests has been completed. 4038 * 4039 * (b) Current value of bfqq->soft_rt_next_start. As pointed out 4040 * above, greedy applications may happen to meet both the 4041 * bandwidth and isochrony requirements under heavy CPU or 4042 * storage-device load. In more detail, in these scenarios, these 4043 * applications happen, only for limited time periods, to do I/O 4044 * slowly enough to meet all the requirements described so far, 4045 * including the filtering in above item (a). These slow-speed 4046 * time intervals are usually interspersed between other time 4047 * intervals during which these applications do I/O at a very high 4048 * speed. Fortunately, exactly because of the high speed of the 4049 * I/O in the high-speed intervals, the values returned by this 4050 * function happen to be so high, near the end of any such 4051 * high-speed interval, to be likely to fall *after* the end of 4052 * the low-speed time interval that follows. These high values are 4053 * stored in bfqq->soft_rt_next_start after each invocation of 4054 * this function. As a consequence, if the last value of 4055 * bfqq->soft_rt_next_start is constantly used to lower-bound the 4056 * next value that this function may return, then, from the very 4057 * beginning of a low-speed interval, bfqq->soft_rt_next_start is 4058 * likely to be constantly kept so high that any I/O request 4059 * issued during the low-speed interval is considered as arriving 4060 * to soon for the application to be deemed as soft 4061 * real-time. Then, in the high-speed interval that follows, the 4062 * application will not be deemed as soft real-time, just because 4063 * it will do I/O at a high speed. And so on. 4064 * 4065 * Getting back to the filtering in item (a), in the following two 4066 * cases this filtering might be easily passed by a greedy 4067 * application, if the reference quantity was just 4068 * bfqd->bfq_slice_idle: 4069 * 1) HZ is so low that the duration of a jiffy is comparable to or 4070 * higher than bfqd->bfq_slice_idle. This happens, e.g., on slow 4071 * devices with HZ=100. The time granularity may be so coarse 4072 * that the approximation, in jiffies, of bfqd->bfq_slice_idle 4073 * is rather lower than the exact value. 4074 * 2) jiffies, instead of increasing at a constant rate, may stop increasing 4075 * for a while, then suddenly 'jump' by several units to recover the lost 4076 * increments. This seems to happen, e.g., inside virtual machines. 4077 * To address this issue, in the filtering in (a) we do not use as a 4078 * reference time interval just bfqd->bfq_slice_idle, but 4079 * bfqd->bfq_slice_idle plus a few jiffies. In particular, we add the 4080 * minimum number of jiffies for which the filter seems to be quite 4081 * precise also in embedded systems and KVM/QEMU virtual machines. 4082 */ 4083static unsigned long bfq_bfqq_softrt_next_start(struct bfq_data *bfqd, 4084 struct bfq_queue *bfqq) 4085{ 4086 return max3(bfqq->soft_rt_next_start, 4087 bfqq->last_idle_bklogged + 4088 HZ * bfqq->service_from_backlogged / 4089 bfqd->bfq_wr_max_softrt_rate, 4090 jiffies + nsecs_to_jiffies(bfqq->bfqd->bfq_slice_idle) + 4); 4091} 4092 4093/** 4094 * bfq_bfqq_expire - expire a queue. 4095 * @bfqd: device owning the queue. 4096 * @bfqq: the queue to expire. 4097 * @compensate: if true, compensate for the time spent idling. 4098 * @reason: the reason causing the expiration. 4099 * 4100 * If the process associated with bfqq does slow I/O (e.g., because it 4101 * issues random requests), we charge bfqq with the time it has been 4102 * in service instead of the service it has received (see 4103 * bfq_bfqq_charge_time for details on how this goal is achieved). As 4104 * a consequence, bfqq will typically get higher timestamps upon 4105 * reactivation, and hence it will be rescheduled as if it had 4106 * received more service than what it has actually received. In the 4107 * end, bfqq receives less service in proportion to how slowly its 4108 * associated process consumes its budgets (and hence how seriously it 4109 * tends to lower the throughput). In addition, this time-charging 4110 * strategy guarantees time fairness among slow processes. In 4111 * contrast, if the process associated with bfqq is not slow, we 4112 * charge bfqq exactly with the service it has received. 4113 * 4114 * Charging time to the first type of queues and the exact service to 4115 * the other has the effect of using the WF2Q+ policy to schedule the 4116 * former on a timeslice basis, without violating service domain 4117 * guarantees among the latter. 4118 */ 4119void bfq_bfqq_expire(struct bfq_data *bfqd, 4120 struct bfq_queue *bfqq, 4121 bool compensate, 4122 enum bfqq_expiration reason) 4123{ 4124 bool slow; 4125 unsigned long delta = 0; 4126 struct bfq_entity *entity = &bfqq->entity; 4127 4128 /* 4129 * Check whether the process is slow (see bfq_bfqq_is_slow). 4130 */ 4131 slow = bfq_bfqq_is_slow(bfqd, bfqq, compensate, reason, &delta); 4132 4133 /* 4134 * As above explained, charge slow (typically seeky) and 4135 * timed-out queues with the time and not the service 4136 * received, to favor sequential workloads. 4137 * 4138 * Processes doing I/O in the slower disk zones will tend to 4139 * be slow(er) even if not seeky. Therefore, since the 4140 * estimated peak rate is actually an average over the disk 4141 * surface, these processes may timeout just for bad luck. To 4142 * avoid punishing them, do not charge time to processes that 4143 * succeeded in consuming at least 2/3 of their budget. This 4144 * allows BFQ to preserve enough elasticity to still perform 4145 * bandwidth, and not time, distribution with little unlucky 4146 * or quasi-sequential processes. 4147 */ 4148 if (bfqq->wr_coeff == 1 && 4149 (slow || 4150 (reason == BFQQE_BUDGET_TIMEOUT && 4151 bfq_bfqq_budget_left(bfqq) >= entity->budget / 3))) 4152 bfq_bfqq_charge_time(bfqd, bfqq, delta); 4153 4154 if (bfqd->low_latency && bfqq->wr_coeff == 1) 4155 bfqq->last_wr_start_finish = jiffies; 4156 4157 if (bfqd->low_latency && bfqd->bfq_wr_max_softrt_rate > 0 && 4158 RB_EMPTY_ROOT(&bfqq->sort_list)) { 4159 /* 4160 * If we get here, and there are no outstanding 4161 * requests, then the request pattern is isochronous 4162 * (see the comments on the function 4163 * bfq_bfqq_softrt_next_start()). Therefore we can 4164 * compute soft_rt_next_start. 4165 * 4166 * If, instead, the queue still has outstanding 4167 * requests, then we have to wait for the completion 4168 * of all the outstanding requests to discover whether 4169 * the request pattern is actually isochronous. 4170 */ 4171 if (bfqq->dispatched == 0) 4172 bfqq->soft_rt_next_start = 4173 bfq_bfqq_softrt_next_start(bfqd, bfqq); 4174 else if (bfqq->dispatched > 0) { 4175 /* 4176 * Schedule an update of soft_rt_next_start to when 4177 * the task may be discovered to be isochronous. 4178 */ 4179 bfq_mark_bfqq_softrt_update(bfqq); 4180 } 4181 } 4182 4183 bfq_log_bfqq(bfqd, bfqq, 4184 "expire (%d, slow %d, num_disp %d, short_ttime %d)", reason, 4185 slow, bfqq->dispatched, bfq_bfqq_has_short_ttime(bfqq)); 4186 4187 /* 4188 * bfqq expired, so no total service time needs to be computed 4189 * any longer: reset state machine for measuring total service 4190 * times. 4191 */ 4192 bfqd->rqs_injected = bfqd->wait_dispatch = false; 4193 bfqd->waited_rq = NULL; 4194 4195 /* 4196 * Increase, decrease or leave budget unchanged according to 4197 * reason. 4198 */ 4199 __bfq_bfqq_recalc_budget(bfqd, bfqq, reason); 4200 if (__bfq_bfqq_expire(bfqd, bfqq, reason)) 4201 /* bfqq is gone, no more actions on it */ 4202 return; 4203 4204 /* mark bfqq as waiting a request only if a bic still points to it */ 4205 if (!bfq_bfqq_busy(bfqq) && 4206 reason != BFQQE_BUDGET_TIMEOUT && 4207 reason != BFQQE_BUDGET_EXHAUSTED) { 4208 bfq_mark_bfqq_non_blocking_wait_rq(bfqq); 4209 /* 4210 * Not setting service to 0, because, if the next rq 4211 * arrives in time, the queue will go on receiving 4212 * service with this same budget (as if it never expired) 4213 */ 4214 } else 4215 entity->service = 0; 4216 4217 /* 4218 * Reset the received-service counter for every parent entity. 4219 * Differently from what happens with bfqq->entity.service, 4220 * the resetting of this counter never needs to be postponed 4221 * for parent entities. In fact, in case bfqq may have a 4222 * chance to go on being served using the last, partially 4223 * consumed budget, bfqq->entity.service needs to be kept, 4224 * because if bfqq then actually goes on being served using 4225 * the same budget, the last value of bfqq->entity.service is 4226 * needed to properly decrement bfqq->entity.budget by the 4227 * portion already consumed. In contrast, it is not necessary 4228 * to keep entity->service for parent entities too, because 4229 * the bubble up of the new value of bfqq->entity.budget will 4230 * make sure that the budgets of parent entities are correct, 4231 * even in case bfqq and thus parent entities go on receiving 4232 * service with the same budget. 4233 */ 4234 entity = entity->parent; 4235 for_each_entity(entity) 4236 entity->service = 0; 4237} 4238 4239/* 4240 * Budget timeout is not implemented through a dedicated timer, but 4241 * just checked on request arrivals and completions, as well as on 4242 * idle timer expirations. 4243 */ 4244static bool bfq_bfqq_budget_timeout(struct bfq_queue *bfqq) 4245{ 4246 return time_is_before_eq_jiffies(bfqq->budget_timeout); 4247} 4248 4249/* 4250 * If we expire a queue that is actively waiting (i.e., with the 4251 * device idled) for the arrival of a new request, then we may incur 4252 * the timestamp misalignment problem described in the body of the 4253 * function __bfq_activate_entity. Hence we return true only if this 4254 * condition does not hold, or if the queue is slow enough to deserve 4255 * only to be kicked off for preserving a high throughput. 4256 */ 4257static bool bfq_may_expire_for_budg_timeout(struct bfq_queue *bfqq) 4258{ 4259 bfq_log_bfqq(bfqq->bfqd, bfqq, 4260 "may_budget_timeout: wait_request %d left %d timeout %d", 4261 bfq_bfqq_wait_request(bfqq), 4262 bfq_bfqq_budget_left(bfqq) >= bfqq->entity.budget / 3, 4263 bfq_bfqq_budget_timeout(bfqq)); 4264 4265 return (!bfq_bfqq_wait_request(bfqq) || 4266 bfq_bfqq_budget_left(bfqq) >= bfqq->entity.budget / 3) 4267 && 4268 bfq_bfqq_budget_timeout(bfqq); 4269} 4270 4271static bool idling_boosts_thr_without_issues(struct bfq_data *bfqd, 4272 struct bfq_queue *bfqq) 4273{ 4274 bool rot_without_queueing = 4275 !blk_queue_nonrot(bfqd->queue) && !bfqd->hw_tag, 4276 bfqq_sequential_and_IO_bound, 4277 idling_boosts_thr; 4278 4279 /* No point in idling for bfqq if it won't get requests any longer */ 4280 if (unlikely(!bfqq_process_refs(bfqq))) 4281 return false; 4282 4283 bfqq_sequential_and_IO_bound = !BFQQ_SEEKY(bfqq) && 4284 bfq_bfqq_IO_bound(bfqq) && bfq_bfqq_has_short_ttime(bfqq); 4285 4286 /* 4287 * The next variable takes into account the cases where idling 4288 * boosts the throughput. 4289 * 4290 * The value of the variable is computed considering, first, that 4291 * idling is virtually always beneficial for the throughput if: 4292 * (a) the device is not NCQ-capable and rotational, or 4293 * (b) regardless of the presence of NCQ, the device is rotational and 4294 * the request pattern for bfqq is I/O-bound and sequential, or 4295 * (c) regardless of whether it is rotational, the device is 4296 * not NCQ-capable and the request pattern for bfqq is 4297 * I/O-bound and sequential. 4298 * 4299 * Secondly, and in contrast to the above item (b), idling an 4300 * NCQ-capable flash-based device would not boost the 4301 * throughput even with sequential I/O; rather it would lower 4302 * the throughput in proportion to how fast the device 4303 * is. Accordingly, the next variable is true if any of the 4304 * above conditions (a), (b) or (c) is true, and, in 4305 * particular, happens to be false if bfqd is an NCQ-capable 4306 * flash-based device. 4307 */ 4308 idling_boosts_thr = rot_without_queueing || 4309 ((!blk_queue_nonrot(bfqd->queue) || !bfqd->hw_tag) && 4310 bfqq_sequential_and_IO_bound); 4311 4312 /* 4313 * The return value of this function is equal to that of 4314 * idling_boosts_thr, unless a special case holds. In this 4315 * special case, described below, idling may cause problems to 4316 * weight-raised queues. 4317 * 4318 * When the request pool is saturated (e.g., in the presence 4319 * of write hogs), if the processes associated with 4320 * non-weight-raised queues ask for requests at a lower rate, 4321 * then processes associated with weight-raised queues have a 4322 * higher probability to get a request from the pool 4323 * immediately (or at least soon) when they need one. Thus 4324 * they have a higher probability to actually get a fraction 4325 * of the device throughput proportional to their high 4326 * weight. This is especially true with NCQ-capable drives, 4327 * which enqueue several requests in advance, and further 4328 * reorder internally-queued requests. 4329 * 4330 * For this reason, we force to false the return value if 4331 * there are weight-raised busy queues. In this case, and if 4332 * bfqq is not weight-raised, this guarantees that the device 4333 * is not idled for bfqq (if, instead, bfqq is weight-raised, 4334 * then idling will be guaranteed by another variable, see 4335 * below). Combined with the timestamping rules of BFQ (see 4336 * [1] for details), this behavior causes bfqq, and hence any 4337 * sync non-weight-raised queue, to get a lower number of 4338 * requests served, and thus to ask for a lower number of 4339 * requests from the request pool, before the busy 4340 * weight-raised queues get served again. This often mitigates 4341 * starvation problems in the presence of heavy write 4342 * workloads and NCQ, thereby guaranteeing a higher 4343 * application and system responsiveness in these hostile 4344 * scenarios. 4345 */ 4346 return idling_boosts_thr && 4347 bfqd->wr_busy_queues == 0; 4348} 4349 4350/* 4351 * For a queue that becomes empty, device idling is allowed only if 4352 * this function returns true for that queue. As a consequence, since 4353 * device idling plays a critical role for both throughput boosting 4354 * and service guarantees, the return value of this function plays a 4355 * critical role as well. 4356 * 4357 * In a nutshell, this function returns true only if idling is 4358 * beneficial for throughput or, even if detrimental for throughput, 4359 * idling is however necessary to preserve service guarantees (low 4360 * latency, desired throughput distribution, ...). In particular, on 4361 * NCQ-capable devices, this function tries to return false, so as to 4362 * help keep the drives' internal queues full, whenever this helps the 4363 * device boost the throughput without causing any service-guarantee 4364 * issue. 4365 * 4366 * Most of the issues taken into account to get the return value of 4367 * this function are not trivial. We discuss these issues in the two 4368 * functions providing the main pieces of information needed by this 4369 * function. 4370 */ 4371static bool bfq_better_to_idle(struct bfq_queue *bfqq) 4372{ 4373 struct bfq_data *bfqd = bfqq->bfqd; 4374 bool idling_boosts_thr_with_no_issue, idling_needed_for_service_guar; 4375 4376 /* No point in idling for bfqq if it won't get requests any longer */ 4377 if (unlikely(!bfqq_process_refs(bfqq))) 4378 return false; 4379 4380 if (unlikely(bfqd->strict_guarantees)) 4381 return true; 4382 4383 /* 4384 * Idling is performed only if slice_idle > 0. In addition, we 4385 * do not idle if 4386 * (a) bfqq is async 4387 * (b) bfqq is in the idle io prio class: in this case we do 4388 * not idle because we want to minimize the bandwidth that 4389 * queues in this class can steal to higher-priority queues 4390 */ 4391 if (bfqd->bfq_slice_idle == 0 || !bfq_bfqq_sync(bfqq) || 4392 bfq_class_idle(bfqq)) 4393 return false; 4394 4395 idling_boosts_thr_with_no_issue = 4396 idling_boosts_thr_without_issues(bfqd, bfqq); 4397 4398 idling_needed_for_service_guar = 4399 idling_needed_for_service_guarantees(bfqd, bfqq); 4400 4401 /* 4402 * We have now the two components we need to compute the 4403 * return value of the function, which is true only if idling 4404 * either boosts the throughput (without issues), or is 4405 * necessary to preserve service guarantees. 4406 */ 4407 return idling_boosts_thr_with_no_issue || 4408 idling_needed_for_service_guar; 4409} 4410 4411/* 4412 * If the in-service queue is empty but the function bfq_better_to_idle 4413 * returns true, then: 4414 * 1) the queue must remain in service and cannot be expired, and 4415 * 2) the device must be idled to wait for the possible arrival of a new 4416 * request for the queue. 4417 * See the comments on the function bfq_better_to_idle for the reasons 4418 * why performing device idling is the best choice to boost the throughput 4419 * and preserve service guarantees when bfq_better_to_idle itself 4420 * returns true. 4421 */ 4422static bool bfq_bfqq_must_idle(struct bfq_queue *bfqq) 4423{ 4424 return RB_EMPTY_ROOT(&bfqq->sort_list) && bfq_better_to_idle(bfqq); 4425} 4426 4427/* 4428 * This function chooses the queue from which to pick the next extra 4429 * I/O request to inject, if it finds a compatible queue. See the 4430 * comments on bfq_update_inject_limit() for details on the injection 4431 * mechanism, and for the definitions of the quantities mentioned 4432 * below. 4433 */ 4434static struct bfq_queue * 4435bfq_choose_bfqq_for_injection(struct bfq_data *bfqd) 4436{ 4437 struct bfq_queue *bfqq, *in_serv_bfqq = bfqd->in_service_queue; 4438 unsigned int limit = in_serv_bfqq->inject_limit; 4439 /* 4440 * If 4441 * - bfqq is not weight-raised and therefore does not carry 4442 * time-critical I/O, 4443 * or 4444 * - regardless of whether bfqq is weight-raised, bfqq has 4445 * however a long think time, during which it can absorb the 4446 * effect of an appropriate number of extra I/O requests 4447 * from other queues (see bfq_update_inject_limit for 4448 * details on the computation of this number); 4449 * then injection can be performed without restrictions. 4450 */ 4451 bool in_serv_always_inject = in_serv_bfqq->wr_coeff == 1 || 4452 !bfq_bfqq_has_short_ttime(in_serv_bfqq); 4453 4454 /* 4455 * If 4456 * - the baseline total service time could not be sampled yet, 4457 * so the inject limit happens to be still 0, and 4458 * - a lot of time has elapsed since the plugging of I/O 4459 * dispatching started, so drive speed is being wasted 4460 * significantly; 4461 * then temporarily raise inject limit to one request. 4462 */ 4463 if (limit == 0 && in_serv_bfqq->last_serv_time_ns == 0 && 4464 bfq_bfqq_wait_request(in_serv_bfqq) && 4465 time_is_before_eq_jiffies(bfqd->last_idling_start_jiffies + 4466 bfqd->bfq_slice_idle) 4467 ) 4468 limit = 1; 4469 4470 if (bfqd->rq_in_driver >= limit) 4471 return NULL; 4472 4473 /* 4474 * Linear search of the source queue for injection; but, with 4475 * a high probability, very few steps are needed to find a 4476 * candidate queue, i.e., a queue with enough budget left for 4477 * its next request. In fact: 4478 * - BFQ dynamically updates the budget of every queue so as 4479 * to accommodate the expected backlog of the queue; 4480 * - if a queue gets all its requests dispatched as injected 4481 * service, then the queue is removed from the active list 4482 * (and re-added only if it gets new requests, but then it 4483 * is assigned again enough budget for its new backlog). 4484 */ 4485 list_for_each_entry(bfqq, &bfqd->active_list, bfqq_list) 4486 if (!RB_EMPTY_ROOT(&bfqq->sort_list) && 4487 (in_serv_always_inject || bfqq->wr_coeff > 1) && 4488 bfq_serv_to_charge(bfqq->next_rq, bfqq) <= 4489 bfq_bfqq_budget_left(bfqq)) { 4490 /* 4491 * Allow for only one large in-flight request 4492 * on non-rotational devices, for the 4493 * following reason. On non-rotationl drives, 4494 * large requests take much longer than 4495 * smaller requests to be served. In addition, 4496 * the drive prefers to serve large requests 4497 * w.r.t. to small ones, if it can choose. So, 4498 * having more than one large requests queued 4499 * in the drive may easily make the next first 4500 * request of the in-service queue wait for so 4501 * long to break bfqq's service guarantees. On 4502 * the bright side, large requests let the 4503 * drive reach a very high throughput, even if 4504 * there is only one in-flight large request 4505 * at a time. 4506 */ 4507 if (blk_queue_nonrot(bfqd->queue) && 4508 blk_rq_sectors(bfqq->next_rq) >= 4509 BFQQ_SECT_THR_NONROT) 4510 limit = min_t(unsigned int, 1, limit); 4511 else 4512 limit = in_serv_bfqq->inject_limit; 4513 4514 if (bfqd->rq_in_driver < limit) { 4515 bfqd->rqs_injected = true; 4516 return bfqq; 4517 } 4518 } 4519 4520 return NULL; 4521} 4522 4523/* 4524 * Select a queue for service. If we have a current queue in service, 4525 * check whether to continue servicing it, or retrieve and set a new one. 4526 */ 4527static struct bfq_queue *bfq_select_queue(struct bfq_data *bfqd) 4528{ 4529 struct bfq_queue *bfqq; 4530 struct request *next_rq; 4531 enum bfqq_expiration reason = BFQQE_BUDGET_TIMEOUT; 4532 4533 bfqq = bfqd->in_service_queue; 4534 if (!bfqq) 4535 goto new_queue; 4536 4537 bfq_log_bfqq(bfqd, bfqq, "select_queue: already in-service queue"); 4538 4539 /* 4540 * Do not expire bfqq for budget timeout if bfqq may be about 4541 * to enjoy device idling. The reason why, in this case, we 4542 * prevent bfqq from expiring is the same as in the comments 4543 * on the case where bfq_bfqq_must_idle() returns true, in 4544 * bfq_completed_request(). 4545 */ 4546 if (bfq_may_expire_for_budg_timeout(bfqq) && 4547 !bfq_bfqq_must_idle(bfqq)) 4548 goto expire; 4549 4550check_queue: 4551 /* 4552 * This loop is rarely executed more than once. Even when it 4553 * happens, it is much more convenient to re-execute this loop 4554 * than to return NULL and trigger a new dispatch to get a 4555 * request served. 4556 */ 4557 next_rq = bfqq->next_rq; 4558 /* 4559 * If bfqq has requests queued and it has enough budget left to 4560 * serve them, keep the queue, otherwise expire it. 4561 */ 4562 if (next_rq) { 4563 if (bfq_serv_to_charge(next_rq, bfqq) > 4564 bfq_bfqq_budget_left(bfqq)) { 4565 /* 4566 * Expire the queue for budget exhaustion, 4567 * which makes sure that the next budget is 4568 * enough to serve the next request, even if 4569 * it comes from the fifo expired path. 4570 */ 4571 reason = BFQQE_BUDGET_EXHAUSTED; 4572 goto expire; 4573 } else { 4574 /* 4575 * The idle timer may be pending because we may 4576 * not disable disk idling even when a new request 4577 * arrives. 4578 */ 4579 if (bfq_bfqq_wait_request(bfqq)) { 4580 /* 4581 * If we get here: 1) at least a new request 4582 * has arrived but we have not disabled the 4583 * timer because the request was too small, 4584 * 2) then the block layer has unplugged 4585 * the device, causing the dispatch to be 4586 * invoked. 4587 * 4588 * Since the device is unplugged, now the 4589 * requests are probably large enough to 4590 * provide a reasonable throughput. 4591 * So we disable idling. 4592 */ 4593 bfq_clear_bfqq_wait_request(bfqq); 4594 hrtimer_try_to_cancel(&bfqd->idle_slice_timer); 4595 } 4596 goto keep_queue; 4597 } 4598 } 4599 4600 /* 4601 * No requests pending. However, if the in-service queue is idling 4602 * for a new request, or has requests waiting for a completion and 4603 * may idle after their completion, then keep it anyway. 4604 * 4605 * Yet, inject service from other queues if it boosts 4606 * throughput and is possible. 4607 */ 4608 if (bfq_bfqq_wait_request(bfqq) || 4609 (bfqq->dispatched != 0 && bfq_better_to_idle(bfqq))) { 4610 struct bfq_queue *async_bfqq = 4611 bfqq->bic && bfqq->bic->bfqq[0] && 4612 bfq_bfqq_busy(bfqq->bic->bfqq[0]) && 4613 bfqq->bic->bfqq[0]->next_rq ? 4614 bfqq->bic->bfqq[0] : NULL; 4615 struct bfq_queue *blocked_bfqq = 4616 !hlist_empty(&bfqq->woken_list) ? 4617 container_of(bfqq->woken_list.first, 4618 struct bfq_queue, 4619 woken_list_node) 4620 : NULL; 4621 4622 /* 4623 * The next four mutually-exclusive ifs decide 4624 * whether to try injection, and choose the queue to 4625 * pick an I/O request from. 4626 * 4627 * The first if checks whether the process associated 4628 * with bfqq has also async I/O pending. If so, it 4629 * injects such I/O unconditionally. Injecting async 4630 * I/O from the same process can cause no harm to the 4631 * process. On the contrary, it can only increase 4632 * bandwidth and reduce latency for the process. 4633 * 4634 * The second if checks whether there happens to be a 4635 * non-empty waker queue for bfqq, i.e., a queue whose 4636 * I/O needs to be completed for bfqq to receive new 4637 * I/O. This happens, e.g., if bfqq is associated with 4638 * a process that does some sync. A sync generates 4639 * extra blocking I/O, which must be completed before 4640 * the process associated with bfqq can go on with its 4641 * I/O. If the I/O of the waker queue is not served, 4642 * then bfqq remains empty, and no I/O is dispatched, 4643 * until the idle timeout fires for bfqq. This is 4644 * likely to result in lower bandwidth and higher 4645 * latencies for bfqq, and in a severe loss of total 4646 * throughput. The best action to take is therefore to 4647 * serve the waker queue as soon as possible. So do it 4648 * (without relying on the third alternative below for 4649 * eventually serving waker_bfqq's I/O; see the last 4650 * paragraph for further details). This systematic 4651 * injection of I/O from the waker queue does not 4652 * cause any delay to bfqq's I/O. On the contrary, 4653 * next bfqq's I/O is brought forward dramatically, 4654 * for it is not blocked for milliseconds. 4655 * 4656 * The third if checks whether there is a queue woken 4657 * by bfqq, and currently with pending I/O. Such a 4658 * woken queue does not steal bandwidth from bfqq, 4659 * because it remains soon without I/O if bfqq is not 4660 * served. So there is virtually no risk of loss of 4661 * bandwidth for bfqq if this woken queue has I/O 4662 * dispatched while bfqq is waiting for new I/O. 4663 * 4664 * The fourth if checks whether bfqq is a queue for 4665 * which it is better to avoid injection. It is so if 4666 * bfqq delivers more throughput when served without 4667 * any further I/O from other queues in the middle, or 4668 * if the service times of bfqq's I/O requests both 4669 * count more than overall throughput, and may be 4670 * easily increased by injection (this happens if bfqq 4671 * has a short think time). If none of these 4672 * conditions holds, then a candidate queue for 4673 * injection is looked for through 4674 * bfq_choose_bfqq_for_injection(). Note that the 4675 * latter may return NULL (for example if the inject 4676 * limit for bfqq is currently 0). 4677 * 4678 * NOTE: motivation for the second alternative 4679 * 4680 * Thanks to the way the inject limit is updated in 4681 * bfq_update_has_short_ttime(), it is rather likely 4682 * that, if I/O is being plugged for bfqq and the 4683 * waker queue has pending I/O requests that are 4684 * blocking bfqq's I/O, then the fourth alternative 4685 * above lets the waker queue get served before the 4686 * I/O-plugging timeout fires. So one may deem the 4687 * second alternative superfluous. It is not, because 4688 * the fourth alternative may be way less effective in 4689 * case of a synchronization. For two main 4690 * reasons. First, throughput may be low because the 4691 * inject limit may be too low to guarantee the same 4692 * amount of injected I/O, from the waker queue or 4693 * other queues, that the second alternative 4694 * guarantees (the second alternative unconditionally 4695 * injects a pending I/O request of the waker queue 4696 * for each bfq_dispatch_request()). Second, with the 4697 * fourth alternative, the duration of the plugging, 4698 * i.e., the time before bfqq finally receives new I/O, 4699 * may not be minimized, because the waker queue may 4700 * happen to be served only after other queues. 4701 */ 4702 if (async_bfqq && 4703 icq_to_bic(async_bfqq->next_rq->elv.icq) == bfqq->bic && 4704 bfq_serv_to_charge(async_bfqq->next_rq, async_bfqq) <= 4705 bfq_bfqq_budget_left(async_bfqq)) 4706 bfqq = bfqq->bic->bfqq[0]; 4707 else if (bfqq->waker_bfqq && 4708 bfq_bfqq_busy(bfqq->waker_bfqq) && 4709 bfqq->waker_bfqq->next_rq && 4710 bfq_serv_to_charge(bfqq->waker_bfqq->next_rq, 4711 bfqq->waker_bfqq) <= 4712 bfq_bfqq_budget_left(bfqq->waker_bfqq) 4713 ) 4714 bfqq = bfqq->waker_bfqq; 4715 else if (blocked_bfqq &am