kernel-land design
------------------
-In the kernel, threads registered in the pwq regulator can be in 4 states:
+In the kernel, threads registered in the pwq regulator can be in 5 states:
blocked::
This is the state of threads that are curently blocked in a syscall.
schedulable threads.
waiting::
- This is the state of threads that are currently in a `PWQR_WAIT` call
- from userspace (see `pwqr_ctl`) but that would not overcommit if
- released by a `PWQR_WAKE` call.
+ This is the state of threads that are currently in a `PWQR_CTL_WAIT`
+ call from userspace (see `pwqr_ctl`) but that would not overcommit if
+ released by a `PWQR_CTL_WAKE` call.
quarantined::
- This is the state of threads that are currently in a `PWQR_WAIT` call
- from userspace (see `pwqr_ctl`) but that would overcommit if released
- by a `PWQR_WAKE` call.
+ This is the state of threads that are currently in a `PWQR_CTL_WAIT`
+ call from userspace (see `pwqr_ctl`) but that would overcommit if
+ released by a `PWQR_CTL_WAKE` call.
+
This state avoids waking a thread to force userland to "park" the thread, this
-is racy, make the scheduler work for nothing useful. Though if `PWQR_WAKE` is
-called, quarantined threads are woken but with a `EDQUOT` errno set, and only
-one by one, no matter how wakes have been asked.
+is racy, make the scheduler work for nothing useful. Though if
+`PWQR_CTL_WAKE` is called, quarantined threads are woken but with a `EDQUOT`
+errno set, and only one by one, no matter how wakes have been asked.
+
-This state actually has only one impact: when `PWQR_WAKE` is called for more
-than one threads, for example 4, and that userland knows that there is 5
+This state actually has only one impact: when `PWQR_CTL_WAKE` is called for
+more than one threads, for example 4, and that userland knows that there is 5
threads in WAIT state, but that actually 3 of them are in the quarantine, only
-2 will be woken up, and the `PWQR_WAKE` call will return 2. Any subsequent
-`PWQR_WAKE` call will wake up one quarantined thread to let it be parked, but
-returning 0 each time to hide that from userland.
+2 will be woken up, and the `PWQR_CTL_WAKE` call will return 2. Any subsequent
+`PWQR_CTL_WAKE` call will wake up one quarantined thread to let it be parked,
+but returning 0 each time to hide that from userland.
parked::
- This is the state of threads currently in a `PWQR_PARK` call from
+ This is the state of threads currently in a `PWQR_CTL_PARK` call from
userspace (see `pwqr_ctl`).
require anything from userland. It's something userland discovers only
when it needs a waiting thread, which may never happen.
+
-If there are no waiting threads, then well, the workqueue overcommits, and
-that's one of the TODO items at the moment (see Notes)
+If there are no waiting threads, then well, the workqueue overcommits, and the
+pwqr file descriptor is made available for reading. When userlands reads, it
+will get the amount of overcommit at the time of the `read(3)` call.
++
+NOTE: the availability for reading is only done after a delay (0.05s roughly
+in the current implementation). And once you've read, this enables a new
+notification in another 0.05s and so on, to allow a slow decrease of the
+number of threads.
When `running + waiting` undercommits::
If waiting is non-zero then well, we don't care, it's that userland
There are several ways to "fix" this:
-in kernel (poll solution)::
+in kernel (poll solution, actually implemented right now)::
Let the file descriptor be pollable, and let it be readable (returning
something like the amount of overcommit at read() time for example) so
that userland is notified that it should try to reduce the amount of
the call, as would be returned by `sysconf(_SC_NPROCESSORS_ONLN)`.
`flags`::
- a mask of flags among `O_CLOEXEC`, and `O_NONBLOCK`.
+ a mask of flags among `PWQR_FL_CLOEXEC`, and `PWQR_FL_NONBLOCK`.
Available operations on the pwqr file descriptor are:
`poll`, `epoll` and friends::
- the PWQR file descriptor can be watched for POLLIN events (not POLLOUT
- ones as it can not be written to).
+ the PWQR file descriptor can be watched for `POLLIN` events (not
+ `POLLOUT` ones as it can not be written to).
`read`::
The file returned can be read upon. The read blocks (or fails setting
`EAGAIN` if in non blocking mode) until the regulator believes the
pool is overcommitting. The buffer passed to read should be able to
- hold an integer. When `read(3)` is successful, it writes the amount of
- overcommiting threads (understand: the number of threads to park so
- that the pool isn't overcommiting anymore).
+ hold an integer. When `read(3)` is successful, it returns in the
+ buffer passed to `read(3)` the amount of overcommiting threads
+ (understand: the number of threads to park so that the pool isn't
+ overcommiting anymore).
RETURN VALUE
~~~~~~~~~~~~
Valid values for the `op` argument are:
-`PWQR_GET_CONC`::
+`PWQR_CTL_GET_CONC`::
Requests the current concurrency level for this regulator.
-`PWQR_SET_CONC`::
+`PWQR_CTL_SET_CONC`::
Modifies the current concurrency level for this regulator. The new
value is passed as the `val` argument. The requests returns the old
concurrency level on success.
A zero or negative value for `val` means 'automatic' and is recomputed as the
current number of online CPUs as `sysconf(_SC_NPROCESSORS_ONLN)` would return.
-`PWQR_REGISTER`::
+`PWQR_CTL_REGISTER`::
Registers the calling thread to be taken into account by the pool
regulator. If the thread is already registered into another regulator,
then it's automatically unregistered from it.
-`PWQR_UNREGISTER`::
+`PWQR_CTL_UNREGISTER`::
Deregisters the calling thread from the pool regulator.
-`PWQR_WAKE`::
+`PWQR_CTL_WAKE`::
Tries to wake `val` threads from the pool. This is done according to
the current concurrency level not to overcommit. On success, a hint of
the number of woken threads is returned, it can be 0.
counter of waiting threads to decrease, but we know the thread won't be usable
so we return 0.
-`PWQR_WAKE_OC`::
+`PWQR_CTL_WAKE_OC`::
Tries to wake `val` threads from the pool. This is done bypassing the
current concurrency level (`OC` stands for `OVERCOMMIT`). On success,
the number of woken threads is returned, it can be 0, but it's the
real count that has been (or will soon be) woken up. If it's less than
required, it's because there aren't enough parked threads.
-`PWQR_WAIT`::
- Puts the thread to wait for a future `PWQR_WAKE` command. If this
+`PWQR_CTL_WAIT`::
+ Puts the thread to wait for a future `PWQR_CTL_WAKE` command. If this
thread must be parked to maintain concurrency below the target, then
the call blocks with no further ado.
+
If the concurrency level is below the target, then the kernel checks if the
address `addr` still contains the value `val` (in the fashion of `futex(2)`).
If it doesn't then the call doesn't block. Else the calling thread is blocked
-until a `PWQR_WAKE` command is received.
+until a `PWQR_CTL_WAKE` command is received.
+
`addr` must of course be a pointer to an aligned integer which stores the
reference ticket in userland.
-`PWQR_PARK`::
+`PWQR_CTL_PARK`::
Puts the thread in park mode. Those are spare threads to avoid
cloning/exiting threads when the pool is regulated. Those threads are
released by the regulator only, and can only be woken from userland
- with the `PWQR_WAKE_OC` command, and once all waiting threads have
+ with the `PWQR_CTL_WAKE_OC` command, and once all waiting threads have
been woken.
+
The call blocks until an overcommiting wake requires the thread, or the kernel
[EINVAL]::
TODO
-Errors specific to `PWQR_REGISTER`:
+Errors specific to `PWQR_CTL_REGISTER`:
[ENOMEM]::
There was insufficient memory to perform the operation.
-Errors specific to `PWQR_WAIT`:
+Errors specific to `PWQR_CTL_WAIT`:
[EWOULDBLOCK]::
When the kernel evaluated if `addr` still contained `val` it didn't.
This works like `futex(2)`.
-Errors specific to `PWQR_WAIT` and `PWQR_PARK`:
+Errors specific to `PWQR_CTL_WAIT` and `PWQR_CTL_PARK`:
[EINTR]::
The call was interrupted by a syscall (note that sometimes the kernel
masks this fact when it has more important "errors" to report like
`EDQUOT`).
[EDQUOT]::
- The thread has been woken by a `PWQR_WAKE` or `PWQR_WAKE_OC` call, but
- is overcommiting.
+ The thread has been woken by a `PWQR_CTL_WAKE` or `PWQR_CTL_WAKE_OC`
+ call, but is overcommiting.
projects / ~madcoder / pwqr.git / blobdiff