implement poll and read

[~madcoder/pwqr.git] / Documentation / pwqr.adoc

diff --git a/Documentation/pwqr.adoc b/Documentation/pwqr.adoc
index 067b2ae..d028a9c 100644 (file)
--- a/Documentation/pwqr.adoc
+++ b/Documentation/pwqr.adoc
@@ -30,7 +30,8 @@ quarantined::
 +
 This state avoids waking a thread to force userland to "park" the thread, this
 is racy, make the scheduler work for nothing useful.  Though if `PWQR_WAKE` is
 +
 This state avoids waking a thread to force userland to "park" the thread, this
 is racy, make the scheduler work for nothing useful.  Though if `PWQR_WAKE` is

-called, quarantined threads are woken but with a `EDQUOT` errno set.
+called, quarantined threads are woken but with a `EDQUOT` errno set, and only
+one by one, no matter how wakes have been asked.

 +
 This state actually has only one impact: when `PWQR_WAKE` is called for more
 than one threads, for example 4, and that userland knows that there is 5
 +
 This state actually has only one impact: when `PWQR_WAKE` is called for more
 than one threads, for example 4, and that userland knows that there is 5


@@ -99,22 +100,14 @@ in kernel (poll solution)::
 +
 It sounds very easy, but it has one major drawback: it meaks the pwqfd must be
 somehow registered into the eventloop, and it's not very suitable for a
 +
 It sounds very easy, but it has one major drawback: it meaks the pwqfd must be
 somehow registered into the eventloop, and it's not very suitable for a

-pthread_workqueue implementation.
-
-in kernel (hack-ish solution)::
-       The kernel could voluntarily unpark/unblock a thread with another
-       errno that would signal overcommiting. Unlike the pollable proposal,
-       this doesn't require hooking in the event loop. Though it requires
-       having one such thread, which may not be the case when userland has
-       reached the peak number of threads it would ever want to use.
+pthread_workqueue implementation. In other words, if you can plug into the
+event-loop because it's a custom one or one that provides thread regulation
+then it's fine, if you can't (glib, libdispatch, ...) then you need a thread
+that will basically just poll() on this file-descriptor, it's really wasteful.

 +
 +

-Is this really a problem? I'm not sure. Especially since when that happens
-userland could pick a victim thread that would call `PWQR_PARK` after each
-processed job, which would allow some kind of poor man's poll.
-+
-The drawback I see in that solution is that we wake up YET ANOTHER thread at a
-moment when we're already overcommiting, which sounds counter productive.
-That's why I didn't implement that.
+NOTE: this has been implemented now, but still it looks "expensive" to hook
+for some users. So if some alternative way to be signalled could exist, it'd
+be really awesome.

 
 in userspace::
        Userspace knows how many "running" threads there are, it's easy to
 
 in userspace::
        Userspace knows how many "running" threads there are, it's easy to


@@ -122,24 +115,21 @@ in userspace::
        already accounted for. When "waiting" is zero, if "registerd - parked"
        is "High" userspace could choose to randomly try to park one thread.
 +
        already accounted for. When "waiting" is zero, if "registerd - parked"
        is "High" userspace could choose to randomly try to park one thread.
 +

-I think `PWQR_PARK` could use `val` to have some "probing" mode, that would
-return `0` if it wouldn't block and `-1/EWOULDBLOCK` if it would in the non
-probing mode. Userspace could maintain some global probing_mode flag, that
-would be a tristate: NONE, SLOW, AGGRESSVE.
+userspace can use non blocking read() to probe if it's overcommiting.

 +
 It's in NONE when userspace belives it's not necessary to probe (e.g. when the
 amount of running + waiting threads isn't that large, say less than 110% of
 the concurrency or any kind of similar rule).
 +
 It's in SLOW mode else. In slow mode each thread does a probe every 32 or 64
 +
 It's in NONE when userspace belives it's not necessary to probe (e.g. when the
 amount of running + waiting threads isn't that large, say less than 110% of
 the concurrency or any kind of similar rule).
 +
 It's in SLOW mode else. In slow mode each thread does a probe every 32 or 64

-jobs to mitigate the cost of the syscall. If the probe returns EWOULDBLOCK
-then the thread goes to PARK mode, and the probing_mode goes to AGGRESSVE.
+jobs to mitigate the cost of the syscall. If the probe returns '1' then ask
+for down-commiting and stay in SLOW mode, if it returns AGAIN all is fine, if
+it returns more than '1' ask for down-commiting and go to AGGRESSIVE.

 +
 When AGGRESSVE threads check if they must park more often and in a more
 controlled fashion (every 32 or 64 jobs isn't nice because jobs can be very
 long), for example based on some poor man's timer (clock_gettime(MONOTONIC)
 +
 When AGGRESSVE threads check if they must park more often and in a more
 controlled fashion (every 32 or 64 jobs isn't nice because jobs can be very
 long), for example based on some poor man's timer (clock_gettime(MONOTONIC)

-sounds fine). As soon as a probe returns 0 or we're in the NONE conditions,
-then the probing_mode goes back to NONE/SLOW.
+sounds fine). State transition works as for SLOW.

 +
 The issue I have with this is that it sounds to add quite some code in the
 fastpath code, hence I dislike it a lot.
 +
 The issue I have with this is that it sounds to add quite some code in the
 fastpath code, hence I dislike it a lot.


@@ -171,7 +161,21 @@ with a concurrency corresponding to the number of online CPUs at the time of
 the call, as would be returned by `sysconf(_SC_NPROCESSORS_ONLN)`.
 
 `flags`::
 the call, as would be returned by `sysconf(_SC_NPROCESSORS_ONLN)`.
 
 `flags`::

-       a mask of flags, currently only O_CLOEXEC.
+       a mask of flags among `O_CLOEXEC`, and `O_NONBLOCK`.
+
+Available operations on the pwqr file descriptor are:
+
+`poll`, `epoll` and friends::
+       the PWQR file descriptor can be watched for POLLIN events (not POLLOUT
+       ones as it can not be written to).
+
+`read`::
+       The file returned can be read upon. The read blocks (or fails setting
+       `EAGAIN` if in non blocking mode) until the regulator believes the
+       pool is overcommitting. The buffer passed to read should be able to
+       hold an integer. When `read(3)` is successful, it writes the amount of
+       overcommiting threads (understand: the number of threads to park so
+       that the pool isn't overcommiting anymore).

 
 RETURN VALUE
 ~~~~~~~~~~~~
 
 RETURN VALUE
 ~~~~~~~~~~~~


@@ -212,9 +216,8 @@ Valid values for the `op` argument are:
        value is passed as the `val` argument. The requests returns the old
        concurrency level on success.
 +
        value is passed as the `val` argument. The requests returns the old
        concurrency level on success.
 +

-       A zero or negative value for `val` means 'automatic' and is recomputed
-       as the current number of online CPUs as
-       `sysconf(_SC_NPROCESSORS_ONLN)` would return.
+A zero or negative value for `val` means 'automatic' and is recomputed as the
+current number of online CPUs as `sysconf(_SC_NPROCESSORS_ONLN)` would return.

 
 `PWQR_REGISTER`::
        Registers the calling thread to be taken into account by the pool
 
 `PWQR_REGISTER`::
        Registers the calling thread to be taken into account by the pool


@@ -226,13 +229,27 @@ Valid values for the `op` argument are:
 
 `PWQR_WAKE`::
        Tries to wake `val` threads from the pool. This is done according to
 
 `PWQR_WAKE`::
        Tries to wake `val` threads from the pool. This is done according to

-       the current concurrency level not to overcommit. On success, the
-       number of woken threads is returned, it can be 0.
+       the current concurrency level not to overcommit. On success, a hint of
+       the number of woken threads is returned, it can be 0.
++
+This is only a hint of the number of threads woken up for two reasons. First,
+the kernel could really have woken up a thread, but when it becomes scheduled,
+it could *then* decide that it would overcommit (because some other thread
+unblocked inbetween for example), and block it again.
++
+But it can also lie in the other direction: userland is supposed to account
+for waiting threads. So when we're overcommiting and userland want a waiting
+thread to be unblocked, we actually say we woke none, but still unblock one
+(the famous quarantined threads we talk about above). This allow the userland
+counter of waiting threads to decrease, but we know the thread won't be usable
+so we return 0.

 
 `PWQR_WAKE_OC`::
        Tries to wake `val` threads from the pool. This is done bypassing the
        current concurrency level (`OC` stands for `OVERCOMMIT`). On success,
 
 `PWQR_WAKE_OC`::
        Tries to wake `val` threads from the pool. This is done bypassing the
        current concurrency level (`OC` stands for `OVERCOMMIT`). On success,

-       the number of woken threads is returned, it can be 0.
+       the number of woken threads is returned, it can be 0, but it's the
+       real count that has been (or will soon be) woken up. If it's less than
+       required, it's because there aren't enough parked threads.

 
 `PWQR_WAIT`::
        Puts the thread to wait for a future `PWQR_WAKE` command. If this
 
 `PWQR_WAIT`::
        Puts the thread to wait for a future `PWQR_WAKE` command. If this


@@ -243,6 +260,9 @@ If the concurrency level is below the target, then the kernel checks if the
 address `addr` still contains the value `val` (in the fashion of `futex(2)`).
 If it doesn't then the call doesn't block. Else the calling thread is blocked
 until a `PWQR_WAKE` command is received.
 address `addr` still contains the value `val` (in the fashion of `futex(2)`).
 If it doesn't then the call doesn't block. Else the calling thread is blocked
 until a `PWQR_WAKE` command is received.

++
+`addr` must of course be a pointer to an aligned integer which stores the
+reference ticket in userland.

 
 `PWQR_PARK`::
        Puts the thread in park mode. Those are spare threads to avoid
 
 `PWQR_PARK`::
        Puts the thread in park mode. Those are spare threads to avoid