schedulable threads.
waiting::
- This is the state of threads that are currently in a `PWQR_WAIT` call
- from userspace (see `pwqr_ctl`) but that would not overcommit if
- released by a `PWQR_WAKE` call.
+ This is the state of threads that are currently in a `PWQR_CTL_WAIT`
+ call from userspace (see `pwqr_ctl`) but that would not overcommit if
+ released by a `PWQR_CTL_WAKE` call.
quarantined::
- This is the state of threads that are currently in a `PWQR_WAIT` call
- from userspace (see `pwqr_ctl`) but that would overcommit if released
- by a `PWQR_WAKE` call.
+ This is the state of threads that are currently in a `PWQR_CTL_WAIT`
+ call from userspace (see `pwqr_ctl`) but that would overcommit if
+ released by a `PWQR_CTL_WAKE` call.
+
This state avoids waking a thread to force userland to "park" the thread, this
-is racy, make the scheduler work for nothing useful. Though if `PWQR_WAKE` is
-called, quarantined threads are woken but with a `EDQUOT` errno set, and only
-one by one, no matter how wakes have been asked.
+is racy, make the scheduler work for nothing useful. Though if
+`PWQR_CTL_WAKE` is called, quarantined threads are woken but with a `EDQUOT`
+errno set, and only one by one, no matter how wakes have been asked.
+
-This state actually has only one impact: when `PWQR_WAKE` is called for more
-than one threads, for example 4, and that userland knows that there is 5
+This state actually has only one impact: when `PWQR_CTL_WAKE` is called for
+more than one threads, for example 4, and that userland knows that there is 5
threads in WAIT state, but that actually 3 of them are in the quarantine, only
-2 will be woken up, and the `PWQR_WAKE` call will return 2. Any subsequent
-`PWQR_WAKE` call will wake up one quarantined thread to let it be parked, but
-returning 0 each time to hide that from userland.
+2 will be woken up, and the `PWQR_CTL_WAKE` call will return 2. Any subsequent
+`PWQR_CTL_WAKE` call will wake up one quarantined thread to let it be parked,
+but returning 0 each time to hide that from userland.
parked::
- This is the state of threads currently in a `PWQR_PARK` call from
+ This is the state of threads currently in a `PWQR_CTL_PARK` call from
userspace (see `pwqr_ctl`).
the call, as would be returned by `sysconf(_SC_NPROCESSORS_ONLN)`.
`flags`::
- a mask of flags among `O_CLOEXEC`, and `O_NONBLOCK`.
+ a mask of flags among `PWQR_FL_CLOEXEC`, and `PWQR_FL_NONBLOCK`.
Available operations on the pwqr file descriptor are:
Valid values for the `op` argument are:
-`PWQR_GET_CONC`::
+`PWQR_CTL_GET_CONC`::
Requests the current concurrency level for this regulator.
-`PWQR_SET_CONC`::
+`PWQR_CTL_SET_CONC`::
Modifies the current concurrency level for this regulator. The new
value is passed as the `val` argument. The requests returns the old
concurrency level on success.
A zero or negative value for `val` means 'automatic' and is recomputed as the
current number of online CPUs as `sysconf(_SC_NPROCESSORS_ONLN)` would return.
-`PWQR_REGISTER`::
+`PWQR_CTL_REGISTER`::
Registers the calling thread to be taken into account by the pool
regulator. If the thread is already registered into another regulator,
then it's automatically unregistered from it.
-`PWQR_UNREGISTER`::
+`PWQR_CTL_UNREGISTER`::
Deregisters the calling thread from the pool regulator.
-`PWQR_WAKE`::
+`PWQR_CTL_WAKE`::
Tries to wake `val` threads from the pool. This is done according to
the current concurrency level not to overcommit. On success, a hint of
the number of woken threads is returned, it can be 0.
counter of waiting threads to decrease, but we know the thread won't be usable
so we return 0.
-`PWQR_WAKE_OC`::
+`PWQR_CTL_WAKE_OC`::
Tries to wake `val` threads from the pool. This is done bypassing the
current concurrency level (`OC` stands for `OVERCOMMIT`). On success,
the number of woken threads is returned, it can be 0, but it's the
real count that has been (or will soon be) woken up. If it's less than
required, it's because there aren't enough parked threads.
-`PWQR_WAIT`::
- Puts the thread to wait for a future `PWQR_WAKE` command. If this
+`PWQR_CTL_WAIT`::
+ Puts the thread to wait for a future `PWQR_CTL_WAKE` command. If this
thread must be parked to maintain concurrency below the target, then
the call blocks with no further ado.
+
If the concurrency level is below the target, then the kernel checks if the
address `addr` still contains the value `val` (in the fashion of `futex(2)`).
If it doesn't then the call doesn't block. Else the calling thread is blocked
-until a `PWQR_WAKE` command is received.
+until a `PWQR_CTL_WAKE` command is received.
+
`addr` must of course be a pointer to an aligned integer which stores the
reference ticket in userland.
-`PWQR_PARK`::
+`PWQR_CTL_PARK`::
Puts the thread in park mode. Those are spare threads to avoid
cloning/exiting threads when the pool is regulated. Those threads are
released by the regulator only, and can only be woken from userland
- with the `PWQR_WAKE_OC` command, and once all waiting threads have
+ with the `PWQR_CTL_WAKE_OC` command, and once all waiting threads have
been woken.
+
The call blocks until an overcommiting wake requires the thread, or the kernel
[EINVAL]::
TODO
-Errors specific to `PWQR_REGISTER`:
+Errors specific to `PWQR_CTL_REGISTER`:
[ENOMEM]::
There was insufficient memory to perform the operation.
-Errors specific to `PWQR_WAIT`:
+Errors specific to `PWQR_CTL_WAIT`:
[EWOULDBLOCK]::
When the kernel evaluated if `addr` still contained `val` it didn't.
This works like `futex(2)`.
-Errors specific to `PWQR_WAIT` and `PWQR_PARK`:
+Errors specific to `PWQR_CTL_WAIT` and `PWQR_CTL_PARK`:
[EINTR]::
The call was interrupted by a syscall (note that sometimes the kernel
masks this fact when it has more important "errors" to report like
`EDQUOT`).
[EDQUOT]::
- The thread has been woken by a `PWQR_WAKE` or `PWQR_WAKE_OC` call, but
- is overcommiting.
+ The thread has been woken by a `PWQR_CTL_WAKE` or `PWQR_CTL_WAKE_OC`
+ call, but is overcommiting.
#include <linux/timer.h>
#include <linux/uaccess.h>
#include <linux/wait.h>
+#include <linux/version.h>
+
+/*
+ * The pthread workqueue regulator code is for now written as a proof of
+ * concept module, meant to work with 2.6.23+ kernels or redhat5 ones.
+ *
+ * For now it uses a device /dev/pwq, which spawns magic file-descriptors
+ * supporting a few ioctl operations (see Documentation/pwqr.adoc shipped in
+ * the same git repository).
+ *
+ * This code is meant to be merged into mainline, but after the following
+ * changes, kept here as a "todolist":
+ *
+ * - get rid of the device stuff (which is 100% of the init code for 2.6.23
+ * kernels);
+ *
+ * - resubmit the patch that makes it possible to call
+ * preempt_notifier_unregister from sched_in/sched_out (just a matter of a
+ * hlist_for_each_safe instead of hlist_for_each), and fix
+ * pwqr_task_release to not require RCU anymore. It makes
+ * pwqr_preempt_noop_ops go away.
+ *
+ * - think about the possibility to add a pwq_notifier pointer directly into
+ * the task_struct, thought it's not *that* necessary, it grows the
+ * structure for a speed gain we don't really need (making pwqr_ctl
+ * faster). I think it's okay to crawl the preempt_notifier list instead.
+ * We may want to add nice "macros" for that though.
+ *
+ * - replace the ioctl with a pwqr_ctl syscall
+ *
+ * - create a pwqr_create() syscall to create a pwqr file-descriptor.
+ *
+ * Summary: most of the code should be untouched or almost not changed,
+ * pwqr_ioctl adapted to become a syscall, and the module boilerplate replaced
+ * with pwqr_create() and file-descriptor creation boilerplate instead. But
+ * looking at fs/eventfd.c this looks rather simple.
+ */
#ifndef CONFIG_PREEMPT_NOTIFIERS
# error PWQ module requires CONFIG_PREEMPT_NOTIFIERS
-#endif
+#else
#include "pwqr.h"
-#define PWQR_HASH_BITS 5
-#define PWQR_HASH_SIZE (1 << PWQR_HASH_BITS)
-
#define PWQR_UC_DELAY (HZ / 10)
#define PWQR_OC_DELAY (HZ / 20)
#define PWQR_STATE_OC 2
#define PWQR_STATE_DEAD (-1)
-struct pwqr_task_bucket {
- spinlock_t lock;
- struct hlist_head tasks;
-};
+/*
+ * This is the first inclusion of CONFIG_PREEMPT_NOTIFIERS in the kernel.
+ *
+ * Though I want it to work on older redhat 5 kernels, that have an emulation
+ * of the feature but not implemented the same way, and instead of linking the
+ * preempt_notifiers from the task_struct directly, they have a private
+ * h-table I don't have access to, so I need my own too.
+ *
+ * For vanilla kernels we crawl through the task_struct::preempt_notifiers
+ * hlist until we find our entry, this list is often very short, and it's no
+ * slower than the global h-table which also crawls a list anyway.
+ */
+#define IS_PRE_2_6_23 (LINUX_VERSION_CODE < KERNEL_VERSION(2, 6, 23))
struct pwqr_sb {
struct kref kref;
struct pwqr_task {
struct preempt_notifier notifier;
- struct hlist_node link;
+ struct pwqr_sb *sb;
struct rcu_head rcu;
+#if IS_PRE_2_6_23
+ struct hlist_node link;
struct task_struct *task;
- struct pwqr_sb *sb;
+#endif
+};
+
+#if IS_PRE_2_6_23
+
+#define PWQR_HASH_BITS 5
+#define PWQR_HASH_SIZE (1 << PWQR_HASH_BITS)
+
+struct pwqr_task_bucket {
+ spinlock_t lock;
+ struct hlist_head tasks;
};
+static struct pwqr_task_bucket pwqr_tasks_hash[PWQR_HASH_SIZE];
+#endif
+
/*
* Global variables
*/
static struct class *pwqr_class;
static int pwqr_major;
-static struct pwqr_task_bucket pwqr_tasks_hash[PWQR_HASH_SIZE];
static struct preempt_ops pwqr_preempt_running_ops;
static struct preempt_ops pwqr_preempt_blocked_ops;
static struct preempt_ops pwqr_preempt_noop_ops;
/*****************************************************************************
* tasks
*/
+#if IS_PRE_2_6_23
static inline struct pwqr_task_bucket *task_hbucket(struct task_struct *task)
{
return &pwqr_tasks_hash[hash_ptr(task, PWQR_HASH_BITS)];
spin_unlock(&b->lock);
return pwqt;
}
+#else
+static struct pwqr_task *pwqr_task_find(struct task_struct *task)
+{
+ struct hlist_node *node;
+ struct preempt_notifier *it;
+ struct pwqr_task *pwqt = NULL;
+
+ hlist_for_each_entry(it, node, &task->preempt_notifiers, link) {
+ if (it->ops == &pwqr_preempt_running_ops ||
+ it->ops == &pwqr_preempt_blocked_ops ||
+ it->ops == &pwqr_preempt_noop_ops)
+ {
+ pwqt = container_of(it, struct pwqr_task, notifier);
+ break;
+ }
+ }
+
+ return pwqt;
+}
+#endif
static struct pwqr_task *pwqr_task_create(struct task_struct *task)
{
- struct pwqr_task_bucket *b = task_hbucket(task);
struct pwqr_task *pwqt;
pwqt = kmalloc(sizeof(*pwqt), GFP_KERNEL);
preempt_notifier_init(&pwqt->notifier, &pwqr_preempt_running_ops);
preempt_notifier_register(&pwqt->notifier);
- pwqt->task = task;
-
- spin_lock(&b->lock);
- hlist_add_head(&pwqt->link, &b->tasks);
- spin_unlock(&b->lock);
-
+#if IS_PRE_2_6_23
+ {
+ struct pwqr_task_bucket *b = task_hbucket(task);
+
+ pwqt->task = task;
+ spin_lock(&b->lock);
+ hlist_add_head(&pwqt->link, &b->tasks);
+ spin_unlock(&b->lock);
+ }
+#endif
return pwqt;
}
__cold
static void pwqr_task_release(struct pwqr_task *pwqt, bool from_notifier)
{
+#if IS_PRE_2_6_23
struct pwqr_task_bucket *b = task_hbucket(pwqt->task);
spin_lock(&b->lock);
hlist_del(&pwqt->link);
spin_unlock(&b->lock);
+#endif
pwqt->notifier.ops = &pwqr_preempt_noop_ops;
if (from_notifier) {
* callbacks if we're not dying, it'll panic on the next
* sched_{in,out} call.
*/
- BUG_ON(!(pwqt->task->state & TASK_DEAD));
+ BUG_ON(!(current->state & TASK_DEAD));
kfree_rcu(pwqt, rcu);
} else {
preempt_notifier_unregister(&pwqt->notifier);
}
static void pwqr_task_sched_out(struct preempt_notifier *notifier,
- struct task_struct *next)
+ struct task_struct *next)
{
struct pwqr_task *pwqt = container_of(notifier, struct pwqr_task, notifier);
struct pwqr_sb *sb = pwqt->sb;
- struct task_struct *p = pwqt->task;
+ struct task_struct *p = current;
if (unlikely(p->state & TASK_DEAD) || unlikely(sb->state < 0)) {
pwqr_task_detach(pwqt, sb);
int rc = 0;
switch (command) {
- case PWQR_GET_CONC:
+ case PWQR_CTL_GET_CONC:
return sb->concurrency;
- case PWQR_SET_CONC:
+ case PWQR_CTL_SET_CONC:
return do_pwqr_set_conc(sb, (int)arg);
- case PWQR_WAKE:
- case PWQR_WAKE_OC:
- return do_pwqr_wake(sb, command == PWQR_WAKE_OC, (int)arg);
+ case PWQR_CTL_WAKE:
+ case PWQR_CTL_WAKE_OC:
+ return do_pwqr_wake(sb, command == PWQR_CTL_WAKE_OC, (int)arg);
- case PWQR_WAIT:
- case PWQR_PARK:
- case PWQR_REGISTER:
- case PWQR_UNREGISTER:
+ case PWQR_CTL_WAIT:
+ case PWQR_CTL_PARK:
+ case PWQR_CTL_REGISTER:
+ case PWQR_CTL_UNREGISTER:
break;
default:
return -EINVAL;
}
pwqt = pwqr_task_find(task);
- if (command == PWQR_UNREGISTER)
+ if (command == PWQR_CTL_UNREGISTER)
return do_pwqr_unregister(sb, pwqt);
if (pwqt == NULL) {
}
switch (command) {
- case PWQR_WAIT:
+ case PWQR_CTL_WAIT:
rc = do_pwqr_wait(sb, pwqt, true, (struct pwqr_ioc_wait __user *)arg);
break;
- case PWQR_PARK:
+ case PWQR_CTL_PARK:
rc = do_pwqr_wait(sb, pwqt, false, NULL);
break;
}
-
- if (unlikely(sb->state < 0)) {
- pwqr_task_detach(pwqt, pwqt->sb);
- return -EBADFD;
- }
return rc;
}
*/
static int __init pwqr_start(void)
{
+#if IS_PRE_2_6_23
int i;
for (i = 0; i < PWQR_HASH_SIZE; i++) {
spin_lock_init(&pwqr_tasks_hash[i].lock);
INIT_HLIST_HEAD(&pwqr_tasks_hash[i].tasks);
}
+#endif
/* Register as a character device */
pwqr_major = register_chrdev(0, "pwqr", &pwqr_dev_fops);
MODULE_LICENSE("GPL");
MODULE_AUTHOR("Pierre Habouzit <pierre.habouzit@intersec.com>");
MODULE_DESCRIPTION("PThreads Work Queues Regulator");
+#endif
// vim:noet:sw=8:cinoptions+=\:0,L-1,=1s:
#define _LINUX_PWQR_H
#include <linux/ioctl.h>
+#ifdef __KERNEL__
+#include <linux/fcntl.h>
+#else
+#include <fcntl.h>
+#endif
+
#define PWQR_DEVICE_NAME "pwq"
#define PWQR_IO '}'
void *pwqr_uaddr;
};
-#define PWQR_GET_CONC _IO (PWQR_IO, 0)
-#define PWQR_SET_CONC _IO (PWQR_IO, 1)
-#define PWQR_REGISTER _IO (PWQR_IO, 2)
-#define PWQR_UNREGISTER _IO (PWQR_IO, 3)
-#define PWQR_WAKE _IO (PWQR_IO, 4)
-#define PWQR_WAKE_OC _IO (PWQR_IO, 5)
-#define PWQR_WAIT _IOW(PWQR_IO, 6, struct pwqr_ioc_wait)
-#define PWQR_PARK _IO (PWQR_IO, 7)
+#define PWQR_CTL_GET_CONC _IO (PWQR_IO, 0)
+#define PWQR_CTL_SET_CONC _IO (PWQR_IO, 1)
+#define PWQR_CTL_REGISTER _IO (PWQR_IO, 2)
+#define PWQR_CTL_UNREGISTER _IO (PWQR_IO, 3)
+#define PWQR_CTL_WAKE _IO (PWQR_IO, 4)
+#define PWQR_CTL_WAKE_OC _IO (PWQR_IO, 5)
+#define PWQR_CTL_WAIT _IOW(PWQR_IO, 6, struct pwqr_ioc_wait)
+#define PWQR_CTL_PARK _IO (PWQR_IO, 7)
+
+#define PWQR_FL_NONBLOCK O_NONBLOCK
+#define PWQR_FL_CLOEXEC O_CLOEXEC
+#define PWQR_FL__FCTNL_FLAGS (PWQR_FL_NONBLOCK | PWQR_FL_CLOEXEC)
+#define PWQR_FL__SET (PWQR_FL__FCTNL_FLAGS)
#endif
static int pwqr_create(int flags)
{
- if (flags & ~(O_NONBLOCK | O_CLOEXEC)) {
+ if (flags & ~PWQR_FL__SET) {
errno = -EINVAL;
return -1;
}
struct pwqr_ioc_wait wait;
switch (op) {
- case PWQR_GET_CONC:
- case PWQR_REGISTER:
- case PWQR_UNREGISTER:
- case PWQR_PARK:
+ case PWQR_CTL_GET_CONC:
+ case PWQR_CTL_REGISTER:
+ case PWQR_CTL_UNREGISTER:
+ case PWQR_CTL_PARK:
return ioctl(fd, op);
- case PWQR_SET_CONC:
- case PWQR_WAKE:
- case PWQR_WAKE_OC:
+ case PWQR_CTL_SET_CONC:
+ case PWQR_CTL_WAKE:
+ case PWQR_CTL_WAKE_OC:
return ioctl(fd, op, val);
- case PWQR_WAIT:
+ case PWQR_CTL_WAIT:
wait.pwqr_ticket = val;
wait.pwqr_uaddr = uaddr;
return ioctl(fd, op, &wait);
atomic_add(&pwqr_g.ticket, 1);
#endif
if (atomic_fetch_and_add(&pwqr_g.waiters, 0))
- pwqr_ctl(pwqr_g.fd, PWQR_WAKE, n, NULL);
+ pwqr_ctl(pwqr_g.fd, PWQR_CTL_WAKE, n, NULL);
}
static ALWAYS_INLINE void pwqr_signal(void)
{
pthread_cleanup_push(&pwqr_do_wait_cleanup, NULL);
pthread_setcanceltype(PTHREAD_CANCEL_ASYNCHRONOUS, &canceltype);
- rc = pwqr_ctl(pwqr_g.fd, PWQR_WAIT, ticket, &pwqr_g.lo);
+ rc = pwqr_ctl(pwqr_g.fd, PWQR_CTL_WAIT, ticket, &pwqr_g.lo);
if (rc < 0) {
if (errno == EINTR) {
rc = INPOOL;
run_job(job);
#endif
- if ((rc = pwqr_ctl(pwqr_g.fd, PWQR_PARK, 0, NULL)) == 0) {
+ if ((rc = pwqr_ctl(pwqr_g.fd, PWQR_CTL_PARK, 0, NULL)) == 0) {
access_once(pwqr_g.overcommit_count) = 0;
if (atomic_fetch_and_sub(&pwqr_g.parked, 1) == 0) {
pwqr_spawn_thread(PARKED, 1);
goto out;
pthread_cleanup_push(&pwqr_main_cleanup, NULL);
- if (pwqr_ctl(pwqr_g.fd, PWQR_REGISTER, 0, NULL) < 0)
+ if (pwqr_ctl(pwqr_g.fd, PWQR_CTL_REGISTER, 0, NULL) < 0)
goto out;
do {
}
pwqr_g.fd = fd;
- n = pwqr_ctl(pwqr_g.fd, PWQR_GET_CONC, 0, NULL) + 4;
+ n = pwqr_ctl(pwqr_g.fd, PWQR_CTL_GET_CONC, 0, NULL) + 4;
pwqr_spawn_thread(INPOOL, n);
pwqr_spawn_thread(PARKED, 4);
projects / ~madcoder / pwqr.git / commitdiff