aio co-op with socket kqueue on Netbsd

[LeiMing]

Hello, I'm just a Windows programmer and now learning *nix APIs. I want to
know how people use aio and socket kqueue together.

Recently I'm learning I/O APIs and noticed that aio_* functions seeem to be
the only way to perform asynchronism I/O on disk files. The interface is
just like overlapped I/O on Windows. But the only usable way I found, to
wait for the operation's completion or to notice me that it finished, which
is usable on NetBSD, is the API aio_suspend.

I searched the Internet and found 4 ways to implement this: realtime signal,
kqueue, callback function and aio_suspend/aio_waitcomplete. Unluckily,
according to this (
http://ftp.netbsd.org/pub/NetBSD/NetBSD-current/src/doc/TODO.kqueue )
document I know that currently NetBSD's kqueue doesn't support aio events.
Sigevent manual ( http://netbsd.gw.com/cgi-bin/man-cgi?sigevent ) shows that
neither realtime signal nor SIGEV_THREAD flag (by which callback
notification implement) is supported on NetBSD.

The existing practise to handle sockets in a network server prefers I/O
completion port on Windows, kqueue on *BSD and epoll on Linux. I/O
completion port handles file read/write operation as well, while kqueue on
FreeBSD seems (I have not yet tried) to work with aio operations. Through
fdevent/fdsignal API Linux handles signals via epoll. By this way there
won't be situation that a process or thread is "waiting for a single
operation" (synchronism I/O). So how could I use kqueue (to handle sockets)
and aio (to handle regular files on disk) on NetBSD to achieve the same goal?

By the way, since the realtime signal feature is not usable on NetBSD and
timer_create sends notice by realtime signal, does it mean that timer_create
will not work on NetBSD? So generally how a time-out feature is implement in
a network server on NetBSD?

Sorry for poor English.
Thanks in advance.

Regards,
LeiMing

[Justin Cormack]

Post by LeiMing
Hello, I'm just a Windows programmer and now learning *nix APIs. I want to
know how people use aio and socket kqueue together.

Generally on Unix, people do not use file aio, unlike on Windows. The
interfaces are new, poorly standardised, only partially async anyway.

Post by LeiMing
Recently I'm learning I/O APIs and noticed that aio_* functions seeem to be
the only way to perform asynchronism I/O on disk files. The interface is
just like overlapped I/O on Windows. But the only usable way I found, to
wait for the operation's completion or to notice me that it finished, which
is usable on NetBSD, is the API aio_suspend.

Yes that is one of the older, less useful APIs. Traditionally these
APIs were usually implemented in userspace using threads, with no
kernel support at all, hence the lack of kernel completion
notification semantics in the specification.

Post by LeiMing
I searched the Internet and found 4 ways to implement this: realtime signal,
kqueue, callback function and aio_suspend/aio_waitcomplete. Unluckily,
according to this (
http://ftp.netbsd.org/pub/NetBSD/NetBSD-current/src/doc/TODO.kqueue )
document I know that currently NetBSD's kqueue doesn't support aio events.
Sigevent manual ( http://netbsd.gw.com/cgi-bin/man-cgi?sigevent ) shows that
neither realtime signal nor SIGEV_THREAD flag (by which callback
notification implement) is supported on NetBSD.

The only one that can provide a sane performant interface is kqueue.
Your best bet would be to implement this on NetBSD if you really want
it.

Post by LeiMing
The existing practise to handle sockets in a network server prefers I/O
completion port on Windows, kqueue on *BSD and epoll on Linux. I/O
completion port handles file read/write operation as well, while kqueue on
FreeBSD seems (I have not yet tried) to work with aio operations. Through
fdevent/fdsignal API Linux handles signals via epoll. By this way there
won't be situation that a process or thread is "waiting for a single
operation" (synchronism I/O). So how could I use kqueue (to handle sockets)
and aio (to handle regular files on disk) on NetBSD to achieve the same goal?

Well most of the event libraries use threads on Unix (eg libuv AFAIK).

Post by LeiMing
By the way, since the realtime signal feature is not usable on NetBSD and
timer_create sends notice by realtime signal, does it mean that timer_create
will not work on NetBSD? So generally how a time-out feature is implement in
a network server on NetBSD?

Using signals is not going to perform very well, I would not do that...

You have to understand that Unix aio has no real history, largely
because IO goes through the page cache and there is no async semantics
for a page fault. This is why when aio has been grafted on it tends to
go via non cached IO, which is not terribly performant. It also tends
to block in metadata operations and so on (eg no Unix supports an
async fsync as far as I know). Windows always had a different
semantics, hence the mismatch you are seeing.

If you want to work on the kernel and make aio kqueue work, by all
means do. If you want to run async applciations, use an existing
library, or ignore file operations and just run a few more threads is
a good strategy for many applications - the best strategy depends on
the application.

Justin

[LeiMing]

Post by Justin Cormack
Generally on Unix, people do not use file aio, unlike on Windows. The
interfaces are new, poorly standardised, only partially async anyway.

Quite surprised that Unix programmers generally don't use asynchronism I/O
APIs on regular files and use threads instead.

Post by Justin Cormack
Yes that is one of the older, less useful APIs. Traditionally these
APIs were usually implemented in userspace using threads, with no
kernel support at all, hence the lack of kernel completion
notification semantics in the specification.
The only one that can provide a sane performant interface is kqueue.
Your best bet would be to implement this on NetBSD if you really want
it.
Well most of the event libraries use threads on Unix (eg libuv AFAIK).
Using signals is not going to perform very well, I would not do that...

For the lack of implementation of realtime signals, maybe "settimer" api can
be used to handle client's timeout. It's not "realtime signals" so kqueue can
monitor it. Could you tell me whether use kqueue on signals will still have
that bad effect on performance? If not, how do unix programmers usually
implement "timeout"?

Post by Justin Cormack
You have to understand that Unix aio has no real history, largely
because IO goes through the page cache and there is no async semantics
for a page fault. This is why when aio has been grafted on it tends to
go via non cached IO, which is not terribly performant. It also tends
to block in metadata operations and so on (eg no Unix supports an
async fsync as far as I know). Windows always had a different
semantics, hence the mismatch you are seeing.
If you want to work on the kernel and make aio kqueue work, by all
means do. If you want to run async applciations, use an existing
library, or ignore file operations and just run a few more threads is
a good strategy for many applications - the best strategy depends on
the application.
Justin

I'll focus my learning on kqueue and epoll.
Thank you for the explaination.

Regards,
LeiMing

[Justin Cormack]

Post by LeiMing

Post by Justin Cormack
Using signals is not going to perform very well, I would not do that...

For the lack of implementation of realtime signals, maybe "settimer" api can
be used to handle client's timeout. It's not "realtime signals" so kqueue can
monitor it. Could you tell me whether use kqueue on signals will still have
that bad effect on performance? If not, how do unix programmers usually
implement "timeout"?

For timers you can use kqueue directly eg
https://wiki.netbsd.org/tutorials/kqueue_tutorial/#index5h2 - you do
not need to get kqueue to pick up a signal.

Justin