Repository: Hexilee/async-io-demo
Branch: master
Commit: 5b8251402577
Files: 19
Total size: 125.9 KB
Directory structure:
gitextract__d09rzo9/
├── .gitignore
├── Cargo.toml
├── LICENSE
├── README.md
├── README.zh.md
├── examples/
│ ├── async-echo.rs
│ ├── fab.rs
│ ├── file-server.rs
│ ├── fs-mio.rs
│ ├── fs.rs
│ ├── poll-timeout.rs
│ ├── spurious-events.rs
│ ├── tcp.rs
│ └── test.txt
└── src/
├── executor.rs
├── fs.rs
├── fs_future.rs
├── fs_mio.rs
└── lib.rs
================================================
FILE CONTENTS
================================================
================================================
FILE: .gitignore
================================================
/target
**/*.rs.bk
Cargo.lock
================================================
FILE: Cargo.toml
================================================
[package]
name = "asyncio"
version = "0.1.0"
authors = ["Hexilee <hexileee@gmail.com>"]
edition = "2018"
[dependencies]
mio = "0.6.16"
crossbeam-channel = "0.3.2"
failure = "0.1.3"
slab = "0.4.1"
log = "0.4"
env_logger = "0.6.0"
================================================
FILE: LICENSE
================================================
MIT License
Copyright (c) 2018 ZJU QSC
Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:
The above copyright notice and this permission notice shall be included in all
copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
SOFTWARE.
================================================
FILE: README.md
================================================
Table of Contents
=================
* [Introduction](#introduction)
* [mio: The Footstone of Asynchronous IO](#mio-the-footstone-of-asynchronous-io)
* [Asynchronous Network IO](#asynchronous-network-io)
* [Spurious Events](#spurious-events)
* [Poll Option](#poll-option)
* [Still Block](#still-block)
* [Custom Event](#custom-event)
* [Callback is Evil](#callback-is-evil)
* [stackless coroutine](#stackless-coroutine)
* [generator](#generator)
* [self-referential structs](#self-referential-structs)
* [Pin](#pin)
* [Reasonable Abstraction](#reasonable-abstraction)
* [Poll<T>](#pollt)
* [await](#await)
* [async](#async)
* [non-blocking coroutine](#non-blocking-coroutine)
* [Executor](#executor)
* [block_on](#block_on)
* [spawn](#spawn)
* [TcpListener](#tcplistener)
* [TcpStream](#tcpstream)
* [echo server](#echo-server)
* [Afterword](#afterword)
### Introduction
2019 is approaching. The rust team keeps their promise about asynchronous IO: `async` is introduced as keywords, `Pin, Future, Poll` and `await` is introduced into standard library.
I have never used rust for asynchronous IO programming earlier, so I almost know nothing about it. However, I would use it for a project recently but couldn't find many documents that are remarkably helpful for newbie of rust asynchronous programming.
Eventually, I wrote several demo and implemented simple asynchronous IO based on `mio` and `coroutine` with the help of both of this blog ([Tokio internals: Understanding Rust's asynchronous I/O framework from the bottom up](https://cafbit.com/post/tokio_internals/)) and source code of "new tokio" [romio](https://github.com/withoutboats/romio) .
This is the final file server:
```rust
// examples/file-server.rs
#[macro_use]
extern crate log;
use asyncio::executor::{block_on, spawn, TcpListener, TcpStream};
use asyncio::fs_future::{read_to_string};
use failure::Error;
fn main() -> Result<(), Error> {
env_logger::init();
block_on(new_server())?
}
const CRLF: &[char] = &['\r', '\n'];
async fn new_server() -> Result<(), Error> {
let mut listener = TcpListener::bind(&"127.0.0.1:7878".parse()?)?;
info!("Listening on 127.0.0.1:7878");
while let Ok((stream, addr)) = listener.accept().await {
info!("connection from {}", addr);
spawn(handle_stream(stream))?;
}
Ok(())
}
async fn handle_stream(mut stream: TcpStream) -> Result<(), Error> {
stream.write_str("Please enter filename: ").await?;
let file_name_vec = stream.read().await?;
let file_name = String::from_utf8(file_name_vec)?.trim_matches(CRLF).to_owned();
let file_contents = read_to_string(file_name).await?;
stream.write_str(&file_contents).await?;
stream.close();
Ok(())
}
```
My purpose of writing this blog is to review and summarize, I will be happy if it can help someone who are interested in rust asynchronous programming. Given that the readability is the primary consideration when I wrote the code appearing in this blog, there may be some performance problem in code, please forgive me. If there are obvious problems in blog or code, you are welcome to point them up.
Most of the code appearing in this blog is collected in this [repo](https://github.com/Hexilee/async-io-demo) (some code is too long, you had better clone and view it in editor), all examples work well at rustc-1.39`.
> When executing examples/async-echo, set environment variable `RUST_LOG=info` for basic runtime information; set `RUST_LOG=debug` for events polling information.
### mio: The Footstone of Asynchronous IO
`mio` is a tidy, low-level asynchronous IO library. Nowadays, almost all asynchronous IO libraries in rust ecosystem are based on `mio`.
As sub modules like `channel, timer` have been marked as deprecated since version-0.6.5, `mio` provides only two core functions:
- basic encapsulation for OS asynchronous network IO
- custom events
The first core function corresponded to API in different OS respectively are:
- Linux(Android) => epoll
- Windows => iocp
- MacOS(iOS), FreeBSD => kqueue
- Fuchsia => \<unknow>
`mio` wraps different asynchronous network API in different OS into a common epoll-like asynchronous API, which supports both of `udp` and `tcp`.
> besides `udp` and `tcp`, `mio` also provides some OS-specific API, like `uds`, we won't talk about them, you can find the usage in source code of mio.
>
>
#### Asynchronous Network IO
This is a demo of asynchronous tcp IO:
```rust
// examples/tcp.rs
use mio::*;
use mio::net::{TcpListener, TcpStream};
use std::io::{Read, Write, self};
use failure::Error;
use std::time::{Duration, Instant};
const SERVER_ACCEPT: Token = Token(0);
const SERVER: Token = Token(1);
const CLIENT: Token = Token(2);
const SERVER_HELLO: &[u8] = b"PING";
const CLIENT_HELLO: &[u8] = b"PONG";
fn main() -> Result<(), Error> {
let addr = "127.0.0.1:13265".parse()?;
// Setup the server socket
let server = TcpListener::bind(&addr)?;
// Create a poll instance
let poll = Poll::new()?;
// Start listening for incoming connections
poll.register(&server, SERVER_ACCEPT, Ready::readable(),
PollOpt::edge())?;
// Setup the client socket
let mut client = TcpStream::connect(&addr)?;
let mut server_handler = None;
// Register the client
poll.register(&client, CLIENT, Ready::readable() | Ready::writable(),
PollOpt::edge())?;
// Create storage for events
let mut events = Events::with_capacity(1024);
let start = Instant::now();
let timeout = Duration::from_millis(10);
'top: loop {
poll.poll(&mut events, None)?;
for event in events.iter() {
if start.elapsed() >= timeout {
break 'top
}
match event.token() {
SERVER_ACCEPT => {
let (handler, addr) = server.accept()?;
println!("accept from addr: {}", &addr);
poll.register(&handler, SERVER, Ready::readable() | Ready::writable(), PollOpt::edge())?;
server_handler = Some(handler);
}
SERVER => {
if event.readiness().is_writable() {
if let Some(ref mut handler) = &mut server_handler {
match handler.write(SERVER_HELLO) {
Ok(_) => {
println!("server wrote");
}
Err(ref err) if err.kind() == io::ErrorKind::WouldBlock => continue,
err => {
err?;
}
}
}
}
if event.readiness().is_readable() {
let mut hello = [0; 4];
if let Some(ref mut handler) = &mut server_handler {
match handler.read_exact(&mut hello) {
Ok(_) => {
assert_eq!(CLIENT_HELLO, &hello);
println!("server received");
}
Err(ref err) if err.kind() == io::ErrorKind::WouldBlock => continue,
err => {
err?;
}
}
}
}
}
CLIENT => {
if event.readiness().is_writable() {
match client.write(CLIENT_HELLO) {
Ok(_) => {
println!("client wrote");
}
Err(ref err) if err.kind() == io::ErrorKind::WouldBlock => continue,
err => {
err?;
}
}
}
if event.readiness().is_readable() {
let mut hello = [0; 4];
match client.read_exact(&mut hello) {
Ok(_) => {
assert_eq!(SERVER_HELLO, &hello);
println!("client received");
}
Err(ref err) if err.kind() == io::ErrorKind::WouldBlock => continue,
err => {
err?;
}
}
}
}
_ => unreachable!(),
}
}
};
Ok(())
}
```
This demo is a little long, let's talk about its main loop first:
```rust
fn main() {
// ...
loop {
poll.poll(&mut events, None).unwrap();
// ...
}
}
```
We need call `Poll::poll` in each loop, the first parameter `events` is used for store events, we set it with capacity 1024 here.
```rust
let mut events = Events::with_capacity(1024);
```
The type of second parameter `timeout` is `Option<Duration>`, method will return `Ok(usize)` when some events occur or timeout if `Some(duration) = timeout`.
> The usize in Ok refers to the number of events, this value is deprecated and will be removed in 0.7.0.
Here we deliver `timeout = None` ，so when the method return without error, there must be some events. Let's traverse events:
```rust
match event.token() {
SERVER_ACCEPT => {
let (handler, addr) = server.accept()?;
println!("accept from addr: {}", &addr);
poll.register(&handler, SERVER, Ready::readable() | Ready::writable(), PollOpt::edge())?;
server_handler = Some(handler);
}
SERVER => {
if event.readiness().is_writable() {
if let Some(ref mut handler) = &mut server_handler {
match handler.write(SERVER_HELLO) {
Ok(_) => {
println!("server wrote");
}
Err(ref err) if err.kind() == io::ErrorKind::WouldBlock => continue,
err => {
err?;
}
}
}
}
if event.readiness().is_readable() {
let mut hello = [0; 4];
if let Some(ref mut handler) = &mut server_handler {
match handler.read_exact(&mut hello) {
Ok(_) => {
assert_eq!(CLIENT_HELLO, &hello);
println!("server received");
}
Err(ref err) if err.kind() == io::ErrorKind::WouldBlock => continue,
err => {
err?;
}
}
}
}
}
CLIENT => {
if event.readiness().is_writable() {
match client.write(CLIENT_HELLO) {
Ok(_) => {
println!("client wrote");
}
Err(ref err) if err.kind() == io::ErrorKind::WouldBlock => continue,
err => {
err?;
}
}
}
if event.readiness().is_readable() {
let mut hello = [0; 4];
match client.read_exact(&mut hello) {
Ok(_) => {
assert_eq!(SERVER_HELLO, &hello);
println!("client received");
}
Err(ref err) if err.kind() == io::ErrorKind::WouldBlock => continue,
err => {
err?;
}
}
}
}
_ => unreachable!(),
}
```
We need match token of each event, these tokens are just those we used to register. For example, we register `server` using token `SERVER_ACCEPT`.
```rust
const SERVER_ACCEPT: Token = Token(0);
...
// Start listening for incoming connections
poll.register(&server, SERVER_ACCEPT, Ready::readable(),
PollOpt::edge()).unwrap();
```
In this case, when we find `event.token() == SERVER_ACCEPT`, we should think it's relevant to `server`, so we try to accept a new `TcpStream` and register it, using token `SERVER`:
```rust
let (handler, addr) = server.accept()?;
println!("accept from addr: {}", &addr);
poll.register(&handler, SERVER, Ready::readable() | Ready::writable(), PollOpt::edge()).unwrap();
server_handler = Some(handler);
```
As the same, if we find `event.token() == SERVER`，we should think it's relevant to `handler`:
```rust
if event.readiness().is_writable() {
if let Some(ref mut handler) = &mut server_handler {
match handler.write(SERVER_HELLO) {
Ok(_) => {
println!("server wrote");
}
Err(ref err) if err.kind() == io::ErrorKind::WouldBlock => continue,
err => {
err?;
}
}
}
}
if event.readiness().is_readable() {
let mut hello = [0; 4];
if let Some(ref mut handler) = &mut server_handler {
match handler.read_exact(&mut hello) {
Ok(_) => {
assert_eq!(CLIENT_HELLO, &hello);
println!("server received");
}
Err(ref err) if err.kind() == io::ErrorKind::WouldBlock => continue,
err => {
err?;
}
}
}
}
```
In this case, we shoud response differently to different `event.readiness()`, this is the third parameter of register, which named `interest`. As its name, `interest` means 'something you are interested', its type is `Ready`. `mio` support four kinds of `Ready`, `readable`, `writable`, `error` and `hup`, you can union them.
We register `handler`with `Ready::readable() | Ready::writable()`, so event can be `readable` or `writable` or both, you can see it in control flow:
using
```rust
if event.readiness().is_writable() {
...
}
if event.readiness().is_readable() {
...
}
```
instead of
```rust
if event.readiness().is_writable() {
...
} else if event.readiness().is_readable() {
...
}
```
#### Spurious Events
The code for dealing with event `SERVER_ACCEPT` above is:
```rust
match event.token() {
SERVER_ACCEPT => {
let (handler, addr) = server.accept()?;
println!("accept from addr: {}", &addr);
poll.register(&handler, SERVER, Ready::readable() | Ready::writable(), PollOpt::edge()).unwrap();
server_handler = Some(handler);
}
```
The result `server.accept()` returned is `io::Result<(TcpStream, SocketAddr)>`. If we trust `event` entirely, using `try` is the right choise (if there should be a new `TcpStream` is ready, `server.accept()` returning `Err` is unforeseen and unmanageable).
However, we should think event may be spurious, the possibility depends on OS and custom implement. There may not be a new `TcpStream` is ready, in this case, `server.accept()` will return `WouldBlock Error`. We should regard `WouldBlock Error ` as a friendly warning: "there isn't a new `TcpStream` is ready, please do it later again." So we should ignore it and continue the loop.
Like the code for dealing with event `SERVER`
```rust
match handler.write(SERVER_HELLO) {
Ok(_) => {
println!("server wrote");
}
Err(ref err) if err.kind() == io::ErrorKind::WouldBlock => continue,
err => {
err?;
}
}
```
#### Poll Option
Now we can execute:
```bash
cargo run --example tcp
```
The terminal print some log:
```bash
client wrote
accept from addr: 127.0.0.1:53205
client wrote
server wrote
server received
...
```
We can see, in 10 milliseconds (`let timeout = Duration::from_millis(10);`), `server` and `client` did dozens of writing and reading!
How should we do if we don't need dozens of writing and reading? In a pretty network environment, `client ` and `server` is almost always writable, so `Poll::poll` may return in dozens of microseconds.
In this case, we should change the forth parameter of `register`:
```rust
poll.register(&server, SERVER_ACCEPT, Ready::readable(),
PollOpt::edge()).unwrap();
```
The type of `PollOpt::edge()` is `PollOpt`, means poll option. There are three kinds of poll options: `level`, `edge` and `oneshot`, what's the difference of them?
For example, in this code:
```rust
if event.readiness().is_readable() {
let mut hello = [0; 4];
match client.read_exact(&mut hello) {
Ok(_) => {
assert_eq!(SERVER_HELLO, &hello);
println!("client received");
}
Err(ref err) if err.kind() == io::ErrorKind::WouldBlock => continue,
err => {
err?;
}
}
}
```
When I receive a readable readiness, I read 4 bytes only. If there are 8 bytes in buffer:
- if I register this `TcpStream` with `PollOpt::level()`, I **MUST** receive a `readable readiness event` in next polling;
- if I register this `TcpStream` with `PollOpt::edge()`, I **MAY** cannot receive a `readable readiness event` in next polling;
So, we can say that readiness using edge-triggered mode is a `Draining readiness`, once a readiness event is received, the corresponding operation must be performed repeatedly until it returns `WouldBlock`. We should alter code above into:
```rust
if event.readiness().is_readable() {
let mut hello = [0; 4];
loop {
match client.read_exact(&mut hello) {
Ok(_) => {
assert_eq!(SERVER_HELLO, &hello);
println!("client received");
}
Err(ref err) if err.kind() == io::ErrorKind::WouldBlock => break,
err => {
err?;
}
}
}
}
```
Then, what's the behavior of `PollOpt::onshot()`? Let's talk about the first question of this section: if we want `handler` to write only once, how should we do? The answer is: register it using `PollOpt::oneshot()`
```rust
let (handler, addr) = server.accept()?;
println!("accept from addr: {}", &addr);
poll.register(&handler, SERVER, Ready::readable() | Ready::writable(), PollOpt::oneshot())?;
server_handler = Some(handler);
```
In this case, you can only receive event `SERVER` once, unless you re-register `handler` using `Poll::reregister`.
> `Poll::reregister` can using different `PollOpt` and `interest` from the last registering
#### Still Block
There is still a problem in the code above: we using blocking IO macro `println!`. We should avoid using blocking IO in the code for dealing events.
Given that FS(file-system) IO(including `stdin, stdout, stderr`) is slow, we can send all FS-IO task to a worker thread.
```rust
use std::sync::mpsc::{Sender, Receiver, channel, SendError};
#[derive(Clone)]
pub struct Fs {
task_sender: Sender<Task>,
}
impl Fs {
pub fn new() -> Self {
let (sender, receiver) = channel();
std::thread::spawn(move || {
loop {
match receiver.recv() {
Ok(task) => {
match task {
Task::Println(ref string) => println!("{}", string),
Task::Exit => return
}
},
Err(_) => {
return;
}
}
}
});
Fs { task_sender: sender }
}
pub fn println(&self, string: String) {
self.task_sender.send(Task::Println(string)).unwrap()
}
}
pub enum Task {
Exit,
Println(String),
}
```
Now, we can replace all `println!` with `Fs::println`.
#### Custom Event
Implementing non-blocking `println` is easy, because this function has no return. How should we do if we want other non-blocking FS-IO functions? For example, opening a file, then reading it to string, then printing the string, how should we do?
The easiest way is using callback, like this:
```rust
// src/fs.rs
use crossbeam_channel::{unbounded, Sender};
use failure::Error;
use std::fs::File;
use std::io::Read;
use std::thread;
#[derive(Clone)]
pub struct Fs {
task_sender: Sender<Task>,
}
pub struct FsHandler {
io_worker: thread::JoinHandle<Result<(), Error>>,
executor: thread::JoinHandle<Result<(), Error>>,
}
pub fn fs_async() -> (Fs, FsHandler) {
let (task_sender, task_receiver) = unbounded();
let (result_sender, result_receiver) = unbounded();
let io_worker = std::thread::spawn(move || {
while let Ok(task) = task_receiver.recv() {
match task {
Task::Println(ref string) => println!("{}", string),
Task::Open(path, callback, fs) => {
result_sender.send(TaskResult::Open(File::open(path)?, callback, fs))?
}
Task::ReadToString(mut file, callback, fs) => {
let mut value = String::new();
file.read_to_string(&mut value)?;
result_sender.send(TaskResult::ReadToString(value, callback, fs))?
}
Task::Exit => {
result_sender.send(TaskResult::Exit)?;
break;
}
}
}
Ok(())
});
let executor = std::thread::spawn(move || {
loop {
let result = result_receiver.recv()?;
match result {
TaskResult::ReadToString(value, callback, fs) => callback(value, fs)?,
TaskResult::Open(file, callback, fs) => callback(file, fs)?,
TaskResult::Exit => break,
};
}
Ok(())
});
(
Fs { task_sender },
FsHandler {
io_worker,
executor,
},
)
}
impl Fs {
pub fn println(&self, string: String) -> Result<(), Error> {
Ok(self.task_sender.send(Task::Println(string))?)
}
pub fn open<F>(&self, path: &str, callback: F) -> Result<(), Error>
where
F: FnOnce(File, Fs) -> Result<(), Error> + Sync + Send + 'static,
{
Ok(self.task_sender.send(Task::Open(
path.to_string(),
Box::new(callback),
self.clone(),
))?)
}
pub fn read_to_string<F>(&self, file: File, callback: F) -> Result<(), Error>
where
F: FnOnce(String, Fs) -> Result<(), Error> + Sync + Send + 'static,
{
Ok(self
.task_sender
.send(Task::ReadToString(file, Box::new(callback), self.clone()))?)
}
pub fn close(&self) -> Result<(), Error> {
Ok(self.task_sender.send(Task::Exit)?)
}
}
impl FsHandler {
pub fn join(self) -> Result<(), Error> {
self.io_worker.join().unwrap()?;
self.executor.join().unwrap()
}
}
type FileCallback = Box<dyn FnOnce(File, Fs) -> Result<(), Error> + Sync + Send>;
type StringCallback = Box<dyn FnOnce(String, Fs) -> Result<(), Error> + Sync + Send>;
pub enum Task {
Exit,
Println(String),
Open(String, FileCallback, Fs),
ReadToString(File, StringCallback, Fs),
}
pub enum TaskResult {
Exit,
Open(File, FileCallback, Fs),
ReadToString(String, StringCallback, Fs),
}
```
```rust
// examples/fs.rs
use asyncio::fs::fs_async;
use failure::Error;
const TEST_FILE_VALUE: &str = "Hello, World!\n";
fn main() -> Result<(), Error> {
let (fs, fs_handler) = fs_async();
fs.open("./examples/test.txt", |file, fs| {
fs.read_to_string(file, |value, fs| {
assert_eq!(TEST_FILE_VALUE, &value);
fs.println(value)?;
fs.close()
})
})?;
fs_handler.join()?;
Ok(())
}
```
running this example:
```bash
cargo run --example fs
```
This implementation work well, but the executor thread is still blocked by the io-worker thread `(result_receiver.recv()`). Can we run a polling loop in executor thread to not be blocked by IO? (executor should execute `(result_receiver.recv()` only when there are some results in result channel).
To implement a non-blocking executor, we can use custom events supported by `mio`.
Altering the code above:
```rust
// src/fs_mio.rs
use crossbeam_channel::{unbounded, Sender, TryRecvError};
use failure::Error;
use mio::*;
use std::fs::File;
use std::io::Read;
use std::thread;
use std::time::Duration;
#[derive(Clone)]
pub struct Fs {
task_sender: Sender<Task>,
}
pub struct FsHandler {
io_worker: thread::JoinHandle<Result<(), Error>>,
executor: thread::JoinHandle<Result<(), Error>>,
}
const FS_TOKEN: Token = Token(0);
pub fn fs_async() -> (Fs, FsHandler) {
let (task_sender, task_receiver) = unbounded();
let (result_sender, result_receiver) = unbounded();
let poll = Poll::new().unwrap();
let (registration, set_readiness) = Registration::new2();
poll.register(
®istration,
FS_TOKEN,
Ready::readable(),
PollOpt::oneshot(),
)
.unwrap();
let io_worker = std::thread::spawn(move || {
while let Ok(task) = task_receiver.recv() {
match task {
Task::Println(ref string) => println!("{}", string),
Task::Open(path, callback, fs) => {
result_sender.send(TaskResult::Open(File::open(path)?, callback, fs))?;
set_readiness.set_readiness(Ready::readable())?;
}
Task::ReadToString(mut file, callback, fs) => {
let mut value = String::new();
file.read_to_string(&mut value)?;
result_sender.send(TaskResult::ReadToString(value, callback, fs))?;
set_readiness.set_readiness(Ready::readable())?;
}
Task::Exit => {
result_sender.send(TaskResult::Exit)?;
set_readiness.set_readiness(Ready::readable())?;
break;
}
}
}
Ok(())
});
let executor = thread::spawn(move || {
let mut events = Events::with_capacity(1024);
'outer: loop {
poll.poll(&mut events, Some(Duration::from_secs(1)))?;
for event in events.iter() {
match event.token() {
FS_TOKEN => {
loop {
match result_receiver.try_recv() {
Ok(result) => match result {
TaskResult::ReadToString(value, callback, fs) => {
callback(value, fs)?
}
TaskResult::Open(file, callback, fs) => callback(file, fs)?,
TaskResult::Exit => break 'outer,
},
Err(e) => match e {
TryRecvError::Empty => break,
TryRecvError::Disconnected => return Err(e.into()),
},
}
}
poll.reregister(
®istration,
FS_TOKEN,
Ready::readable(),
PollOpt::oneshot(),
)?;
}
_ => unreachable!(),
}
}
}
Ok(())
});
(
Fs { task_sender },
FsHandler {
io_worker,
executor,
},
)
}
impl Fs {
pub fn println(&self, string: String) -> Result<(), Error> {
Ok(self.task_sender.send(Task::Println(string))?)
}
pub fn open<F>(&self, path: &str, callback: F) -> Result<(), Error>
where
F: FnOnce(File, Fs) -> Result<(), Error> + Sync + Send + 'static,
{
Ok(self.task_sender.send(Task::Open(
path.to_string(),
Box::new(callback),
self.clone(),
))?)
}
pub fn read_to_string<F>(&self, file: File, callback: F) -> Result<(), Error>
where
F: FnOnce(String, Fs) -> Result<(), Error> + Sync + Send + 'static,
{
Ok(self
.task_sender
.send(Task::ReadToString(file, Box::new(callback), self.clone()))?)
}
pub fn close(&self) -> Result<(), Error> {
Ok(self.task_sender.send(Task::Exit)?)
}
}
impl FsHandler {
pub fn join(self) -> Result<(), Error> {
self.io_worker.join().unwrap()?;
self.executor.join().unwrap()
}
}
type FileCallback = Box<dyn FnOnce(File, Fs) -> Result<(), Error> + Sync + Send>;
type StringCallback = Box<dyn FnOnce(String, Fs) -> Result<(), Error> + Sync + Send>;
pub enum Task {
Exit,
Println(String),
Open(String, FileCallback, Fs),
ReadToString(File, StringCallback, Fs),
}
pub enum TaskResult {
Exit,
Open(File, FileCallback, Fs),
ReadToString(String, StringCallback, Fs),
}
// examples/fs-mio.rs
use asyncio::fs_mio::fs_async;
use failure::Error;
const TEST_FILE_VALUE: &str = "Hello, World!\n";
fn main() -> Result<(), Error> {
let (fs, fs_handler) = fs_async();
fs.open("./examples/test.txt", |file, fs| {
fs.read_to_string(file, |value, fs| {
assert_eq!(TEST_FILE_VALUE, &value);
fs.println(value)?;
fs.close()
})
})?;
fs_handler.join()?;
Ok(())
}
```
running this example:
```bash
cargo run --example fs-mio
```
We can see the difference between two implementations. On the one hand, executor will never be blocking by `result_receiver.recv()`, instead, it will wait for `Poll::poll` returning; on the other hand, io worker thread will execute `set_readiness.set_readiness(Ready::readable())?` after executing `result_sender.send`, to inform executor there are some events happens.
In this case, executor will never be blocked by io worker, because we can register all events in executor, and `mio::Poll` will listen to all events (eg. combine `fs-mio` with `tcp` into a file server).
#### Callback is Evil
Before writing a file server, we should talk about the problems of callback.
Callback can make code confusing, you can realize it more clearly when handling error, like this:
```rust
use asyncio::fs_mio::fs_async;
use failure::Error;
const TEST_FILE_VALUE: &str = "Hello, World!";
fn main() -> Result<(), Error> {
let (fs, fs_handler) = fs_async();
fs.open("./examples/test.txt",
|file, fs| {
fs.read_to_string(file,
|value, fs| {
assert_eq!(TEST_FILE_VALUE, &value);
fs.println(value,
|err| {
...
}
);
fs.close()
},
|err| {
...
}
)
},
|err| {
...
}
)?;
fs_handler.join()?;
Ok(())
}
```
Moreover, there is a lifetime problem in rust when we use closure, which means, we have to clone a environment variable if we want to borrow it in closure (if it implements `Clone`), otherwise we should deliver its reference as a parameter of closure (you should change signature of closure as you need, what a shit!).
Considering a variety of reasons, `rust` eventully uses `coroutine` as its asynchronous API abstraction.
### stackless coroutine
`coroutine` in this blog refers to `stackless coroutine` based on `rust generator` instead of `green thread(stackful coroutine)` which is obsoleted earlier.
#### generator
`rust` introduced `generator` at May of this year, however, it's still unstable and unsafe. Here is a typical Fibonacci sequence generator:
```rust
// examples/fab.rs
// cargo +nightly run --example fab
#![feature(generators, generator_trait)]
use std::ops::{Generator, GeneratorState};
use std::pin::Pin;
fn main() {
let mut gen = fab(5);
loop {
match unsafe { Pin::new_unchecked(&mut gen).resume() } {
GeneratorState::Yielded(value) => println!("yield {}", value),
GeneratorState::Complete(ret) => {
println!("return {}", ret);
break;
}
}
}
}
fn fab(mut n: u64) -> impl Generator<Yield = u64, Return = u64> {
move || {
let mut last = 0u64;
let mut current = 1;
yield last;
while n > 0 {
yield current;
let tmp = last;
last = current;
current = tmp + last;
n -= 1;
}
return last;
}
}
```
Because of the "interrupt behaviors" of `generator`, we will naturally consider to combine it with `mio`: assign a `token` for each `generator`, then poll, resume corresponding `generator` when receive a event; register an awaking event and yield before each generator is going to block. Can we implement non-blocking IO in "synchronous code" on this way?
It seems to work well in theory, but there are still "two dark clouds"。
#### self-referential structs
The first "dark cloud" is relevant to memory management of `rust`.
If you write a `generator` like this:
```rust
fn self_ref_generator() -> impl Generator<Yield=u64, Return=()> {
|| {
let x: u64 = 1;
let ref_x: &u64 = &x;
yield 0;
yield *ref_x;
}
}
```
`rustc` will refuse to compile and tell you "borrow may still be in use when generator yields". You may be a little panic because `rustc` doesn't tell you how to fix it. Going to google it, you will find it is relevant to implementation of `generator`.
As memtioned earlier, `generator` is stackless, which means `rustc` doesn't reserve a complete "stack" for each generator, instead, only variables and values required by a specific "state" will be reserved.
This code is valid:
```rust
fn no_ref_generator() -> impl Generator<Yield=u64, Return=()> {
|| {
let x: u64 = 1;
let ref_x: &u64 = &x;
yield *ref_x;
yield 0;
}
}
```
Because `rustc` knows the only variable or value need be reserved after the first "yield" is literal `0` . However, for the `self_ref_generator`, `rustc` should reserve both of variable `x` and its reference `ref_x` after the first "yield". In this case, generator should be compiled into a structure like this:
```rust
enum SomeGenerator<'a> {
...
SomeState {
x: u64
ref_x: &'a u64
}
...
}
```
This is the notorious "self-referential structs" in `rust`, what will happen when you try to compile code like this?
```rust
struct A<'a> {
b: u64,
ref_b: Option<&'a u64>
}
impl<'a> A<'a> {
fn new() -> Self {
let mut a = A{b: 1, ref_b: None};
a.ref_b = Some(&a.b);
a
}
}
```
Of course, `rustc` will refuse to compile it. It's reasonable, variable `a` on stack will be copied and dropped and its field ref_b will be invalid when function `new` returns. Lifetime rules of `rust` helps you avoid this memory problem.
However, even if you write code like this:
```rust
use std::borrow::{BorrowMut};
struct A<'a> {
b: u64,
ref_b: Option<&'a u64>
}
impl<'a> A<'a> {
fn boxed() -> Box<Self> {
let mut a = Box::new(A{b: 1, ref_b: None});
let mut_ref: &mut A = a.borrow_mut();
mut_ref.ref_b = Some(&mut_ref.b);
a
}
}
```
`rustc` still refuses to compile it. It's unreasonable，variable `a` on heap will not be dropped after function `new` returns, and its field `ref_b` should be always valid. However, `rustc` doesn't know, and you cannot prove it in the language that compiler can understand.
Moreover, you even cannot mutably borrow self-referential structs, like this:
```rust
struct A<'a> {
b: u64,
ref_b: Option<&'a u64>
}
impl<'a> A<'a> {
fn new() -> Self {
A{b: 1, ref_b: None}
}
fn mute(&mut self) {
}
}
fn main() {
let mut a = A::new();
a.ref_b = Some(&a.b);
a.mute();
}
```
`rustc` still refuses to compile it. It's awful, because the signature of earlier `Future::poll` is:
```rust
fn poll(&mut self) -> Poll<Self::Item, Self::Error>;
```
and the signature of `Generator::resume` is still:
```rust
unsafe fn resume(&mut self) -> GeneratorState<Self::Yield, Self::Return>;
```
As a result, self-reference will lead to unable implementation of `trait Generator` and `trait Future` . In this case, we can use `NonNull` to avoid compiler checking:
```rust
use std::ptr::NonNull;
struct A {
b: u64,
ref_b: NonNull<u64>
}
impl A {
fn new() -> Self {
A{b: 1, ref_b: NonNull::dangling()}
}
}
fn main() {
let mut a = A::new();
a.ref_b = NonNull::from(&a.b);
}
```
However, you should guarantee memory safety by yourself (self-referential structs **MUST NOT** be moved, and you **MUST NOT** deliver its mutable reference to `mem::replace` or `mem::swap`), it's not a nice solution.
Can we find some ways to guarantee its moving and mutably borrowing cannot be safe? `rust` introduces `Pin` to hold this job. Specifications of `pin` can be found in this [RFC](https://github.com/rust-lang/rfcs/blob/master/text/2349-pin.md), this blog will only introduce it simply.
##### Pin
`rust` implement `trait std::marker::Unpin` for almost all types by default. It's only a marker to indicate safely moving of a type. For types marked as `Unpin`, `Pin<&'a mut T>` and `&'a mut T` have no difference, you can safely exchange them by `Pin::new(&mut T)` and `Pin::get_mut(this: Pin<&mut T>)`.
However, for types that cannot be moved safely, like the `A` mentioned earlier, we should mark it as `!Unpin` first, a safe way is to give it a field whose type is marked `!Unpin`, for example, `Pinned`.
```rust
#![feature(pin)]
use std::marker::{Pinned};
use std::ptr::NonNull;
struct A {
b: u64,
ref_b: NonNull<u64>,
_pin: Pinned,
}
impl A {
fn new() -> Self {
A {
b: 1,
ref_b: NonNull::dangling(),
_pin: Pinned,
}
}
}
fn main() {
let mut a = A::new();
let mut pinned = unsafe { Pin::new_unchecked(&mut a) };
let ref_b = NonNull::from(&pinned.b);
let mut_ref: Pin<&mut A> = pinned.as_mut();
unsafe {Pin::get_mut_unchecked(mut_ref).ref_b = ref_b};
let unmoved = pinned;
assert_eq!(unmoved.ref_b, NonNull::from(&unmoved.b));
}
```
For types marked as `!Unpin`, `Pin<&'a mut T>` and `&'a mut T` cannot be safely exchanged, you can unsafely exchange them by `Pin::new_unchecked` and `Pin::get_mut_unchecked`. We can always guarantee the safety in the scope we construct it, so after calling two unsafe methods, we can guarantee:
- we can never get mutable reference safely: `Pin::get_mut_unchecked` is unsafe
- we can never move it: because `Pin` only owns a mutable reference, and `Pin::get_mut_unchecked` is unsafe, so deliver the mutable reference into `mem::replace` and `mem::swap` is unsafe
Of course, if you don't want to construct `Pin` in unsafe way or you want `Pin` to own the ownership of the instance, you can use `Box::pin` thus allocate instance on the heap.
```rust
struct A {
b: u64,
ref_b: NonNull<u64>,
_pin: Pinned,
}
impl A {
fn boxed() -> Pin<Box<Self>> {
let mut boxed = Box::pin(A {
b: 1,
ref_b: NonNull::dangling(),
_pin: Pinned,
});
let ref_b = NonNull::from(&boxed.b);
let mut_ref: Pin<&mut A> = boxed.as_mut();
unsafe { Pin::get_mut_unchecked(mut_ref).ref_b = ref_b };
boxed
}
}
fn main() {
let boxed = A::boxed();
let unmoved = boxed;
assert_eq!(unmoved.ref_b, NonNull::from(&unmoved.b));
}
```
After introducing of `Pin`, the new `Future` is defined as:
```rust
pub trait Future {
type Output;
fn poll(self: Pin<&mut Self>, cx: &mut Context<'_>) -> Poll<Self::Output>;
}
```
#### Reasonable Abstraction
The second "dark cloud" is relevant to API abstraction.
Now that `rust` chooses `coroutine` as API abstraction of non-blocking IO, what should be introduced into key words, what should be introduced into standard library and what should be implemented by community? Let developers call unsafe `Generator::resume` is inapposite, using `mio` as the only specified low-level non-blocking IO implementation is also unreasonable.
Up to now, `rust` supports:
- key words
- `async`
- `await`
- standard libraries
- `std::future`
- `trait Future`
- `trait GenFuture`
- `std::task`
- `enum Poll<T>`
- `struct Context`
- `struct Waker`
- `struct RawWaker`
`RawWaker` consists of an data ptr and a virtual table, you can construct one with `SetReadiness` and implement `fn wake(*const ())` by `SetReadiness::set_readiness`. Then you should wrap your raw waker in `Waker ` and `Context`.
##### Poll\<T>
`Poll<T>` is defined as:
```rust
pub enum Poll<T> {
Ready(T),
Pending,
}
```
##### await
`await` is the first suffix keyword in rust, which can only be used in `async` block or function.
`future.await` will be expanded into:
```rust
loop {
if let Poll::Ready(x) = std::future::poll_with_tls_context(unsafe{
Pin::new_unchecked(&mut future)
}) {
break x;
}
yield
}
```
`std::future::poll_with_tls_context` will poll with "thread local context" via a thread-local variable `TLS_CX`.
##### async
`async` is used to wrap a `Generator` into a `GenFuture`. `GenFuture` is defined as:
```rust
struct GenFuture<T: Generator<Yield = ()>>(T);
impl<T: Generator<Yield = ()>> !Unpin for GenFuture<T> {}
impl<T: Generator<Yield = ()>> Future for GenFuture<T> {
type Output = T::Return;
fn poll(self: Pin<&mut Self>, cx: &mut Context<'_>) -> Poll<Self::Output> {
let gen = unsafe { Pin::map_unchecked_mut(self, |s| &mut s.0) };
set_task_context(cx, || match gen.resume() {
GeneratorState::Yielded(()) => Poll::Pending,
GeneratorState::Complete(x) => Poll::Ready(x),
})
}
}
pub fn from_generator<T: Generator<Yield = ()>>(x: T) -> impl Future<Output = T::Return> {
GenFuture(x)
}
```
We can see, `GenFuture` will call `set_task_context` before calling `self.0.resume`, thus code in generator can get this `Context` via `TLS_CX`.
So, code like this:
```rust
async fn async_recv(string_channel: Receiver<String>) -> String {
string_channel.recv_future().await
}
```
will be expanded into:
```rust
fn async_recv(string_channel: Receiver<String>) -> impl Future<Output = T::Return> {
from_generator(move || {
let recv_future = string_channel.recv_future();
loop {
if let Poll::Ready(x) = std::future::poll_with_tls_context(unsafe{
Pin::new_unchecked(&mut recv_future)
}) {
break x;
}
yield
}
})
}
```
#### non-blocking coroutine
Mastering all the basic knowledge mentioned above, we can do some practices.
`coroutine` doesn't mean "non-blocking", you can invoke blocking API in async blocks or functions. The key to non-blocking IO is, when `GenFuture` is going to block (eg. an API returns `io::ErrorKind::WouldBlock`), register a source task by local waker and sleep (`yield`), lower-level non-blocking scheduler will awake this `GenFuture` after task completed.
In [src/executor.rs](https://github.com/Hexilee/async-io-demo/blob/master/src/executor.rs), I implement `Executor`, `block_on`, `spawn`, `TcpListener` and `TcpStream`. Code is a little long, you had better clone and view it in editor.
> Be careful to distinguish `Poll`(`mio::Poll`) from `task::Poll` and to distinguish `net::{TcpListener, TcpStream}`(`mio::net::{TcpListener, TcpStream}`) from `TcpListener, TcpStream`
##### Executor
`Executor` is a struct containing `mio::Poll`, main task waker and two `Slab`s for managing `Task` and `Source`. I don't implement any special methods for it, its only duty is to be initialized as thread local variable `EXECUTOR` and to be borrowed by other functions.
##### block_on
This function will block current thread, the only parameter is `main_task: Future<Output=T>`; type of return value is `T`. This function is generally called by main function.
`block_on` borrows thread local variable `EXECUTOR`, its main logic loop will call `mio::Poll::poll` to wait for events. I classify all tokens (`0 - MAX_RESOURCE_NUM(1 << 31)`) into three kinds:
- main task token
receiving `Token` whose value is `MAIN_TASK_TOKEN (1 << 31)` means main task need be awaked, `main_task.poll` will be called, `block_on` will return `Ok(ret)` if `main_task.poll` returns `task::Poll::Ready(ret)`.
- task token
odd `Token` means corresponding task (spawned by function `spawn`) need be awaked, `task.poll` will be invoked, `block_on` will return `Err(err)` if `task.poll` returns `Err(err)`.
- source token
even `Token` means corresponding source (registered by function `register_source`) is completed, `source.task_waker.waker` will be invoked to awake task which registered it.
##### spawn
Function to spawn tasks.
##### TcpListener
`wrapper` for `mio::net::TcpListener`, method `accept` will return a `Future`.
##### TcpStream
`wrapper` for `mio::net::TcpStream`, method `read` and `write` will both return a `Future`.
##### echo server
Implemented `executor`, we can write a simple echo server:
```rust
// examples/async-echo
#[macro_use]
extern crate log;
use asyncio::executor::{block_on, spawn, TcpListener};
use failure::Error;
fn main() -> Result<(), Error> {
env_logger::init();
block_on(
async {
let mut listener = TcpListener::bind(&"127.0.0.1:7878".parse()?)?;
info!("Listening on 127.0.0.1:7878");
while let Ok((mut stream, addr)) = listener.accept().await {
info!("connection from {}", addr);
spawn(
async move {
let client_hello = stream.read().await?;
let read_length = client_hello.len();
let write_length =
stream.write(client_hello).await?;
assert_eq!(read_length, write_length);
stream.close();
Ok(())
},
)?;
};
Ok(())
},
)?
}
```
run
```bash
RUST_LOG=info cargo run --example async-echo
```
You can test it using `telnet`.
### Afterword
However, to run the example metioned at the beginning, we should implement a non-blocking FS-IO based on `mio` and `std::future`. Here is the final implementation: [src/fs_future.rs](https://github.com/Hexilee/async-io-demo/blob/master/src/fs_future.rs).
Now, let's run:
```bash
RUST_LOG=info cargo run --example file-server
```
Test it using `telnet`:
```bash
[~] telnet 127.0.0.1 7878
Trying 127.0.0.1...
Connected to localhost.
Escape character is '^]'.
Please enter filename: examples/test.txt
Hello, World!
Connection closed by foreign host.
```
You can look up source code by yourself if you are interested in it.
================================================
FILE: README.zh.md
================================================
Table of Contents
=================
* [引言](#引言)
* [异步 IO 的基石 - mio](#异步-io-的基石---mio)
* [异步网络 IO](#异步网络-io)
* [容错性原则](#容错性原则)
* [Poll Option](#poll-option)
* [Still Block](#still-block)
* [自定义事件](#自定义事件)
* [Callback is evil](#callback-is-evil)
* [coroutine](#coroutine)
* [generator](#generator)
* [自引用](#自引用)
* [Pin](#pin)
* [合理的抽象](#合理的抽象)
* [Poll<T>](#pollt)
* [await!](#await)
* [async](#async)
* [non-blocking coroutine](#non-blocking-coroutine)
* [Executor](#executor)
* [block_on](#block_on)
* [spawn](#spawn)
* [TcpListener](#tcplistener)
* [TcpStream](#tcpstream)
* [echo-server](#echo-server)
* [后记](#后记)
### 引言
2018 年接近尾声，`rust` 团队勉强立住了异步 `IO` 的 flag，`async` 成为了关键字，`Pin`, `Future`, `Poll` 和 `await!` 也进入了标准库。不过一直以来实际项目中用不到这套东西，所以也没有主动去了解过。
最近心血来潮想用 `rust` 写点东西，但并找不到比较能看的文档（可能是因为 `rust` 发展太快了，很多都过时了），最后参考[这篇文章](https://cafbit.com/post/tokio_internals/)和 `"new tokio"`( [romio](https://github.com/withoutboats/romio) ) 写了几个 `demo`，并基于 `mio` 在 `coroutine` 中实现了简陋的异步 `IO`。
最终实现的 file-server 如下：
```rust
// examples/async-echo.rs
#![feature(async_await)]
#![feature(await_macro)]
#![feature(futures_api)]
#[macro_use]
extern crate log;
use asyncio::executor::{block_on, spawn, TcpListener, TcpStream};
use asyncio::fs_future::{read_to_string};
use failure::Error;
fn main() -> Result<(), Error> {
env_logger::init();
block_on(new_server())?
}
const CRLF: &[char] = &['\r', '\n'];
async fn new_server() -> Result<(), Error> {
let mut listener = TcpListener::bind(&"127.0.0.1:7878".parse()?)?;
info!("Listening on 127.0.0.1:7878");
while let Ok((stream, addr)) = await!(listener.accept()) {
info!("connection from {}", addr);
spawn(handle_stream(stream))?;
}
Ok(())
}
async fn handle_stream(mut stream: TcpStream) -> Result<(), Error> {
await!(stream.write_str("Please enter filename: "))?;
let file_name_vec = await!(stream.read())?;
let file_name = String::from_utf8(file_name_vec)?.trim_matches(CRLF).to_owned();
let file_contents = await!(read_to_string(file_name))?;
await!(stream.write_str(&file_contents))?;
stream.close();
Ok(())
}
```
写这篇文章的主要目的是梳理和总结，同时也希望能给对这方面有兴趣的 `Rustacean` 作为参考。本文代码以易于理解为主要编码原则，某些地方并没有太考虑性能，还请见谅；但如果文章和代码中有明显错误，欢迎指正。
本文代码仓库在 [Github](https://github.com/Hexilee/async-io-demo) （部分代码较长，建议 `clone` 下来用编辑器看），所有 `examples` 在 `nightly-x86_64-apple-darwin 2018 Edition` 上均能正常运行。运行 `example/async-echo` 时设置 `RUST_LOG` 为 `info` 可以在 terminal 看到基本的运行信息，`debug` 则可见事件循环中的事件触发顺序。
### 异步 `IO` 的基石 - `mio`
`mio` 是一个极简的底层异步 `IO` 库，如今 `rust` 生态中几乎所有的异步 `IO` 程序都基于它。
随着 `channel`, `timer` 等 `sub module` 在 `0.6.5` 版本被标为 `deprecated`，如今的 mio 提供的唯二两个核心功能分别是：
- 对操作系统异步网络 `IO` 的封装
- 用户自定义事件队列
第一个核心功能对应到不同操作系统分别是
- `Linux(Android) => epoll`
- `Windows => iocp`
- `MacOS(iOS), FreeBSD => kqueue`
- `Fuchsia => <unknown>`
mio 把这些不同平台上的 API 封装出了一套 `epoll like` 的异步网络 API，支持 `udp 和 tcp`。
> 除此之外还封装了一些不同平台的拓展 API，比如 `uds`，本文不对这些 API 做介绍。
#### 异步网络 IO
下面是一个 `tcp` 的 `demo`
```rust
// examples/tcp.rs
use mio::*;
use mio::net::{TcpListener, TcpStream};
use std::io::{Read, Write, self};
use failure::Error;
use std::time::{Duration, Instant};
const SERVER_ACCEPT: Token = Token(0);
const SERVER: Token = Token(1);
const CLIENT: Token = Token(2);
const SERVER_HELLO: &[u8] = b"PING";
const CLIENT_HELLO: &[u8] = b"PONG";
fn main() -> Result<(), Error> {
let addr = "127.0.0.1:13265".parse()?;
// Setup the server socket
let server = TcpListener::bind(&addr)?;
// Create a poll instance
let poll = Poll::new()?;
// Start listening for incoming connections
poll.register(&server, SERVER_ACCEPT, Ready::readable(),
PollOpt::edge())?;
// Setup the client socket
let mut client = TcpStream::connect(&addr)?;
let mut server_handler = None;
// Register the client
poll.register(&client, CLIENT, Ready::readable() | Ready::writable(),
PollOpt::edge())?;
// Create storage for events
let mut events = Events::with_capacity(1024);
let start = Instant::now();
let timeout = Duration::from_millis(10);
'top: loop {
poll.poll(&mut events, None)?;
for event in events.iter() {
if start.elapsed() >= timeout {
break 'top
}
match event.token() {
SERVER_ACCEPT => {
let (handler, addr) = server.accept()?;
println!("accept from addr: {}", &addr);
poll.register(&handler, SERVER, Ready::readable() | Ready::writable(), PollOpt::edge())?;
server_handler = Some(handler);
}
SERVER => {
if event.readiness().is_writable() {
if let Some(ref mut handler) = &mut server_handler {
match handler.write(SERVER_HELLO) {
Ok(_) => {
println!("server wrote");
}
Err(ref err) if err.kind() == io::ErrorKind::WouldBlock => continue,
err => {
err?;
}
}
}
}
if event.readiness().is_readable() {
let mut hello = [0; 4];
if let Some(ref mut handler) = &mut server_handler {
match handler.read_exact(&mut hello) {
Ok(_) => {
assert_eq!(CLIENT_HELLO, &hello);
println!("server received");
}
Err(ref err) if err.kind() == io::ErrorKind::WouldBlock => continue,
err => {
err?;
}
}
}
}
}
CLIENT => {
if event.readiness().is_writable() {
match client.write(CLIENT_HELLO) {
Ok(_) => {
println!("client wrote");
}
Err(ref err) if err.kind() == io::ErrorKind::WouldBlock => continue,
err => {
err?;
}
}
}
if event.readiness().is_readable() {
let mut hello = [0; 4];
match client.read_exact(&mut hello) {
Ok(_) => {
assert_eq!(SERVER_HELLO, &hello);
println!("client received");
}
Err(ref err) if err.kind() == io::ErrorKind::WouldBlock => continue,
err => {
err?;
}
}
}
}
_ => unreachable!(),
}
}
};
Ok(())
}
```
这个 `demo` 稍微有点长，接下来我们把它一步步分解。
直接看主循环
```rust
fn main() {
// ...
loop {
poll.poll(&mut events, None).unwrap();
// ...
}
}
```
每次循环都得执行 `poll.poll`，第一个参数是用来存 `events` 的 `Events`， 容量是 `1024`；
```rust
let mut events = Events::with_capacity(1024);
```
第二个参数是 `timeout`，即一个 `Option<Duration>`，超时会直接返回。返回类型是 `io::Result<usize>`。
> 其中的 `usize` 代表 `events` 的数量，这个返回值是 `deprecated` 并且会在之后的版本移除，仅供参考
这里我们设置了 `timeout = None`，所以当这个函数返回时，必然是某些事件被触发了。让我们遍历 `events`：
```rust
match event.token() {
SERVER_ACCEPT => {
let (handler, addr) = server.accept()?;
println!("accept from addr: {}", &addr);
poll.register(&handler, SERVER, Ready::readable() | Ready::writable(), PollOpt::edge())?;
server_handler = Some(handler);
}
SERVER => {
if event.readiness().is_writable() {
if let Some(ref mut handler) = &mut server_handler {
match handler.write(SERVER_HELLO) {
Ok(_) => {
println!("server wrote");
}
Err(ref err) if err.kind() == io::ErrorKind::WouldBlock => continue,
err => {
err?;
}
}
}
}
if event.readiness().is_readable() {
let mut hello = [0; 4];
if let Some(ref mut handler) = &mut server_handler {
match handler.read_exact(&mut hello) {
Ok(_) => {
assert_eq!(CLIENT_HELLO, &hello);
println!("server received");
}
Err(ref err) if err.kind() == io::ErrorKind::WouldBlock => continue,
err => {
err?;
}
}
}
}
}
CLIENT => {
if event.readiness().is_writable() {
match client.write(CLIENT_HELLO) {
Ok(_) => {
println!("client wrote");
}
Err(ref err) if err.kind() == io::ErrorKind::WouldBlock => continue,
err => {
err?;
}
}
}
if event.readiness().is_readable() {
let mut hello = [0; 4];
match client.read_exact(&mut hello) {
Ok(_) => {
assert_eq!(SERVER_HELLO, &hello);
println!("client received");
}
Err(ref err) if err.kind() == io::ErrorKind::WouldBlock => continue,
err => {
err?;
}
}
}
}
_ => unreachable!(),
}
```
我们匹配每一个 `event` 的 `token`，这里的 `token` 就是我用来注册的那些 `token`。比如我在上面注册了 `server`
```rust
// Start listening for incoming connections
poll.register(&server, SERVER_ACCEPT, Ready::readable(),
PollOpt::edge()).unwrap();
```
第二个参数就是 `token`
```rust
const SERVER_ACCEPT: Token = Token(0);
```
这样当 `event.token() == SERVER_ACCEPT` 时，就说明这个事件跟我们注册的 `server` 有关，于是我们试图 `accept` 一个新的连接并把它注册进 `poll`，使用的 `token` 是 `SERVER`。
```rust
let (handler, addr) = server.accept()?;
println!("accept from addr: {}", &addr);
poll.register(&handler, SERVER, Ready::readable() | Ready::writable(), PollOpt::edge())?;
server_handler = Some(handler);
```
这样我们之后如果发现 `event.token() == SERVER`，我们就认为它和注册的 `handler` 有关：
```rust
if event.readiness().is_writable() {
if let Some(ref mut handler) = &mut server_handler {
match handler.write(SERVER_HELLO) {
Ok(_) => {
println!("server wrote");
}
Err(ref err) if err.kind() == io::ErrorKind::WouldBlock => continue,
err => {
err?;
}
}
}
}
if event.readiness().is_readable() {
let mut hello = [0; 4];
if let Some(ref mut handler) = &mut server_handler {
match handler.read_exact(&mut hello) {
Ok(_) => {
assert_eq!(CLIENT_HELLO, &hello);
println!("server received");
}
Err(ref err) if err.kind() == io::ErrorKind::WouldBlock => continue,
err => {
err?;
}
}
}
}
```
这时候我们还需要判断 `event.readiness()`，这就是 `register` 函数的第三个参数，叫做 `interest`，顾名思义，就是“感兴趣的事”。它的类型是 `Ready`，一共四种，`readable, writable, error 和 hup`，可进行并运算。
在上面我们给 `handler` 注册了 `Ready::readable() | Ready::writable()`，所以 `event` 可能是 `readable` 也可能是 `writable`，所以我们要经过判断来执行相应的逻辑。注意这里的判断是
```rust
if ... {
...
}
if ... {
...
}
```
而非
```rust
if ... {
...
} else if ... {
...
}
```
因为一个事件可能同时是 `readable` 和 `writable`。
#### 容错性原则
大概逻辑先讲到这儿，这里先讲一下 `mio` 的“容错性原则”，即不能完全相信 `event`。
可以看到我上面有一段代码是这么写的
```rust
match event.token() {
SERVER_ACCEPT => {
let (handler, addr) = server.accept()?;
println!("accept from addr: {}", &addr);
poll.register(&handler, SERVER, Ready::readable() | Ready::writable(), PollOpt::edge())?;
server_handler = Some(handler);
}
```
`server.accept()` 返回的是 `io::Result<(TcpStream, SocketAddr)>`。如果我们选择完全相信 `event` 的话，在这里 `unwrap()` 并没有太大问题 —— 如果真的有一个新的连接就绪，`accept()` 产生的 `io::Result` 是我们无法预料且无法处理的，我们应该抛给调用者或者直接 `panic`。
但问题就是，我们可以认为 `event` 的伪消息是可预料的，可能并没有一个新的连接准备就绪，这时候我们 `accept()` 会引发 `WouldBlock Error`。但我们不应该认为 `WouldBlock` 是一种错误 —— 这是一种友善的提醒。`server` 告诉我们：“并没有新的连接，请下次再来吧。”，所以在这里我们应该忽略（可以打个 `log`）它并重新进入循环。
像我后面写的那样：
```rust
match client.write(CLIENT_HELLO) {
Ok(_) => {
println!("client wrote");
}
Err(ref err) if err.kind() == io::ErrorKind::WouldBlock => continue,
err => {
err?;
}
}
```
#### Poll Option
好了，现在我们可以运行：
```bash
[async-io-demo] cargo run --example tcp
```
terminal 里打印出了
```bash
client wrote
accept from addr: 127.0.0.1:53205
client wrote
server wrote
server received
...
```
我们可以发现，在短短的 `10 millis` 内（`let timeout = Duration::from_millis(10);`），`server` 和 `client` 分别进行了数十次的读写！
如果我们不想进行这么多次读写呢？比如，我们只想让 `server` 写一次。在网络比较通畅的情况下，`client` 和 `server` 几乎一直是可写的，所以 `Poll::poll` 在数微秒内就返回了。
这时候就要看 `register` 的第四个参数了。
```rust
poll.register(&server, SERVER_ACCEPT, Ready::readable(),
PollOpt::edge()).unwrap();
```
`PollOpt::edge()` 的类型是 `PollOpt`，一共有 `level, edge, oneshot` 三种，他们有什么区别呢？
比如在我上面的代码里，
```rust
if event.readiness().is_readable() {
let mut hello = [0; 4];
match client.read_exact(&mut hello) {
Ok(_) => {
assert_eq!(SERVER_HELLO, &hello);
println!("client received");
}
Err(ref err) if err.kind() == io::ErrorKind::WouldBlock => continue,
err => {
err?;
}
}
}
```
我在收到一个 `readable readiness` 时，只读了四个字节。如果这时候缓冲区里有八字节的数据，那么：
- 如果我注册时使用 `PollOpt::level()`，我在下次 `poll` 时 **一定** 还能收到一次 `readable readiness event` （只要我没有主动执行 `set_readiness(Read::empty())`）；
- 如果我注册时使用 `PollOpt::edge()`，我在下次 `poll` 时 **不一定** 还能收到一次 `readable readiness event`；
所以，使用 `PollOpt::edge()` 时有一个“排尽原则（`Draining readiness`）”，即每次触发 `event` 时一定要操作到资源耗尽返回 `WouldBlock`，即上面的代码要改成：
```rust
if event.readiness().is_readable() {
let mut hello = [0; 4];
loop {
match client.read_exact(&mut hello) {
Ok(_) => {
assert_eq!(SERVER_HELLO, &hello);
println!("client received");
}
Err(ref err) if err.kind() == io::ErrorKind::WouldBlock => break,
err => {
err?;
}
}
}
}
```
那么，`oneshot` 又是怎样的行为呢？让我们回到上面的问题，如果我们只想让 `handler` 写一次，怎么办 —— 注册时使用 `PollOpt::oneshot()`，即
```rust
let (handler, addr) = server.accept()?;
println!("accept from addr: {}", &addr);
poll.register(&handler, SERVER, Ready::readable() | Ready::writable(), PollOpt::oneshot())?;
server_handler = Some(handler);
```
这样的话，你只能收到一次 `SERVER` 事件，除非你使用 `Poll::reregister` 重新注册 `handler`。
> `Poll::reregister` 可以更改 `PollOpt` 和 `interest`
#### Still Block
其实上面这个 `demo` 还存在一个问题，即我们在回调代码块中使用了同步的 `IO` 操作 `println!`。我们要尽可能避免在回调的代码块里使用耗时的 `IO` 操作。
考虑到文件 `IO` (包括 `Stdin, Stdout, Stderr`) 速度很慢，我们只需要把所有的文件 `IO` 交给一个线程进行即可。
```rust
use std::sync::mpsc::{Sender, Receiver, channel, SendError};
#[derive(Clone)]
pub struct Fs {
task_sender: Sender<Task>,
}
impl Fs {
pub fn new() -> Self {
let (sender, receiver) = channel();
std::thread::spawn(move || {
loop {
match receiver.recv() {
Ok(task) => {
match task {
Task::Println(ref string) => println!("{}", string),
Task::Exit => return
}
},
Err(_) => {
return;
}
}
}
});
Fs { task_sender: sender }
}
pub fn println(&self, string: String) {
self.task_sender.send(Task::Println(string)).unwrap()
}
}
pub enum Task {
Exit,
Println(String),
}
```
之后，可以使用 `Fs::println` 替换所有的 `println!`。
#### 自定义事件
上面我们实现异步 `println` 比较简单，这是因为 `println` 并没有返回值，不需要进行后续操作。设想一下，如果要我们实现 `open` 和 `read_to_string`，先异步地 `open` 一个文件，然后异步地 `read_to_string`，最后再异步地 `println`, 我们要怎么做？
最简单的写法是回调，像这样：
```rust
// src/fs.rs
use crossbeam_channel::{unbounded, Sender};
use std::fs::File;
use std::io::Read;
use std::boxed::FnBox;
use std::thread;
use failure::Error;
#[derive(Clone)]
pub struct Fs {
task_sender: Sender<Task>,
}
pub struct FsHandler {
io_worker: thread::JoinHandle<Result<(), Error>>,
executor: thread::JoinHandle<Result<(), Error>>,
}
pub fn fs_async() -> (Fs, FsHandler) {
let (task_sender, task_receiver) = unbounded();
let (result_sender, result_receiver) = unbounded();
let io_worker = std::thread::spawn(move || {
loop {
match task_receiver.recv() {
Ok(task) => {
match task {
Task::Println(ref string) => println!("{}", string),
Task::Open(path, callback, fs) => {
result_sender
.send(TaskResult::Open(File::open(path)?, callback, fs))?
}
Task::ReadToString(mut file, callback, fs) => {
let mut value = String::new();
file.read_to_string(&mut value)?;
result_sender
.send(TaskResult::ReadToString(value, callback, fs))?
}
Task::Exit => {
result_sender
.send(TaskResult::Exit)?;
break;
}
}
}
Err(_) => {
break;
}
}
}
Ok(())
});
let executor = std::thread::spawn(move || {
loop {
let result = result_receiver.recv()?;
match result {
TaskResult::ReadToString(value, callback, fs) => callback.call_box((value, fs))?,
TaskResult::Open(file, callback, fs) => callback.call_box((file, fs))?,
TaskResult::Exit => break
};
};
Ok(())
});
(Fs { task_sender }, FsHandler { io_worker, executor })
}
impl Fs {
pub fn println(&self, string: String) -> Result<(), Error> {
Ok(self.task_sender.send(Task::Println(string))?)
}
pub fn open<F>(&self, path: &str, callback: F) -> Result<(), Error>
where F: FnOnce(File, Fs) -> Result<(), Error> + Sync + Send + 'static {
Ok(self.task_sender.send(Task::Open(path.to_string(), Box::new(callback), self.clone()))?)
}
pub fn read_to_string<F>(&self, file: File, callback: F) -> Result<(), Error>
where F: FnOnce(String, Fs) -> Result<(), Error> + Sync + Send + 'static {
Ok(self.task_sender.send(Task::ReadToString(file, Box::new(callback), self.clone()))?)
}
pub fn close(&self) -> Result<(), Error> {
Ok(self.task_sender.send(Task::Exit)?)
}
}
impl FsHandler {
pub fn join(self) -> Result<(), Error> {
self.io_worker.join().unwrap()?;
self.executor.join().unwrap()
}
}
type FileCallback = Box<FnBox(File, Fs) -> Result<(), Error> + Sync + Send>;
type StringCallback = Box<FnBox(String, Fs) -> Result<(), Error> + Sync + Send>;
pub enum Task {
Exit,
Println(String),
Open(String, FileCallback, Fs),
ReadToString(File, StringCallback, Fs),
}
pub enum TaskResult {
Exit,
Open(File, FileCallback, Fs),
ReadToString(String, StringCallback, Fs),
}
```
```rust
// examples/fs.rs
use asyncio::fs::fs_async;
use failure::Error;
const TEST_FILE_VALUE: &str = "Hello, World!";
fn main() -> Result<(), Error> {
let (fs, fs_handler) = fs_async();
fs.open("./examples/test.txt", |file, fs| {
fs.read_to_string(file, |value, fs| {
assert_eq!(TEST_FILE_VALUE, &value);
fs.println(value)?;
fs.close()
})
})?;
fs_handler.join()?;
Ok(())
}
```
测试
```bash
[async-io-demo] cargo run --example fs
```
这样写在逻辑上的确是对的，但是负责跑 `callback` 的 `executor` 线程其实被负责 `io` 的线程阻塞住了（`result_receiver.recv()`）。那我们能不能在 `executor` 线程里跑一个事件循环，以达到不被 `io` 线程阻塞的目的呢？（即确定 `result_receiver` 中有 `result` 时，`executor` 才会进行 `result_receiver.recv()`）.
这就到了体现 `mio` 强大可拓展性的时候：注册用户态的事件队列。
把上面的代码稍加修改，就成了这样：
```rust
// src/fs_mio.rs
use crossbeam_channel::{unbounded, Sender, TryRecvError};
use std::fs::File;
use std::io::{Read};
use std::boxed::FnBox;
use std::thread;
use failure::Error;
use std::time::Duration;
use mio::*;
#[derive(Clone)]
pub struct Fs {
task_sender: Sender<Task>,
}
pub struct FsHandler {
io_worker: thread::JoinHandle<Result<(), Error>>,
executor: thread::JoinHandle<Result<(), Error>>,
}
const FS_TOKEN: Token = Token(0);
pub fn fs_async() -> (Fs, FsHandler) {
let (task_sender, task_receiver) = unbounded();
let (result_sender, result_receiver) = unbounded();
let poll = Poll::new().unwrap();
let (registration, set_readiness) = Registration::new2();
poll.register(®istration, FS_TOKEN, Ready::readable(), PollOpt::oneshot()).unwrap();
let io_worker = std::thread::spawn(move || {
loop {
match task_receiver.recv() {
Ok(task) => {
match task {
Task::Println(ref string) => println!("{}", string),
Task::Open(path, callback, fs) => {
result_sender
.send(TaskResult::Open(File::open(path)?, callback, fs))?;
set_readiness.set_readiness(Ready::readable())?;
}
Task::ReadToString(mut file, callback, fs) => {
let mut value = String::new();
file.read_to_string(&mut value)?;
result_sender
.send(TaskResult::ReadToString(value, callback, fs))?;
set_readiness.set_readiness(Ready::readable())?;
}
Task::Exit => {
result_sender
.send(TaskResult::Exit)?;
set_readiness.set_readiness(Ready::readable())?;
break;
}
}
}
Err(_) => {
break;
}
}
}
Ok(())
});
let executor = thread::spawn(move || {
let mut events = Events::with_capacity(1024);
'outer: loop {
poll.poll(&mut events, Some(Duration::from_secs(1)))?;
for event in events.iter() {
match event.token() {
FS_TOKEN => {
loop {
match result_receiver.try_recv() {
Ok(result) => {
match result {
TaskResult::ReadToString(value, callback, fs) => callback.call_box((value, fs))?,
TaskResult::Open(file, callback, fs) => callback.call_box((file, fs))?,
TaskResult::Exit => break 'outer
}
}
Err(e) => {
match e {
TryRecvError::Empty => break,
TryRecvError::Disconnected => Err(e)?
}
}
}
}
poll.reregister(®istration, FS_TOKEN, Ready::readable(), PollOpt::oneshot())?;
}
_ => unreachable!()
}
}
};
Ok(())
});
(Fs { task_sender }, FsHandler { io_worker, executor })
}
impl Fs {
pub fn println(&self, string: String) -> Result<(), Error> {
Ok(self.task_sender.send(Task::Println(string))?)
}
pub fn open<F>(&self, path: &str, callback: F) -> Result<(), Error>
where F: FnOnce(File, Fs) -> Result<(), Error> + Sync + Send + 'static {
Ok(self.task_sender.send(Task::Open(path.to_string(), Box::new(callback), self.clone()))?)
}
pub fn read_to_string<F>(&self, file: File, callback: F) -> Result<(), Error>
where F: FnOnce(String, Fs) -> Result<(), Error> + Sync + Send + 'static {
Ok(self.task_sender.send(Task::ReadToString(file, Box::new(callback), self.clone()))?)
}
pub fn close(&self) -> Result<(), Error> {
Ok(self.task_sender.send(Task::Exit)?)
}
}
impl FsHandler {
pub fn join(self) -> Result<(), Error> {
self.io_worker.join().unwrap()?;
self.executor.join().unwrap()
}
}
type FileCallback = Box<FnBox(File, Fs) -> Result<(), Error> + Sync + Send>;
type StringCallback = Box<FnBox(String, Fs) -> Result<(), Error> + Sync + Send>;
pub enum Task {
Exit,
Println(String),
Open(String, FileCallback, Fs),
ReadToString(File, StringCallback, Fs),
}
pub enum TaskResult {
Exit,
Open(File, FileCallback, Fs),
ReadToString(String, StringCallback, Fs),
}
```
```rust
// examples/fs-mio.rs
use asyncio::fs_mio::fs_async;
use failure::Error;
const TEST_FILE_VALUE: &str = "Hello, World!";
fn main() -> Result<(), Error> {
let (fs, fs_handler) = fs_async();
fs.open("./examples/test.txt", |file, fs| {
fs.read_to_string(file, |value, fs| {
assert_eq!(TEST_FILE_VALUE, &value);
fs.println(value)?;
fs.close()
})
})?;
fs_handler.join()?;
Ok(())
}
```
可以注意到，上面的代码发生的改变就是，`executor` 不再被 `result_receiver.recv` 阻塞，而变成了注册事件（`registration`）后等待 `Poll::poll` 返回事件；只有等到了新的事件，才会进行 `result_receiver.try_recv`。同时，`io_worker` 线程在 `send result` 之后会执行 `set_readiness.set_readiness(Ready::readable())?;`，以通知 `executor` 线程对相应结果做处理。
这样的话，`executor` 就不会被 `io worker` 阻塞了，因为我们可以把所有的事件都注册到 `executor` 上，`mio::Poll` 会同时监听多个事件（比如把 `fs` 和 `tcp` 结合起来）。
测试
```bash
[async-io-demo] cargo run --example fs-mio
```
#### Callback is evil
既然文件 `IO` 的 `executor` 不再会被 `io worker` 线程阻塞了，那我们来试试让 `fs` 和 `tcp` 共用一个 `poll` 然后建立一个简单的文件服务器吧。
但可以先等等，因为我已经开始觉得写 `callback` 有点难受了 —— 如果我们还想处理错误的话，会觉得更难受，像这样
```rust
use asyncio::fs_mio::fs_async;
use failure::Error;
const TEST_FILE_VALUE: &str = "Hello, World!";
fn main() -> Result<(), Error> {
let (fs, fs_handler) = fs_async();
fs.open("./examples/test.txt",
|file, fs| {
fs.read_to_string(file,
|value, fs| {
assert_eq!(TEST_FILE_VALUE, &value);
fs.println(value,
|err| {
...
}
);
fs.close()
},
|err| {
...
}
)
},
|err| {
...
}
)?;
fs_handler.join()?;
Ok(())
}
```
而且对 `rust` 来说，更加艰难的是闭包中的生命周期问题（闭包几乎不能通过捕获来借用环境变量）。这就意味着，如果我要借用环境中的某个变量，我要么 `clone` 它（如果它实现了 `Clone` 的话），要么把它作为闭包参数传入（意味着你要根据需要改每一层回调函数的签名，这太屎了）。
考虑到各种原因，`rust` 最终选择用 `coroutine` 作为异步 `IO` 的 `API` 抽象。
### coroutine
这里所说的 `coroutine` 是指基于 `rust generator` 的 `stackless coroutine` 而非早期被 `rust` 抛弃的 `green thread(stackful coroutine)`。
#### generator
`rust` 大概在今年五月份引入了 `generator`，但到现在还是 unstable 的 —— 虽说也没多少人用 stable（误
一个典型的斐波那契 `generator` 如下
```rust
// examples/fab.rs
#![feature(generators, generator_trait)]
use std::ops::{Generator, GeneratorState};
fn main() {
let mut gen = fab(5);
loop {
match unsafe { gen.resume() } {
GeneratorState::Yielded(value) => println!("yield {}", value),
GeneratorState::Complete(ret) => {
println!("return {}", ret);
break;
}
}
}
}
fn fab(mut n: u64) -> impl Generator<Yield=u64, Return=u64> {
move || {
let mut last = 0u64;
let mut current = 1;
yield last;
while n > 0 {
yield current;
let tmp = last;
last = current;
current = tmp + last;
n -= 1;
}
return last;
}
}
```
由于 `generator` 的“中断特性”，我们很自然的可以想到，如果用 `generator` 搭配 `mio`，给每个 `generator` 分配一个 `token`，然后 `poll mio` 的事件循环，收到一个唤醒事件就 `resume` 相应的 `generator`；每个 `generator` 在要阻塞的时候拿自己的 `token` 注册一个唤醒事件然后 `yield`，不就实现了“同步代码”的异步 `IO` 吗？
这样看来原理上来说已经稳了，但 `rust` 异步 `IO` 的天空依旧漂浮着两朵乌云。
#### 自引用
第一朵乌云和 `rust` 自身的内存管理机制有关。
如果你写出这样的 `generator`
```rust
fn self_ref_generator() -> impl Generator<Yield=u64, Return=()> {
|| {
let x: u64 = 1;
let ref_x: &u64 = &x;
yield 0;
yield *ref_x;
}
}
```
`rust` 一定会给你抛个错然后告诉你 "borrow may still be in use when generator yields"。编译器没有教你怎么修正可能会让你有些恐慌，去不存在的搜索引擎上查了查，你发现这和 `generator` 的实现有关。
前文中提到，`rust generator` 是 `stackless` 的，即它并不会保留一个完整的栈，而是根据不同的状态保留需要的变量。如果你把上面的代码改成
```rust
fn no_ref_generator() -> impl Generator<Yield=u64, Return=()> {
|| {
let x: u64 = 1;
let ref_x: &u64 = &x;
yield *ref_x;
yield 0;
}
}
```
在第一次 `yield` 结束之后，编译器会发现 `generator` 唯一需要保留的是字面量 `0`，所以这段代码可以顺利编译通过。但是，对于前面的 `generator`，第一次 `yield` 过后，编译器发现你需要同时保留 `x` 和它的引用 `ref_x`，这样的话 `generator` 就会变成类似这样的结构（仅供参考）：
```rust
enum SomeGenerator<'a> {
...
SomeState {
x: u64
ref_x: &'a u64
}
...
}
```
这就是 `rust` 中“臭名昭著” 的自引用，下面这段代码会发生什么呢
```rust
struct A<'a> {
b: u64,
ref_b: Option<&'a u64>
}
impl<'a> A<'a> {
fn new() -> Self {
let mut a = A{b: 1, ref_b: None};
a.ref_b = Some(&a.b);
a
}
}
```
你会发现它编译不过，当然这是很合理的，栈上的 a 变量拷贝出去之后其成员 b 的引用会失效，`rust`的生命周期机制帮你规避了这个问题。但即使你改成这样
```rust
use std::borrow::{BorrowMut};
struct A<'a> {
b: u64,
ref_b: Option<&'a u64>
}
impl<'a> A<'a> {
fn boxed() -> Box<Self> {
let mut a = Box::new(A{b: 1, ref_b: None});
let mut_ref: &mut A = a.borrow_mut();
mut_ref.ref_b = Some(&mut_ref.b);
a
}
}
```
这样按道理来说是没问题的，因为 a 的实体已经在堆上了，即使你拷贝它在栈上的引用，也不会改变其成员 b 的地址，引用一直是有效的 —— 但问题是，你没法跟编译器解释这事，编译器认为函数里面的 `&mut_ref.b`只能活到函数结束，这样含有这个引用的 a 自然也不能 move 出来。
那你可能会想，那我就在外面再取引用就好了
```rust
struct A<'a> {
b: u64,
ref_b: Option<&'a u64>
}
impl<'a> A<'a> {
fn new() -> Self {
A{b: 1, ref_b: None}
}
}
fn main() {
let mut a = A::new();
a.ref_b = Some(&a.b);
}
```
这样的确没啥毛病，但是，你会发现自引用不仅阻止了 move，还阻止了你对 A 可变引用。。比如这样就编译不过
```rust
struct A<'a> {
b: u64,
ref_b: Option<&'a u64>
}
impl<'a> A<'a> {
fn new() -> Self {
A{b: 1, ref_b: None}
}
fn mute(&mut self) {
}
}
fn main() {
let mut a = A::new();
a.ref_b = Some(&a.b);
a.mute();
}
```
但远古的 `Future::poll` 签名就长这样
```rust
fn poll(&mut self) -> Poll<Self::Item, Self::Error>;
```
而直到现在 `Generator::resume` 的签名还是这样
```rust
unsafe fn resume(&mut self) -> GeneratorState<Self::Yield, Self::Return>;
```
这样的话自引用会导致 `generator` 无法实现 `Generator` 和 `Future`
在这种情况下，我们可以使用 `NonNull`来避过编译器的检查
```rust
use std::ptr::NonNull;
struct A {
b: u64,
ref_b: NonNull<u64>
}
impl A {
fn new() -> Self {
A{b: 1, ref_b: NonNull::dangling()}
}
}
fn main() {
let mut a = A::new();
a.ref_b = NonNull::from(&a.b);
}
```
这样的确没有了烦人的生命周期约束，但也意味着你要自己保证内存安全 —— 绝对不能 move，也不能对其可变引用使用 `mem::replace` 或 `mem::swap` ，这样非常不妙。
##### Pin
那有没有办法通过其它方式来保证能保证它不能被 move 或者取可变引用呢？这就是 `pin`的应用场景了。`pin`具体的内容可以看这篇 [RFC](https://github.com/rust-lang/rfcs/blob/master/text/2349-pin.md)，本文只是简要说明一下。
`rust` 默认给大部分类型实现了 `trait std::marker::Unpin`，这只是一个标记，表示这个类型 move 是安全的，这时候，`Pin<&'a mut T>` 跟 `&'a mut T` 没有区别，你也可以安全地通过 `Pin::new(&mut T)` 和 `Pin::get_mut(this: Pin<&mut T>)`相互转换。
但对于不能安全 move 的类型，比如上面的 `A`，我们得先把它标记为 `!Unpin`，安全的标记方法是给它一个 `!Unpin`的成员，比如 `Pinned`。
```rust
#![feature(pin)]
use std::marker::{Pinned};
use std::ptr::NonNull;
struct A {
b: u64,
ref_b: NonNull<u64>,
_pin: Pinned,
}
impl A {
fn new() -> Self {
A {
b: 1,
ref_b: NonNull::dangling(),
_pin: Pinned,
}
}
}
fn main() {
let mut a = A::new();
let mut pinned = unsafe { Pin::new_unchecked(&mut a) };
let ref_b = NonNull::from(&pinned.b);
let mut_ref: Pin<&mut A> = pinned.as_mut();
unsafe {Pin::get_mut_unchecked(mut_ref).ref_b = ref_b};
let unmoved = pinned;
assert_eq!(unmoved.ref_b, NonNull::from(&unmoved.b));
}
```
从 `!Unpin` 的类型构建 `Pin` 总是 `unsafe` 的，它们通过 `Pin::new_unchecked` 和 `Pin::get_mut_unchecked` 相互转换。当然，我们在构建时是可以保证它是 `safe` ，我们只要完成这两个 `unsafe`的操作，就可以保证：
- 永远不能 `safe` 地获得可变引用： `Pin::get_mut_unchecked` 是 `unsafe` 的
- 永远不能 `safe` 地 move：因为 `Pin` 只拥有可变引用，且由于`Pin::get_mut_unchecked` 是 `unsafe` 的，你不能 `safe` 地对其可变引用使用 `mem::replace` 或 `mem::swap`
当然，如果你不想在构建时使用 `unsafe`或者想获得 `a` 的所有权以便在函数间传递，你可以使用 `Box::pinned`从而把它分配在堆上
```rust
struct A {
b: u64,
ref_b: NonNull<u64>,
_pin: Pinned,
}
impl A {
fn boxed() -> Pin<Box<Self>> {
let mut boxed = Box::pinned(A {
b: 1,
ref_b: NonNull::dangling(),
_pin: Pinned,
});
let ref_b = NonNull::from(&boxed.b);
let mut_ref: Pin<&mut A> = boxed.as_mut();
unsafe { Pin::get_mut_unchecked(mut_ref).ref_b = ref_b };
boxed
}
}
fn main() {
let boxed = A::boxed();
let unmoved = boxed;
assert_eq!(unmoved.ref_b, NonNull::from(&unmoved.b));
}
```
有了 `Pin` 之后，新版 `Future` 的定义就是这样的了
```rust
pub trait Future {
type Output;
fn poll(self: Pin<&mut Self>, lw: &LocalWaker) -> Poll<Self::Output>;
}
```
#### 合理的抽象
既然已经打算钦定了 `coroutine` 作为异步 `IO` 的 `API` 抽象，那应该把哪些东西加入标准库、哪些东西加入语法支持、哪些东西交给第三方实现呢？让开发者手动调用 `unsafe` 的 `Generator::resume` 终归不是很妙，也不好把 `mio` 作为唯一的底层异步 `IO` 实现（如果这样的话不如把 `mio` 也并入标准库）。
现在的 `rust` 提供了 `async` 的语法支持（以前是用过程宏的实现的）、`await!`的标准库宏支持，标准库 `std::future` 的 `trait Future` 和 `struct GenFuture` ， 标准库 `std::task` 的 `enum Poll<T>, struct LocalWaker, struct Waker ` 和 `trait UnsafeWaker`。
你需要给你的 `MyWaker` 实现 `trait UnsafeWaker`，用 `mio` 的话就用 `SetReadiness`，`unsafe fn wake(&self)` 用 `SetReadiness::set_readiness` 实现。然后把 `MyWaker` 包在 `Waker, LocalWaker` 里面。
##### Poll\<T\>
`Poll<T>` 的定义为
```rust
pub enum Poll<T> {
Ready(T),
Pending,
}
```
##### await!
`await!` 宏只能在 `async` 函数或者块里面用，传入一个 `Future`
`await!(future)`会被展开成
```rust
loop {
if let Poll::Ready(x) = ::future::poll_with_tls(unsafe{
Pin::new_unchecked(&mut future)
}) {
break x;
}
yield
}
```
`::future::poll_with_tls` 即 `thread-local waker`，就是你传给这个 `GenFuture::poll` 的 `LocalWaker`，
##### async
`async`则会把 `Generator` 包装成 `Future(GenFuture)` 。
`GenFuture` 的相关定义如下
```rust
struct GenFuture<T: Generator<Yield = ()>>(T);
impl<T: Generator<Yield = ()>> !Unpin for GenFuture<T> {}
impl<T: Generator<Yield = ()>> Future for GenFuture<T> {
type Output = T::Return;
fn poll(self: Pin<&mut Self>, lw: &LocalWaker) -> Poll<Self::Output> {
set_task_waker(lw, || match unsafe { Pin::get_mut_unchecked(self).0.resume() } {
GeneratorState::Yielded(()) => Poll::Pending,
GeneratorState::Complete(x) => Poll::Ready(x),
})
}
}
pub fn from_generator<T: Generator<Yield = ()>>(x: T) -> impl Future<Output = T::Return> {
GenFuture(x)
}
```
这里可以看到，`GenFuture` 在每次调用 `self.0.resume` 之前会 `set_task_waker`，通过一个 `thread_local` 的变量中转，从而 `generator` 里面的 `future::poll` 能通过 `poll_with_tls` 拿到这个 `LocalWaker`。
所以，下面的代码
```rust
async fn async_recv(string_channel: Receiver<String>) -> String {
await!(string_channel.recv_future())
}
```
会被类似地展开为这样
```rust
fn async_recv(string_channel: Receiver<String>) -> impl Future<Output = T::Return> {
from_generator(move || {
let recv_future = string_channel.recv_future();
loop {
if let Poll::Ready(x) = ::future::poll_with_tls(unsafe{
Pin::new_unchecked(&mut recv_future)
}) {
break x;
}
yield
}
})
}
```
#### non-blocking coroutine
掌握了上文的基础知识后，我们就可以开始实践了。
coroutine 本身并不意味着“非阻塞”，你完全可以在两次 `yield` 之间调用阻塞 `IO` 的 `API` 从而导致阻塞。 非阻塞的关键在于，在将要阻塞的时候（比如某个 `API` 返回了 `io::ErrorKind::WouldBlock`），在 `GenFuture::poll` 中用底层异步接口注册一个事件和唤醒回调（`waker`）然后自身休眠（`yield`），底层异步调度在特定事件发生的时候回调唤醒这个 `Future`。
下面我参照 `romio` 的异步调度实现了 `Executor` `block_on, spawn, TcpListener` 和 `TcpStream`，代码较长，建议 `clone` 后用编辑器看。（请注意区分 `Poll(mio::Poll)` 与 `task::Poll` 以及 `net::{TcpListener, TcpStream}(mio::net::{TcpListener, TcpStream})` 与 `TcpListener, TcpStream`）
[src/executor.rs](https://github.com/Hexilee/async-io-demo/blob/master/src/executor.rs)
##### Executor
`Executor` 中包含 `mio::Poll`，`main task waker` 及用来管理 `task` 和 `source` 的 `Slab` 各一个。其本身并没有实现什么特别的方法，主要是初始化为 `thread_local` 的 `EXECUTOR` 供其它函数借用。
##### block_on
`block_on` 函数会阻塞当前线程，传入参数是一个 `future: Future<Output=T>`，被称为 `main task`；返回值类型是 `T`。该函数一般在最外层被调用。
`block_on` 会引用 `thread_local EXECUTOR`，主要逻辑是调用 `mio::Poll::poll` 来响应事件。`block_on` 把 `0 - MAX_RESOURCE_NUM(1 << 31)` 个 `Token` 分为三类。
- `main task token`
收到 `Token` 为 `MAIN_TASK_TOKEN` 的事件即表示需要唤醒 `main task`，执行 `main_task.poll`，返回 `task::Poll::Ready(T)` 则 `block_on` 函数返回。
- `task token`
奇数 `token` 表示由 `spawn` 函数分发的其它任务需要被唤醒，执行相应的 `task.poll`，`token` 和该事件在 `EXECUTOR.tasks` 中的 `index` 一一映射。
- `source token`
偶数 `token` 表示由 `register_source` 函数注册的 `source`需要被分发，执行相应 `source` 的 `waker()` 以唤醒分发它们的 `task`。
##### spawn
分发任务
##### TcpListener
包装了 `mio::net::TcpListener`，`accept` 方法返回一个 `Future`。
##### TcpStream
包装了 `mio::net::TcpStream`, `read`和 `write` 方法均返回 `Future`。
##### echo-server
实现了 `executor` 之后，我们可以就写一个简单的 `echo-server` 了
```rust
// examples/async-echo
#![feature(async_await)]
#![feature(await_macro)]
#[macro_use]
extern crate log;
use asyncio::executor::{block_on, spawn, TcpListener};
use failure::Error;
fn main() -> Result<(), Error> {
env_logger::init();
block_on(
async {
let mut listener = TcpListener::bind(&"127.0.0.1:7878".parse()?)?;
info!("Listening on 127.0.0.1:7878");
while let Ok((mut stream, addr)) = await!(listener.accept()) {
info!("connection from {}", addr);
spawn(
async move {
let client_hello = await!(stream.read())?;
let read_length = client_hello.len();
let write_length =
await!(stream.write(client_hello))?;
assert_eq!(read_length, write_length);
stream.close();
Ok(())
},
)?;
};
Ok(())
},
)?
}
```
```bash
RUST_LOG=info cargo run --example async-echo
```
可以用 `telnet` 连上试试看。
### 后记
当然最后还留了一个 demo，就是把文件 `IO` 也封装为 `coroutine` 的非阻塞 `IO`，实现在 `src/fs_future.rs` 中，这时可以运行本文开头给的 example 了。
```bash
RUST_LOG=info cargo run --example file-server
```
用 `telnet` 测试
```bash
[~] telnet 127.0.0.1 7878
Trying 127.0.0.1...
Connected to localhost.
Escape character is '^]'.
Please enter filename: examples/test.txt
Hello, World!
Connection closed by foreign host.
```
读者有兴趣的话可以看一下 [src/fs_future.rs](https://github.com/Hexilee/async-io-demo/blob/master/src/fs_future.rs) 中的实现，这里就不细说，接下来我们再谈谈现在 `coroutine API` 的不足。
我目前发现的主要问题就是不能在 `Future::poll` 中使用 `try`，导致出现 `Result` 的地方只能 `match`，希望之后会有比较好的解决方案（比如给 `task::Poll<Result<R, E>>` 实现 `Try`）。
第二个问题是 `Waker` 最里面装的是 `UnsafeWaker`的 `NonNull` 指针，当然我能理解 `rust` 团队有性能等其它方面的考虑，但如果用 `mio` 的 `set_readiness` 封装出 `MyWaker` 的话，`clone` 完全不需要 `NonNull`，而且我在实际编码时因为这个出过空指针错误。。希望以后能提供一个更安全的选择。
================================================
FILE: examples/async-echo.rs
================================================
#[macro_use]
extern crate log;
use asyncio::executor::{block_on, spawn, TcpListener};
use failure::Error;
fn main() -> Result<(), Error> {
env_logger::init();
block_on(
async {
let mut listener = TcpListener::bind(&"127.0.0.1:7878".parse()?)?;
info!("Listening on 127.0.0.1:7878");
while let Ok((mut stream, addr)) = listener.accept().await {
info!("connection from {}", addr);
spawn(
async move {
let client_hello = stream.read().await?;
let read_length = client_hello.len();
let write_length =
stream.write(client_hello).await?;
assert_eq!(read_length, write_length);
stream.close();
Ok(())
},
)?;
};
Ok(())
},
)?
}
================================================
FILE: examples/fab.rs
================================================
#![feature(generators, generator_trait)]
use std::ops::{Generator, GeneratorState};
use std::pin::Pin;
fn main() {
let mut gen = fab(5);
loop {
match unsafe { Pin::new_unchecked(&mut gen).resume() } {
GeneratorState::Yielded(value) => println!("yield {}", value),
GeneratorState::Complete(ret) => {
println!("return {}", ret);
break;
}
}
}
}
fn fab(mut n: u64) -> impl Generator<Yield = u64, Return = u64> {
move || {
let mut last = 0u64;
let mut current = 1;
yield last;
while n > 0 {
yield current;
let tmp = last;
last = current;
current = tmp + last;
n -= 1;
}
return last;
}
}
================================================
FILE: examples/file-server.rs
================================================
#[macro_use]
extern crate log;
use asyncio::executor::{block_on, spawn, TcpListener, TcpStream};
use asyncio::fs_future::{read_to_string};
use failure::Error;
fn main() -> Result<(), Error> {
env_logger::init();
block_on(new_server())?
}
const CRLF: &[char] = &['\r', '\n'];
async fn new_server() -> Result<(), Error> {
let mut listener = TcpListener::bind(&"127.0.0.1:7878".parse()?)?;
info!("Listening on 127.0.0.1:7878");
while let Ok((stream, addr)) = listener.accept().await {
info!("connection from {}", addr);
spawn(handle_stream(stream))?;
}
Ok(())
}
async fn handle_stream(mut stream: TcpStream) -> Result<(), Error> {
stream.write_str("Please enter filename: ").await?;
let file_name_vec = stream.read().await?;
let file_name = String::from_utf8(file_name_vec)?.trim_matches(CRLF).to_owned();
let file_contents = read_to_string(file_name).await?;
stream.write_str(&file_contents).await?;
stream.close();
Ok(())
}
================================================
FILE: examples/fs-mio.rs
================================================
use asyncio::fs_mio::fs_async;
use failure::Error;
const TEST_FILE_VALUE: &str = "Hello, World!\n";
fn main() -> Result<(), Error> {
let (fs, fs_handler) = fs_async();
fs.open("./examples/test.txt", |file, fs| {
fs.read_to_string(file, |value, fs| {
assert_eq!(TEST_FILE_VALUE, &value);
fs.println(value)?;
fs.close()
})
})?;
fs_handler.join()?;
Ok(())
}
================================================
FILE: examples/fs.rs
================================================
use asyncio::fs::fs_async;
use failure::Error;
const TEST_FILE_VALUE: &str = "Hello, World!\n";
fn main() -> Result<(), Error> {
let (fs, fs_handler) = fs_async();
fs.open("./examples/test.txt", |file, fs| {
fs.read_to_string(file, |value, fs| {
assert_eq!(TEST_FILE_VALUE, &value);
fs.println(value)?;
fs.close()
})
})?;
fs_handler.join()?;
Ok(())
}
================================================
FILE: examples/poll-timeout.rs
================================================
use mio::*;
use std::time::{Duration};
use failure::Error;
fn main() -> Result<(), Error> {
let poll = Poll::new()?;
let timeout = Duration::from_millis(10);
let mut events = Events::with_capacity(1024);
match poll.poll(&mut events, Some(timeout)) {
Ok(num) => assert_eq!(0, num),
Err(_) => unreachable!()
}
Ok(())
}
================================================
FILE: examples/spurious-events.rs
================================================
use mio::*;
use std::time::Duration;
use crossbeam_channel::unbounded;
use std::io;
use failure::Error;
const TOKEN: Token = Token(0);
fn main() -> Result<(), Error> {
let poll = Poll::new()?;
let (registration, set_readiness) = Registration::new2();
poll.register(®istration, TOKEN, Ready::readable(), PollOpt::edge())?;
let (task_sender, task_receiver) = unbounded();
let worker_handler = std::thread::spawn(move || -> io::Result<()> {
let mut task_counter = 0;
loop {
match task_receiver.recv() {
Ok(_) => {
task_counter += 1;
if task_counter > 50 {
break;
}
set_readiness.set_readiness(Ready::readable())?;
}
Err(err) => {
println!("err: {}", err);
break;
}
}
}
println!("task counter: {}", task_counter);
Ok(())
});
let mut events = Events::with_capacity(1024);
let mut total_event_counter = 0;
let mut token_event_counter = 0;
task_sender.send(())?;
loop {
match poll.poll(&mut events, Some(Duration::from_secs(1))) {
Ok(n) => {
if n == 0 {
// timeout
break;
}
for event in events.iter() {
total_event_counter += 1;
match event.token() {
TOKEN => {
token_event_counter += 1;
task_sender.send(())?;
}
_ => unreachable!()
}
}
}
_ => unreachable!()
}
}
worker_handler.join().unwrap()?;
println!("total_event_counter: {}, token_event_counter: {}", total_event_counter, token_event_counter);
Ok(())
}
================================================
FILE: examples/tcp.rs
================================================
use mio::*;
use mio::net::{TcpListener, TcpStream};
use std::io::{Read, Write, self};
use failure::Error;
use std::time::{Duration, Instant};
const SERVER_ACCEPT: Token = Token(0);
const SERVER: Token = Token(1);
const CLIENT: Token = Token(2);
const SERVER_HELLO: &[u8] = b"PING";
const CLIENT_HELLO: &[u8] = b"PONG";
fn main() -> Result<(), Error> {
let addr = "127.0.0.1:13265".parse()?;
// Setup the server socket
let server = TcpListener::bind(&addr)?;
// Create a poll instance
let poll = Poll::new()?;
// Start listening for incoming connections
poll.register(&server, SERVER_ACCEPT, Ready::readable(),
PollOpt::edge())?;
// Setup the client socket
let mut client = TcpStream::connect(&addr)?;
let mut server_handler = None;
// Register the client
poll.register(&client, CLIENT, Ready::readable() | Ready::writable(),
PollOpt::edge())?;
// Create storage for events
let mut events = Events::with_capacity(1024);
let start = Instant::now();
let timeout = Duration::from_millis(10);
'top: loop {
poll.poll(&mut events, None)?;
for event in events.iter() {
if start.elapsed() >= timeout {
break 'top
}
match event.token() {
SERVER_ACCEPT => {
let (handler, addr) = server.accept()?;
println!("accept from addr: {}", &addr);
poll.register(&handler, SERVER, Ready::readable() | Ready::writable(), PollOpt::edge())?;
server_handler = Some(handler);
}
SERVER => {
if event.readiness().is_writable() {
if let Some(ref mut handler) = &mut server_handler {
match handler.write(SERVER_HELLO) {
Ok(_) => {
println!("server wrote");
}
Err(ref err) if err.kind() == io::ErrorKind::WouldBlock => continue,
err => {
err?;
}
}
}
}
if event.readiness().is_readable() {
let mut hello = [0; 4];
if let Some(ref mut handler) = &mut server_handler {
match handler.read_exact(&mut hello) {
Ok(_) => {
assert_eq!(CLIENT_HELLO, &hello);
println!("server received");
}
Err(ref err) if err.kind() == io::ErrorKind::WouldBlock => continue,
err => {
err?;
}
}
}
}
}
CLIENT => {
if event.readiness().is_writable() {
match client.write(CLIENT_HELLO) {
Ok(_) => {
println!("client wrote");
}
Err(ref err) if err.kind() == io::ErrorKind::WouldBlock => continue,
err => {
err?;
}
}
}
if event.readiness().is_readable() {
let mut hello = [0; 4];
match client.read_exact(&mut hello) {
Ok(_) => {
assert_eq!(SERVER_HELLO, &hello);
println!("client received");
}
Err(ref err) if err.kind() == io::ErrorKind::WouldBlock => continue,
err => {
err?;
}
}
}
}
_ => unreachable!(),
}
}
};
Ok(())
}
================================================
FILE: examples/test.txt
================================================
Hello, World!
================================================
FILE: src/executor.rs
================================================
use failure::Error;
use mio::*;
use slab::Slab;
use std::borrow::Borrow;
use std::cell::RefCell;
use std::future::Future;
use std::io::{self, Read, Write};
use std::mem::transmute;
use std::net::SocketAddr;
use std::pin::Pin;
use std::rc::Rc;
use std::task::{self, Context, RawWaker, RawWakerVTable, Waker};
const MAX_RESOURCE_NUM: usize = 1 << 31;
const MAIN_TASK_TOKEN: Token = Token(MAX_RESOURCE_NUM);
const EVENT_CAP: usize = 1024;
//const POLL_TIME_OUT_MILL: u64 = 100;
const RAW_WAKER_V_TABLE: RawWakerVTable = RawWakerVTable::new(clone_raw, wake, wake, drop_raw);
const fn get_source_token(index: usize) -> Token {
Token(index * 2)
}
const fn get_task_token(index: usize) -> Token {
Token(index * 2 + 1)
}
// panic when token is ord
unsafe fn index_from_source_token(token: Token) -> usize {
if !is_source(token) {
panic!(format!("not a source token: {}", token.0));
}
token.0 / 2
}
// panic when token is not ord
unsafe fn index_from_task_token(token: Token) -> usize {
if !is_task(token) {
panic!(format!("not a task token: {}", token.0));
}
(token.0 - 1) / 2
}
const fn is_source(token: Token) -> bool {
token.0 % 2 == 0
}
const fn is_task(token: Token) -> bool {
token.0 % 2 == 1
}
type PinFuture<T> = Pin<Box<dyn Future<Output = T>>>;
struct Executor {
poll: Poll,
main_awaker: Awaker,
tasks: RefCell<Slab<Task>>,
sources: RefCell<Slab<Source>>,
}
struct Awaker {
awake_readiness: SetReadiness,
awake_registration: Registration,
}
struct Source {
waker: Waker,
evented: Box<dyn Evented>,
}
struct Task {
waker: Awaker,
inner_task: PinFuture<Result<(), Error>>,
}
#[derive(Clone)]
pub struct TcpListener {
inner: Rc<net::TcpListener>,
accept_source_token: Option<Token>,
}
#[derive(Clone)]
pub struct TcpStream {
inner: Rc<RefCell<net::TcpStream>>,
source_token: Option<Token>,
readiness: Ready,
}
pub struct TcpAcceptState<'a> {
listener: &'a mut TcpListener,
}
pub struct StreamReadState<'a> {
stream: &'a mut TcpStream,
}
pub struct StreamWriteState<'a> {
stream: &'a mut TcpStream,
data: Vec<u8>,
}
fn clone_raw(ptr: *const ()) -> RawWaker {
RawWaker::new(ptr, &RAW_WAKER_V_TABLE)
}
fn drop_raw(_ptr: *const ()) {}
fn wake(ptr: *const ()) {
let readiness: SetReadiness = unsafe { transmute(ptr) };
readiness.set_readiness(Ready::readable()).unwrap();
}
impl Awaker {
fn gen_waker(&self) -> Waker {
unsafe {
Waker::from_raw(RawWaker::new(
transmute(self.awake_readiness.clone()),
&RAW_WAKER_V_TABLE,
))
}
}
}
impl Executor {
pub fn new() -> Result<Self, Error> {
let poll = Poll::new()?;
let (awake_registration, awake_readiness) = Registration::new2();
poll.register(
&awake_registration,
MAIN_TASK_TOKEN,
Ready::all(),
PollOpt::edge(),
)?;
Ok(Executor {
poll,
main_awaker: Awaker {
awake_registration,
awake_readiness,
},
tasks: RefCell::new(Slab::new()),
sources: RefCell::new(Slab::new()),
})
}
}
thread_local! {
static EXECUTOR: Executor = Executor::new().expect("initializing executor failed!")
}
pub fn block_on<R, F>(main_task: F) -> Result<R, Error>
where
R: Sized,
F: Future<Output = R>,
{
EXECUTOR.with(move |executor: &Executor| -> Result<R, Error> {
let mut pinned_task = Box::pin(main_task);
let mut events = Events::with_capacity(EVENT_CAP);
match pinned_task
.as_mut()
.poll(&mut Context::from_waker(&executor.main_awaker.gen_waker()))
{
task::Poll::Ready(result) => {
debug!("main task complete");
Ok(result)
}
task::Poll::Pending => {
debug!("main task pending");
loop {
// executor.poll.poll(&mut events, Some(Duration::from_millis(POLL_TIME_OUT_MILL))).expect("polling failed");
executor.poll.poll(&mut events, None)?;
debug!("events empty: {}", events.is_empty());
for event in events.iter() {
debug!("get event: {:?}", event.token());
match event.token() {
MAIN_TASK_TOKEN => {
debug!("receive a main task event");
match pinned_task.as_mut().poll(&mut Context::from_waker(
&executor.main_awaker.gen_waker(),
)) {
task::Poll::Ready(result) => {
return Ok(result);
}
task::Poll::Pending => {
debug!("main task pending again");
continue;
}
}
}
token if is_source(token) => {
debug!("receive a source event: {:?}", token);
let index = unsafe { index_from_source_token(token) };
debug!("source: Index({})", index);
let source = &executor.sources.borrow()[index];
debug!("source addr: {:p}", source);
source.waker.wake_by_ref();
}
token if is_task(token) => {
debug!("receive a task event: {:?}", token);
let index = unsafe { index_from_task_token(token) };
let mut tasks = executor.tasks.borrow_mut();
let task = &mut tasks[index];
match task
.inner_task
.as_mut()
.poll(&mut Context::from_waker(&task.waker.gen_waker()))
{
task::Poll::Ready(result) => {
debug!("task({:?}) complete", token);
executor.poll.deregister(&task.waker.awake_registration)?;
tasks.remove(index);
result?;
}
task::Poll::Pending => {
debug!("task({:?}) pending", token);
continue;
}
}
}
_ => {}
}
}
}
}
}
})
}
pub fn spawn<F: Future<Output = Result<(), Error>> + 'static>(task: F) -> Result<(), Error> {
EXECUTOR.with(move |executor: &Executor| {
let (awake_registration, awake_readiness) = Registration::new2();
let index = executor.tasks.borrow_mut().insert(Task {
inner_task: Box::pin(task),
waker: Awaker {
awake_readiness,
awake_registration,
},
});
let token = get_task_token(index);
let mut tasks = executor.tasks.borrow_mut();
let task = &mut tasks[index];
debug!("task({:?}) spawn", token);
match task
.inner_task
.as_mut()
.poll(&mut Context::from_waker(&task.waker.gen_waker()))
{
task::Poll::Ready(_) => {
debug!("task({:?}) complete when spawn", token);
tasks.remove(index);
}
task::Poll::Pending => {
executor.poll.register(
&task.waker.awake_registration,
token,
Ready::all(),
PollOpt::edge(),
)?;
debug!("task({:?}) pending", token);
}
};
Ok(())
})
}
pub fn register_source<T: Evented + 'static>(
evented: T,
waker: Waker,
interest: Ready,
) -> Result<Token, Error> {
EXECUTOR.with(move |executor: &Executor| {
let index = executor.sources.borrow_mut().insert(Source {
waker,
evented: Box::new(evented),
});
debug!("new sources: Index({})", index);
let token = get_source_token(index);
let source = &executor.sources.borrow()[index];
executor
.poll
.register(&source.evented, token, interest, PollOpt::edge())?;
debug!("register source: {:?}", token);
Ok(token)
})
}
// panic when token is ord
unsafe fn reregister_source(token: Token, interest: Ready) -> Result<(), Error> {
EXECUTOR.with(move |executor: &Executor| {
let index = index_from_source_token(token);
debug!("reregister source: Index({})", index);
let source = &executor.sources.borrow()[index];
executor
.poll
.reregister(&source.evented, token, interest, PollOpt::edge())?;
debug!("source addr: {:p}", source);
Ok(())
})
}
// panic when token is ord
unsafe fn drop_source(token: Token) -> Result<(), Error> {
EXECUTOR.with(move |executor: &Executor| {
let index = index_from_source_token(token);
let mut sources = executor.sources.borrow_mut();
let source = &sources[index];
executor.poll.deregister(&source.evented)?;
sources.remove(index);
Ok(())
})
}
impl Evented for TcpListener {
fn register(
&self,
poll: &Poll,
token: Token,
interest: Ready,
opts: PollOpt,
) -> io::Result<()> {
self.inner.register(poll, token, interest, opts)
}
fn reregister(
&self,
poll: &Poll,
token: Token,
interest: Ready,
opts: PollOpt,
) -> io::Result<()> {
self.inner.reregister(poll, token, interest, opts)
}
fn deregister(&self, poll: &Poll) -> io::Result<()> {
self.inner.deregister(poll)
}
}
impl Evented for TcpStream {
fn register(
&self,
poll: &Poll,
token: Token,
interest: Ready,
opts: PollOpt,
) -> io::Result<()> {
let ref_cell: &RefCell<net::TcpStream> = self.inner.borrow();
let stream = ref_cell.borrow();
stream.register(poll, token, interest, opts)
}
fn reregister(
&self,
poll: &Poll,
token: Token,
interest: Ready,
opts: PollOpt,
) -> io::Result<()> {
let ref_cell: &RefCell<net::TcpStream> = self.inner.borrow();
let stream = ref_cell.borrow();
stream.reregister(poll, token, interest, opts)
}
fn deregister(&self, poll: &Poll) -> io::Result<()> {
let ref_cell: &RefCell<net::TcpStream> = self.inner.borrow();
let stream = ref_cell.borrow();
stream.deregister(poll)
}
}
impl TcpListener {
fn new(listener: mio::net::TcpListener) -> TcpListener {
TcpListener {
inner: Rc::new(listener),
accept_source_token: None,
}
}
fn poll_accept(&mut self, waker: Waker) -> task::Poll<Result<(TcpStream, SocketAddr), Error>> {
if self.accept_source_token.is_none() {
self.accept_source_token = Some(
match register_source(self.clone(), waker, Ready::readable()) {
Ok(token) => token,
Err(err) => return task::Poll::Ready(Err(err)),
},
)
};
match self.inner.accept() {
Ok((stream, addr)) => {
debug!("accept stream from: {}", addr);
task::Poll::Ready(Ok((TcpStream::new(stream), addr)))
}
Err(ref err) if err.kind() == io::ErrorKind::WouldBlock => {
debug!("accept would block");
task::Poll::Pending
}
Err(err) => task::Poll::Ready(Err(err.into())),
}
}
pub fn bind(addr: &SocketAddr) -> io::Result<TcpListener> {
let l = mio::net::TcpListener::bind(addr)?;
Ok(TcpListener::new(l))
}
pub fn local_addr(&self) -> io::Result<SocketAddr> {
self.inner.local_addr()
}
pub fn ttl(&self) -> io::Result<u32> {
self.inner.ttl()
}
pub fn set_ttl(&self, ttl: u32) -> io::Result<()> {
self.inner.set_ttl(ttl)
}
pub fn accept(&mut self) -> TcpAcceptState {
TcpAcceptState { listener: self }
}
}
impl TcpStream {
pub fn new(connected: mio::net::TcpStream) -> TcpStream {
TcpStream {
inner: Rc::new(RefCell::new(connected)),
source_token: None,
readiness: Ready::empty(),
}
}
pub fn read(&mut self) -> StreamReadState {
StreamReadState { stream: self }
}
pub fn write(&mut self, data: Vec<u8>) -> StreamWriteState {
StreamWriteState { stream: self, data }
}
pub fn write_str(&mut self, data: &str) -> StreamWriteState {
StreamWriteState {
stream: self,
data: data.as_bytes().to_vec(),
}
}
pub fn close(self) {
if let Some(token) = self.source_token {
unsafe { drop_source(token).unwrap() };
}
}
}
impl TcpStream {
fn read_poll(&mut self, waker: Waker) -> task::Poll<Result<Vec<u8>, Error>> {
match self.source_token {
None => {
self.readiness = Ready::readable();
self.source_token =
Some(match register_source(self.clone(), waker, self.readiness) {
Ok(token) => token,
Err(err) => return task::Poll::Ready(Err(err)),
})
}
Some(token) => {
if !self.readiness.is_readable() {
self.readiness |= Ready::readable();
if let Err(err) = unsafe { reregister_source(token, self.readiness) } {
return task::Poll::Ready(Err(err));
}
}
}
}
let mut ret = [0u8; 1024];
let ref_cell: &RefCell<net::TcpStream> = self.inner.borrow();
match ref_cell.borrow_mut().read(&mut ret) {
Ok(n) => {
debug!("stream read {} bytes", n);
task::Poll::Ready(Ok(ret[..n].to_vec()))
}
Err(ref err) if err.kind() == io::ErrorKind::WouldBlock => {
debug!("stream read pending");
task::Poll::Pending
}
Err(err) => task::Poll::Ready(Err(err.into())),
}
}
fn write_poll(&mut self, data: &[u8], waker: Waker) -> task::Poll<Result<usize, Error>> {
match self.source_token {
None => {
self.readiness = Ready::writable();
self.source_token =
Some(match register_source(self.clone(), waker, self.readiness) {
Ok(token) => token,
Err(err) => return task::Poll::Ready(Err(err)),
})
}
Some(token) => {
if !self.readiness.is_writable() {
self.readiness |= Ready::writable();
if let Err(err) = unsafe { reregister_source(token, self.readiness) } {
return task::Poll::Ready(Err(err));
};
}
}
}
let ref_cell: &RefCell<net::TcpStream> = self.inner.borrow();
match ref_cell.borrow_mut().write(data) {
Ok(n) => task::Poll::Ready(Ok(n)),
Err(ref err) if err.kind() == io::ErrorKind::WouldBlock => task::Poll::Pending,
Err(err) => task::Poll::Ready(Err(err.into())),
}
}
}
impl<'a> Future for TcpAcceptState<'a> {
type Output = Result<(TcpStream, SocketAddr), Error>;
fn poll(
mut self: Pin<&mut Self>,
cx: &mut Context<'_>,
) -> task::Poll<<Self as Future>::Output> {
self.listener.poll_accept(cx.waker().clone())
}
}
impl<'a> Future for StreamReadState<'a> {
type Output = Result<Vec<u8>, Error>;
fn poll(
mut self: Pin<&mut Self>,
cx: &mut Context<'_>,
) -> task::Poll<<Self as Future>::Output> {
self.stream.read_poll(cx.waker().clone())
}
}
impl<'a> Future for StreamWriteState<'a> {
type Output = Result<usize, Error>;
fn poll(
mut self: Pin<&mut Self>,
cx: &mut Context<'_>,
) -> task::Poll<<Self as Future>::Output> {
let data = self.data.clone();
self.stream.write_poll(data.as_slice(), cx.waker().clone())
}
}
================================================
FILE: src/fs.rs
================================================
use crossbeam_channel::{unbounded, Sender};
use failure::Error;
use std::fs::File;
use std::io::Read;
use std::thread;
#[derive(Clone)]
pub struct Fs {
task_sender: Sender<Task>,
}
pub struct FsHandler {
io_worker: thread::JoinHandle<Result<(), Error>>,
executor: thread::JoinHandle<Result<(), Error>>,
}
pub fn fs_async() -> (Fs, FsHandler) {
let (task_sender, task_receiver) = unbounded();
let (result_sender, result_receiver) = unbounded();
let io_worker = std::thread::spawn(move || {
while let Ok(task) = task_receiver.recv() {
match task {
Task::Println(ref string) => println!("{}", string),
Task::Open(path, callback, fs) => {
result_sender.send(TaskResult::Open(File::open(path)?, callback, fs))?
}
Task::ReadToString(mut file, callback, fs) => {
let mut value = String::new();
file.read_to_string(&mut value)?;
result_sender.send(TaskResult::ReadToString(value, callback, fs))?
}
Task::Exit => {
result_sender.send(TaskResult::Exit)?;
break;
}
}
}
Ok(())
});
let executor = std::thread::spawn(move || {
loop {
let result = result_receiver.recv()?;
match result {
TaskResult::ReadToString(value, callback, fs) => callback(value, fs)?,
TaskResult::Open(file, callback, fs) => callback(file, fs)?,
TaskResult::Exit => break,
};
}
Ok(())
});
(
Fs { task_sender },
FsHandler {
io_worker,
executor,
},
)
}
impl Fs {
pub fn println(&self, string: String) -> Result<(), Error> {
Ok(self.task_sender.send(Task::Println(string))?)
}
pub fn open<F>(&self, path: &str, callback: F) -> Result<(), Error>
where
F: FnOnce(File, Fs) -> Result<(), Error> + Sync + Send + 'static,
{
Ok(self.task_sender.send(Task::Open(
path.to_string(),
Box::new(callback),
self.clone(),
))?)
}
pub fn read_to_string<F>(&self, file: File, callback: F) -> Result<(), Error>
where
F: FnOnce(String, Fs) -> Result<(), Error> + Sync + Send + 'static,
{
Ok(self
.task_sender
.send(Task::ReadToString(file, Box::new(callback), self.clone()))?)
}
pub fn close(&self) -> Result<(), Error> {
Ok(self.task_sender.send(Task::Exit)?)
}
}
impl FsHandler {
pub fn join(self) -> Result<(), Error> {
self.io_worker.join().unwrap()?;
self.executor.join().unwrap()
}
}
type FileCallback = Box<dyn FnOnce(File, Fs) -> Result<(), Error> + Sync + Send>;
type StringCallback = Box<dyn FnOnce(String, Fs) -> Result<(), Error> + Sync + Send>;
pub enum Task {
Exit,
Println(String),
Open(String, FileCallback, Fs),
ReadToString(File, StringCallback, Fs),
}
pub enum TaskResult {
Exit,
Open(File, FileCallback, Fs),
ReadToString(String, StringCallback, Fs),
}
================================================
FILE: src/fs_future.rs
================================================
use crate::executor::register_source;
use crossbeam_channel::{bounded, unbounded, Receiver, Sender, TryRecvError};
use failure::Error;
use log::debug;
use mio::{Ready, Registration, SetReadiness, Token};
use std::fs;
use std::future::Future;
use std::io;
use std::pin::Pin;
use std::task::{self, Context};
use std::thread;
struct BlockTaskWorker {
task_sender: Sender<Box<dyn BlockTask>>,
}
impl BlockTaskWorker {
fn new() -> Self {
let (task_sender, task_receiver) = unbounded();
let worker = BlockTaskWorker { task_sender };
thread::spawn(move || loop {
match task_receiver.recv() {
Ok(mut task) => task.exec(),
Err(err) => panic!("{}", err),
}
});
worker
}
}
thread_local! {
static TASK_WORKER: BlockTaskWorker = BlockTaskWorker::new();
}
fn send_block_task<T: BlockTask + 'static>(task: T) {
let boxed_task = Box::new(task);
TASK_WORKER.with(move |task_worker| {
task_worker.task_sender.send(boxed_task).unwrap();
})
}
trait BlockTask: Send {
fn exec(&mut self);
}
struct ReadFileTask {
file_name: String,
string_sender: Sender<io::Result<String>>,
set_readiness: SetReadiness,
}
struct ReadFileState {
source_token: Option<Token>,
registration: Option<Registration>,
string_receiver: Receiver<io::Result<String>>,
}
impl BlockTask for ReadFileTask {
fn exec(&mut self) {
debug!("ready to open file: {}", &self.file_name);
self.string_sender
.send(fs::read_to_string(&self.file_name))
.unwrap();
debug!("sent file named {}", &self.file_name);
self.set_readiness.set_readiness(Ready::readable()).unwrap();
}
}
pub fn read_to_string(file_name: String) -> impl Future<Output = Result<String, Error>> {
let (registration, set_readiness) = Registration::new2();
let (string_sender, string_receiver) = bounded(1);
send_block_task(ReadFileTask {
file_name,
string_sender,
set_readiness,
});
ReadFileState {
source_token: None,
registration: Some(registration),
string_receiver,
}
}
impl Future for ReadFileState {
type Output = Result<String, Error>;
fn poll(
mut self: Pin<&mut Self>,
cx: &mut Context<'_>,
) -> task::Poll<<Self as Future>::Output> {
if self.source_token.is_none() {
self.source_token = Some(
match register_source(
self.registration.take().unwrap(),
cx.waker().clone(),
Ready::readable(),
) {
Ok(token) => token,
Err(err) => return task::Poll::Ready(Err(err)),
},
)
};
match self.string_receiver.try_recv() {
Ok(read_result) => match read_result {
Ok(value) => {
debug!("read value {}", &value);
task::Poll::Ready(Ok(value))
}
Err(err) => {
debug!("read err {}", &err);
task::Poll::Ready(Err(err.into()))
}
},
Err(TryRecvError::Empty) => {
debug!("read file pending");
task::Poll::Pending
}
Err(err) => {
debug!("read file disconnecting");
task::Poll::Ready(Err(err.into()))
}
}
}
}
================================================
FILE: src/fs_mio.rs
================================================
use crossbeam_channel::{unbounded, Sender, TryRecvError};
use failure::Error;
use mio::*;
use std::fs::File;
use std::io::Read;
use std::thread;
use std::time::Duration;
#[derive(Clone)]
pub struct Fs {
task_sender: Sender<Task>,
}
pub struct FsHandler {
io_worker: thread::JoinHandle<Result<(), Error>>,
executor: thread::JoinHandle<Result<(), Error>>,
}
const FS_TOKEN: Token = Token(0);
pub fn fs_async() -> (Fs, FsHandler) {
let (task_sender, task_receiver) = unbounded();
let (result_sender, result_receiver) = unbounded();
let poll = Poll::new().unwrap();
let (registration, set_readiness) = Registration::new2();
poll.register(
®istration,
FS_TOKEN,
Ready::readable(),
PollOpt::oneshot(),
)
.unwrap();
let io_worker = std::thread::spawn(move || {
while let Ok(task) = task_receiver.recv() {
match task {
Task::Println(ref string) => println!("{}", string),
Task::Open(path, callback, fs) => {
result_sender.send(TaskResult::Open(File::open(path)?, callback, fs))?;
set_readiness.set_readiness(Ready::readable())?;
}
Task::ReadToString(mut file, callback, fs) => {
let mut value = String::new();
file.read_to_string(&mut value)?;
result_sender.send(TaskResult::ReadToString(value, callback, fs))?;
set_readiness.set_readiness(Ready::readable())?;
}
Task::Exit => {
result_sender.send(TaskResult::Exit)?;
set_readiness.set_readiness(Ready::readable())?;
break;
}
}
}
Ok(())
});
let executor = thread::spawn(move || {
let mut events = Events::with_capacity(1024);
'outer: loop {
poll.poll(&mut events, Some(Duration::from_secs(1)))?;
for event in events.iter() {
match event.token() {
FS_TOKEN => {
loop {
match result_receiver.try_recv() {
Ok(result) => match result {
TaskResult::ReadToString(value, callback, fs) => {
callback(value, fs)?
}
TaskResult::Open(file, callback, fs) => callback(file, fs)?,
TaskResult::Exit => break 'outer,
},
Err(e) => match e {
TryRecvError::Empty => break,
TryRecvError::Disconnected => return Err(e.into()),
},
}
}
poll.reregister(
®istration,
FS_TOKEN,
Ready::readable(),
PollOpt::oneshot(),
)?;
}
_ => unreachable!(),
}
}
}
Ok(())
});
(
Fs { task_sender },
FsHandler {
io_worker,
executor,
},
)
}
impl Fs {
pub fn println(&self, string: String) -> Result<(), Error> {
Ok(self.task_sender.send(Task::Println(string))?)
}
pub fn open<F>(&self, path: &str, callback: F) -> Result<(), Error>
where
F: FnOnce(File, Fs) -> Result<(), Error> + Sync + Send + 'static,
{
Ok(self.task_sender.send(Task::Open(
path.to_string(),
Box::new(callback),
self.clone(),
))?)
}
pub fn read_to_string<F>(&self, file: File, callback: F) -> Result<(), Error>
where
F: FnOnce(String, Fs) -> Result<(), Error> + Sync + Send + 'static,
{
Ok(self
.task_sender
.send(Task::ReadToString(file, Box::new(callback), self.clone()))?)
}
pub fn close(&self) -> Result<(), Error> {
Ok(self.task_sender.send(Task::Exit)?)
}
}
impl FsHandler {
pub fn join(self) -> Result<(), Error> {
self.io_worker.join().unwrap()?;
self.executor.join().unwrap()
}
}
type FileCallback = Box<dyn FnOnce(File, Fs) -> Result<(), Error> + Sync + Send>;
type StringCallback = Box<dyn FnOnce(String, Fs) -> Result<(), Error> + Sync + Send>;
pub enum Task {
Exit,
Println(String),
Open(String, FileCallback, Fs),
ReadToString(File, StringCallback, Fs),
}
pub enum TaskResult {
Exit,
Open(File, FileCallback, Fs),
ReadToString(String, StringCallback, Fs),
}
================================================
FILE: src/lib.rs
================================================
#[macro_use]
extern crate log;
pub mod executor;
pub mod fs;
pub mod fs_mio;
pub mod fs_future;
gitextract__d09rzo9/
├── .gitignore
├── Cargo.toml
├── LICENSE
├── README.md
├── README.zh.md
├── examples/
│ ├── async-echo.rs
│ ├── fab.rs
│ ├── file-server.rs
│ ├── fs-mio.rs
│ ├── fs.rs
│ ├── poll-timeout.rs
│ ├── spurious-events.rs
│ ├── tcp.rs
│ └── test.txt
└── src/
├── executor.rs
├── fs.rs
├── fs_future.rs
├── fs_mio.rs
└── lib.rs
SYMBOL INDEX (112 symbols across 12 files)
FILE: examples/async-echo.rs
function main (line 7) | fn main() -> Result<(), Error> {
FILE: examples/fab.rs
function main (line 6) | fn main() {
function fab (line 19) | fn fab(mut n: u64) -> impl Generator<Yield = u64, Return = u64> {
FILE: examples/file-server.rs
function main (line 8) | fn main() -> Result<(), Error> {
constant CRLF (line 14) | const CRLF: &[char] = &['\r', '\n'];
function new_server (line 16) | async fn new_server() -> Result<(), Error> {
function handle_stream (line 26) | async fn handle_stream(mut stream: TcpStream) -> Result<(), Error> {
FILE: examples/fs-mio.rs
constant TEST_FILE_VALUE (line 4) | const TEST_FILE_VALUE: &str = "Hello, World!\n";
function main (line 6) | fn main() -> Result<(), Error> {
FILE: examples/fs.rs
constant TEST_FILE_VALUE (line 4) | const TEST_FILE_VALUE: &str = "Hello, World!\n";
function main (line 6) | fn main() -> Result<(), Error> {
FILE: examples/poll-timeout.rs
function main (line 5) | fn main() -> Result<(), Error> {
FILE: examples/spurious-events.rs
constant TOKEN (line 7) | const TOKEN: Token = Token(0);
function main (line 9) | fn main() -> Result<(), Error> {
FILE: examples/tcp.rs
constant SERVER_ACCEPT (line 7) | const SERVER_ACCEPT: Token = Token(0);
constant SERVER (line 8) | const SERVER: Token = Token(1);
constant CLIENT (line 9) | const CLIENT: Token = Token(2);
constant SERVER_HELLO (line 10) | const SERVER_HELLO: &[u8] = b"PING";
constant CLIENT_HELLO (line 11) | const CLIENT_HELLO: &[u8] = b"PONG";
function main (line 13) | fn main() -> Result<(), Error> {
FILE: src/executor.rs
constant MAX_RESOURCE_NUM (line 14) | const MAX_RESOURCE_NUM: usize = 1 << 31;
constant MAIN_TASK_TOKEN (line 15) | const MAIN_TASK_TOKEN: Token = Token(MAX_RESOURCE_NUM);
constant EVENT_CAP (line 16) | const EVENT_CAP: usize = 1024;
constant RAW_WAKER_V_TABLE (line 18) | const RAW_WAKER_V_TABLE: RawWakerVTable = RawWakerVTable::new(clone_raw,...
function get_source_token (line 20) | const fn get_source_token(index: usize) -> Token {
function get_task_token (line 24) | const fn get_task_token(index: usize) -> Token {
function index_from_source_token (line 29) | unsafe fn index_from_source_token(token: Token) -> usize {
function index_from_task_token (line 37) | unsafe fn index_from_task_token(token: Token) -> usize {
function is_source (line 44) | const fn is_source(token: Token) -> bool {
function is_task (line 48) | const fn is_task(token: Token) -> bool {
type PinFuture (line 52) | type PinFuture<T> = Pin<Box<dyn Future<Output = T>>>;
type Executor (line 54) | struct Executor {
method new (line 125) | pub fn new() -> Result<Self, Error> {
type Awaker (line 61) | struct Awaker {
method gen_waker (line 114) | fn gen_waker(&self) -> Waker {
type Source (line 66) | struct Source {
type Task (line 71) | struct Task {
type TcpListener (line 77) | pub struct TcpListener {
method new (line 372) | fn new(listener: mio::net::TcpListener) -> TcpListener {
method poll_accept (line 379) | fn poll_accept(&mut self, waker: Waker) -> task::Poll<Result<(TcpStrea...
method bind (line 401) | pub fn bind(addr: &SocketAddr) -> io::Result<TcpListener> {
method local_addr (line 406) | pub fn local_addr(&self) -> io::Result<SocketAddr> {
method ttl (line 409) | pub fn ttl(&self) -> io::Result<u32> {
method set_ttl (line 412) | pub fn set_ttl(&self, ttl: u32) -> io::Result<()> {
method accept (line 416) | pub fn accept(&mut self) -> TcpAcceptState {
type TcpStream (line 83) | pub struct TcpStream {
method new (line 422) | pub fn new(connected: mio::net::TcpStream) -> TcpStream {
method read (line 430) | pub fn read(&mut self) -> StreamReadState {
method write (line 434) | pub fn write(&mut self, data: Vec<u8>) -> StreamWriteState {
method write_str (line 438) | pub fn write_str(&mut self, data: &str) -> StreamWriteState {
method close (line 445) | pub fn close(self) {
method read_poll (line 453) | fn read_poll(&mut self, waker: Waker) -> task::Poll<Result<Vec<u8>, Er...
method write_poll (line 488) | fn write_poll(&mut self, data: &[u8], waker: Waker) -> task::Poll<Resu...
type TcpAcceptState (line 89) | pub struct TcpAcceptState<'a> {
type StreamReadState (line 93) | pub struct StreamReadState<'a> {
type StreamWriteState (line 97) | pub struct StreamWriteState<'a> {
function clone_raw (line 102) | fn clone_raw(ptr: *const ()) -> RawWaker {
function drop_raw (line 106) | fn drop_raw(_ptr: *const ()) {}
function wake (line 108) | fn wake(ptr: *const ()) {
function block_on (line 150) | pub fn block_on<R, F>(main_task: F) -> Result<R, Error>
function spawn (line 229) | pub fn spawn<F: Future<Output = Result<(), Error>> + 'static>(task: F) -...
function register_source (line 266) | pub fn register_source<T: Evented + 'static>(
function reregister_source (line 288) | unsafe fn reregister_source(token: Token, interest: Ready) -> Result<(),...
function drop_source (line 302) | unsafe fn drop_source(token: Token) -> Result<(), Error> {
method register (line 314) | fn register(
method reregister (line 324) | fn reregister(
method deregister (line 334) | fn deregister(&self, poll: &Poll) -> io::Result<()> {
method register (line 340) | fn register(
method reregister (line 352) | fn reregister(
method deregister (line 364) | fn deregister(&self, poll: &Poll) -> io::Result<()> {
type Output (line 518) | type Output = Result<(TcpStream, SocketAddr), Error>;
method poll (line 519) | fn poll(
type Output (line 528) | type Output = Result<Vec<u8>, Error>;
method poll (line 529) | fn poll(
type Output (line 538) | type Output = Result<usize, Error>;
method poll (line 539) | fn poll(
FILE: src/fs.rs
type Fs (line 8) | pub struct Fs {
method println (line 62) | pub fn println(&self, string: String) -> Result<(), Error> {
method open (line 66) | pub fn open<F>(&self, path: &str, callback: F) -> Result<(), Error>
method read_to_string (line 77) | pub fn read_to_string<F>(&self, file: File, callback: F) -> Result<(),...
method close (line 86) | pub fn close(&self) -> Result<(), Error> {
type FsHandler (line 12) | pub struct FsHandler {
method join (line 92) | pub fn join(self) -> Result<(), Error> {
function fs_async (line 17) | pub fn fs_async() -> (Fs, FsHandler) {
type FileCallback (line 98) | type FileCallback = Box<dyn FnOnce(File, Fs) -> Result<(), Error> + Sync...
type StringCallback (line 99) | type StringCallback = Box<dyn FnOnce(String, Fs) -> Result<(), Error> + ...
type Task (line 101) | pub enum Task {
type TaskResult (line 108) | pub enum TaskResult {
FILE: src/fs_future.rs
type BlockTaskWorker (line 13) | struct BlockTaskWorker {
method new (line 18) | fn new() -> Self {
function send_block_task (line 35) | fn send_block_task<T: BlockTask + 'static>(task: T) {
type BlockTask (line 42) | trait BlockTask: Send {
method exec (line 43) | fn exec(&mut self);
method exec (line 59) | fn exec(&mut self) {
type ReadFileTask (line 46) | struct ReadFileTask {
type ReadFileState (line 52) | struct ReadFileState {
function read_to_string (line 69) | pub fn read_to_string(file_name: String) -> impl Future<Output = Result<...
type Output (line 85) | type Output = Result<String, Error>;
method poll (line 86) | fn poll(
FILE: src/fs_mio.rs
type Fs (line 10) | pub struct Fs {
method println (line 102) | pub fn println(&self, string: String) -> Result<(), Error> {
method open (line 106) | pub fn open<F>(&self, path: &str, callback: F) -> Result<(), Error>
method read_to_string (line 117) | pub fn read_to_string<F>(&self, file: File, callback: F) -> Result<(),...
method close (line 126) | pub fn close(&self) -> Result<(), Error> {
type FsHandler (line 14) | pub struct FsHandler {
method join (line 132) | pub fn join(self) -> Result<(), Error> {
constant FS_TOKEN (line 19) | const FS_TOKEN: Token = Token(0);
function fs_async (line 21) | pub fn fs_async() -> (Fs, FsHandler) {
type FileCallback (line 138) | type FileCallback = Box<dyn FnOnce(File, Fs) -> Result<(), Error> + Sync...
type StringCallback (line 139) | type StringCallback = Box<dyn FnOnce(String, Fs) -> Result<(), Error> + ...
type Task (line 141) | pub enum Task {
type TaskResult (line 148) | pub enum TaskResult {
Condensed preview — 19 files, each showing path, character count, and a content snippet. Download the .json file or copy for the full structured content (143K chars).
[
{
"path": ".gitignore",
"chars": 30,
"preview": "/target\n**/*.rs.bk\nCargo.lock\n"
},
{
"path": "Cargo.toml",
"chars": 229,
"preview": "[package]\nname = \"asyncio\"\nversion = \"0.1.0\"\nauthors = [\"Hexilee <hexileee@gmail.com>\"]\nedition = \"2018\"\n\n[dependencies]"
},
{
"path": "LICENSE",
"chars": 1063,
"preview": "MIT License\n\nCopyright (c) 2018 ZJU QSC\n\nPermission is hereby granted, free of charge, to any person obtaining a copy\nof"
},
{
"path": "README.md",
"chars": 47588,
"preview": "Table of Contents\n=================\n\n* [Introduction](#introduction)\n* [mio: The Footstone of Asynchronous IO](#mio-the-"
},
{
"path": "README.zh.md",
"chars": 40761,
"preview": "Table of Contents\n=================\n\n* [引言](#引言)\n* [异步 IO 的基石 - mio](#异步-io-的基石---mio)\n * [异步网络 IO](#异步网络-io)\n * ["
},
{
"path": "examples/async-echo.rs",
"chars": 968,
"preview": "#[macro_use]\nextern crate log;\n\nuse asyncio::executor::{block_on, spawn, TcpListener};\nuse failure::Error;\n\nfn main() ->"
},
{
"path": "examples/fab.rs",
"chars": 797,
"preview": "#![feature(generators, generator_trait)]\n\nuse std::ops::{Generator, GeneratorState};\nuse std::pin::Pin;\n\nfn main() {\n "
},
{
"path": "examples/file-server.rs",
"chars": 1001,
"preview": "#[macro_use]\nextern crate log;\n\nuse asyncio::executor::{block_on, spawn, TcpListener, TcpStream};\nuse asyncio::fs_future"
},
{
"path": "examples/fs-mio.rs",
"chars": 429,
"preview": "use asyncio::fs_mio::fs_async;\nuse failure::Error;\n\nconst TEST_FILE_VALUE: &str = \"Hello, World!\\n\";\n\nfn main() -> Resul"
},
{
"path": "examples/fs.rs",
"chars": 425,
"preview": "use asyncio::fs::fs_async;\nuse failure::Error;\n\nconst TEST_FILE_VALUE: &str = \"Hello, World!\\n\";\n\nfn main() -> Result<()"
},
{
"path": "examples/poll-timeout.rs",
"chars": 357,
"preview": "use mio::*;\nuse std::time::{Duration};\nuse failure::Error;\n\nfn main() -> Result<(), Error> {\n let poll = Poll::new()?"
},
{
"path": "examples/spurious-events.rs",
"chars": 1987,
"preview": "use mio::*;\nuse std::time::Duration;\nuse crossbeam_channel::unbounded;\nuse std::io;\nuse failure::Error;\n\nconst TOKEN: To"
},
{
"path": "examples/tcp.rs",
"chars": 4291,
"preview": "use mio::*;\nuse mio::net::{TcpListener, TcpStream};\nuse std::io::{Read, Write, self};\nuse failure::Error;\nuse std::time:"
},
{
"path": "examples/test.txt",
"chars": 14,
"preview": "Hello, World!\n"
},
{
"path": "src/executor.rs",
"chars": 17238,
"preview": "use failure::Error;\nuse mio::*;\nuse slab::Slab;\nuse std::borrow::Borrow;\nuse std::cell::RefCell;\nuse std::future::Future"
},
{
"path": "src/fs.rs",
"chars": 3218,
"preview": "use crossbeam_channel::{unbounded, Sender};\nuse failure::Error;\nuse std::fs::File;\nuse std::io::Read;\nuse std::thread;\n\n"
},
{
"path": "src/fs_future.rs",
"chars": 3522,
"preview": "use crate::executor::register_source;\nuse crossbeam_channel::{bounded, unbounded, Receiver, Sender, TryRecvError};\nuse f"
},
{
"path": "src/fs_mio.rs",
"chars": 4866,
"preview": "use crossbeam_channel::{unbounded, Sender, TryRecvError};\nuse failure::Error;\nuse mio::*;\nuse std::fs::File;\nuse std::io"
},
{
"path": "src/lib.rs",
"chars": 99,
"preview": "#[macro_use]\nextern crate log;\n\npub mod executor;\n\npub mod fs;\n\npub mod fs_mio;\n\npub mod fs_future;"
}
]
About this extraction
This page contains the full source code of the Hexilee/async-io-demo GitHub repository, extracted and formatted as plain text for AI agents and large language models (LLMs). The extraction includes 19 files (125.9 KB), approximately 32.6k tokens, and a symbol index with 112 extracted functions, classes, methods, constants, and types. Use this with OpenClaw, Claude, ChatGPT, Cursor, Windsurf, or any other AI tool that accepts text input. You can copy the full output to your clipboard or download it as a .txt file.
Extracted by GitExtract — free GitHub repo to text converter for AI. Built by Nikandr Surkov.