Showing preview only (992K chars total). Download the full file or copy to clipboard to get everything.
Repository: pretzelhammer/rust-blog
Branch: master
Commit: 3d33acbb0dc1
Files: 26
Total size: 963.4 KB
Directory structure:
gitextract_5feznwjv/
├── .gitignore
├── assets/
│ └── main.rs
├── license-apache
├── license-mit
├── posts/
│ ├── chat-server.md
│ ├── common-rust-lifetime-misconceptions.md
│ ├── learning-rust-in-2020.md
│ ├── learning-rust-in-2024.md
│ ├── restful-api-in-sync-and-async-rust.md
│ ├── rust-in-non-rust-servers.md
│ ├── sizedness-in-rust.md
│ ├── too-many-brainfuck-compilers.md
│ ├── tour-of-rusts-standard-library-traits.md
│ ├── translations/
│ │ ├── jp/
│ │ │ └── common-rust-lifetime-misconceptions.md
│ │ ├── rus/
│ │ │ └── common-rust-lifetime-misconceptions.md
│ │ ├── tr/
│ │ │ └── why-blog.md
│ │ └── zh-hans/
│ │ ├── chat-server.md
│ │ ├── common-rust-lifetime-misconceptions.md
│ │ ├── learning-rust-in-2020.md
│ │ ├── learning-rust-in-2024.md
│ │ ├── rust-in-non-rust-servers.md
│ │ ├── sizedness-in-rust.md
│ │ ├── tour-of-rusts-standard-library-traits.md
│ │ └── why-blog.md
│ └── why-blog.md
└── readme.md
================================================
FILE CONTENTS
================================================
================================================
FILE: .gitignore
================================================
.DS_Store
wip
================================================
FILE: assets/main.rs
================================================
fn main() {
println!("hopefully github detects this file and categorizes my repo as a Rust repo");
}
================================================
FILE: license-apache
================================================
Apache License
Version 2.0, January 2004
http://www.apache.org/licenses/
TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION
1. Definitions.
"License" shall mean the terms and conditions for use, reproduction,
and distribution as defined by Sections 1 through 9 of this document.
"Licensor" shall mean the copyright owner or entity authorized by
the copyright owner that is granting the License.
"Legal Entity" shall mean the union of the acting entity and all
other entities that control, are controlled by, or are under common
control with that entity. For the purposes of this definition,
"control" means (i) the power, direct or indirect, to cause the
direction or management of such entity, whether by contract or
otherwise, or (ii) ownership of fifty percent (50%) or more of the
outstanding shares, or (iii) beneficial ownership of such entity.
"You" (or "Your") shall mean an individual or Legal Entity
exercising permissions granted by this License.
"Source" form shall mean the preferred form for making modifications,
including but not limited to software source code, documentation
source, and configuration files.
"Object" form shall mean any form resulting from mechanical
transformation or translation of a Source form, including but
not limited to compiled object code, generated documentation,
and conversions to other media types.
"Work" shall mean the work of authorship, whether in Source or
Object form, made available under the License, as indicated by a
copyright notice that is included in or attached to the work
(an example is provided in the Appendix below).
"Derivative Works" shall mean any work, whether in Source or Object
form, that is based on (or derived from) the Work and for which the
editorial revisions, annotations, elaborations, or other modifications
represent, as a whole, an original work of authorship. For the purposes
of this License, Derivative Works shall not include works that remain
separable from, or merely link (or bind by name) to the interfaces of,
the Work and Derivative Works thereof.
"Contribution" shall mean any work of authorship, including
the original version of the Work and any modifications or additions
to that Work or Derivative Works thereof, that is intentionally
submitted to Licensor for inclusion in the Work by the copyright owner
or by an individual or Legal Entity authorized to submit on behalf of
the copyright owner. For the purposes of this definition, "submitted"
means any form of electronic, verbal, or written communication sent
to the Licensor or its representatives, including but not limited to
communication on electronic mailing lists, source code control systems,
and issue tracking systems that are managed by, or on behalf of, the
Licensor for the purpose of discussing and improving the Work, but
excluding communication that is conspicuously marked or otherwise
designated in writing by the copyright owner as "Not a Contribution."
"Contributor" shall mean Licensor and any individual or Legal Entity
on behalf of whom a Contribution has been received by Licensor and
subsequently incorporated within the Work.
2. Grant of Copyright License. Subject to the terms and conditions of
this License, each Contributor hereby grants to You a perpetual,
worldwide, non-exclusive, no-charge, royalty-free, irrevocable
copyright license to reproduce, prepare Derivative Works of,
publicly display, publicly perform, sublicense, and distribute the
Work and such Derivative Works in Source or Object form.
3. Grant of Patent License. Subject to the terms and conditions of
this License, each Contributor hereby grants to You a perpetual,
worldwide, non-exclusive, no-charge, royalty-free, irrevocable
(except as stated in this section) patent license to make, have made,
use, offer to sell, sell, import, and otherwise transfer the Work,
where such license applies only to those patent claims licensable
by such Contributor that are necessarily infringed by their
Contribution(s) alone or by combination of their Contribution(s)
with the Work to which such Contribution(s) was submitted. If You
institute patent litigation against any entity (including a
cross-claim or counterclaim in a lawsuit) alleging that the Work
or a Contribution incorporated within the Work constitutes direct
or contributory patent infringement, then any patent licenses
granted to You under this License for that Work shall terminate
as of the date such litigation is filed.
4. Redistribution. You may reproduce and distribute copies of the
Work or Derivative Works thereof in any medium, with or without
modifications, and in Source or Object form, provided that You
meet the following conditions:
(a) You must give any other recipients of the Work or
Derivative Works a copy of this License; and
(b) You must cause any modified files to carry prominent notices
stating that You changed the files; and
(c) You must retain, in the Source form of any Derivative Works
that You distribute, all copyright, patent, trademark, and
attribution notices from the Source form of the Work,
excluding those notices that do not pertain to any part of
the Derivative Works; and
(d) If the Work includes a "NOTICE" text file as part of its
distribution, then any Derivative Works that You distribute must
include a readable copy of the attribution notices contained
within such NOTICE file, excluding those notices that do not
pertain to any part of the Derivative Works, in at least one
of the following places: within a NOTICE text file distributed
as part of the Derivative Works; within the Source form or
documentation, if provided along with the Derivative Works; or,
within a display generated by the Derivative Works, if and
wherever such third-party notices normally appear. The contents
of the NOTICE file are for informational purposes only and
do not modify the License. You may add Your own attribution
notices within Derivative Works that You distribute, alongside
or as an addendum to the NOTICE text from the Work, provided
that such additional attribution notices cannot be construed
as modifying the License.
You may add Your own copyright statement to Your modifications and
may provide additional or different license terms and conditions
for use, reproduction, or distribution of Your modifications, or
for any such Derivative Works as a whole, provided Your use,
reproduction, and distribution of the Work otherwise complies with
the conditions stated in this License.
5. Submission of Contributions. Unless You explicitly state otherwise,
any Contribution intentionally submitted for inclusion in the Work
by You to the Licensor shall be under the terms and conditions of
this License, without any additional terms or conditions.
Notwithstanding the above, nothing herein shall supersede or modify
the terms of any separate license agreement you may have executed
with Licensor regarding such Contributions.
6. Trademarks. This License does not grant permission to use the trade
names, trademarks, service marks, or product names of the Licensor,
except as required for reasonable and customary use in describing the
origin of the Work and reproducing the content of the NOTICE file.
7. Disclaimer of Warranty. Unless required by applicable law or
agreed to in writing, Licensor provides the Work (and each
Contributor provides its Contributions) on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
implied, including, without limitation, any warranties or conditions
of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A
PARTICULAR PURPOSE. You are solely responsible for determining the
appropriateness of using or redistributing the Work and assume any
risks associated with Your exercise of permissions under this License.
8. Limitation of Liability. In no event and under no legal theory,
whether in tort (including negligence), contract, or otherwise,
unless required by applicable law (such as deliberate and grossly
negligent acts) or agreed to in writing, shall any Contributor be
liable to You for damages, including any direct, indirect, special,
incidental, or consequential damages of any character arising as a
result of this License or out of the use or inability to use the
Work (including but not limited to damages for loss of goodwill,
work stoppage, computer failure or malfunction, or any and all
other commercial damages or losses), even if such Contributor
has been advised of the possibility of such damages.
9. Accepting Warranty or Additional Liability. While redistributing
the Work or Derivative Works thereof, You may choose to offer,
and charge a fee for, acceptance of support, warranty, indemnity,
or other liability obligations and/or rights consistent with this
License. However, in accepting such obligations, You may act only
on Your own behalf and on Your sole responsibility, not on behalf
of any other Contributor, and only if You agree to indemnify,
defend, and hold each Contributor harmless for any liability
incurred by, or claims asserted against, such Contributor by reason
of your accepting any such warranty or additional liability.
END OF TERMS AND CONDITIONS
================================================
FILE: license-mit
================================================
Permission is hereby granted, free of charge, to any
person obtaining a copy of this software and associated
documentation files (the "Software"), to deal in the
Software without restriction, including without
limitation the rights to use, copy, modify, merge,
publish, distribute, sublicense, and/or sell copies of
the Software, and to permit persons to whom the Software
is furnished to do so, subject to the following
conditions:
The above copyright notice and this permission notice
shall be included in all copies or substantial portions
of the Software.
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF
ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED
TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A
PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT
SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY
CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION
OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR
IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER
DEALINGS IN THE SOFTWARE.
================================================
FILE: posts/chat-server.md
================================================
# Beginner's Guide to Concurrent Programming: Coding a Multithreaded Chat Server using Tokio
_04 May 2024 · #rust · #async · #concurrency · #tokio_
<details>
<summary><b>Table of contents</b></summary>
[Introduction](#introduction)<br>
[01\) Simplest possible echo server](#01-simplest-possible-echo-server)<br>
[02\) Handling multiple connections serially](#02-handling-multiple-connections-serially)<br>
[03\) Modifying messages](#03-modifying-messages)<br>
[04\) Parsing a stream of bytes as lines](#04-parsing-a-stream-of-bytes-as-lines)<br>
[05\) Adding `/help` & `/quit` server commands](#05-adding-help--quit-server-commands)<br>
[06\) Handling multiple connections concurrently](#06-handling-multiple-connections-concurrently)<br>
[07\) Letting users kinda chat](#07-letting-users-kinda-chat)<br>
[08\) Letting users actually chat](#08-letting-users-actually-chat)<br>
[09\) Assigning names to users](#09-assigning-names-to-users)<br>
[10\) Letting users edit their names with `/name`](#10-letting-users-edit-their-names-with-name)<br>
[11\) Freeing user's name if they disconnect](#11-freeing-users-name-if-they-disconnect)<br>
[12\) Adding a main room](#12-adding-a-main-room)<br>
[13\) Letting users join or create rooms with `/join`](#13-letting-users-join-or-create-rooms-with-join)<br>
[14\) Listing all rooms with `/rooms`](#14-listing-all-rooms-with-rooms)<br>
[15\) Removing empty rooms](#15-removing-empty-rooms)<br>
[16\) Listing users in the room with `/users`](#16-listing-users-in-the-room-with-users)<br>
[17\) Optimizing performance](#17-optimizing-performance)<br>
[18\) Finishing touches](#18-finishing-touches)<br>
[Conclusion](#conclusion)<br>
[Discuss](#discuss)<br>
[Further reading](#further-reading)<br>
[Notifications](#notifications)<br>
</details>
## Introduction
I recently finished coding a multithreaded chat server using Tokio and I'm pretty happy with it. I'd like to share what I learned in this easy-to-follow step-by-step tutorial-style article. Let's get into it.
> [!NOTE]
> The full source code for every step can be found in [the examples directory](https://github.com/pretzelhammer/chat-server/tree/main/examples) of [this repository](https://github.com/pretzelhammer/chat-server).
## 01\) Simplest possible echo server
Let's start by writing the simplest possible echo server.
```rust
use tokio::{io::{AsyncReadExt, AsyncWriteExt}, net::TcpListener};
#[tokio::main]
async fn main() -> anyhow::Result<()> {
let server = TcpListener::bind("127.0.0.1:42069").await?;
let (mut tcp, _) = server.accept().await?;
let mut buffer = [0u8; 16];
loop {
let n = tcp.read(&mut buffer).await?;
if n == 0 {
break;
}
let _ = tcp.write(&buffer[..n]).await?;
}
Ok(())
}
```
`#[tokio::main]` is a procedural macro that removes some of the boilerplate of building a tokio runtime, and turns this:
```rust
#[tokio::main]
async fn my_async_fn() {
todo!()
}
```
Roughly into this:
```rust
fn main() {
tokio::runtime::Builder::new_multi_thread()
.enable_all()
.build()
.unwrap()
.block_on(my_async_fn)
}
async fn my_async_fn() {
todo!()
}
```
And as a quick refresher if we have an async function like this:
```rust
async fn my_async_fn<T>(t: T) -> T {
todo!()
}
```
It roughly desugars into:
```rust
fn my_async_fn<T>(t: T) -> impl Future<Output = T> {
todo!()
}
```
And a `Future` represents some form of asynchronous computation that we can `await` to get the result.
Let's also use the `anyhow` crate for carefree error propagation. Any place where we might want to return `Result<T, Box<dyn std::err::Error>>` we can substitute it with `anyhow::Result<T>` and get the same behavior.
In this line we're binding a TCP listener:
```rust
let server = TcpListener::bind("127.0.0.1:42069").await?;
```
> [!IMPORTANT]
> This is `tokio::net::TcpListener` and not `std::net::TcpListener`. The former is async and the latter is sync. Also calling `bind` returns a `Future` which we **must** `await` for anything to happen because futures are lazy in Rust!
As a general rule of thumb, if there's a type that handles IO by the same name in both `tokio` and `std` we want to use the one in `tokio`.
The rest of the code should hopefully be straight-forward:
```rust
let (mut tcp, _) = server.accept().await?;
let mut buffer = [0u8; 16];
loop {
let n = tcp.read(&mut buffer).await?;
if n == 0 {
break;
}
let _ = tcp.write(&buffer[..n]).await?;
}
```
We accept a connection, create a buffer, and then we read bytes from the connection into the buffer and write those bytes back to the connection in a loop until the connection closes.
We can connect to this server using a tool like `telnet` to see that it does in fact echo everything back to us:
```console
$ telnet 127.0.0.1 42069
> my first e c h o server!
my first e c h o server!
> hooray!
hooray!
```
> [!TIP]
> To quit `telnet` type `^]` (control + right square bracket) to enter command mode and type "quit" + ENTER.
If you'd like to mess around with the code yourself just `git clone` [this repository](https://github.com/pretzelhammer/chat-server) and you'll be able to quickly run any example with `just example {number}`. Then you can tinker with the source code at `examples/server-{number}.rs` to your heart's content. Once an example is running you can interact with it by running `just telnet`.
## 02\) Handling multiple connections serially
There's an annoying bug in our server: it quits after handling just one connection! If we try to `just telnet` more than once we get `telnet: Unable to connect to remote host: Connection refused` at which point we have to manually restart the server with `just example 01` again. 🤦
Here's how we would fix it:
```rust
use tokio::{io::{AsyncReadExt, AsyncWriteExt}, net::TcpListener};
#[tokio::main]
async fn main() -> anyhow::Result<()> {
let server = TcpListener::bind("127.0.0.1:42069").await?;
loop {
let (mut tcp, _) = server.accept().await?;
let mut buffer = [0u8; 16];
loop {
let n = tcp.read(&mut buffer).await?;
if n == 0 {
break;
}
let _ = tcp.write(&buffer[..n]).await?;
}
}
}
```
We just had to add another `loop` around our `server.accept()` line! Pretty easy, now we can run the updated example with `just example 02` and the server stays up regardless of how many times we run `just telnet` in a row.
## 03\) Modifying messages
As exciting as an echo server is, it would be even more exciting if it modified messages somehow. So how about we try adding a ❤️ emoji at the end of every echoed line? Here's what that would look like:
```rust
use tokio::{io::{AsyncReadExt, AsyncWriteExt}, net::TcpListener};
#[tokio::main]
async fn main() -> anyhow::Result<()> {
let server = TcpListener::bind("127.0.0.1:42069").await?;
loop {
let (mut tcp, _) = server.accept().await?;
let mut buffer = [0u8; 16];
loop {
let n = tcp.read(&mut buffer).await?;
if n == 0 {
break;
}
// convert byte slice to a String
let mut line = String::from_utf8(buffer[..n].to_vec())?;
// remove line terminating chars added by telnet
line.pop(); // remove \n char
line.pop(); // remove \r char
// add our own line terminator :)
line.push_str(" ❤️\n");
let _ = tcp.write(line.as_bytes()).await?;
}
}
}
```
Demo of our hearty echo server:
```console
$ just telnet
> hello
hello ❤️
> it works!
it works! ❤️
```
However if we write a message that's a little too long we'll see this bug:
```console
> this is the best day ever!
this is the be ❤️
day ever ❤️
```
Welp! Nobody said this was gonna be easy. We can increase the fixed size of our buffer but by how much? We can use a growable buffer like a `Vec` but what if the client sends a _really, really long line_? We could solve these problems ourselves, but they're pretty common, so we can also offload them to someone else too.
## 04\) Parsing a stream of bytes as lines
Tokio offers a convenient and robust solution to our line problem in the `tokio-util` crate. Here's how we can use it:
```rust
use futures::{SinkExt, StreamExt};
use tokio::net::TcpListener;
use tokio_util::codec::{FramedRead, FramedWrite, LinesCodec};
#[tokio::main]
async fn main() -> anyhow::Result<()> {
let server = TcpListener::bind("127.0.0.1:42069").await?;
loop {
let (mut tcp, _) = server.accept().await?;
let (reader, writer) = tcp.split();
let mut stream = FramedRead::new(reader, LinesCodec::new());
let mut sink = FramedWrite::new(writer, LinesCodec::new());
while let Some(Ok(mut msg)) = stream.next().await {
msg.push_str(" ❤️");
sink.send(msg).await?;
}
}
}
```
There's a lot of new stuff in this example so let's go over it. The `split` method splits a `TcpStream` into a `ReadHalf` and `WriteHalf`. This is useful if we want to add these halves to different structs, or send them to different threads, or read and write to the same `TcpStream` concurrently (which we'll be doing later).
`ReadHalf` implements `AsyncRead` and `WriteHalf` implements `AsyncWrite`, however as mentioned previously, these can be tedious and error-prone to work with directly, which why we bring in `LinesCodec`, `FramedRead`, and `FramedWrite`.
`LinesCodec` handles the low-level details of converting a stream of bytes into a stream of UTF-8 strings delimited by newlines, and using it together with `FramedRead` we can wrap a `ReadHalf` to get an implementation of `Stream<Item = Result<String, _>>`, which is much easier to work with than an `AsyncRead`. A `Stream` is like the async version of an `Iterator`. For example, if we had a sync function like this:
```rust
fn iterate<T>(items: impl Iterator<Item = T>) {
for item in items {
todo!()
}
}
```
The refactored async version would be:
```rust
use futures::{Stream, StreamExt};
async fn iterate<T>(mut items: impl Stream<Item = T> + Unpin) {
while let Some(item) = items.next().await {
todo!()
}
}
```
We also use `LinesCodec` together with `FramedWrite` to wrap a `WriteHalf` to get an implementation of `Sink<String, Error = _>`, which is much easier to work with than an `AsyncWrite`. As you've probably guessed, a `Sink` is the opposite of a `Stream`, it consumes values instead of producing values.
The rest of the code is straight-forward:
```rust
while let Some(Ok(mut msg)) = stream.next().await {
msg.push_str(" ❤️");
sink.send(msg).await?;
}
```
We get message from the stream, add a heart to it, and then send it to the sink. If we wanted to be fancy we could have also mapped the stream and forward it to the sink like this:
```rust
stream.map(|msg| {
let mut msg = msg?;
msg.push_str(" ❤️");
Ok(msg)
}).forward(sink).await?
```
`forward` returns a `Future` that completes when the `Stream` has been fully processed into the `Sink` and the `Sink` has been closed and flushed.
Now our server correctly appends the heart regardless of message length:
```console
$ just telnet
> this is a really really really long message kinda
this is a really really really long message kinda ❤️
```
## 05\) Adding `/help` & `/quit` server commands
Telnet is annoying to quit. The usual tricks of `esc`, `^C`, and `^D` don't work. We have to type `^]` to enter command mode and then type `quit` + ENTER. 🤦
We can make our server more user-friendly by implementing our own commands, so let's start with `/help` and `/quit`. `/help` will print out a list and description of all the commands our server supports and `/quit` will cause the server to close the connection (which will also cause telnet to quit).
So that these commands are discoverable lets send them immediately to every client that connects. Here's what everything put together looks like:
```rust
use futures::{SinkExt, StreamExt};
use tokio::net::TcpListener;
use tokio_util::codec::{FramedRead, FramedWrite, LinesCodec};
const HELP_MSG: &str = include_str!("help.txt");
#[tokio::main]
async fn main() -> anyhow::Result<()> {
let server = TcpListener::bind("127.0.0.1:42069").await?;
loop {
let (mut tcp, _) = server.accept().await?;
let (reader, writer) = tcp.split();
let mut stream = FramedRead::new(reader, LinesCodec::new());
let mut sink = FramedWrite::new(writer, LinesCodec::new());
// send list of server commands to
// the user as soon as they connect
sink.send(HELP_MSG).await?;
while let Some(Ok(mut msg)) = stream.next().await {
// handle new /help command
if msg.starts_with("/help") {
sink.send(HELP_MSG).await?;
// handle new /quit command
} else if msg.starts_with("/quit") {
break;
// handle regular message
} else {
msg.push_str(" ❤️");
sink.send(msg).await?;
}
}
}
}
```
Let's give it a spin:
```console
$ just telnet
Server commands
/help - prints this message
/quit - quits server
> /help # new command
Server commands
/help - prints this message
/quit - quits server
> woohoo it works
woohoo it works ❤️
> /quit # new command
Connection closed by foreign host.
```
## 06\) Handling multiple connections concurrently
The biggest downside of our server is that it only handles one connection at a time! If we run `just telnet` in two separate terminals we'll notice our server will only respond to the first connection, and won't start responding to the second connection until the first quits. Although we've been using a lot of async APIs our current implementation behaves no differently than a sync single-threaded server. Let's change that:
```rust
use futures::{SinkExt, StreamExt};
use tokio::net::{TcpListener, TcpStream};
use tokio_util::codec::{FramedRead, FramedWrite, LinesCodec};
const HELP_MSG: &str = include_str!("help.txt");
#[tokio::main]
async fn main() -> anyhow::Result<()> {
let server = TcpListener::bind("127.0.0.1:42069").await?;
loop {
let (tcp, _) = server.accept().await?;
// spawn a separate task for
// to handle every connection
tokio::spawn(handle_user(tcp));
}
}
async fn handle_user(mut tcp: TcpStream) -> anyhow::Result<()> {
let (reader, writer) = tcp.split();
let mut stream = FramedRead::new(reader, LinesCodec::new());
let mut sink = FramedWrite::new(writer, LinesCodec::new());
sink.send(HELP_MSG).await?;
while let Some(Ok(mut msg)) = stream.next().await {
if msg.starts_with("/help") {
sink.send(HELP_MSG).await?;
} else if msg.starts_with("/quit") {
break;
} else {
msg.push_str(" ❤️");
sink.send(msg).await?;
}
}
Ok(())
}
```
`tokio::spawn` takes a `Future` and spawns an asynchronous "task" to complete it. The execution begins immediately so we don't have to `await` the returned join handle like we would a future. A "task" is like a native thread except instead of being managed by the OS it's managed by Tokio. You're probably already familiar with this concept by some of these other names: lightweight threads, green threads, user-space threads.
## 07\) Letting users kinda chat
To really get the party started we need to upgrade our echo server to a chat server that lets separate concurrent connections communicate with each other:
> [!NOTE]
> The code is starting to get long and difficult to read. All following examples will be presented as a heavily abbreviated diff highlighting the key changes, but you can still find the full source code for any example in [the examples directory](https://github.com/pretzelhammer/chat-server/tree/main/examples) of [this repository](https://github.com/pretzelhammer/chat-server). You can see a diff between any two examples by running `just diff {number} {number}`. For instance, to see the diff between this example and the previous example you would run `just diff 06 07`.
```rust
// ...
async fn main() -> anyhow::Result<()> {
// ...
// create broadcast channel
let (tx, _) = broadcast::channel::<String>(32);
// ...
// clone it for every connected client
tokio::spawn(handle_user(tcp, tx.clone()));
}
async fn handle_user(
mut tcp: TcpStream,
tx: Sender<String>
) -> anyhow::Result<()> {
// ...
// get a receiver from the sender
let mut rx = tx.subscribe();
// ...
while let Some(Ok(mut user_msg)) = stream.next().await {
// ...
// send all messages to the channel
tx.send(user_msg)?;
// ...
// receive all of our and others'
// messages from the channel
let peer_msg = rx.recv().await?;
sink.send(peer_msg).await?;
}
// ...
}
```
We communicate between different clients using a broadcast channel. After creating the channel we get a `Sender` and a `Receiver` which we can `clone` any number of times and send to different threads. Each value sent via a `Sender` gets received by every `Receiver`, so the value type must implement `Clone`.
Before we would get a message from the client's stream and immediately echo it out to the client's sink. Now when we get a message from the client's stream we pass it through the broadcast channel before we get it back and then send it to the client's sink. Every client will receive their own and others' messages from the shared channel.
Let's try out our new code by connecting with two clients at once:
```console
$ just telnet # concurrent client 1
> 1: hello # msg 1
1: hello ❤️
> 1: anybody there? # msg 2
1: anybody there? ❤️
$ just telnet # concurrent client 2
> 2: hey there # msg 3
1: hello ❤️
> 2: how are you # msg 4
1: anybody there? ❤️
> 2: i am right here # msg 5
2: hey there ❤️
> 2: wtf # msg 6
2: how are you ❤️
```
Each client is seeing each other's messages but they seem kinda delayed and staggered for some reason. Something just isn't quite right here.
The bug in our code is here:
```rust
// the client must first send a message
while let Some(Ok(mut user_msg)) = stream.next().await {
// in order to receive a message
let peer_msg = rx.recv().await?;
// and these two things always alternate
}
```
In order to receive a message from a peer, we must first send a message. What if we want to be a lurker? Or what if our conversation partner is much more chatty than us? On the other hand, if we're the chatty one then we'll barely see any messages from our partner since we'll mostly see our own echoed output.
To solve this problem we need to be able to `await` two futures at once. In this case those futures are the ones created by `stream.next()`, which gets the next message from the client, and `rx.recv()`, which gets the next message from the channel.
## 08\) Letting users actually chat
`tokio::select!` allows us to poll multiple futures at once:
```rust
async fn handle_user(
mut tcp: TcpStream,
tx: Sender<String>
) -> anyhow::Result<()> {
// ...
loop {
tokio::select! {
user_msg = stream.next() => {
// ...
},
peer_msg = rx.recv() => {
// ...
},
}
}
// ...
}
```
We execute the match arm of whatever future completes first. The other future is dropped.
Now if we try our server:
```console
$ just telnet # concurrent client 1
> 1: hello # msg 1
1: hello ❤️
> 1: anybody there? # msg 2
1: anybody there? ❤️
2: i am right here ❤️
2: how are you ❤️
> 1: i am doing great # msg 5
$ just telnet # concurrent client 2
1: hello ❤️
1: anybody there? ❤️
> 2: i am right here # msg 3
2: i am right here ❤️
> 2: how are you? # msg 4
2: how are you ❤️
1: i am doing great ❤️
```
It works! Anyway, celebrations aside, we need to talk about cancel safety. As mentioned before, Rust futures are lazy, and they only make progress while being polled. Polling is a little bit different than awaiting. To await a future means to poll it to completion. To poll a future means to ask it to make some progress, but it may not necessarily complete.
On one hand, this is great, because if we start polling a future and later decide we don't need its result anymore, we can stop polling it and we won't waste anymore CPU on doing useless work. On the other hand, this may not be so great if the future we're cancelling is in the middle of an important operation that if not completed may drop important data or may leave data in a corrupt state.
Let's look at an example of "cancelling" a future. Cancelling is in quotes because it's not an explicit operation, it just means we started to poll a future but then stopped polling it before it completed.
```rust
use tokio::time::sleep;
use std::time::Duration;
async fn count_to(num: u8) {
for i in 1..=num {
sleep(Duration::from_millis(100)).await;
println!("{i}");
}
}
#[tokio::main]
async fn main() {
println!("start counting");
// the select! macro polls each
// future until one of them completes,
// and then we execute the match arm
// of the completed future and drop
// all of the other futures
tokio::select! {
_ = count_to(3) => {
println!("counted to 3");
},
_ = count_to(10) => {
println!("counted to 10");
},
};
println!("stop counting");
// this sleep is here to demonstrate
// that the count_to(10) doesn't make
// any progress after we stop polling
// it, even if we go to sleep and do
// nothing else for a while
sleep(Duration::from_millis(1000)).await;
}
```
This program outputs:
```
start counting
1
1
2
2
3
3
counted to 3
stop counting
```
We "cancelled" the `count_to(10)` future. In this simple toy example we don't care if the count completes or not so this future is cancel-safe in that sense, but if finishing the count was critical to our application then cancelling this future would be a problem. To make sure the future completes we can `await` it after the `tokio::select!`:
```rust
// ...
async fn main() {
println!("start counting");
let count_to_10 = count_to(10);
tokio::select! {
_ = count_to(3) => {
println!("counted to 3");
},
_ = count_to_10 => { // ❌
println!("counted to 10");
},
};
println!("stop counting");
println!("jk, keep counting");
count_to_10.await; // ❌
println!("finished counting to 10");
}
```
Throws:
```
error[E0382]: use of moved value: count_to_10
```
Oh duh, we made the simplest mistake in the book, trying to use a value after we moved it. Let's pass a mutable reference instead:
```rust
// ...
async fn main() {
println!("start counting");
let count_to_10 = count_to(10);
tokio::select! {
_ = count_to(3) => {
println!("counted to 3");
},
_ = &mut count_to_10 => { // ❌
println!("counted to 10");
},
};
println!("stop counting");
println!("jk, keep counting");
count_to_10.await;
println!("counted to 10");
}
```
Now throws:
```
error[E0277]: {async fn body@src/main.rs:23:28: 28:2}
cannot be unpinned
-> src/main.rs:34:5
|
23 | async fn count_to(num: u8) {
| ----------------- within this impl futures::Future<Output = ()>
...
34 | / tokio::select! {
35 | | _ = count_to(3) => {
36 | | println!("counted to 3");
37 | | },
...
41 | | };
| | ^
| | |
| |_____within impl futures::Future<Output = ()>,
| the trait Unpin is not implemented for
| {async fn body@src/main.rs:23:28: 28:2},
| which is required by &mut impl
| futures::Future<Output = ()>: futures::Future
| required by a bound introduced by this call
|
= note: consider using the pin! macro
consider using Box::pin if you need to access
the pinned value outside of the current scope
```
We need to "pin" our future. Okay, let's do what the compiler suggested:
```rust
#[tokio::main]
async fn main() {
println!("start counting");
let count_to_10 = count_to(10);
tokio::pin!(count_to_10); // ✔️
tokio::select! {
_ = count_to(3) => {
println!("counted to 3");
},
_ = &mut count_to_10 => {
println!("counted to 10");
},
};
println!("stop counting");
println!("jk, keep counting");
count_to_10.await;
println!("finished counting to 10");
}
```
Compiles and outputs:
```
start counting
1
1
2
2
3
3
counted to 3
stop counting
jk, keep counting
4
5
6
7
8
9
10
finished counting to 10
```
To pin something in Rust means to pin its location in memory. Once it is pinned it cannot be moved. The reason some futures need to be pinned before being polled is because under-the-hood they can contain self-referential pointers that would be invalidated if the future was ever moved.
If that last part flew over your head don't worry, I don't fully get it either. But fear not, here's a general algorithm we can follow to solve these kinds of problems when they arise:
**1\)** If we're writing generic code that takes a future or something that produces futures we can add `+ Unpin` to the trait bounds. So for example this doesn't compile:
```rust
use futures::{Stream, StreamExt};
async fn iterate<T>(
mut items: impl Stream<Item = T>
) {
while let Some(item) = items.next().await { // ❌
todo!()
}
}
```
Throws:
```
error[E0277]: impl Stream<Item = T> cannot be unpinned
```
But if we sprinkle `Unpin` into the function signature it works:
```rust
async fn iterate<T>(
mut items: impl Stream<Item = T> + Unpin // ✔️
) {
while let Some(item) = items.next().await {
todo!()
}
}
```
**2\)** However, let's say that causes compile errors elsewhere in our code, because we are passing a stream to this function isn't `Unpin`. We can remove the `Unpin` from the function signature and use the `pin!` macro to pin the stream within the function:
```rust
async fn iterate<T>(
mut items: impl Stream<Item = T>
) {
tokio::pin!(items); // ✔️
while let Some(item) = items.next().await {
todo!()
}
}
```
This pins it to the stack, so it cannot escape the current scope.
**3\)** If the pinned object needs to escape the current scope there's `Box::pin` to pin it in the heap:
```rust
async fn iterate<T>(
mut items: impl Stream<Item = T>
) {
let mut items = Box::pin(items); // ✔️
while let Some(item) = items.next().await {
todo!()
}
}
```
**4\)** Or we can ask the caller to figure out this detail for us:
```rust
async fn iterate<T, S: Stream<Item = T> + ?Sized>(
mut items: Pin<&mut S>
) {
while let Some(item) = items.next().await {
todo!()
}
}
```
However in this case the caller is also us, so this doesn't help that much over solutions 2 & 3.
> [!IMPORTANT]
> In summary: we need to be mindful which futures are and aren't cancel-safe when we pass them to code that may not poll them to completion, e.g. `tokio::select!`. If you're writing a Rust library that polls futures you need to document if your library will poll the futures to completion or not. If you're writing a Rust library that produces futures you need to document which of the futures are and aren't cancel-safe. If you're using a Rust library that either polls or returns futures you need to carefully read its docs.
## 09\) Assigning names to users
In the current iteration of our chat server it's hard to follow who said what. We could prepend each connection's socket address to their message to disambiguate them, and if we did it would look something like this:
```console
$ just telnet
> hello
127.0.0.1:51270: hello
```
However that is both ugly and boring. Let's generate random names by combining an adjective with an animal and assign them to users when they join:
```rust
pub static ADJECTIVES: [&str; 628] = [
"Mushy",
"Starry",
"Peaceful",
"Phony",
"Amazing",
"Queasy",
// ...
];
pub static ANIMALS: [&str; 243] = [
"Owl",
"Mantis",
"Gopher",
"Robin",
"Vulture",
"Prawn",
// ...
];
pub fn random_name() -> String {
let adjective = fastrand::choice(ADJECTIVES).unwrap();
let animal = fastrand::choice(ANIMALS).unwrap();
format!("{adjective}{animal}")
}
```
Here's a sampling of some of the names this creates:
```
HushedLlama
DimpledDinosaur
UrbanMongoose
YawningMinotaur
RomanticRhino
DapperPeacock
PlasticCentaur
BubblyChicken
AnxiousGriffin
SpicyAlpaca
MindlessOctopus
WealthyPelican
CruelCapybara
RegalFrog
PinkPoodle
QuirkyGazelle
PoshGopher
CarelessBobcat
SomberWeasel
ZenMammoth
DazzlingSquid
```
To keep our main file clean and concise, let's put this functionality into our lib file and import it:
```rust
use chat_server::random_name;
// ...
async fn handle_user(
mut tcp: TcpStream,
tx: Sender<String>
) -> anyhow::Result<()> {
// ...
// generate random name
let name = random_name();
// ...
// tell user their name
sink.send(format!("You are {name}")).await?;
// ...
user_msg = stream.next() => {
// ...
// prepend user's name to their messages
tx.send(format!("{name}: {user_msg}"))?;
},
// ...
}
```
Let's give it a try:
```console
$ just chat
You are MeatyPuma
> hello
MeatyPuma: hello
PeacefulGibbon: howdy
```
Excellent.
> [!NOTE]
> I switched from using `just telnet` to `just chat` because I got tired of using telnet and built a TUI chat client that's easier to use and looks nicer, which is what `just chat` runs.
## 10\) Letting users edit their names with `/name`
We'd like names to be unique across the server. We can enforce this by maintaining the names in a `HashSet<String>`. However, since we'd also like to let users edit their names using the `/name` command we need to share this set between users running in different threads.
> [!TIP]
> To share mutable data across many threads we can wrap it with `Arc<Mutex<T>>`, which is like the thread-safe version of `Rc<RefCell<T>>` if you've ever used that before.
Let's wrap our `Arc<Mutex<HashSet<T>>>` in a new type to make using it more ergonomic:
```rust
// ...
#[derive(Clone)]
struct Names(Arc<Mutex<HashSet<String>>>);
impl Names {
fn new() -> Self {
Self(Arc::new(Mutex::new(HashSet::new())))
}
// returns true if name was inserted,
// i.e. the name is unique
fn insert(&self, name: String) -> bool {
self.0.lock().unwrap().insert(name)
}
// returns unique name
fn get_unique(&self) -> String {
let mut name = random_name();
let mut guard = self.0.lock().unwrap();
while !guard.insert(name.clone()) {
name = random_name();
}
name
}
}
#[tokio::main]
async fn main() -> anyhow::Result<()> {
// ...
let names = Names::new();
// ...
tokio::spawn(handle_user(tcp, tx.clone(), names.clone()));
}
async fn handle_user(
mut tcp: TcpStream,
tx: Sender<String>,
names: Names,
) -> anyhow::Result<()> {
// ...
// get a unique name for new user
let mut name = names.get_unique();
// ...
// tell them their name
sink.send(format!("You are {name}")).await?;
// ...
user_msg = stream.next() => {
// ...
// handle new /name command
if user_msg.starts_with("/name") {
let new_name = user_msg
.split_ascii_whitespace()
.nth(1)
.unwrap()
.to_owned();
// check if name is unique
let changed_name = names.insert(new_name.clone());
if changed_name {
// notify everyone that user
// changed their name
tx.send(format!("{name} is now {new_name}"))?;
// remove previous name
names.remove(&name);
// set new name
name = new_name;
} else {
// tell user that name is
// already taken
sink.send(
format!("{new_name} is already taken")
).await?;
}
}
// ...
},
// ...
}
```
Let's try it out:
```console
$ just chat
Server commands
/help - prints this message
/name {name} - change name
/quit - quits server
You are FancyYak
> hello
FancyYak: hello
> /name pretzelhammer # new command
FancyYak is now pretzelhammer
> 🦀🦀🦀
pretzelhammer: 🦀🦀🦀
```
> [!CAUTION]
> Rust promises that compiling safe programs are free of memory vulnerabilities, but it makes no promises that they will be free of deadlocks. When we add locks to our program we need to be careful to avoid creating deadlock scenarios.
Here's some tips for avoiding deadlocks:
**1\)** Don't hold locks across await points
An `await` point is anywhere in an async function where `await` is called. When we call `await` we yield control back to the Tokio scheduler. If our yielded future holds a lock that means any executing futures won't be able to acquire it, in which case they will block forever while waiting on it, and the yielded lock-holding future won't get an opportunity to run again, and so we have a deadlock.
That was a bit abstract so let's run through a concrete example. Imagine we have a single-threaded Tokio runtime, with three futures ready to be polled:
```
Tokio scheduler, future queue:
+---------+---------+---------+
| fut A | fut B | fut C |
+---------+---------+---------+
```
Since this is a single-threaded runtime we can only execute one future at a time. Tokio polls the first future, future A, which runs code that looks like this:
```rust
async do_stuff<T: Debug>(mutex: Mutex<T>) {
// acquires lock
let guard = mutex.lock().unwrap();
// hits await point, i.e. yields to scheduler
other_async_fn().await?;
// releases lock
dbg!(guard);
}
```
When it hits the `await` point, the future goes back to the end of the queue:
```
Tokio scheduler, future queue:
+---------+---------+---------+
| fut B | fut C | fut A* |
+---------+---------+---------+
* holding lock
```
Then Tokio tries to poll the next future, future B, and that future runs through the same code path, trying to acquire a lock to the same mutex that future A is currently holding! It will block forever! Future B cannot make progress until future A releases the lock, but future A cannot release the lock until future B yields back to the scheduler. We have a deadlock.
_"But what if we use an async mutex instead of a sync mutex?"_
It is true that tip 1 applies to sync mutexes, like `std::sync::Mutex`, but not to async mutexes, like `tokio::sync::Mutex`. For mutexes designed to be used in an async context, we can hold their locks across `await` points, however they are slower. To quote the Tokio docs:
> Contrary to popular belief, it is ok and often preferred to use the ordinary Mutex from the standard library in asynchronous code.
>
> The feature that the async mutex offers over the blocking mutex is the ability to keep it locked across an await point. This makes the async mutex more expensive than the blocking mutex, so the blocking mutex should be preferred in the cases where it can be used. The primary use case for the async mutex is to provide shared mutable access to IO resources such as a database connection. If the value behind the mutex is just data, it's usually appropriate to use a blocking mutex such as the one in the standard library.
Generally speaking, if we can structure our code to never to hold a lock across an `await` point it's better to use a sync mutex, and if we absolutely have to hold a lock across an `await` point then we would switch to an async mutex.
**2\)** Don't reaquire the same lock multiple times
Trivial example:
```rust
fn main() {
let mut mutex = Mutex::new(5);
// acquire lock
let g = mutex.lock().unwrap();
// try to acquire lock again
mutex.lock().unwrap(); // deadlocks
}
```
Although the mistake is super obvious in the example above when it happens in Real Code<sup>TM</sup> it's much harder to spot and debug.
_"I won't have to worry about this if I'm using a read-write lock, right? Since those are suppose to be able to give out many read locks to concurrent threads."_
Surprisingly no, even reacquiring a read lock on a read-write lock twice in the same thread can produce a deadlock. To borrow a diagram from the standard library `RwLock` docs:
```
// Thread 1 | // Thread 2
let _rg = lock.read(); |
| // will block
| let _wg = lock.write();
// may deadlock |
let _rg = lock.read(); |
```
And to quote the `RwLock` docs from the `parking_lot` crate:
> This lock uses a task-fair locking policy which avoids both reader and writer starvation. This means that readers trying to acquire the lock will block even if the lock is unlocked when there are writers waiting to acquire the lock. Because of this, attempts to recursively acquire a read lock within a single thread may result in a deadlock.
I guess we just have to be really careful 🤷
**3\)** Acquire locks in the same order everywhere
If we need to acquire multiple locks to safely perform some operation, we need to always acquire those locks in the same order, otherwise deadlocks can trivially occur, like here:
```
// Thread 1 | // Thread 2
let _a = a.lock(); | let _b = b.lock();
// ... | // ...
let _b = b.lock(); | let _a = a.lock();
```
_"My head is spinning from all of these gotchas. Surely there has to be an easier or better way to do all of this locking stuff?"_
**4\)** Use lockfree data structures
Doing this allows you to disregard tips 1-3, since lockfree data structures cannot deadlock. However, in general, lockfree data structures are slower than most of their lock-based counterparts.
**5\)** Use channels for everything
Doing this also allows you to disregard tips 1-3, since channels cannot deadlock. I'm not well-read enough on this subject to comment on whether using channels for everything can degrade or improve the performance of a concurrent program vs using locks. I imagine the answer, like the answer to most computer science questions, is _"it depends."_
This approach is also sometimes called the "actor pattern" and if you search for "actor" on cargo you'll find a lot of actor framework crates that supposedly help with structuring your program to follow this pattern.
Anyway, that was long detour. Let's get back to our chat server.
## 11\) Freeing user's name if they disconnect
We have a bug in our code. Names are not removed from the set when a user disconnects, so after a name is taken it can never be used again, not until we restart the server. Unfortunately there's a tricky obstacle we have to overcome before we can fix this.
The obstacle is the user may disconnect due to an error, and we're using `?` everywhere in our `handle_user` function, which propagates the error up to `main`, but it shouldn't be `main`'s responsibility to clean up names, that's a detail that should remain within `handle_user`. We can get rid of `?` and pattern match on `Result`s everywhere but that's verbose and ugly. So what do we do when we have to get rid of a bunch of repetitive, ugly, verbose code? We use macros.
As a quick recap, remember that all blocks in Rust are expressions, and we can `break` out of a block with a value. A couple examples:
```rust
fn main() {
// breaking from loop with value
let value = loop {
break "value";
};
assert_eq!(value, "value");
// to break from a non-loop block
// it needs to be labelled
let value = 'label: {
break 'label "value";
};
assert_eq!(value, "value");
}
```
Also, the `?` operator is not magic and can be implemented as a macro:
```rust
macro_rules! question_mark {
($result:expr) => {
match $result {
Ok(ok) => ok,
Err(err) => return Err(err.into()),
}
}
}
```
Which is everything we want, except the `return` should be a `break`, since we'd like to handle the errors within our function and not propagate them to the caller. So let's write a new macro and call it `b!` which is short for `break`:
```rust
macro_rules! b {
($result:expr) => {
match $result {
Ok(ok) => ok,
Err(err) => break Err(err.into()),
}
}
}
```
And then we can refactor a function that propagates errors to its caller like this:
```rust
fn some_function() -> anyhow::Result<()> {
// initialize state here
loop {
fallible_statement_1?;
fallible_statement_2?;
// etc
}
// clean up state here, but
// this may never be reached
// because the ? returns from
// the function instead of
// breaking from the loop
Ok(())
}
```
Into a function which catches and handles its own errors:
```rust
fn some_function() {
// initialize state here
let result = loop {
b!(fallible_statement_1);
b!(fallible_statement_2);
// etc
};
// clean up state here, always reached
if let Err(err) = result {
// handle errors if necessary
}
// nothing to return anymore since
// we take care of everything within
// the function :)
}
```
So with all of that context out of the way, here's the updated code:
```rust
// ...
async fn handle_user(
mut tcp: TcpStream,
tx: Sender<String>,
names: Names,
) -> anyhow::Result<()> {
// ...
// we now catch errors here
let result: anyhow::Result<()> = loop {
// all fallible statements
// from before are now wrapped
// with our b!() macro
};
// the line below is always reached
// and the user's name is always freed,
// regardless if they quit normally or
// abruptly disconnected due to an error
names.remove(&name);
// return result to caller if they want
// to do anything extra
result
}
```
Now if a user disconnects for any reason we will always reclaim their name.
## 12\) Adding a main room
Right now all users are dumped into the same room and have nowhere else to go. It will be difficult to keep the conversation on one topic and hard to follow the discussion if multiple side-conversations start happening concurrently. We should add the ability to create and join different rooms within our server. As a first step let's refactor our current code to add everyone who joins the server into a default room, called `main`. The updated code:
```rust
// ...
struct Room {
tx: Sender<String>,
}
impl Room {
fn new() -> Self {
let (tx, _) = broadcast::channel(32);
Self {
tx,
}
}
}
const MAIN: &str = "main";
#[derive(Clone)]
struct Rooms(Arc<RwLock<HashMap<String, Room>>>);
impl Rooms {
fn new() -> Self {
Self(Arc::new(RwLock::new(HashMap::new())))
}
fn join(&self, room_name: &str) -> Sender<String> {
// get read access
let read_guard = self.0.read().unwrap();
// check if room already exists
if let Some(room) = read_guard.get(room_name) {
return room.tx.clone();
}
// must drop read before acquiring write
drop(read_guard);
// create room if it doesn't yet exist
// get write access
let mut write_guard = self.0.write().unwrap();
let room = write_guard
.entry(room_name.to_owned())
.or_insert(Room::new());
room.tx.clone()
}
}
#[tokio::main]
async fn main() -> anyhow::Result<()> {
// ...
let rooms = Rooms::new();
// ...
tokio::spawn(handle_user(tcp, names.clone(), rooms.clone()));
}
async fn handle_user(
mut tcp: TcpStream,
names: Names,
rooms: Rooms,
) -> anyhow::Result<()> {
// ...
// when user connects to server
// automatically put them in
// the main room
let room_name = MAIN.to_owned();
let room_tx = rooms.join(&room_name);
let mut room_rx = room_tx.subscribe();
// notify everyone in room that
// a new user has joined
let _ = room_tx.send(format!("{name} joined {room_name}"));
// ...
tokio::select! {
user_msg = stream.next() => {
// ...
// send messages to the room
// we're currently in
b!(room_tx.send(format!("{name}: {user_msg}")));
},
// receive messages from the
// room we're currently in
peer_msg = room_rx.recv() => {
// ...
},
}
// ...
// notify everyone in room that
// we have left
let _ = room_tx.send(format!("{name} left {room_name}"));
// ...
}
```
A `Room` is a wrapper around a `broadcast::Sender<String>`, and `Rooms` is a wrapper around an `Arc<RwLock<HashMap<String, Room>>>` because we need to maintain a map of room names to broadcast channels and we'd like to share and modify this map across many threads.
We also added notification messages for when a user joins and leaves a room. Let's see what it looks like altogether:
```console
$ just chat
You are AwesomeVulture
AwesomeVulture joined main
JealousHornet joined main
JealousHornet: we are at the main room!
> can we create other rooms?
AwesomeVulture: can we create other rooms?
JealousHornet: not yet, back to work we go
JealousHornet left main
```
## 13\) Letting users join or create rooms with `/join`
Since we implemented the `join` method earlier this will be pretty easy:
```rust
// ...
async fn handle_user(
mut tcp: TcpStream,
names: Names,
rooms: Rooms,
) -> anyhow::Result<()> {
// ...
// automatically join main room
// on connect, as before
let mut room_name = MAIN.to_owned();
let mut room_tx = rooms.join(&room_name);
let mut room_rx = room_tx.subscribe();
// ...
if user_msg.starts_with("/join") {
let new_room = user_msg
.split_ascii_whitespace()
.nth(1)
.unwrap()
.to_owned();
// check if user is already in the room
// they're trying to join
if new_room == room_name {
b!(sink.send(format!("You are in {room_name}")).await);
continue;
}
// notify current room that we've left
b!(room_tx.send(format!("{name} left {room_name}")));
// join new room, this creates
// the room if it doesn't
// already exist
room_tx = rooms.join(&new_room);
room_rx = room_tx.subscribe();
room_name = new_room;
// notify new room that we have joined
b!(room_tx.send(format!("{name} joined {room_name}")));
}
// ...
// notify our current room that we've left
// on disconnect, as before
let _ = room_tx.send(format!("{name} left {room_name}"));
// ...
}
```
Now we can start pizza parties:
```console
$ just chat
Server commands
/help - prints this message
/name {name} - change name
/join {room} - joins room
/quit - quits server
You are ElasticBonobo
ElasticBonobo joined main
BlondCyclops joined main
> /join pizza # new command
ElasticBonobo joined pizza
BlondCyclops joined pizza
> let's have a pizza party
ElasticBonobo: let's have a pizza party
BlondCyclops: 🍕🥳
```
## 14\) Listing all rooms with `/rooms`
Right now pizza parties on the server are not very discoverable. If a user lands in the `main` room there's no way for them to know all the other users on the server are in `pizza`. Let's add a `/rooms` command that will list all of the rooms on server:
```rust
// ...
#[derive(Clone)]
struct Rooms(Arc<RwLock<HashMap<String, Room>>>);
impl Rooms {
fn list(&self) -> Vec<(String, usize)> {
// iterate over rooms map
let mut list: Vec<_> = self
.0
.read()
.unwrap()
.iter()
// receiver_count tells us
// the # of users in the room
.map(|(name, room)| (
name.to_owned(),
room.tx.receiver_count(),
))
.collect();
list.sort_by(|a, b| {
use std::cmp::Ordering::*;
// sort rooms by # of users first
match b.1.cmp(&a.1) {
// and by alphabetical order second
Equal => a.0.cmp(&b.0),
ordering => ordering,
}
});
list
}
}
// ...
async fn handle_user(
mut tcp: TcpStream,
names: Names,
rooms: Rooms,
) -> anyhow::Result<()> {
// ...
// handle new /rooms command
if user_msg.starts_with("/rooms") {
let rooms_list = rooms.list();
let rooms_list = rooms_list
.into_iter()
.map(|(name, count)| format!("{name} ({count})"))
.collect::<Vec<_>>()
.join(", ");
b!(sink.send(format!("Rooms - {rooms_list}")).await);
}
// ...
}
```
Now everyone is invited to our pizza parties:
```console
$ just chat
Server commands
/help - prints this message
/name {name} - change name
/rooms - list rooms
/join {room} - joins room
/quit - quits server
You are SilentYeti
SilentYeti joined main
> /rooms # new command
Rooms - pizza (2), main (1)
> /join pizza
SilentYeti joined pizza
> can i be part of this pizza party? 🥺
SilentYeti: can i be part of this pizza party? 🥺
BulkyApe: of course ❤️
AmazingDragon: 🔥🔥🔥
```
## 15\) Removing empty rooms
We have a bug, after a room is created it's never deleted, even after it becomes empty. After a while our server rooms list will look like this:
```console
> /rooms
Rooms - a (0), bunch (0), of (0), abandoned (0), rooms (0)
```
Let's fix that:
```rust
// ...
#[derive(Clone)]
struct Rooms(Arc<RwLock<HashMap<String, Room>>>);
impl Rooms {
// ...
fn leave(&self, room_name: &str) {
let read_guard = self.0.read().unwrap();
let mut delete_room = false;
if let Some(room) = read_guard.get(room_name) {
// if the receiver count is 1 then
// we're the last person in the room
// and can remove it
delete_room = room.tx.receiver_count() <= 1;
}
drop(read_guard);
if delete_room {
let mut write_guard = self.0.write().unwrap();
write_guard.remove(room_name);
}
}
fn change(
&self,
prev_room: &str,
next_room: &str
) -> Sender<String> {
self.leave(prev_room);
self.join(next_room)
}
// ...
}
async fn handle_user(
mut tcp: TcpStream,
names: Names,
rooms: Rooms,
) -> anyhow::Result<()> {
// ...
if user_msg.starts_with("/join") {
// ...
// now correctly deletes the room
// we're leaving if it becomes empty
room_tx = rooms.change(&room_name, &new_room);
// ...
}
// ...
// when we disconnect we also
// need to leave and delete the
// room if it's empty
rooms.leave(&room_name);
// ...
}
```
## 16\) Listing users in the room with `/users`
Users within a room are not discoverable. Let's add a `/users` command that will list the users in the current room. To do that we'll have to add a `HashSet<String>` to the `Room` struct and update many of the `Rooms` methods to also take a user name when joining, changing, or leaving a room:
```rust
// ...
struct Room {
// ...
// keep track of the names
// of the users in the room
users: HashSet<String>,
}
impl Room {
fn new() -> Self {
// ...
let users = HashSet::new();
Self {
// ...
users,
}
}
}
#[derive(Clone)]
struct Rooms(Arc<RwLock<HashMap<String, Room>>>);
impl Rooms {
// ...
fn join(&self, room_name: &str, user_name: &str) -> Sender<String> {
// ...
room.users.insert(user_name.to_owned());
// ...
}
fn leave(&self, room_name: &str, user_name: &str) {
// ...
room.users.remove(user_name);
// ...
}
// update user's name in room if they
// changed it using the /name command
fn change_name(
&self,
room_name: &str,
prev_name: &str,
new_name: &str
) {
let mut write_guard = self.0.write().unwrap();
if let Some(room) = write_guard.get_mut(room_name) {
room.users.remove(prev_name);
room.users.insert(new_name.to_owned());
}
}
// returns list of users' names in the room
fn list_users(&self, room_name: &str) -> Option<Vec<String>> {
self
.0
.read()
.unwrap()
.get(room_name)
.map(|room| {
// get users in room
let mut users = room
.users
.iter()
.cloned()
.collect::<Vec<_>>();
// alphabetically sort
// users by names
users.sort();
users
})
}
}
// ...
async fn handle_user(
mut tcp: TcpStream,
names: Names,
rooms: Rooms,
) -> anyhow::Result<()> {
// ...
// send our name when joining a room
room_tx = rooms.join(&room_name, &name);
// ...
if user_msg.starts_with("/name") {
// ...
if changed_name {
// let room know we changed our name
rooms.change_name(&room_name, &name, &new_name);
// ...
}
// ...
} else if user_msg.starts_with("/join") {
// ...
// send our name when changing rooms
room_tx = rooms.change(&room_name, &new_room, &name);
// ...
// handle new /users command
} else if user_msg.starts_with("/users") {
let users_list = rooms
.list_users(&room_name)
.unwrap()
.join(", ");
b!(sink.send(format!("Users - {users_list}")).await);
}
// ...
rooms.leave(&room_name, &name);
// ...
}
```
Now we can find our friends within rooms:
```console
$ just chat
Server commands
/help - prints this message
/name {name} - change name
/rooms - list rooms
/join {room} - joins room
/users - lists users in current room
/quit - quits server
You are StarryDolphin
StarryDolphin joined main
> /users # new command
Users - ColorfulSheep, PaleHedgehog, StarryDolphin
> hey colorful sheep! 👋
StarryDolphin: hey colorful sheep! 👋
ColorfulSheep: good to see you again starry dolphin! 🙌
```
## 17\) Optimizing performance
We're gonna bucket our performance optimizations into three categories: reducing heap allocations, reducing lock contention, and compiling for speed.
There's other categories, but I think these are the most relevant for our particular program.
### Reducing heap allocations
The fastest code is the code that never runs. If we don't have to allocate something on the heap then we don't have to call the allocator.
#### `String` -> `CompactString`
We have a lot of short strings. We can have thousands of users and hundreds of rooms on the server and user names and room names are both almost always shorter than 24 characters. Instead of storing them as `String`s we can store them as `CompactString`s. `String`s always store their data on the heap, but `CompactString`s will store strings shorter than 24 bytes on the stack, and will only heap allocate the string if it's longer than 24 bytes. If we enforce a maximum length of 24 ASCII characters for user and room names then we can guarantee we'll never have to perform any heap allocations for them.
#### `Sender<String>` -> `Sender<Arc<str>>`
As you may remember, when we `send` something to a broadcast channel every call to `recv` clones that data. That means if a user sends a `String` message that is five paragraphs long in a room with 1000 other users we're going to have to clone that message 1000 times which means doing 1000 heap allocations. We know that after a message is sent it's immutable, so we don't need to send a `String`, we can convert the `String` to an `Arc<str>` instead and send that, because cloning an `Arc<str>` is very cheap since all it does is increment an atomic counter.
#### Miscellaneous micro optimizations
After combing through the code I found a couple places where we were carelessly allocating unnecessary `Vec`s and `String`s, mostly for the `/rooms` and `/users` commands, and made them both only allocate a single `String` when generating a response.
### Reducing lock contention
High lock contention increases how long threads need to wait for a lock to be free. Reducing lock contention reduces thread waiting time and increases program throughput.
#### `Mutex<HashSet<String>>` -> `DashSet<CompactString>`
We store names in a `Mutex<HashSet<String>>`. That's one lock for potentially thousands of keys. Instead of putting a lock around the entire set, what if we could put a lock around every individual key in the set? A `DashSet` doesn't necessarily go that far, but it does split the data into several shards internally and each shard gets its own lock. Some ASCII diagrams to help explain:
```
+-------------------------------+
| Mutex |
| +---------------------------+ |
| | HashSet | |
| | +-----+-----+-----+-----+ | |
| | | key | key | key | key | | |
| | +-----+-----+-----+-----+ | |
| +---------------------------+ |
+-------------------------------+
+-----------------------------------+
| DashSet |
| +---------------+---------------+ |
| | RwLock | RwLock | |
| | +-----+-----+ | +-----+-----+ | |
| | | key | key | | | key | key | | |
| | +-----+-----+ | +-----+-----+ | |
| +-------------------+-----------+ |
+-----------------------------------+
```
In this case guarding the same amount of data but with more locks means each lock will get less contention between threads.
#### `RwLock<HashMap<String, Room>>` -> `DashMap<CompactString, Room>`
We store rooms in a `RwLock<HashMap<String, Room>>` but for the same reasons mentioned above we can use a `DashMap` to reduce lock contention.
#### Better random name generation
There's a big issue in our random name generation. It's a [Birthday Problem](https://en.wikipedia.org/wiki/Birthday_problem) just in different clothes. Even if we have 600 unique adjectives and 250 unique animals and we can generate 150k unique names using those, we'd expect the probability of a collision in the first 1k generated names to be very low, right? Unfortunately no, after generating just 460 names there's already a >50% chance that there will be a collision, and after generating just 1k names the probability of a collision is >96%. More collisions means more time threads will spend fighting to find a unique name for every user that joins the server as the number of active users on the server climbs.
I refactored the name generation to iterate through all possible name combinations in a pseudo-random fashion, so the generated names still appear random but now we can guarantee that for 600 unique adjectives and 250 unique animals that we will generate 150k unique names in a row without any collisions.
#### System allocator -> jemalloc
jemalloc is suppose to be faster for multithreaded programs because it uses per-thread arenas which reduces contention for memory allocation. That sounds good to me, so let's change our allocator to jemalloc.
### Compiling for speed
The default cargo `build` command is configured to quickly compile slow programs. Instead, we would like to slowly compile a fast program. In order to do that we need to add this to our Cargo.toml:
```toml
[profile.release]
codegen-units = 1
lto = "fat"
```
And then execute the `build` command with these flags:
```console
$ RUSTFLAGS="-C target-cpu=native" cargo build --release
```
Anyway, this section was a lot. You can see the full source code [here](https://github.com/pretzelhammer/chat-server/blob/main/examples/server-17.rs), and you can see a diff between the non-optimized and optimized versions by running `just diff 16 17`.
## 18\) Finishing touches
We've neglected logging, parsing command line arguments, and error handling thus far because they're boring and most people don't like reading about them. Let's speed through them.
Here's how we can setup `tracing` to log to `stdout`:
```rust
use std::io;
use tracing_subscriber::{fmt, EnvFilter, layer::SubscriberExt};
fn setup_logging() {
let subscriber = tracing_subscriber::registry()
.with(EnvFilter::from_default_env())
.with(fmt::Layer::new()
.without_time()
.compact()
.with_ansi(true)
.with_writer(io::stdout)
);
tracing::subscriber::set_global_default(subscriber)
.expect("Unable to set a global subscriber");
}
```
And then after running `setup_logging` somewhere early in our main function we can call the `trace!`, `debug!`, `info!`, `warn!`, and `error!` macros from `tracing`, which all function similarly to `println!`. We also can customize the logging level via the environment, using the `RUST_LOG` environment variable.
Right now our server always runs at `127.0.0.1` on port `42069`. We should let server admins be able to customize this without having to recompile our code. We can accept these parameters as command line arguments and parse them using the `clap` crate:
```rust
use std::net::{IpAddr, SocketAddr, Ipv4Addr};
use clap::Parser;
const DEFAULT_IP: IpAddr = IpAddr::V4(Ipv4Addr::new(127, 0, 0, 1));
const DEFAULT_PORT: u16 = 42069;
#[derive(Parser)]
#[command(long_about = None)]
struct Cli {
#[arg(short, long, default_value_t = DEFAULT_IP)]
ip: IpAddr,
#[arg(short, long, default_value_t = DEFAULT_PORT)]
port: u16,
}
fn parse_socket_addr() -> SocketAddr {
let cli = Cli::parse();
SocketAddr::new(cli.ip, cli.port)
}
```
For error handling there's a bunch of little mundane things that we neglected, none of which are particularly interesting to write about.
You can see the full source code with all logging and error handling [here](https://github.com/pretzelhammer/chat-server/blob/main/examples/server-18.rs). To see a diff against the previous version of the code run `just diff 17 18`.
## Conclusion
We learned a lot! The final full code for the server is [here](https://github.com/pretzelhammer/chat-server/blob/main/src/bin/chat-server.rs). You can run it with `just server`. To chat run `just chat`. And if it gets lonely run `just bots`.
## Discuss
Discuss this article on
- [Github](https://github.com/pretzelhammer/rust-blog/discussions/75)
- [official Rust users forum](https://users.rust-lang.org/t/beginners-guide-to-concurrent-programming-coding-a-multithreaded-chat-server-using-tokio/110976)
- [learnrust subreddit](https://www.reddit.com/r/learnrust/comments/1cmebbo/beginners_guide_to_concurrent_programming_coding/)
- [rust subreddit](https://www.reddit.com/r/rust/comments/1cnz9p9/beginners_guide_to_concurrent_programming_coding/)
## Further reading
- [Common Rust Lifetime Misconceptions](./common-rust-lifetime-misconceptions.md)
- [Tour of Rust's Standard Library Traits](./tour-of-rusts-standard-library-traits.md)
- [Learning Rust in 2024](./learning-rust-in-2024.md)
- [Using Rust in Non-Rust Servers to Improve Performance](./rust-in-non-rust-servers.md)
- [Sizedness in Rust](./sizedness-in-rust.md)
- [RESTful API in Sync & Async Rust](./restful-api-in-sync-and-async-rust.md)
- [Learn Assembly with Entirely Too Many Brainfuck Compilers](./too-many-brainfuck-compilers.md)
## Notifications
Get notified when a new blog post gets published by
- Subscribing to this repo's [releases RSS feed](https://github.com/pretzelhammer/rust-blog/releases.atom) or
- Watching this repo's releases (click `Watch` → click `Custom` → select `Releases` → click `Apply`)
================================================
FILE: posts/common-rust-lifetime-misconceptions.md
================================================
# Common Rust Lifetime Misconceptions
_19 May 2020 · #rust · #lifetimes_
**Table of Contents**
- [Intro](#intro)
- [The Misconceptions](#the-misconceptions)
- [1) `T` only contains owned types](#1-t-only-contains-owned-types)
- [2) if `T: 'static` then `T` must be valid for the entire program](#2-if-t-static-then-t-must-be-valid-for-the-entire-program)
- [3) `&'a T` and `T: 'a` are the same thing](#3-a-t-and-t-a-are-the-same-thing)
- [4) my code isn't generic and doesn't have lifetimes](#4-my-code-isnt-generic-and-doesnt-have-lifetimes)
- [5) if it compiles then my lifetime annotations are correct](#5-if-it-compiles-then-my-lifetime-annotations-are-correct)
- [6) boxed trait objects don't have lifetimes](#6-boxed-trait-objects-dont-have-lifetimes)
- [7) compiler error messages will tell me how to fix my program](#7-compiler-error-messages-will-tell-me-how-to-fix-my-program)
- [8) lifetimes can grow and shrink at run-time](#8-lifetimes-can-grow-and-shrink-at-run-time)
- [9) downgrading mut refs to shared refs is safe](#9-downgrading-mut-refs-to-shared-refs-is-safe)
- [10) closures follow the same lifetime elision rules as functions](#10-closures-follow-the-same-lifetime-elision-rules-as-functions)
- [Conclusion](#conclusion)
- [Discuss](#discuss)
- [Further Reading](#further-reading)
- [Notifications](#notifications)
## Intro
I've held all of these misconceptions at some point and I see many beginners struggle with these misconceptions today. Some of my terminology might be non-standard, so here's a table of shorthand phrases I use and what I intend for them to mean.
| Phrase | Shorthand for |
|-|-|
| `T` | 1) a set containing all possible types _or_<br>2) some type within that set |
| owned type | some non-reference type, e.g. `i32`, `String`, `Vec`, etc |
| 1) borrowed type _or_<br>2) ref type | some reference type regardless of mutability, e.g. `&i32`, `&mut i32`, etc |
| 1) mut ref _or_<br>2) exclusive ref | exclusive mutable reference, i.e. `&mut T` |
| 1) immut ref _or_<br>2) shared ref | shared immutable reference, i.e. `&T` |
## The Misconceptions
In a nutshell: A variable's lifetime is how long the data it points to can be statically verified by the compiler to be valid at its current memory address. I'll now spend the next ~6500 words going into more detail about where people commonly get confused.
### 1) `T` only contains owned types
This misconception is more about generics than lifetimes but generics and lifetimes are tightly intertwined in Rust so it's not possible to talk about one without also talking about the other. Anyway:
When I first started learning Rust I understood that `i32`, `&i32`, and `&mut i32` are different types. I also understood that some generic type variable `T` represents a set which contains all possible types. However, despite understanding both of these things separately, I wasn't able to understand them together. In my newbie Rust mind this is how I thought generics worked:
| | | | |
|-|-|-|-|
| **Type Variable** | `T` | `&T` | `&mut T` |
| **Examples** | `i32` | `&i32` | `&mut i32` |
`T` contains all owned types. `&T` contains all immutably borrowed types. `&mut T` contains all mutably borrowed types. `T`, `&T`, and `&mut T` are disjoint finite sets. Nice, simple, clean, easy, intuitive, and completely totally wrong. This is how generics actually work in Rust:
| | | | |
|-|-|-|-|
| **Type Variable** | `T` | `&T` | `&mut T` |
| **Examples** | `i32`, `&i32`, `&mut i32`, `&&i32`, `&mut &mut i32`, ... | `&i32`, `&&i32`, `&&mut i32`, ... | `&mut i32`, `&mut &mut i32`, `&mut &i32`, ... |
`T`, `&T`, and `&mut T` are all infinite sets, since it's possible to borrow a type ad-infinitum. `T` is a superset of both `&T` and `&mut T`. `&T` and `&mut T` are disjoint sets. Here's a couple examples which validate these concepts:
```rust
trait Trait {}
impl<T> Trait for T {}
impl<T> Trait for &T {} // ❌
impl<T> Trait for &mut T {} // ❌
```
The above program doesn't compile as expected:
```none
error[E0119]: conflicting implementations of trait `Trait` for type `&_`
--> src/lib.rs:3:1
|
2 | impl<T> Trait for T {}
| ------------------- first implementation here
3 | impl<T> Trait for &T {}
| ^^^^^^^^^^^^^^^^^^^^ conflicting implementation for `&_`
error[E0119]: conflicting implementations of trait `Trait` for type `&mut _`
--> src/lib.rs:4:1
|
2 | impl<T> Trait for T {}
| ------------------- first implementation here
3 | impl<T> Trait for &T {}
4 | impl<T> Trait for &mut T {}
| ^^^^^^^^^^^^^^^^^^^^^^^^ conflicting implementation for `&mut _`
```
The compiler doesn't allow us to define an implementation of `Trait` for `&T` and `&mut T` since it would conflict with the implementation of `Trait` for `T` which already includes all of `&T` and `&mut T`. The program below compiles as expected, since `&T` and `&mut T` are disjoint:
```rust
trait Trait {}
impl<T> Trait for &T {} // ✅
impl<T> Trait for &mut T {} // ✅
```
Although it could probably go without saying, but for the sake of making sure nobody erroneously extrapolates the last couple examples, implementations for concrete types cannot overlap and this compiles just fine:
```rust
trait Trait {}
struct Struct;
impl Trait for Struct {} // ✅
impl Trait for &Struct {} // ✅
impl Trait for &mut Struct {} // ✅
```
**Key Takeaways**
- `T` is a superset of both `&T` and `&mut T`
- `&T` and `&mut T` are disjoint sets
### 2) if `T: 'static` then `T` must be valid for the entire program
**Misconception Corollaries**
- `T: 'static` should be read as _"`T` has a `'static` lifetime"_
- `&'static T` and `T: 'static` are the same thing
- if `T: 'static` then `T` must be immutable
- if `T: 'static` then `T` can only be created at compile time
Most Rust beginners get introduced to the `'static` lifetime for the first time in a code example that looks something like this:
```rust
fn main() {
let str_literal: &'static str = "str literal";
}
```
They get told that `"str literal"` is hardcoded into the compiled binary and is loaded into read-only memory at run-time so it's immutable and valid for the entire program and that's what makes it `'static`. These concepts are further reinforced by the rules surrounding defining `static` variables using the `static` keyword.
```rust
// Note: This example is purely for illustrative purposes.
// Never use `static mut`. It's a footgun. There are
// safe patterns for global mutable singletons in Rust but
// those are outside the scope of this article.
static BYTES: [u8; 3] = [1, 2, 3];
static mut MUT_BYTES: [u8; 3] = [1, 2, 3];
fn main() {
MUT_BYTES[0] = 99; // ❌ - mutating static is unsafe
unsafe {
MUT_BYTES[0] = 99;
assert_eq!(99, MUT_BYTES[0]);
}
}
```
Regarding `static` variables
- they can only be created at compile-time
- they should be immutable, mutating them is unsafe
- they're valid for the entire program
The `'static` lifetime was probably named after the default lifetime of `static` variables, right? So it makes sense that the `'static` lifetime has to follow all the same rules, right?
Well yes, but a type _with_ a `'static` lifetime is different from a type _bounded by_ a `'static` lifetime. The latter can be dynamically allocated at run-time, can be safely and freely mutated, can be dropped, and can live for arbitrary durations.
It's important at this point to distinguish `&'static T` from `T: 'static`.
`&'static T` is an immutable reference to some `T` that can be safely held indefinitely long, including up until the end of the program. This is only possible if `T` itself is immutable and does not move _after the reference was created_. `T` does not need to be created at compile-time. It's possible to generate random dynamically allocated data at run-time and return `'static` references to it via a memory leak, e.g.
```rust
use rand;
// generate random 'static str refs at run-time
fn rand_str_generator() -> &'static str {
let rand_string = rand::random::<u64>().to_string();
Box::leak(rand_string.into_boxed_str())
}
```
`T: 'static` is some `T` that can be safely held indefinitely long, including up until the end of the program. `T: 'static` includes all `&'static T` however it also includes all owned types, like `String`, `Vec`, etc. The owner of some data is guaranteed that data will never get invalidated as long as the owner holds onto it, therefore the owner can safely hold onto the data indefinitely long, including up until the end of the program. `T: 'static` should be read as _"`T` can live at least as long as a `'static` lifetime"_ not _"`T` has a `'static` lifetime"_. A program to help illustrate these concepts:
```rust
use rand;
fn drop_static<T: 'static>(t: T) {
std::mem::drop(t);
}
fn main() {
let mut strings: Vec<String> = Vec::new();
for _ in 0..10 {
if rand::random() {
// all the strings are randomly generated
// and dynamically allocated at run-time
let string = rand::random::<u64>().to_string();
strings.push(string);
}
}
// strings are owned types so they can
// live at least as long as 'static
for mut string in strings {
// all the strings are mutable
string.push_str("a mutation");
// all the strings are droppable
drop_static(string); // ✅
}
// all the strings have been invalidated before the end of the program
println!("I am the end of the program");
}
```
**Key Takeaways**
- `T: 'static` should be read as _"`T` can live at least as long as a `'static` lifetime"_
- if `T: 'static` then `T` can be a borrowed type with a `'static` lifetime _or_ an owned type
- since `T: 'static` includes owned types that means `T`
- can be dynamically allocated at run-time
- does not have to be valid for the entire program
- can be safely and freely mutated
- can be dynamically dropped at run-time
- can have lifetimes of different durations
### 3) `&'a T` and `T: 'a` are the same thing
This misconception is a generalized version of the one above.
`&'a T` requires and implies `T: 'a` since a reference to `T` of lifetime `'a` cannot be valid for `'a` if `T` itself is not valid for `'a`. For example, the Rust compiler will never allow the construction of the type `&'static Ref<'a, T>` because if `Ref` is only valid for `'a` we can't make a `'static` reference to it.
`T: 'a` includes all `&'a T` but the reverse is not true.
```rust
// only takes ref types that can outlive 'a
fn t_ref<'a, T: 'a>(t: &'a T) {}
// takes any types that can outlive 'a
fn t_bound<'a, T: 'a>(t: T) {}
// owned type which contains a reference
struct Ref<'a, T: 'a>(&'a T);
fn main() {
let string = String::from("string");
t_bound(&string); // ✅
t_bound(Ref(&string)); // ✅
t_bound(&Ref(&string)); // ✅
t_ref(&string); // ✅
t_ref(Ref(&string)); // ❌ - expected ref, found struct
t_ref(&Ref(&string)); // ✅
// string can outlive 'static which is longer than 'a
t_bound(string); // ✅
}
```
**Key Takeaways**
- `T: 'a` is more general and more flexible than `&'a T`
- `T: 'a` accepts owned types, owned types which contain references, and references
- `&'a T` only accepts references
- if `T: 'static` then `T: 'a` since `'static` >= `'a` for all `'a`
### 4) my code isn't generic and doesn't have lifetimes
**Misconception Corollaries**
- it's possible to avoid using generics and lifetimes
This comforting misconception is kept alive thanks to Rust's lifetime elision rules, which allow you to omit lifetime annotations in functions because the Rust borrow checker will infer them following these rules:
- every input ref to a function gets a distinct lifetime
- if there's exactly one input lifetime it gets applied to all output refs
- if there's multiple input lifetimes but one of them is `&self` or `&mut self` then the lifetime of `self` is applied to all output refs
- otherwise output lifetimes have to be made explicit
That's a lot to take in so let's look at some examples:
```rust
// elided
fn print(s: &str);
// expanded
fn print<'a>(s: &'a str);
// elided
fn trim(s: &str) -> &str;
// expanded
fn trim<'a>(s: &'a str) -> &'a str;
// illegal, can't determine output lifetime, no inputs
fn get_str() -> &str;
// explicit options include
fn get_str<'a>() -> &'a str; // generic version
fn get_str() -> &'static str; // 'static version
// illegal, can't determine output lifetime, multiple inputs
fn overlap(s: &str, t: &str) -> &str;
// explicit (but still partially elided) options include
fn overlap<'a>(s: &'a str, t: &str) -> &'a str; // output can't outlive s
fn overlap<'a>(s: &str, t: &'a str) -> &'a str; // output can't outlive t
fn overlap<'a>(s: &'a str, t: &'a str) -> &'a str; // output can't outlive s & t
fn overlap(s: &str, t: &str) -> &'static str; // output can outlive s & t
fn overlap<'a>(s: &str, t: &str) -> &'a str; // no relationship between input & output lifetimes
// expanded
fn overlap<'a, 'b>(s: &'a str, t: &'b str) -> &'a str;
fn overlap<'a, 'b>(s: &'a str, t: &'b str) -> &'b str;
fn overlap<'a>(s: &'a str, t: &'a str) -> &'a str;
fn overlap<'a, 'b>(s: &'a str, t: &'b str) -> &'static str;
fn overlap<'a, 'b, 'c>(s: &'a str, t: &'b str) -> &'c str;
// elided
fn compare(&self, s: &str) -> &str;
// expanded
fn compare<'a, 'b>(&'a self, &'b str) -> &'a str;
```
If you've ever written
- a function which takes or returns references
- a struct method which takes or returns references
- a generic function
- a trait object (more on this later)
- a closure (more on this later)
then your code has generic elided lifetime annotations all over it.
**Key Takeaways**
- almost all Rust code is generic code and there's elided lifetime annotations everywhere
### 5) if it compiles then my lifetime annotations are correct
**Misconception Corollaries**
- Rust's lifetime elision rules for functions are always right
- Rust's borrow checker is always right, technically _and semantically_
- Rust knows more about the semantics of my program than I do
It's possible for a Rust program to be technically compilable but still semantically wrong. Take this for example:
```rust
struct ByteIter<'a> {
remainder: &'a [u8]
}
impl<'a> ByteIter<'a> {
fn next(&mut self) -> Option<&u8> {
if self.remainder.is_empty() {
None
} else {
let byte = &self.remainder[0];
self.remainder = &self.remainder[1..];
Some(byte)
}
}
}
fn main() {
let mut bytes = ByteIter { remainder: b"1" };
assert_eq!(Some(&b'1'), bytes.next());
assert_eq!(None, bytes.next());
}
```
`ByteIter` is an iterator that iterates over a slice of bytes. We're skipping the `Iterator` trait implementation for conciseness. It seems to work fine, but what if we want to check a couple bytes at a time?
```rust
fn main() {
let mut bytes = ByteIter { remainder: b"1123" };
let byte_1 = bytes.next();
let byte_2 = bytes.next();
if byte_1 == byte_2 { // ❌
// do something
}
}
```
Uh oh! Compile error:
```none
error[E0499]: cannot borrow `bytes` as mutable more than once at a time
--> src/main.rs:20:18
|
19 | let byte_1 = bytes.next();
| ----- first mutable borrow occurs here
20 | let byte_2 = bytes.next();
| ^^^^^ second mutable borrow occurs here
21 | if byte_1 == byte_2 {
| ------ first borrow later used here
```
I guess we can copy each byte. Copying is okay when we're working with bytes but if we turned `ByteIter` into a generic slice iterator that can iterate over any `&'a [T]` then we might want to use it in the future with types that may be very expensive or impossible to copy and clone. Oh well, I guess there's nothing we can do about that, the code compiles so the lifetime annotations must be right, right?
Nope, the current lifetime annotations are actually the source of the bug! It's particularly hard to spot because the buggy lifetime annotations are elided. Let's expand the elided lifetimes to get a clearer look at the problem:
```rust
struct ByteIter<'a> {
remainder: &'a [u8]
}
impl<'a> ByteIter<'a> {
fn next<'b>(&'b mut self) -> Option<&'b u8> {
if self.remainder.is_empty() {
None
} else {
let byte = &self.remainder[0];
self.remainder = &self.remainder[1..];
Some(byte)
}
}
}
```
That didn't help at all. I'm still confused. Here's a hot tip that only Rust pros know: give your lifetime annotations descriptive names. Let's try again:
```rust
struct ByteIter<'remainder> {
remainder: &'remainder [u8]
}
impl<'remainder> ByteIter<'remainder> {
fn next<'mut_self>(&'mut_self mut self) -> Option<&'mut_self u8> {
if self.remainder.is_empty() {
None
} else {
let byte = &self.remainder[0];
self.remainder = &self.remainder[1..];
Some(byte)
}
}
}
```
Each returned byte is annotated with `'mut_self` but the bytes are clearly coming from `'remainder`! Let's fix it.
```rust
struct ByteIter<'remainder> {
remainder: &'remainder [u8]
}
impl<'remainder> ByteIter<'remainder> {
fn next(&mut self) -> Option<&'remainder u8> {
if self.remainder.is_empty() {
None
} else {
let byte = &self.remainder[0];
self.remainder = &self.remainder[1..];
Some(byte)
}
}
}
fn main() {
let mut bytes = ByteIter { remainder: b"1123" };
let byte_1 = bytes.next();
let byte_2 = bytes.next();
std::mem::drop(bytes); // we can even drop the iterator now!
if byte_1 == byte_2 { // ✅
// do something
}
}
```
Now that we look back on the previous version of our program it was obviously wrong, so why did Rust compile it? The answer is simple: it was memory safe.
The Rust borrow checker only cares about the lifetime annotations in a program to the extent it can use them to statically verify the memory safety of the program. Rust will happily compile programs even if the lifetime annotations have semantic errors, and the consequence of this is that the program becomes unnecessarily restrictive.
Here's a quick example that's the opposite of the previous example: Rust's lifetime elision rules happen to be semantically correct in this instance but we unintentionally write a very restrictive method with our own unnecessary explicit lifetime annotations.
```rust
#[derive(Debug)]
struct NumRef<'a>(&'a i32);
impl<'a> NumRef<'a> {
// my struct is generic over 'a so that means I need to annotate
// my self parameters with 'a too, right? (answer: no, not right)
fn some_method(&'a mut self) {}
}
fn main() {
let mut num_ref = NumRef(&5);
num_ref.some_method(); // mutably borrows num_ref for the rest of its lifetime
num_ref.some_method(); // ❌
println!("{:?}", num_ref); // ❌
}
```
If we have some struct generic over `'a` we almost never want to write a method with a `&'a mut self` receiver. What we're communicating to Rust is _"this method will mutably borrow the struct for the entirety of the struct's lifetime"_. In practice this means Rust's borrow checker will only allow at most one call to `some_method` before the struct becomes permanently mutably borrowed and thus unusable. The use-cases for this are extremely rare but the code above is very easy for confused beginners to write and it compiles. The fix is to not add unnecessary explicit lifetime annotations and let Rust's lifetime elision rules handle it:
```rust
#[derive(Debug)]
struct NumRef<'a>(&'a i32);
impl<'a> NumRef<'a> {
// no more 'a on mut self
fn some_method(&mut self) {}
// above line desugars to
fn some_method_desugared<'b>(&'b mut self){}
}
fn main() {
let mut num_ref = NumRef(&5);
num_ref.some_method();
num_ref.some_method(); // ✅
println!("{:?}", num_ref); // ✅
}
```
**Key Takeaways**
- Rust's lifetime elision rules for functions are not always right for every situation
- Rust does not know more about the semantics of your program than you do
- give your lifetime annotations descriptive names
- try to be mindful of where you place explicit lifetime annotations and why
### 6) boxed trait objects don't have lifetimes
Earlier we discussed Rust's lifetime elision rules _for functions_. Rust also has lifetime elision rules for trait objects, which are:
- if a trait object is used as a type argument to a generic type then its lifetime bound is inferred from the containing type
- if there's a unique bound from the containing then that's used
- if there's more than one bound from the containing type then an explicit bound must be specified
- if the above doesn't apply then
- if the trait is defined with a single lifetime bound then that bound is used
- if `'static` is used for any lifetime bound then `'static` is used
- if the trait has no lifetime bounds then its lifetime is inferred in expressions and is `'static` outside of expressions
All of that sounds super complicated but can be simply summarized as _"a trait object's lifetime bound is inferred from context."_ After looking at a handful of examples we'll see the lifetime bound inferences are pretty intuitive so we don't have to memorize the formal rules:
```rust
use std::cell::Ref;
trait Trait {}
// elided
type T1 = Box<dyn Trait>;
// expanded, Box<T> has no lifetime bound on T, so inferred as 'static
type T2 = Box<dyn Trait + 'static>;
// elided
impl dyn Trait {}
// expanded
impl dyn Trait + 'static {}
// elided
type T3<'a> = &'a dyn Trait;
// expanded, &'a T requires T: 'a, so inferred as 'a
type T4<'a> = &'a (dyn Trait + 'a);
// elided
type T5<'a> = Ref<'a, dyn Trait>;
// expanded, Ref<'a, T> requires T: 'a, so inferred as 'a
type T6<'a> = Ref<'a, dyn Trait + 'a>;
trait GenericTrait<'a>: 'a {}
// elided
type T7<'a> = Box<dyn GenericTrait<'a>>;
// expanded
type T8<'a> = Box<dyn GenericTrait<'a> + 'a>;
// elided
impl<'a> dyn GenericTrait<'a> {}
// expanded
impl<'a> dyn GenericTrait<'a> + 'a {}
```
Concrete types which implement traits can have references and thus they also have lifetime bounds, and so their corresponding trait objects have lifetime bounds. Also you can implement traits directly for references which obviously have lifetime bounds:
```rust
trait Trait {}
struct Struct {}
struct Ref<'a, T>(&'a T);
impl Trait for Struct {}
impl Trait for &Struct {} // impl Trait directly on a ref type
impl<'a, T> Trait for Ref<'a, T> {} // impl Trait on a type containing refs
```
Anyway, this is worth going over because it often confuses beginners when they refactor a function from using trait objects to generics or vice versa. Take this program for example:
```rust
use std::fmt::Display;
fn dynamic_thread_print(t: Box<dyn Display + Send>) {
std::thread::spawn(move || {
println!("{}", t);
}).join();
}
fn static_thread_print<T: Display + Send>(t: T) { // ❌
std::thread::spawn(move || {
println!("{}", t);
}).join();
}
```
It throws this compile error:
```none
error[E0310]: the parameter type `T` may not live long enough
--> src/lib.rs:10:5
|
9 | fn static_thread_print<T: Display + Send>(t: T) {
| -- help: consider adding an explicit lifetime bound...: `T: 'static +`
10 | std::thread::spawn(move || {
| ^^^^^^^^^^^^^^^^^^
|
note: ...so that the type `[closure@src/lib.rs:10:24: 12:6 t:T]` will meet its required lifetime bounds
--> src/lib.rs:10:5
|
10 | std::thread::spawn(move || {
| ^^^^^^^^^^^^^^^^^^
```
Okay great, the compiler tells us how to fix the issue so let's fix the issue.
```rust
use std::fmt::Display;
fn dynamic_thread_print(t: Box<dyn Display + Send>) {
std::thread::spawn(move || {
println!("{}", t);
}).join();
}
fn static_thread_print<T: Display + Send + 'static>(t: T) { // ✅
std::thread::spawn(move || {
println!("{}", t);
}).join();
}
```
It compiles now but these two functions look awkward next to each other, why does the second function require a `'static` bound on `T` where the first function doesn't? That's a trick question. Using the lifetime elision rules Rust automatically infers a `'static` bound in the first function so both actually have `'static` bounds. This is what the Rust compiler sees:
```rust
use std::fmt::Display;
fn dynamic_thread_print(t: Box<dyn Display + Send + 'static>) {
std::thread::spawn(move || {
println!("{}", t);
}).join();
}
fn static_thread_print<T: Display + Send + 'static>(t: T) {
std::thread::spawn(move || {
println!("{}", t);
}).join();
}
```
**Key Takeaways**
- all trait objects have some inferred default lifetime bounds
### 7) compiler error messages will tell me how to fix my program
**Misconception Corollaries**
- Rust's lifetime elision rules for trait objects are always right
- Rust knows more about the semantics of my program than I do
This misconception is the previous two misconceptions combined into one example:
```rust
use std::fmt::Display;
fn box_displayable<T: Display>(t: T) -> Box<dyn Display> { // ❌
Box::new(t)
}
```
Throws this error:
```none
error[E0310]: the parameter type `T` may not live long enough
--> src/lib.rs:4:5
|
3 | fn box_displayable<T: Display>(t: T) -> Box<dyn Display> {
| -- help: consider adding an explicit lifetime bound...: `T: 'static +`
4 | Box::new(t)
| ^^^^^^^^^^^
|
note: ...so that the type `T` will meet its required lifetime bounds
--> src/lib.rs:4:5
|
4 | Box::new(t)
| ^^^^^^^^^^^
```
Okay, let's fix it how the compiler is telling us to fix it, nevermind the fact that it's automatically inferring a `'static` lifetime bound for our boxed trait object without telling us and its recommended fix is based on that unstated fact:
```rust
use std::fmt::Display;
fn box_displayable<T: Display + 'static>(t: T) -> Box<dyn Display> { // ✅
Box::new(t)
}
```
So the program compiles now... but is this what we actually want? Probably, but maybe not. The compiler didn't mention any other fixes but this would have also been appropriate:
```rust
use std::fmt::Display;
fn box_displayable<'a, T: Display + 'a>(t: T) -> Box<dyn Display + 'a> { // ✅
Box::new(t)
}
```
This function accepts all the same arguments as the previous version plus a lot more! Does that make it better? Not necessarily, it depends on the requirements and constraints of our program. This example is a bit abstract so let's take a look at a simpler and more obvious case:
```rust
fn return_first(a: &str, b: &str) -> &str { // ❌
a
}
```
Throws:
```none
error[E0106]: missing lifetime specifier
--> src/lib.rs:1:38
|
1 | fn return_first(a: &str, b: &str) -> &str {
| ---- ---- ^ expected named lifetime parameter
|
= help: this function's return type contains a borrowed value, but the signature does not say whether it is borrowed from `a` or `b`
help: consider introducing a named lifetime parameter
|
1 | fn return_first<'a>(a: &'a str, b: &'a str) -> &'a str {
| ^^^^ ^^^^^^^ ^^^^^^^ ^^^
```
The error message recommends annotating both inputs and the output with the same lifetime. If we did this our program would compile but this function would overly-constrain the return type. What we actually want is this:
```rust
fn return_first<'a>(a: &'a str, b: &str) -> &'a str { // ✅
a
}
```
**Key Takeaways**
- Rust's lifetime elision rules for trait objects are not always right for every situation
- Rust does not know more about the semantics of your program than you do
- Rust compiler error messages suggest fixes which will make your program compile which is not that same as fixes which will make you program compile _and_ best suit the requirements of your program
### 8) lifetimes can grow and shrink at run-time
**Misconception Corollaries**
- container types can swap references at run-time to change their lifetime
- Rust borrow checker does advanced control flow analysis
This does not compile:
```rust
struct Has<'lifetime> {
lifetime: &'lifetime str,
}
fn main() {
let long = String::from("long");
let mut has = Has { lifetime: &long };
assert_eq!(has.lifetime, "long");
{
let short = String::from("short");
// "switch" to short lifetime
has.lifetime = &short;
assert_eq!(has.lifetime, "short");
// "switch back" to long lifetime (but not really)
has.lifetime = &long;
assert_eq!(has.lifetime, "long");
// `short` dropped here
}
assert_eq!(has.lifetime, "long"); // ❌ - `short` still "borrowed" after drop
}
```
It throws:
```none
error[E0597]: `short` does not live long enough
--> src/main.rs:11:24
|
11 | has.lifetime = &short;
| ^^^^^^ borrowed value does not live long enough
...
15 | }
| - `short` dropped here while still borrowed
16 | assert_eq!(has.lifetime, "long");
| --------------------------------- borrow later used here
```
This also does not compile, throws the exact same error as above:
```rust
struct Has<'lifetime> {
lifetime: &'lifetime str,
}
fn main() {
let long = String::from("long");
let mut has = Has { lifetime: &long };
assert_eq!(has.lifetime, "long");
// this block will never run
if false {
let short = String::from("short");
// "switch" to short lifetime
has.lifetime = &short;
assert_eq!(has.lifetime, "short");
// "switch back" to long lifetime (but not really)
has.lifetime = &long;
assert_eq!(has.lifetime, "long");
// `short` dropped here
}
assert_eq!(has.lifetime, "long"); // ❌ - `short` still "borrowed" after drop
}
```
Lifetimes have to be statically verified at compile-time and the Rust borrow checker only does very basic control flow analysis, so it assumes every block in an `if-else` statement and every match arm in a `match` statement can be taken and then chooses the shortest possible lifetime for the variable. Once a variable is bounded by a lifetime it is bounded by that lifetime _forever_. The lifetime of a variable can only shrink, and all the shrinkage is determined at compile-time.
**Key Takeaways**
- lifetimes are statically verified at compile-time
- lifetimes cannot grow or shrink or change in any way at run-time
- Rust borrow checker will always choose the shortest possible lifetime for a variable assuming all code paths can be taken
### 9) downgrading mut refs to shared refs is safe
**Misconception Corollaries**
- re-borrowing a reference ends its lifetime and starts a new one
You can pass a mut ref to a function expecting a shared ref because Rust will implicitly re-borrow the mut ref as immutable:
```rust
fn takes_shared_ref(n: &i32) {}
fn main() {
let mut a = 10;
takes_shared_ref(&mut a); // ✅
takes_shared_ref(&*(&mut a)); // above line desugared
}
```
Intuitively this makes sense, since there's no harm in re-borrowing a mut ref as immutable, right? Surprisingly no, as the program below does not compile:
```rust
fn main() {
let mut a = 10;
let b: &i32 = &*(&mut a); // re-borrowed as immutable
let c: &i32 = &a;
dbg!(b, c); // ❌
}
```
Throws this error:
```none
error[E0502]: cannot borrow `a` as immutable because it is also borrowed as mutable
--> src/main.rs:4:19
|
3 | let b: &i32 = &*(&mut a);
| -------- mutable borrow occurs here
4 | let c: &i32 = &a;
| ^^ immutable borrow occurs here
5 | dbg!(b, c);
| - mutable borrow later used here
```
A mutable borrow does occur, but it's immediately and unconditionally re-borrowed as immutable and then dropped. Why is Rust treating the immutable re-borrow as if it still has the mut ref's exclusive lifetime? While there's no issue in the particular example above, allowing the ability to downgrade mut refs to shared refs does indeed introduce potential memory safety issues:
```rust
use std::sync::Mutex;
struct Struct {
mutex: Mutex<String>
}
impl Struct {
// downgrades mut self to shared str
fn get_string(&mut self) -> &str {
self.mutex.get_mut().unwrap()
}
fn mutate_string(&self) {
// if Rust allowed downgrading mut refs to shared refs
// then the following line would invalidate any shared
// refs returned from the get_string method
*self.mutex.lock().unwrap() = "surprise!".to_owned();
}
}
fn main() {
let mut s = Struct {
mutex: Mutex::new("string".to_owned())
};
let str_ref = s.get_string(); // mut ref downgraded to shared ref
s.mutate_string(); // str_ref invalidated, now a dangling pointer
dbg!(str_ref); // ❌ - as expected!
}
```
The point here is that when you re-borrow a mut ref as a shared ref you don't get that shared ref without a big gotcha: it extends the mut ref's lifetime for the duration of the re-borrow even if the mut ref itself is dropped. Using the re-borrowed shared ref is very difficult because it's immutable but it can't overlap with any other shared refs. The re-borrowed shared ref has all the cons of a mut ref and all the cons of a shared ref and has the pros of neither. I believe re-borrowing a mut ref as a shared ref should be considered a Rust anti-pattern. Being aware of this anti-pattern is important so that you can easily spot it when you see code like this:
```rust
// downgrades mut T to shared T
fn some_function<T>(some_arg: &mut T) -> &T;
struct Struct;
impl Struct {
// downgrades mut self to shared self
fn some_method(&mut self) -> &Self;
// downgrades mut self to shared T
fn other_method(&mut self) -> &T;
}
```
Even if you avoid re-borrows in function and method signatures Rust still does automatic implicit re-borrows so it's easy to bump into this problem without realizing it like so:
```rust
use std::collections::HashMap;
type PlayerID = i32;
#[derive(Debug, Default)]
struct Player {
score: i32,
}
fn start_game(player_a: PlayerID, player_b: PlayerID, server: &mut HashMap<PlayerID, Player>) {
// get players from server or create & insert new players if they don't yet exist
let player_a: &Player = server.entry(player_a).or_default();
let player_b: &Player = server.entry(player_b).or_default();
// do something with players
dbg!(player_a, player_b); // ❌
}
```
The above fails to compile. `or_default()` returns a `&mut Player` which we're implicitly re-borrowing as `&Player` because of our explicit type annotations. To do what we want we have to:
```rust
use std::collections::HashMap;
type PlayerID = i32;
#[derive(Debug, Default)]
struct Player {
score: i32,
}
fn start_game(player_a: PlayerID, player_b: PlayerID, server: &mut HashMap<PlayerID, Player>) {
// drop the returned mut Player refs since we can't use them together anyway
server.entry(player_a).or_default();
server.entry(player_b).or_default();
// fetch the players again, getting them immutably this time, without any implicit re-borrows
let player_a = server.get(&player_a);
let player_b = server.get(&player_b);
// do something with players
dbg!(player_a, player_b); // ✅
}
```
Kinda awkward and clunky but this is the sacrifice we make at the Altar of Memory Safety.
**Key Takeaways**
- try not to re-borrow mut refs as shared refs, or you're gonna have a bad time
- re-borrowing a mut ref doesn't end its lifetime, even if the ref is dropped
### 10) closures follow the same lifetime elision rules as functions
This is more of a Rust Gotcha than a misconception.
Closures, despite being functions, do not follow the same lifetime elision rules as functions.
```rust
fn function(x: &i32) -> &i32 {
x
}
fn main() {
let closure = |x: &i32| x; // ❌
}
```
Throws:
```none
error: lifetime may not live long enough
--> src/main.rs:6:29
|
6 | let closure = |x: &i32| x;
| - - ^ returning this value requires that `'1` must outlive `'2`
| | |
| | return type of closure is &'2 i32
| let's call the lifetime of this reference `'1`
```
After desugaring we get:
```rust
// input lifetime gets applied to output
fn function<'a>(x: &'a i32) -> &'a i32 {
x
}
fn main() {
// input and output each get their own distinct lifetimes
let closure = for<'a, 'b> |x: &'a i32| -> &'b i32 { x };
// note: the above line is not valid syntax, but we need it for illustrative purposes
}
```
There's no good reason for this discrepancy. Closures were first implemented with different type inference semantics than functions and now we're stuck with it forever because to unify them at this point would be a breaking change. So how can we explicitly annotate a closure's type? Our options include:
```rust
fn main() {
// cast to trait object, becomes unsized, oops, compile error
let identity: dyn Fn(&i32) -> &i32 = |x: &i32| x;
// can allocate it on the heap as a workaround but feels clunky
let identity: Box<dyn Fn(&i32) -> &i32> = Box::new(|x: &i32| x);
// can skip the allocation and just create a static reference
let identity: &dyn Fn(&i32) -> &i32 = &|x: &i32| x;
// previous line desugared :)
let identity: &'static (dyn for<'a> Fn(&'a i32) -> &'a i32 + 'static) = &|x: &i32| -> &i32 { x };
// this would be ideal but it's invalid syntax
let identity: impl Fn(&i32) -> &i32 = |x: &i32| x;
// this would also be nice but it's also invalid syntax
let identity = for<'a> |x: &'a i32| -> &'a i32 { x };
// since "impl trait" works in the function return position
fn return_identity() -> impl Fn(&i32) -> &i32 {
|x| x
}
let identity = return_identity();
// more generic version of the previous solution
fn annotate<T, F>(f: F) -> F where F: Fn(&T) -> &T {
f
}
let identity = annotate(|x: &i32| x);
}
```
As I'm sure you've already noticed from the examples above, when closure types are used as trait bounds they do follow the usual function lifetime elision rules.
There's no real lesson or insight to be had here, it just is what it is.
**Key Takeaways**
- every language has gotchas 🤷
## Conclusion
- `T` is a superset of both `&T` and `&mut T`
- `&T` and `&mut T` are disjoint sets
- `T: 'static` should be read as _"`T` can live at least as long as a `'static` lifetime"_
- if `T: 'static` then `T` can be a borrowed type with a `'static` lifetime _or_ an owned type
- since `T: 'static` includes owned types that means `T`
- can be dynamically allocated at run-time
- does not have to be valid for the entire program
- can be safely and freely mutated
- can be dynamically dropped at run-time
- can have lifetimes of different durations
- `T: 'a` is more general and more flexible than `&'a T`
- `T: 'a` accepts owned types, owned types which contain references, and references
- `&'a T` only accepts references
- if `T: 'static` then `T: 'a` since `'static` >= `'a` for all `'a`
- almost all Rust code is generic code and there's elided lifetime annotations everywhere
- Rust's lifetime elision rules are not always right for every situation
- Rust does not know more about the semantics of your program than you do
- give your lifetime annotations descriptive names
- try to be mindful of where you place explicit lifetime annotations and why
- all trait objects have some inferred default lifetime bounds
- Rust compiler error messages suggest fixes which will make your program compile which is not that same as fixes which will make you program compile _and_ best suit the requirements of your program
- lifetimes are statically verified at compile-time
- lifetimes cannot grow or shrink or change in any way at run-time
- Rust borrow checker will always choose the shortest possible lifetime for a variable assuming all code paths can be taken
- try not to re-borrow mut refs as shared refs, or you're gonna have a bad time
- re-borrowing a mut ref doesn't end its lifetime, even if the ref is dropped
- every language has gotchas 🤷
## Discuss
Discuss this article on
- [learnrust subreddit](https://www.reddit.com/r/learnrust/comments/gmrcrq/common_rust_lifetime_misconceptions/)
- [official Rust users forum](https://users.rust-lang.org/t/blog-post-common-rust-lifetime-misconceptions/42950)
- [rust subreddit](https://www.reddit.com/r/rust/comments/golrsx/common_rust_lifetime_misconceptions/)
- [Hackernews](https://news.ycombinator.com/item?id=23279731)
- [Github](https://github.com/pretzelhammer/rust-blog/discussions)
## Further Reading
- [Tour of Rust's Standard Library Traits](./tour-of-rusts-standard-library-traits.md)
- [Beginner's Guide to Concurrent Programming: Coding a Multithreaded Chat Server using Tokio](./chat-server.md)
- [Learning Rust in 2024](./learning-rust-in-2024.md)
- [Using Rust in Non-Rust Servers to Improve Performance](./rust-in-non-rust-servers.md)
- [Sizedness in Rust](./sizedness-in-rust.md)
- [RESTful API in Sync & Async Rust](./restful-api-in-sync-and-async-rust.md)
- [Learn Assembly with Entirely Too Many Brainfuck Compilers](./too-many-brainfuck-compilers.md)
## Notifications
Get notified when a new blog post gets published by
- Subscribing to this repo's [releases RSS feed](https://github.com/pretzelhammer/rust-blog/releases.atom) or
- Watching this repo's releases (click `Watch` → click `Custom` → select `Releases` → click `Apply`)
================================================
FILE: posts/learning-rust-in-2020.md
================================================
# Learning Rust in 2020
_09 May 2020 · #rust · #programming · #exercises_
**Table of Contents**
- [Intro](#intro)
- [TL;DR](#tldr)
- [Practical Rust Resource Reviews](#practical-rust-resource-reviews)
- [HackerRank](#hackerrank)
- [Project Euler](#project-euler)
- [LeetCode](#leetcode)
- [Codewars](#codewars)
- [Advent of Code](#advent-of-code)
- [Rustlings](#rustlings)
- [Exercism](#exercism)
- [Conclusion](#conclusion)
- [Discuss](#discuss)
- [Further Reading](#further-reading)
- [Notifications](#notifications)
## Intro
When I started learning Rust I made the mistake of following the advice to read [The Book](https://doc.rust-lang.org/book/title-page.html) first. While it's a great resource, it's pretty overwhelming for a beginner to get told _"If you'd like to learn this programming language the best way to start is to read this 20 chapter book!"_ Most people give up before they even get started when they get advice like this. Nobody ever told someone to read a 20 chapter book just to get started with Javascript or Python. Rust's learning curve is no joke but you gotta give the people what they want, and they want to program, not read about programming. Programming is fun and reading about programming is not as fun.
The first 10% of this article is gonna be me giving you advice on how to learn Rust in 2020 following a _practical hands-on coding_ approach. This is the good part of the article. You can safely exit after this part (I'll tell you when). The remaining 90% of this article is me ranting about how most online coding challenge sites have poor support for Rust.
## TL;DR
If you're a total Rust newbie and want to learn as much as possible in just one day you should read fasterthanlime's excellent [A half-hour to learn Rust](https://fasterthanli.me/blog/2020/a-half-hour-to-learn-rust/) and then checkout the awesome [Rustlings](https://github.com/rust-lang/rustlings) repo and complete the exercises.
If you're a Rust beginner you should get started on [Exercism's Rust Track](https://exercism.io/tracks/rust). If you get stuck you should ask your friends Google and StackOverflow for help. I recommend taking the time to get comfortable reading and navigating the [Rust Standard Library Docs](https://doc.rust-lang.org/std/) which is amazing and has simple practical examples for how to use everything inside of it. [Rust by Example](https://doc.rust-lang.org/rust-by-example/) is also a really good high-level reference that you can use to quickly learn Rust syntax and features. If you want to gain a deeper understanding of a certain Rust concept only then do I recommend finding the appropriate chapter in [The Book](https://doc.rust-lang.org/book/title-page.html) to read. The best part of completing an exercise on Exercism is that you get access to all the solutions by other members which you can sort by most-starred to see particularly idiomatic or clever solutions. This is a great way to learn!
At this point you're probably an advanced beginner and can find your own path. If you need more guidance and would like to continue working on small simple programs I recommend doing the exercises from the [Advent of Code 2018 Calendar](https://adventofcode.com/2018). The reason why I specifically recommended the 2018 calendar is because once you're finished with an exercise you can compare your solution to [BurntSushi's Advent of Code 2018 Rust solutions](https://github.com/BurntSushi/advent-of-code). BurntSushi writes really clean, readable, idiomatic Rust code. Reading the code of an experienced Rustacean will teach you as much as the exercises themselves.
Exit now, the good part of the article is over.
## Practical Rust Resource Reviews
_Alternative title: Reviews of Free Online Resources a Rust Beginner can use to Practice Writing Small Simple Rust Programs_
Most of these resources weren't specifically created for the purpose of teaching Rust, however they can all be used to learn and practice Rust and many of them explicitly support Rust submissions and provide Rust-specific versions of problems.
The resources are ordered from worst to best.
### [HackerRank](https://www.hackerrank.com)
Rust is a supported language on HackerRank except you aren't allowed to submit Rust solutions to most of the problems on their site. I tried to upload my solution directly and they refused it:
This is really strange because I was able to browse Rust solutions for the problem above submitted by other HackerRank users, so it's possible to submit a Rust solution somehow. I tried Googling this issue but Google didn't return any useful results. There's no way for me to evaluate HackerRank other than to tell you not to waste your time with it like I did.
### [Project Euler](https://projecteuler.net/archives)
When I first started to learn programming back in 2012 I commonly heard _"If you wanna get up to speed quickly in a new programming language solve some Project Euler problems with it!"_ which was okay advice at the time since there were not many other alternatives but in my opinion Project Euler has very little to do with programming. Project Euler problems are more math problems than they are programming problems. Their challenge lies almost entirely in the mathematical reasoning required to reach the solution as the programming required is usually trivial. I would not recommend solving Project Euler problems as a way to learn Rust unless you're very mathematically inclined and have some nostalgia for the site.
### [LeetCode](https://leetcode.com/problemset/all/)
Rust is a supported language on LeetCode. For every problem on LeetCode you get a solution template which usually contains a single unimplemented function which you then have to implement and submit in order to solve the problem. For more involved problems the solution template might include a `struct` and an `impl` block with several unimplemented methods. Unfortunately, these solution templates are not created by humans, they are automatically generated, which results in a lot of really awkward and unidiomatic Rust code. Examples:
| LeetCode generated Rust | Idiomatic Rust |
|-|-|
| tree problems represent links as `Option<Rc<RefCell<Node>>>` | `Option<Rc<RefCell<Node>>>` is overkill for tree links and `Option<Box<Node>>` works just as well and is much easier to work with |
| methods which obviously mutate self still borrow it immutably, e.g. `fn insert(&self, val: i32)` | methods that mutate self need to borrow it mutably, e.g. `fn insert(&mut self, val: i32)` |
| signed 32-bit integers are used for all numbers, even if the problem is undefined for negative integers, e.g. `fn nth_fib(n: i32) -> i32` | problems which are undefined for negative integers should use unsigned integers, e.g. `fn nth_fib(n: u32) -> u32` |
| functions always take ownership of their arguments, even if it's unnecessary, e.g. `fn sum(nums: Vec<i32>) -> i32` | if you don't need ownership then borrow `fn sum(nums: &[i32]) -> i32` |
| functions sometimes ignore basic error cases, e.g. for `fn get_max(nums: Vec<i32>) -> i32` what `i32` should be returned if `nums` is empty? | if a result might be undefined the return type should be wrapped in an `Option`, e.g. `fn get_max(nums: &[i32]) -> Option<i32>` |
Other LeetCode issues, specific to Rust:
- LeetCode doesn't allow you to pull in 3rd-party dependencies in solutions. Normally I think this is okay for most languages but Rust in particular has a pretty slim standard library which doesn't even include regex support so a lot of the more complex string parsing problems on LeetCode are pointlessly difficult to solve in Rust but have otherwise trivial solutions in other languages which have regex support in their standard libraries.
- None of the problems in the `concurrency` category accept solutions in Rust. What? Fearless concurrency is one of Rust's major selling points!
- After solving a problem you can go to the problem's comments section to see other user's solutions (as many users like to publish their solutions there) but because Rust isn't very popular on LeetCode sometimes you won't find any Rust solutions ;(
General LeetCode issues:
- LeetCode has a surprising amount of very low quality problems. Problems can be liked and disliked by users but problems are never removed even if they hit very high dislike ratios. I've seen lots of problems with 100+ votes and 80%+ dislike ratios and I don't understand why they are kept on the site.
- Problem difficulty ratings are kinda off. Problems are rated as Easy, Medium, or Hard but there are many Easy problems with lower solve rates than many Hard problems.
- Not all problems accept solutions in all languages, and you can't filter problems by which languages they accept. None of the graph problems on LeetCode accept Rust solutions, for example.
- LeetCode blocks "premium" problems behind a steep monthly paywall but doesn't offer any kind of premium free-trial so there's no telling if the quality is actually any better than the free problems.
Things LeetCode does right:
- Solutions to problems are tested against a suite of secret unit tests, but if you fail a particular test case they show you the failed case.
- All of the generated Rust code at least follows rustfmt conventions.
### [Codewars](https://www.codewars.com/join?language=rust)
Codewars is a misleading name. There's no war going on at Codewars. There's no time limit to solve problems and your solutions aren't judged on their speed of execution or memory usage. You aren't in competition with anyone else. This isn't a bad thing, just worth pointing out.
Rust is a supported language on Codewars. For every problem on Codewars you get a solution template which usually contains a single unimplemented function which you then have to implement and submit in order to solve the problem. These solution templates are created by humans, including humans who aren't familiar with Rust, so you occasionally get some awkward and unidiomatic Rust. Examples:
| Codewars' Rust Problems | Idiomatic Rust |
|-|-|
| sometimes don't follow rustfmt conventions, e.g. `fn makeUppercase(s:&str)->String` | always follows rustfmt conventions, e.g. `fn make_uppercase(s: &str) -> String` |
| sometimes takes signed integer arguments for problems that aren't defined for negative integers, e.g. `fn nth_fib(n: i32) -> i32` | if a problem isn't defined for negative integers use unsigned integer arguments, e.g. `fn nth_fib(n: u32) -> u32` |
| sometimes a problem asks you to return `-1` for the null case, e.g. `fn get_index(needle: i32, haystack: &[i32]) -> i32` | if a result can be null the return type should be wrapped in an `Option`, e.g. `fn get_index(needle: i32, haystack: &[i32]) -> Option<usize>` |
| sometimes don't take advantage of deref coercion, e.g. `fn do_stuff(s: &String, list: &Vec<i32>)` | takes advantage of deref coercion, e.g. `fn do_stuff(s: &str, list: &[i32])` |
All of the issues above only happen sometimes since there are Rustaceans of various skill-levels on Codewars translating problems to Rust. This is a huge step up from LeetCode where all of the generated Rust problem code is consistently unidiomatic. However, the Rust community on Codewars as a whole might lean towards the inexperienced side since I've seen some highly upvoted "idiomatic" solutions that were also a bit on the awkward side. Examples:
| Codewars' highest upvoted Rust solutions | Idiomatic Rust |
|-|-|
| sometimes use an explicit return at the end of a function block, e.g. `return result;` | blocks are evaluated as expressions and implicitly return their last item, an explicit return at the end of a function block is unnecessary, e.g. `result` |
| often use compact formatting to make the solution look more concise | should follow rustfmt conventions |
| sometimes make unnecessary allocations, e.g. `str_slice.to_string().chars()` | if you don't need to allocate then don't, e.g. `str_slice.chars()` |
| often try to solve the problem using nothing but iterators at the cost of everything else | iterators are expressive and idiomatic, but if you have to chain 15 of them in a row and there are multiple levels of nested iterators in-between then perhaps you should consider refactoring to use some helper functions, intermediate variables, and maybe even a for-loop |
Again, the issues above only happen sometimes. An experienced Rustacean can spot them easily but there are a lot of Rust newbies on these sites who have no clue they are learning anti-patterns.
Other Codewars issues, specific to Rust:
- Rust doesn't seem that popular on Codewars, the site has 9000 exercises but only 300 of them have been translated to Rust ;(
Other general Codewars issues:
- Your solution is tested against a suite of secret unit tests, if you fail one of the secret unit tests you aren't shown the failed test case. This is especially annoying if the test case tests for an edge case that wasn't clearly communicated in the problem description.
Things Codewars does right:
- There's a small whitelist of 3rd-party dependencies you can use to help solve problems with Rust. This whitelist includes: rand, chrono, regex, serde, itertools, and lazy_static which helps round out Rust's standard library and puts it more on par with other languages.
- You can filter problems by language.
- Submitting a solution to a problem also automatically publishes the solution. You can view and upvote other members' solutions. You can sort solutions by most upvotes to see particularly concise and clever solutions, which sometimes will also be very idiomatic (but sometimes not, as explained above).
- Problem difficulty grading is pretty good! Instead of grading problems as Easy, Medium, or Hard like LeetCode, Codewars chooses to grade problems from easiest to hardest as: 8 kyu, 7 kyu, 6 kyu, 5 kyu, 4 kyu, 3 kyu, 2 kyu, 1 kyu. I completed 60 problems in the 8 kyu - 4 kyu range and every level felt a little more difficult than the last, which aligned with my expectations.
### [Advent of Code](https://adventofcode.com/)
Advent of Code is totally language-agnostic. This would seem like a minus at first but seeing how horribly HackerRank, LeetCode, and Codewars handle their support for Rust on their sites it's actually a plus. Advent of Code also gets placed above the previously mentioned sites because AoC's exercises are really interesting, diverse, and high quality in my opinion.
General AoC issues:
- After you finish an exercise there's no way to see other people's Rust solutions unless you search from them on Google, and even after you find some there's no telling how good or idiomatic they are.
To solve the above issue I recommend going through the 2018 Calendar problems and comparing your solutions to [BurntSushi's AoC 2018 Rust solutions](https://github.com/BurntSushi/advent-of-code). BurntSushi writes really clean, readable, idiomatic Rust code. If you want to go through the 2019 Calendar then I recommend comparing your solutions to [bcmyers' AoC 2019 Rust solutions](https://github.com/bcmyers/aoc2019). The reason I specifically suggest bcmyers' is because he made a [youtube playlist of him coding up the solutions](https://www.youtube.com/playlist?list=PLQXBtq4j4Ozkx3r4eoMstdkkOG98qpBfg) and he does a great job of explaining his thought process and why he's doing what he's doing while he's coding.
Things AoC got right:
- High quality, interesting, curated exercises that are tied together with a narrative.
- Language agnostic, so while it doesn't teach you any Rust patterns it at least doesn't teach you any Rust anti-patterns either.
### [Rustlings](https://github.com/rust-lang/rustlings)
Rustlings is sooo good. All Rustlings exercises are hand-crafted for Rust with love and it's a wonderful breath of fresh air. Finally, a set of exercises that really teach you idiomatic Rust!
If you're a total Rust newbie you should absolutely checkout [Rustlings](https://github.com/rust-lang/rustlings) and get started on the exercises. I highly recommend reading fasterthanlime's [A half-hour to learn Rust](https://fasterthanli.me/blog/2020/a-half-hour-to-learn-rust/) first as it'll get you up to speed on a lot of Rust syntax and concepts super quickly.
I have only 1 tiny Rustlings criticism: there are some sudden difficulty spikes in the "error-handling" and "conversions" exercises that I could see some users getting overwhelmed by. I assume most probably make it through, or at least I hope.
I also have 1 tiny non-criticism: it's too short. This is a non-criticism because it's one of Rustlings design goals to be a quick and gentle introduction to Rust but it's so good that of course I wish it was somehow longer.
### [Exercism](https://exercism.io/tracks/rust)
Exercism has a Rust track, which is a collection of exercises roughly ordered by subject and difficulty. The Rust track shares a lot of exercises in common with other tracks, but all of the exercises were translated to Rust by experienced Rustaceans and don't suffer from any of the awkward unidiomatic Rust issues that are common on LeetCode and Codewars. There are about a dozen Rust-specific problems that require you to implement a standard library trait, or write a macro, or write a parallel solution using multiple threads, or write unsafe Rust code. These exercises are by far the highlights of the track and I wish there were more of them. Exercism is second only to Rustlings as a resource for learning Rust. The only reason I placed it above Rustlings is Rustlings can be completed in an evening and Exercism's Rust track will take at least a month to complete so it just has a lot more content.
Exercism issues, specific to the Rust track:
- "Mentored mode" is useless, as most of the Rust mentors on the site are inactive, and the students heavily outnumber them, so it's much better to go through a track in "practice mode".
- There are 92 exercises but a good chunk of them don't really teach you anything new so they kinda feel like busywork. They could probably cut ~20 exercises from the track to make it feel a lot tighter.
Things Exercism does right:
- All problems are translated to Rust or written for Rust by experienced Rustaceans.
- There are problems which specifically teach Rust's idioms, design patterns, and unique features.
- Problem difficulties are fairly graded, easy problems are easy, medium problems are medium, hard problems are hard.
- You can include whatever 3rd-party dependencies that you want in your solutions.
- All unit tests are public, if you're failing a test you know exactly why.
- After you submit a solution you can browse other user's solutions, and you can sort solutions by which received the most stars.
## Conclusion
Same as the [TL;DR](#tldr) :)
## Discuss
Discuss this article on
- [learnrust subreddit](https://www.reddit.com/r/learnrust/comments/ggj8tf/learning_rust_in_2020/)
- [official Rust users forum](https://users.rust-lang.org/t/blog-post-learning-rust-in-2020/42373)
- [rust subreddit](https://www.reddit.com/r/rust/comments/gie64f/learning_rust_in_2020/)
- [Hackernews](https://news.ycombinator.com/item?id=23160975)
- [Github](https://github.com/pretzelhammer/rust-blog/discussions)
## Further Reading
- [Common Rust Lifetime Misconceptions](./common-rust-lifetime-misconceptions.md)
- [Tour of Rust's Standard Library Traits](./tour-of-rusts-standard-library-traits.md)
- [Beginner's Guide to Concurrent Programming: Coding a Multithreaded Chat Server using Tokio](./chat-server.md)
- [Learning Rust in 2024](./learning-rust-in-2024.md)
- [Using Rust in Non-Rust Servers to Improve Performance](./rust-in-non-rust-servers.md)
- [Sizedness in Rust](./sizedness-in-rust.md)
- [RESTful API in Sync & Async Rust](./restful-api-in-sync-and-async-rust.md)
- [Learn Assembly with Entirely Too Many Brainfuck Compilers](./too-many-brainfuck-compilers.md)
## Notifications
Get notified when a new blog post gets published by
- Subscribing to this repo's [releases RSS feed](https://github.com/pretzelhammer/rust-blog/releases.atom) or
- Watching this repo's releases (click `Watch` → click `Custom` → select `Releases` → click `Apply`)
================================================
FILE: posts/learning-rust-in-2024.md
================================================
# Learning Rust in 2024
_21 September 2024 · #rust · #coding · #exercises_
**Table of contents**
- [TL;DR](#tldr)
- [0\) Reference material](#0-reference-material)
- [1\) Read A half hour to learn Rust](#1-read-a-half-hour-to-learn-rust)
- [2\) Complete rustlings](#2-complete-rustlings)
- [3\) Spend 10 hours coding in Rust](#3-spend-10-hours-coding-in-rust)
- [100 Exercises to Learn Rust](#100-exercises-to-learn-rust)
- [The Rust track on Exercism](#the-rust-track-on-exercism)
- [Advent of Code](#advent-of-code)
- [Tutorials](#tutorials)
- [4\) Read Common Rust Lifetime Misconceptions](#4-read-common-rust-lifetime-misconceptions)
- [5\) Spend another 10 hours coding in Rust](#5-spend-another-10-hours-coding-in-rust)
- [6\) Read Tour of Rust's Standard Library Traits](#6-read-tour-of-rusts-standard-library-traits)
- [What's next?](#whats-next)
- [Honorable mentions](#honorable-mentions)
- [Discuss](#discuss)
- [Further Reading](#further-reading)
- [Notifications](#notifications)
This is my opinionated guide on how to go from knowing nothing about Rust to being kinda okay at Rust quickly as possible. This is what I would tell myself if I was starting today.
## TL;DR
1. Read [A half hour to learn Rust](https://fasterthanli.me/articles/a-half-hour-to-learn-rust) (30 - 60 mins)
2. Complete [rustlings](https://github.com/rust-lang/rustlings) (2 - 3 hours)
3. Spend 10 hours coding in Rust (8 - 12 hours)
4. Read [Common Rust Lifetime Misconceptions](./common-rust-lifetime-misconceptions.md) (30 - 60 mins)
5. Spend another 10 hours coding in Rust (8 - 12 hours)
6. Read [Tour of Rust's Standard Library Traits](./tour-of-rusts-standard-library-traits.md) (2 - 4 hours)
And that's it, after a short 19 to 30 hours you'll go from being a Rust beginner to a Rust advanced beginner 😎
## 0\) Reference material
While learning Rust it's useful to have [Rust by Example](https://doc.rust-lang.org/rust-by-example/index.html) and the [Rust Standard Library docs](https://doc.rust-lang.org/std/) open in a couple browser tabs. If you get stuck on something you can probably find the answer by searching in either of those two resources, as both have a search bar at the top if you need to quickly find something.
Other good alternatives are StackOverflow and ChatGPT (or whatever LLM you prefer). Almost every Rust beginner question has been asked and answered on StackOverflow and is indexed on Google. As for ChatGPT, it may get the answer wrong like half the time, but it will at least point you in the general direction of where you should search next to find a good answer.
If you've tried all of those things but still cannot get unstuck, then here are some beginner-friendly online Rust communities where you can ask questions: the [/r/rust](https://www.reddit.com/r/rust/) subreddit, the [/r/learnrust](https://www.reddit.com/r/learnrust/) subreddit, and the [official Rust users forum](https://users.rust-lang.org/).
Also it's a good idea to have [Rust Playground](https://play.rust-lang.org/) open in a browser tab. It's an online code editor and compiler for Rust code. It's great for tinkering with small examples to reinforce what you're learning while reading.
## 1\) Read [A half hour to learn Rust](https://fasterthanli.me/articles/a-half-hour-to-learn-rust)
I recommend [A half hour to learn Rust](https://fasterthanli.me/articles/a-half-hour-to-learn-rust) above all other resources similar to it because it introduces Rust syntax and concepts in a very systematic and ground-up approach. It starts with the simplest possible example and then incrementally adds a little bit more complexity in every following example. The sequencing is very logical and easy to follow.
## 2\) Complete [rustlings](https://github.com/rust-lang/rustlings)
[Rustlings](https://github.com/rust-lang/rustlings) is an ordered collection of tiny Rust programming exercises. Each exercise is a tiny Rust program that doesn't compile or is failing its unit tests. Your job for every exercise is to fix the program so it compiles and passes its unit tests.
Before starting rustlings you need to have Rust installed. I recommend using [rustup](https://rustup.rs/) to manage Rust installations. Once you have `rustup` you just have to run `rustup update` to update Rust on your machine any time.
When coding Rust, you can use whatever code editor you want, but I recommend using one that supports [rust-analyzer](https://rust-analyzer.github.io/). I use [VS Code](https://code.visualstudio.com/) with [this extension](https://marketplace.visualstudio.com/items?itemName=rust-lang.rust-analyzer) to enable rust-analyzer.
Getting started with rustlings is very easy because we can install it using `cargo`. Cargo is Rust's dependency manager and build tool, and comes with all Rust installations. If you installed Rust using `rustup` then you already have it.
Run `cargo install rustlings`. By default, it will put the rustlings binary in `$HOME/.cargo/bin`, so you should add that path to your `$PATH` variable if it's not in there already, e.g.
```sh
export PATH="$HOME/.cargo/bin:$PATH"
```
Now run `rustlings`, which will create a `rustlings/` directory for you with all of the exercises in it. Open that directory in a code editor and then run the `rustlings` command within the directory and follow its instructions. It will tell you which exercise to work on. Complete the exercise and then the command will tell which exercise to work on next. Rinse and repeat until you're done.
## 3\) Spend 10 hours coding in Rust
This can be whatever you want, but for your first 10 hours I would suggest sticking to single-threaded sync Rust. There's enough novel and challenging Rust concepts\* to learn within those parameters to easily keep you busy for at least 10 hours, and many people take much longer to fully wrap their heads around them. Multithreaded async Rust adds a bunch of additional complex concepts\*\*, so starting with it before you have the basics down will likely be overwhelming.
\*: ownership, borrowing, lifetimes, lifetime elision, mutable vs immutable references, sizedness, deref coercion, traits, trait objects, generics, const generics, static vs dynamic dispatch, error handling, iterators, modules, declarative macros, procedural macros, just to name a few.
\*\*: futures, pinning, polling, awaiting, send, sync, cancellation, streams, sinks, locks, atomics, channels, actors, just to name a few.
If you have no ideas on what to code then I have a few suggestions.
### [100 Exercises to Learn Rust](https://rust-exercises.com/100-exercises/)
If you enjoyed the difficulty and pacing of rustlings then you will enjoy 100 Exercises to Learn Rust (100ELR). Although made by different authors, 100ELR feels like a natural extension of rustlings and clearly drew a lot of inspiration from it.
To get started clone the [mainmatter/100-exercises-to-learn-rust](https://github.com/mainmatter/100-exercises-to-learn-rust) repo to your machine. Install their test suite runner using `cargo install --locked workshop-runner`. Then start reading the [accompanying book](https://rust-exercises.com/100-exercises/). It has 100 chapters and at the end of each chapter it tells you what exercise to do in the repo. Once you complete an exercise run `wr` in the project root to check that your solution passes the unit tests, and if it does continue on to the next chapter. Rinse and repeat until you're done.
Although 100 chapters sounds like a lot they're all quite short so you'll get through them faster than you think. Completing the entire book likely take most people about 15 to 25 hours.
However, if you found rustlings too easy or too slow and would like to try something more challenging, then you may want to skip 100ELR.
### The [Rust track](https://exercism.org/tracks/rust) on Exercism
The [Rust track](https://exercism.org/tracks/rust) on Exercism is the best collection of small Rust exercises on the internet aside from rustlings and 100ELR. What makes it so good is that many of the exercises were crafted by Rustaceans, and are meant to teach you the aspects of Rust that make it unique and interesting compared to other programming languages. There's an exercise where you have to write a declarative macro! There's even exercise where you have to write unsafe code!
To get started you have to make an account on Exercism and enroll in the Rust track (both are free). You can then complete the exercises online using Exercism's online code editor or by downloading the exercises to your machine, completing them using your code editor, and submitting your solutions to Exercism using the `exercism` cli tool.
I recommend using the `exercism` cli tool. There are many advantages. First, your code editor has rust-analyzer and Exercism's online code editor doesn't. Second, you can use dark mode in your editor but you can't use dark mode on Exercism unless you pay (it's a premium feature). Third, you can track your solutions using git. Fourth, you can choose which unit tests to run while you're developing your solution, instead of running the full test suite every time. The official Exercism instructions on how to do the exercises locally on your machine are [here](https://exercism.org/docs/using/solving-exercises/working-locally).
Each track on Exercism has a "learning mode" and a "practice mode." At the moment the Rust track's "learning mode" is disabled while the Exercism team reworks it. Regardless, I think the "practice mode" is better because it lets you do the exercises in whatever order you want. So while you may not have the option right now, if the option between a "learning mode" and a "practice mode" comes back to the Rust track in the future I'd recommend picking practice mode.
While the exercises in the Rust track are ordered, the order is arbitrary and following it isn't important. With the exception of the [Luhn](https://exercism.org/tracks/rust/exercises/luhn), [Luhn From](https://exercism.org/tracks/rust/exercises/luhn-from), and [Luhn Trait](https://exercism.org/tracks/rust/exercises/luhn-trait) exercises, all of the other exercises are totally unrelated to each other and can be done in whatever order.
The best part of completing an exercise is that it gives you the ability to see and star other people's solutions. After completing an exercise I recommend checking these out, especially the most starred solutions, as they can teach you elegant and idiomatic parts of Rust that you may have not discovered on your own.
Furthermore, the exercises are rated by difficulty: easy, medium, or hard. Currently there's 99 total exercises in the Rust track and I would say that about 30 are really good and the other 59 are just generic exercises that were copy & pasted from other tracks. Less than 0.1% of people complete the entire track. I wouldn't be surprised if less than 1% of people even complete half of the track. I have done all of the exercises so I'll recommend which ones I think are the best, since you're likely not going to do all of them anyway.
You should start with the easy exercises. Difficulty-wise they're slightly more challenging than what you'd find in rustlings or 100ELR. Quality-wise all of them are okay. None of them will take you longer than 20 minutes to complete. Also, if completing the easy exercises stops being rewarding then move on to the medium exercises, don't push yourself to complete all the easy ones if you don't feel like you're getting anything out of them anymore.
Anyway, as for the medium exercises, these are the best ones in my opinion (in no particular order):
- [PaaS I/O](https://exercism.org/tracks/rust/exercises/paasio)
- [Simple Linked List](https://exercism.org/tracks/rust/exercises/simple-linked-list)
- [Fizzy](https://exercism.org/tracks/rust/exercises/fizzy)
- [Robot Name](https://exercism.org/tracks/rust/exercises/robot-name)
- [Triangle](https://exercism.org/tracks/rust/exercises/triangle)
- [Robot Simulator](https://exercism.org/tracks/rust/exercises/robot-simulator)
- [Accumulate](https://exercism.org/tracks/rust/exercises/accumulate)
- [Word Count](https://exercism.org/tracks/rust/exercises/word-count)
- [Grep](https://exercism.org/tracks/rust/exercises/grep)
- [Luhn](https://exercism.org/tracks/rust/exercises/luhn)
- [Luhn From](https://exercism.org/tracks/rust/exercises/luhn-from)
- [Luhn Trait](https://exercism.org/tracks/rust/exercises/luhn-trait)
- [Clock](https://exercism.org/tracks/rust/exercises/clock)
- [Space Age](https://exercism.org/tracks/rust/exercises/space-age)
- [Sublist](https://exercism.org/tracks/rust/exercises/sublist)
- [Binary Search](https://exercism.org/tracks/rust/exercises/binary-search)
- [ETL](https://exercism.org/tracks/rust/exercises/etl)
- [Grade School](https://exercism.org/tracks/rust/exercises/grade-school)
- [Hamming](https://exercism.org/tracks/rust/exercises/hamming)
- [Isogram](https://exercism.org/tracks/rust/exercises/isogram)
- [Nucleotide Count](https://exercism.org/tracks/rust/exercises/nucleotide-count)
Depending on which ones you do they can take you anywhere from 10 to 45 minutes.
Once you're ready to bump up the difficulty again, these are the best hard exercises in my opinion (in no particular order):
- [Macros](https://exercism.org/tracks/rust/exercises/macros)
- [Parallel Letter Frequency](https://exercism.org/tracks/rust/exercises/parallel-letter-frequency)
- [Xorcism](https://exercism.org/tracks/rust/exercises/xorcism)
- [React](https://exercism.org/tracks/rust/exercises/react)
- [Circular Buffer](https://exercism.org/tracks/rust/exercises/circular-buffer)
- [Forth](https://exercism.org/tracks/rust/exercises/forth)
- [Doubly Linked List](https://exercism.org/tracks/rust/exercises/doubly-linked-list)
Depending on which ones you do they can take anywhere from 30 to 90 minutes.
If you do a bunch of easy exercises, plus the medium and hard ones I recommend, it will probably take you around 20 hours. I think completing the entire track will probably be double that, at around 40 hours. With that said don't push yourself to finish the entire track if you feel yourself starting to lose interest, it'd be better to change gears and continue your Rust coding journey trying something else next, because the most important thing is that you keep having fun while learning.
### [Advent of Code](https://adventofcode.com/)
[Advent of Code](https://adventofcode.com/) (AoC) is a collection of programming puzzles. Every year since 2015 a new puzzle has been released every day from December 1st to 25th. Each puzzle has 2 parts. The first 10-15 puzzles of each year are usually easy to medium difficulty, and the last 10-15 puzzles of each year are usually hard. The puzzles are designed to be language agnostic, meaning you can solve them using any programming language. So if your goal is to learn Rust then just doing the puzzles is not going to help you very much, but if you complete a puzzle and then review an experienced Rustacean's solution that could help teach you a lot of Rust.
The experienced Rustacean in this case would be fasterthanlime. On top of publishing his solutions for the 2020 and 2022 AoC puzzles he also wrote beginner-friendly blog posts explaining his thought process while solving each one. If you're going to do AoC to learn Rust then you should start with the 2020 puzzles and after completing each one read the corresponding fasterthanlime blog post on it. I wouldn't recommend going far past puzzle 10 for any given year because they can start to get really challenging and you'll be spending most of your time busting your brain trying to figure out how to solve the puzzle instead of learning Rust.
#### Setting up
First, create an account on [AoC](https://adventofcode.com/). Second, since AoC is language agnostic you have to set up your own cargo project with your own scaffolding in order to solve the puzzles. Doing this can be very educational but it's also pretty boring. I'd suggest skipping that and using this Github template instead: [fspoettel/advent-of-code-rust](https://github.com/fspoettel/advent-of-code-rust). Click on `Use this template` -> `Create a new repository`, then `git clone` the created repository to your machine. In `.cargo/config.toml` set `AOC_YEAR` to `2020`.
Also you can install the `aoc-cli` tool to fetch all instructions and inputs from your AoC account to your machine. To do that run `cargo install aoc-cli --version 0.12.0`, then create the file `$HOME/.adventofcode.session` and paste your AoC session cookie into it. To get your session cookie, press `F12` while anywhere on the AoC website to open your browser's developer tools, then look for `Cookies` under the `Application` or `Storage` tabs and copy the `session` cookie value.
Once all of that is done you can run `cargo download {day}` within your project directory to generate the scaffolding and download the puzzle for that day. Once you think you've solved it you can run `cargo solve {day}` and then submit your final result via the AoC website to verify. For day 1 the commands would be `cargo download 1` and `cargo solve 1`.
#### Year 2020
As mentioned before, make sure to set `AOC_YEAR` to `2020` in `.cargo/config.toml`. After completing a puzzle read fasterthanlime's solution on his blog, here's direct links to his solutions to the first 10 puzzles:
1. [AoC 2020 Day 1](https://fasterthanli.me/series/advent-of-code-2020/part-1)
1. [AoC 2020 Day 2](https://fasterthanli.me/series/advent-of-code-2020/part-2)
1. [AoC 2020 Day 3](https://fasterthanli.me/series/advent-of-code-2020/part-3)
1. [Aoc 2020 Day 4](https://fasterthanli.me/series/advent-of-code-2020/part-4)
1. [AoC 2020 Day 5](https://fasterthanli.me/series/advent-of-code-2020/part-5)
1. [AoC 2020 Day 6](https://fasterthanli.me/series/advent-of-code-2020/part-6)
1. [AoC 2020 Day 7](https://fasterthanli.me/series/advent-of-code-2020/part-7)
1. [AoC 2020 Day 8](https://fasterthanli.me/series/advent-of-code-2020/part-8)
1. [AoC 2020 Day 9](https://fasterthanli.me/series/advent-of-code-2020/part-9)
1. [AoC 2020 Day 10](https://fasterthanli.me/series/advent-of-code-2020/part-10)
#### Year 2022
If you completed at least 10 of the 2020 puzzles then year 2022 is the next best year to do. Clone another copy of [fspoettel/advent-of-code-rust](https://github.com/fspoettel/advent-of-code-rust) and set `AOC_YEAR` to `2022` in `.cargo/config.toml` in the project directory. Then here's direct links to fasterthanlime's 2022 solutions:
1. [AoC 2022 Day 1](https://fasterthanli.me/series/advent-of-code-2022/part-1)
1. [AoC 2022 Day 2](https://fasterthanli.me/series/advent-of-code-2022/part-2)
1. [AoC 2022 Day 3](https://fasterthanli.me/series/advent-of-code-2022/part-3)
1. [AoC 2022 Day 4](https://fasterthanli.me/series/advent-of-code-2022/part-4)
1. [AoC 2022 Day 5](https://fasterthanli.me/series/advent-of-code-2022/part-5)
1. [AoC 2022 Day 6](https://fasterthanli.me/series/advent-of-code-2022/part-6)
1. [AoC 2022 Day 7](https://fasterthanli.me/series/advent-of-code-2022/part-7)
1. [AoC 2022 Day 8](https://fasterthanli.me/series/advent-of-code-2022/part-8)
1. [AoC 2022 Day 9](https://fasterthanli.me/series/advent-of-code-2022/part-9)
1. [AoC 2022 Day 10](https://fasterthanli.me/series/advent-of-code-2022/part-10)
### Tutorials
The goal of most Rust tutorials is to show you how some piece of software could be implemented in Rust, not to necessarily to teach you Rust, and so many tutorial authors assume their audience will already be somewhat competent with the language and will omit a bunch of beginner explanations for the sake of keeping the tutorial focused and concise.
With that said, writing a non-trivial piece of software can be a lot more fun and rewarding than completing a bunch of small exercises, so this approach can be preferable for keeping your motivations high, especially if you're building something you're genuinely interested and excited about.
I wish I could recommend some tutorials here but there's so many of them out there that it would be hard for me to go through all of them and rank them by their quality. The quality probably isn't that important though, what's important is that you're writing Rust code and enjoying your time, so pick any tutorial that looks fun and follow along without sweating the details.
I will give a shoutout to [Codecrafters](https://codecrafters.io) though. Their tutorials are clearly very high quality and they have Rust starter templates for all of them. In every step of every tutorial they give you instructions and hints on how to implement the next part of the program, and once you're done you submit your code and it runs against a test suite to check if everything was implemented correctly.
A big downside of Codecrafters is that it costs $40/month. They occasionally run promotions where a tutorial will be free for a month, so depending on when you check their site you may get lucky and a tutorial that catches your eye will be free. However, their pricing suggests they target software companies looking to train their engineers, and not individual engineers themselves. If you're employed at such a company, check if your company provides you a discretionary training allowance for courses like Codecrafters, which you could use to cover the cost.
## 4\) Read [Common Rust Lifetime Misconceptions](./common-rust-lifetime-misconceptions.md)
Disclaimer: I wrote this.
Reading [Common Rust Lifetime Misconceptions](./common-rust-lifetime-misconceptions.md) will help you understand why half of the stuff you tried in your last 10 hours of Rust coding didn't work 🙂
Jokes aside, understanding lifetimes is probably the biggest stumbling block that many beginners struggle to get over when learning Rust, and this article dispels all of the most common misconceptions that people have about lifetimes that cause them confusion and frustration.
The reason I recommend it after at least 10 hours of practical Rust coding experience is because I think it might be too much technical information too soon for absolute beginners, who may find it more overwhelming than helpful.
## 5\) Spend another 10 hours coding in Rust
As before if you're not sure what to code my suggestions remain the same: [100 Exercises to Learn Rust](https://rust-exercises.com/100-exercises/), the [Rust track](https://exercism.org/tracks/rust) on Exercism, [Advent of Code](https://adventofcode.com/), or tutorials.
Also since you now have some Rust experience under your belt feel free to try coding something in multithreaded async Rust if you're feeling adventurous.
## 6\) Read [Tour of Rust's Standard Library Traits](./tour-of-rusts-standard-library-traits.md)
Disclaimer: I wrote this.
Traits are the main way to write polymorphic code in Rust, so they're used everywhere, especially the ones from the standard library. [Tour of Rust's Standard Library Traits](./tour-of-rusts-standard-library-traits.md) gives a guided tour of the most popular standard library traits, their methods, how to use them, and when to implement them for your own types. It's pretty thorough, and shares a ton of useful tips. And although the article is lengthy you don't have to read the entire thing to get value out of it, so don't let its size daunt you.
## What's next?
Congrats. You made it. You definitely know enough Rust to forge your own path forward. Good luck and have fun.
## Honorable mentions
These are other Rust beginner resources that I really like but wasn't able to find a spot for them in my Rust learning guide.
### [Tour of Rust](https://tourofrust.com/)
It's really good. However, it covers a lot of the same ground that other resources in the guide already cover, so it was omitted for the sake of keeping the guide streamlined.
### [Comprehensive Rust](https://google.github.io/comprehensive-rust/)
This is a Rust course developed by the Android team at Google. It's a collection of slides which are meant to be presented by a speaker experienced with Rust, but the book version has the speaker notes at the bottom of each page so that it can be learned from as a standalone resource.
It's very concise and fast-paced, and also has sections covering parts of Rust that are typically omitted from other beginner resources, such as [bare metal Rust](https://google.github.io/comprehensive-rust/bare-metal.html) and [concurrency in Rust](https://google.github.io/comprehensive-rust/concurrency/welcome.html). However, since it's better presented by a speaker it didn't make sense to put it into my guide for self-learners.
### [Clear explanation of Rust's module system](https://www.sheshbabu.com/posts/rust-module-system/)
This is a great article. I wasn't sure where to put it in the guide because it could go anywhere, but ideally it comes right when the learner starts breaking their Rust code across multiple files, which comes at a different point for every person in their Rust coding journey. Anyway, whenever that point comes for you, read this article then.
## Discuss
Discuss this article on
- [Github](https://github.com/pretzelhammer/rust-blog/discussions/84)
- [official Rust users forum](https://users.rust-lang.org/t/learning-rust-in-2024)
- [learnrust subreddit](https://www.reddit.com/r/learnrust/comments/1fnlvd8/learning_rust_in_2024/)
- [rust subreddit](https://www.reddit.com/r/rust/comments/1fod8u9/learning_rust_in_2024/)
## Further reading
- [Common Rust Lifetime Misconceptions](./common-rust-lifetime-misconceptions.md)
- [Tour of Rust's Standard Library Traits](./tour-of-rusts-standard-library-traits.md)
- [Beginner's Guide to Concurrent Programming: Coding a Multithreaded Chat Server using Tokio](./chat-server.md)
- [Using Rust in Non-Rust Servers to Improve Performance](./rust-in-non-rust-servers.md)
- [Sizedness in Rust](./sizedness-in-rust.md)
- [RESTful API in Sync & Async Rust](./restful-api-in-sync-and-async-rust.md)
- [Learn Assembly with Entirely Too Many Brainfuck Compilers](./too-many-brainfuck-compilers.md)
## Notifications
Get notified when a new blog post gets published by
- Subscribing to this repo's [releases RSS feed](https://github.com/pretzelhammer/rust-blog/releases.atom) or
- Watching this repo's releases (click `Watch` → click `Custom` → select `Releases` → click `Apply`)
================================================
FILE: posts/restful-api-in-sync-and-async-rust.md
================================================
# RESTful API in Sync & Async Rust
_11 May 2021 · #rust · #diesel · #rocket · #sqlx · #actix-web_
**Table of Contents**
- [Intro](#intro)
- [General](#general)
- [Project Setup](#project-setup)
- [Loading Environment Variables w/dotenv](#loading-environment-variables-wdotenv)
- [Handling Dates & Times w/chrono](#handling-dates--times-wchrono)
- [Logging w/fern](#logging-wfern)
- [JSON Serialization w/serde](#json-serialization-wserde)
- [Domain Modeling](#domain-modeling)
- [Sync Implementation](#sync-implementation)
- [SQL Schema Migrations w/diesel-cli](#sql-schema-migrations-wdiesel-cli)
- [Executing SQL Queries w/Diesel](#executing-sql-queries-wdiesel)
- [Mapping DB Enums to Rust Enums](#mapping-db-enums-to-rust-enums)
- [Fetching Data](#fetching-data)
- [Inserting Data](#inserting-data)
- [Updating Data](#updating-data)
- [Deleting Data](#deleting-data)
- [Using a Connection Pool w/r2d2](#using-a-connection-pool-wr2d2)
- [Refactoring DB Operations Into a Module](#refactoring-db-operations-into-a-module)
- [HTTP Routing w/Rocket](#http-routing-wrocket)
- [Routing Basics](#routing-basics)
- [GET Requests](#get-requests)
- [POST & PATCH Requests](#post--patch-requests)
- [DELETE Requests](#delete-requests)
- [Refactoring API Routes Into a Module](#refactoring-api-routes-into-a-module)
- [Authentication](#authentication)
- [Async Implementation](#async-implementation)
- [SQL Schema Migrations w/sqlx-cli](#sql-schema-migrations-wsqlx-cli)
- [Executing SQL Queries w/sqlx](#executing-sql-queries-wsqlx)
- [Fetching Data](#fetching-data-1)
- [Inserting Data](#inserting-data-1)
- [Updating Data](#updating-data-1)
- [Deleting Data](#deleting-data-1)
- [Compile-Time Verification of SQL Queries](#compile-time-verification-of-sql-queries)
- [Using a Connection Pool w/sqlx](#using-a-connection-pool-wsqlx)
- [Refactoring DB Operations Into a Module](#refactoring-db-operations-into-a-module-1)
- [HTTP Routing w/actix-web](#http-routing-wactix-web)
- [Routing Basics](#routing-basics-1)
- [GET Requests](#get-requests-1)
- [POST & PATCH Requests](#post--patch-requests-1)
- [DELETE Requests](#delete-requests-1)
- [Refactoring API Routes Into a Module](#refactoring-api-routes-into-a-module-1)
- [Authentication](#authentication-1)
- [Benchmarks](#benchmarks)
- [Servers](#servers)
- [Methodology](#methodology)
- [Measuring Resource Usage](#measuring-resource-usage)
- [Results](#results)
- [Read-Only Workload](#read-only-workload)
- [Reads + Writes Workload](#reads--writes-workload)
- [Concluding Thoughts](#concluding-thoughts)
- [Diesel vs sqlx](#diesel-vs-sqlx)
- [Rocket vs actix-web](#rocket-vs-actix-web)
- [Sync Rust vs Async Rust](#sync-rust-vs-async-rust)
- [Rust vs JS](#rust-vs-js)
- [In Summary](#in-summary)
- [Discuss](#discuss)
- [Further Reading](#further-reading)
- [Notifications](#notifications)
## Intro
Let's implement a RESTful API server in Rust for an imaginary Kanban-style project management app. A popular real-world example of such an app is Trello:
On its surface Kanban is pretty simple: there's a board and cards. The board represents a project. The cards represent tasks. The position of the cards on the board represents the state and progress of the tasks. The simplest boards have 3 columns for tasks which are: queued (to do), in progress (doing), and done (done).
Despite being simple on its surface, Kanban, and all kinds of project management software in general, is a literal bottomless pit of complexity. There's a million things we could implement, and after we finish the first million things there would be a million more. However, since I'm trying to write a single article and not an entire book series let's keep the feature scope tiny.
The server should support the ability to:
- Create boards
- Boards have names
- Get a list of all boards
- Delete boards
- Create cards
- Can associate cards with boards
- Cards have descriptions and statuses
- Get a list of all the cards on a board
- Get a board summary: count of all the cards on the board grouped by their status
- Update cards
- Delete cards
And that's it! To make this project slightly more interesting let's also include token-based authentication for all of the server's endpoints, but let's keep it simple: as long as a request contains a valid token it has access to all of the boards and cards.
Furthermore, to satisfy my own curiosity, and to maximize the educationalness of this article, we're going to write two implementations together: one using sync Rust and the other using async Rust. The first implementation will use r2d2, Diesel, and Rocket. The second implementation will use sqlx, and actix-web. Here's a quick preview of the crates we'll be using for this project:
General crates
- dotenv (loading environment variables)
- log + fern (logging)
- chrono (date & time handling)
- serde + serde_json (JSON de/serialization)
Sync crates
- diesel-cli (DB schema migrations)
- diesel + diesel-derive-enum (ORM / building SQL queries)
- r2d2 (DB connection pool)
- rocket + rocket_contrib (HTTP routing)
Async crates
- sqlx-cli (DB schema migrations)
- sqlx (executing SQL queries & DB connection pool)
- actix-web (HTTP routing)
- futures (general future-related utilities)
After finishing both sync and async implementations we'll run some benchmarks to see which has better performance, because everyone loves benchmarks.
## General
### Project Setup
All of the boring instructions for setting this project up, like installing Docker and running locally, are in the [companion code repository](https://github.com/pretzelhammer/kanban). For this article let's focus entirely on the fun part: the Rust!
After the initial setup we have this empty `Cargo.toml` file:
```toml
# Cargo.toml
[package]
name = "kanban"
version = "0.1.0"
edition = "2018"
```
And this empty main file:
```rust
// src/main.rs
fn main() {
println!("Hello, world!");
}
```
### Loading Environment Variables w/dotenv
crates
- dotenv
```diff
# Cargo.toml
[package]
name = "kanban"
version = "0.1.0"
edition = "2018"
+[dependencies]
+dotenv = "0.15"
```
This crate does one small simple job: it loads variables from an `.env` in the current working directory and adds them to the program's environment variables. Here's the general `.env` file we'll be using:
```bash
# .env
LOG_LEVEL=INFO
LOG_FILE=server.log
DATABASE_URL=postgres://postgres@localhost:5432/postgres
```
Updated main file which uses `dotenv`:
```rust
// src/main.rs
type StdErr = Box<dyn std::error::Error>;
fn main() -> Result<(), StdErr> {
// loads env variables from .env
dotenv::dotenv()?;
// example
assert_eq!("INFO", std::env::var("LOG_LEVEL").unwrap());
Ok(())
}
```
### Handling Dates & Times w/chrono
crates
- chrono
```diff
# Cargo.toml
[package]
name = "kanban"
version = "0.1.0"
edition = "2018"
[dependencies]
dotenv = "0.15"
+chrono = "0.4"
```
Rust's go-to library for handling dates & times is chrono. We're not using the dependency in our project _just yet_ but will very soon after we add a few more dependencies.
### Logging w/fern
crates
- log
- fern
```diff
# Cargo.toml
[package]
name = "kanban"
version = "0.1.0"
edition = "2018"
[dependencies]
dotenv = "0.15"
chrono = "0.4"
+log = "0.4"
+fern = "0.6"
```
Log is Rust's logging facade library. It provides the high-level logging API but we still need to pick an implementation, and the implementation we're going to use is the fern crate. Fern allows us to easily customize the logging format and also chain multiple outputs so we can log to stderr and a file if we wanted to. After adding log and fern let's encapsulate all of the logging configuration and initialization into its own module:
```rust
// src/logger.rs
use std::env;
use std::fs;
use log::{debug, error, info, trace, warn};
pub fn init() -> Result<(), fern::InitError> {
// pull log level from env
let log_level = env::var("LOG_LEVEL").unwrap_or("INFO".into());
let log_level = log_level
.parse::<log::LevelFilter>()
.unwrap_or(log::LevelFilter::Info);
let mut builder = fern::Dispatch::new()
.format(|out, message, record| {
out.finish(format_args!(
"[{}][{}][{}] {}",
chrono::Local::now().format("%H:%M:%S"),
record.target(),
record.level(),
message
))
})
.level(log_level)
// log to stderr
.chain(std::io::stderr());
// also log to file if one is provided via env
if let Ok(log_file) = env::var("LOG_FILE") {
let log_file = fs::File::create(log_file)?;
builder = builder.chain(log_file);
}
// globally apply logger
builder.apply()?;
trace!("TRACE output enabled");
debug!("DEBUG output enabled");
info!("INFO output enabled");
warn!("WARN output enabled");
error!("ERROR output enabled");
Ok(())
}
```
And then add that module to our main file:
```diff
// src/main.rs
+mod logger;
type StdErr = Box<dyn std::error::Error>;
fn main() -> Result<(), StdErr> {
dotenv::dotenv()?;
+ logger::init()?;
Ok(())
}
```
If we run the program now, since `INFO` is the default logging level, here's what we'd see:
```bash
$ cargo run
[08:36:30][kanban::logger][INFO] INFO output enabled
[08:36:30][kanban::logger][WARN] WARN output enabled
[08:36:30][kanban::logger][ERROR] ERROR output enabled
```
### JSON Serialization w/serde
crates
- serde
- serde_json
```diff
# Cargo.toml
[package]
name = "kanban"
version = "0.1.0"
edition = "2018"
[dependencies]
dotenv = "0.15"
- chrono = "0.4"
+ chrono = { version = "0.4", features = ["serde"] }
log = "0.4"
fern = "0.6"
+ serde = { version = "1.0", features = ["derive"] }
+ serde_json = "1.0"
```
Pro-tip: when adding a new dependency to a project it's good to look through existing dependencies to see if they have the new dependency as a feature flag. In this case chrono has serde as a feature flag, which if enabled, adds `serde::Serialize` and `serde::Deserialize` impls to all of chrono's types. This will allow us to use chrono types in our own structs later which we will also derive `serde::Serialize` and `serde::Deserialize` impls for.
### Domain Modeling
Okay, let's start modeling our domain. We know we will have boards so:
```rust
#[derive(serde::Serialize)]
#[serde(rename_all = "camelCase")]
pub struct Board {
pub id: i64,
pub name: String,
pub created_at: chrono::DateTime<chrono::Utc>,
}
```
Unpacking the new stuff:
- `#[derive(serde::Serialize)]` derives a `serde::Serialize` impl for `Board` which will allow us to serialize it to JSON using the `serde_json` crate.
- `#[serde(rename_all = "camelCase")]` renames all of the snake_case member identifiers to camelCase when serializing (or vice versa when deserializing). This is because it's a convention to use snake_case names in Rust but JSON is often produced and consumed by JS code and the JS convention is to use camelCase for member identifiers.
- Making `id` an `i64` instead of an `u64` might seem like an odd choice but since we're using PostgreSQL as our DB we have to do this because PostgreSQL only supports signed integer types.
- A `created_at` member is always useful to have, if for no other reason than to be able to sort entities by chronological order when no better sort order is available.
Okay, let's add cards and statuses:
```rust
#[derive(serde::Serialize)]
#[serde(rename_all = "camelCase")]
pub struct Card {
pub id: i64,
pub board_id: i64,
pub description: String,
pub status: Status,
pub created_at: chrono::DateTime<chrono::Utc>,
}
#[derive(serde::Serialize, serde::Deserialize)]
#[serde(rename_all = "camelCase")]
pub enum Status {
Todo,
Doing,
Done,
}
```
Since we'd also like to support returning a "board summary" which contains the count of all of the cards on a board grouped by their status here's the model for that:
```rust
#[derive(serde::Serialize)]
pub struct BoardSummary {
pub todo: i64,
pub doing: i64,
pub done: i64,
}
```
When using the API to create a new board users can provide the board name but not its id, since that will be set by the DB, so we need a model for that as well:
```rust
#[derive(serde::Deserialize)]
pub struct CreateBoard {
pub name: String,
}
```
Likewise users can also create cards. When creating a card let's assume we only want users to provide the new card's description and what board it should be associated with. The new card will get the default todo status and will get its id set by the DB:
```rust
#[derive(serde::Deserialize)]
#[serde(rename_all = "camelCase")]
pub struct CreateCard {
pub board_id: i64,
pub description: String,
}
```
When updating a card let's assume we want users to only be able to update the description or the status. It would be pretty weird if we allowed them to move cards between boards which is a pretty unusual feature in most project management apps:
```rust
#[derive(serde::Deserialize)]
pub struct UpdateCard {
pub description: String,
pub status: Status,
}
```
Throw all of those into their own module and we get:
```rust
// src/models.rs
// for GET requests
#[derive(serde::Serialize)]
#[serde(rename_all = "camelCase")]
pub struct Board {
pub id: i64,
pub name: String,
pub created_at: chrono::DateTime<chrono::Utc>,
}
#[derive(serde::Serialize)]
#[serde(rename_all = "camelCase")]
pub struct Card {
pub id: i64,
pub board_id: i64,
pub description: String,
pub status: Status,
pub created_at: chrono::DateTime<chrono::Utc>,
}
#[derive(serde::Serialize, serde::Deserialize)]
#[serde(rename_all = "camelCase")]
pub enum Status {
Todo,
Doing,
Done,
}
#[derive(serde::Serialize)]
pub struct BoardSummary {
pub todo: i64,
pub doing: i64,
pub done: i64,
}
// for POST requests
#[derive(serde::Deserialize)]
pub struct CreateBoard {
pub name: String,
}
#[derive(serde::Deserialize)]
#[serde(rename_all = "camelCase")]
pub struct CreateCard {
pub board_id: i64,
pub description: String,
}
// for PATCH requests
#[derive(serde::Deserialize)]
pub struct UpdateCard {
pub description: String,
pub status: Status,
}
```
And the updated main file:
```diff
// src/main.rs
mod logger;
+mod models;
type StdErr = Box<dyn std::error::Error>;
fn main() -> Result<(), StdErr> {
dotenv::dotenv()?;
logger::init()?;
Ok(())
}
```
## Sync Implementation
### SQL Schema Migrations w/diesel-cli
crates
- diesel-cli
```bash
cargo install diesel_cli
```
If the above command doesn't work at first, it's likely because we don't have all the development libraries for all of diesel-cli's supported databases. Since we're just using PostgreSQL, we can make sure the development libraries are installed with these commands:
```bash
# macOS
brew install postgresql
# ubuntu
apt-get install postgresql libpq-dev
```
And then we can tell cargo to only install diesel-cli with support for PostgreSQL:
```bash
cargo install diesel_cli --no-default-features --features postgres
```
Once we have diesel-cli installed we can use it to create new migrations and execute pending migrations. diesel-cli figures out which DB to connect to by checking the `DATABASE_URL` environment variable, which it will also load from an `.env` file if one exists in the current working directory.
Assuming the DB is currently running and a `DATABASE_URL` environment variable is present, here's the first diesel-cli command we'd run to bootstrap our project:
```bash
diesel setup
```
With this diesel-cli creates a `migrations` directory where we can generate and write our DB schema migrations. Let's generate our first migration:
```bash
diesel migration generate create_boards
```
This will create a new directory, e.g. `migrations/<year>-<month>-<day>-<time>_create_boards`, with an `up.sql` and `down.sql` which is where we'll write our SQL queries. The "up" query is for creating or modifying our DB schema, in this case creating a boards table:
```sql
-- create_boards up.sql
CREATE TABLE IF NOT EXISTS boards (
id BIGSERIAL PRIMARY KEY,
name TEXT NOT NULL,
created_at TIMESTAMP WITH TIME ZONE NOT NULL DEFAULT (CURRENT_TIMESTAMP AT TIME ZONE 'utc')
);
-- seed db with some test data for local dev
INSERT INTO boards
(name)
VALUES
('Test board 1'),
('Test board 2'),
('Test board 3');
```
And the "down" query is for reverting the schema changes made in the "up" query, in this case dropping the created boards table:
```sql
-- create_boards down.sql
DROP TABLE IF EXISTS boards;
```
We will also need to store some cards:
```bash
diesel migration generate create_cards
```
The up query for cards:
```sql
-- create_cards up.sql
CREATE TYPE STATUS_ENUM AS ENUM ('todo', 'doing', 'done');
CREATE TABLE IF NOT EXISTS cards (
id BIGSERIAL PRIMARY KEY,
board_id BIGINT NOT NULL,
description TEXT NOT NULL,
status STATUS_ENUM NOT NULL DEFAULT 'todo',
created_at TIMESTAMP WITH TIME ZONE NOT NULL DEFAULT (CURRENT_TIMESTAMP AT TIME ZONE 'utc'),
CONSTRAINT board_fk
FOREIGN KEY (board_id)
REFERENCES boards(id)
ON DELETE CASCADE
);
-- seed db with some test data for local dev
INSERT INTO cards
(board_id, description, status)
VALUES
(1, 'Test card 1', 'todo'),
(1, 'Test card 2', 'doing'),
(1, 'Test card 3', 'done'),
(2, 'Test card 4', 'todo'),
(2, 'Test card 5', 'todo'),
(3, 'Test card 6', 'done'),
(3, 'Test card 7', 'done');
```
And the down query for cards:
```sql
-- create_cards down.sql
DROP TABLE IF EXISTS cards;
```
After writing our migrations we can run them with this command:
```bash
diesel migration run
```
This executes the migrations in chronological order and also writes a Diesel schema file which should look like something like this at this point:
```rust
// src/schema.rs
table! {
boards (id) {
id -> Int8,
name -> Text,
created_at -> Timestamptz,
}
}
table! {
cards (id) {
id -> Int8,
board_id -> Int8,
description -> Text,
status -> Status_enum,
created_at -> Timestamptz,
}
}
joinable!(cards -> boards (board_id));
allow_tables_to_appear_in_same_query!(
boards,
cards,
);
```
The above file will always be generated by diesel-cli and is not something we should ever try to edit by hand, however the `diesel setup` command from earlier also generates a `diesel.toml` configuration file which we can edit if we need to configure or modify how the Diesel schema is generated:
```toml
# diesel.toml
[print_schema]
file = "src/schema.rs"
```
This will be useful to know for later.
### Executing SQL Queries w/Diesel
crates
- diesel
```diff
# Cargo.toml
[package]
name = "kanban"
version = "0.1.0"
edition = "2018"
[dependencies]
dotenv = "0.15"
chrono = { version = "0.4", features = ["serde"] }
log = "0.4"
fern = "0.6"
serde = { version = "1.0", features = ["derive"] }
serde_json = "1.0"
+diesel = { version = "1.4", features = ["postgres", "chrono"] }
```
We enable the postgres and chrono feature flags since we'll be connecting to PostgreSQL and deserializing PostgreSQL's timestamp types to chrono types. Updated main file:
```diff
// src/main.rs
+#[macro_use]
+extern crate diesel;
mod logger;
mod models;
+mod schema;
type StdErr = Box<dyn std::error::Error>;
fn main() -> Result<(), StdErr> {
dotenv::dotenv()?;
logger::init()?;
Ok(())
}
```
Before decorating our models with Diesel's derive macros we have to bring the generated types from the Diesel schema file into scope:
```diff
// src/models.rs
+use crate::schema::*;
// etc
```
However, if we currently try to run `cargo check` we'd get this:
```none
error[E0412]: cannot find type `Status_enum` in this scope
--> src/schema.rs:14:19
|
14 | status -> Status_enum,
| ^^^^^^^^^^^ not found in this scope
```
Uh oh, now what?
#### Mapping DB Enums to Rust Enums
As you may recall from the preceding section we defined an enum type in PostgreSQL like so:
```sql
CREATE TYPE STATUS_ENUM AS ENUM ('todo', 'doing', 'done');
```
Unfortunately Diesel does not support mapping DB enums to Rust enums out-of-the-box in a convenient way, so we have to pull in an unofficial 3rd-party library to do this for us:
```diff
# Cargo.toml
[package]
name = "kanban"
version = "0.1.0"
edition = "2018"
[dependencies]
dotenv = "0.15"
chrono = { version = "0.4", features = ["serde"] }
log = "0.4"
fern = "0.6"
serde = { version = "1.0", features = ["derive"] }
serde_json = "1.0"
diesel = { version = "1.4", features = ["postgres", "chrono"] }
+diesel-derive-enum = { version = "1.1", features = ["postgres"] }
```
Then we decorate the enum type with the following derive macro and attribute:
```diff
-#[derive(serde::Serialize, serde::Deserialize)]
+#[derive(serde::Serialize, serde::Deserialize, diesel_derive_enum::DbEnum)]
#[serde(rename_all = "camelCase")]
+#[DieselType = "Status_enum"]
pub enum Status {
Todo,
Doing,
Done,
}
```
The derive macro generates a new type called `Status_enum` within the same scope, and this type can be used by Diesel to map the `STATUS_ENUM` DB enum to our `Status` Rust enum.
Then we update `diesel.toml` with this:
```diff
[print_schema]
file = "src/schema.rs"
+import_types = ["diesel::sql_types::*", "crate::models::Status_enum"]
```
Then we re-generate the Diesel schema file using this command:
```bash
diesel print-schema > ./src/schema.rs
```
Which updates our Diesel schema file to this:
```diff
// src/schema.rs
table! {
+ use diesel::sql_types::*;
+ use crate::models::Status_enum;
boards (id) {
id -> Int8,
name -> Text,
created_at -> Timestamptz,
}
}
table! {
+ use diesel::sql_types::*;
+ use crate::models::Status_enum;
cards (id) {
id -> Int8,
board_id -> Int8,
description -> Text,
status -> Status_enum,
created_at -> Timestamptz,
}
}
joinable!(cards -> boards (board_id));
allow_tables_to_appear_in_same_query!(
boards,
cards,
);
```
And now when we run `cargo check` everything type checks just fine. Wow, that was quite the annoying hassle, but thankfully it's over.
#### Fetching Data
Alright, now let's impl some Diesel traits for our models by using the handy derive macros:
```diff
-#[derive(serde::Serialize)]
+#[derive(serde::Serialize, diesel::Queryable)]
#[serde(rename_all = "camelCase")]
pub struct Board {
pub id: i64,
pub name: String,
pub created_at: chrono::DateTime<chrono::Utc>,
}
```
Deriving `diesel::Queryable` for a struct allows it to be returned as the result of a Diesel query. The order of the struct's members must match the order of the columns in the rows returned from the Diesel query.
The schema file generated by Diesel exports a DSL we can use to craft DB queries with. One of the major benefits of using the DSL is that it gives us verification at compile-time that all of our SQL queries are syntactically and semantically correct! Here's some commented examples:
```rust
use diesel::prelude::*;
use diesel::PgConnection;
// generated by diesel-cli from DB schema
use crate::schema::boards;
// handwritten by us
use crate::models::Board;
// example of connecting to PostgreSQL
fn get_connection() -> PgConnection {
dotenv::dotenv().unwrap();
let db_url = env::var("DATABASE_URL").unwrap();
PgConnection::establish(&db_url).unwrap()
}
// fetches all boards from the boards table,
// diesel can map the DB rows to the Board model
// because we derived Queryable for it and the
// DB row data matches the Board's members
fn all_boards(conn: &PgConnection) -> Vec<Board> {
boards::table
.load(conn)
.unwrap()
}
// fetching boards in chronological order
fn all_boards_chronological(conn: &PgConnection) -> Vec<Board> {
boards::table
.order_by(boards::created_at.asc())
.load(conn)
.unwrap()
}
// fetching a board by its id
fn board_by_id(conn: &PgConnection, board_id: i64) -> Board {
boards::table
.filter(boards::id.eq(board_id))
.first(conn)
.unwrap()
}
// fetching boards by their exact name
fn board_by_name(conn: &PgConnection, board_name: &str) -> Vec<Board> {
boards::table
.filter(boards::name.eq(board_name))
.load(conn)
.unwrap()
}
// fetching boards if their names contain some string
fn board_name_contains(conn: &PgConnection, contains: &str) -> Vec<Board> {
// in LIKE queries "%" means "match zero or more of any character"
let contains = format!("%{}%", contains);
boards::table
.filter(boards::name.ilike(contains))
.load(conn)
.unwrap()
}
// fetching boards created within the past 24 hours
fn recent_boards(conn: &PgConnection) -> Vec<Board> {
let past_day = chrono::Utc::now() - chrono::Duration::days(1);
boards::table
.filter(boards::created_at.ge(past_day))
.load(conn)
.unwrap()
}
// fetching boards create within the past 24 hours
// AND whose name contains some string
// by chaining filter methods
fn recent_boards_and_name_contains(conn: &PgConnection, contains: &str) -> Vec<Board> {
let contains = format!("%{}%", contains);
let past_day = chrono::Utc::now() - chrono::Duration::days(1);
boards::table
.filter(boards::name.ilike(contains))
.filter(boards::created_at.ge(past_day))
.load(conn)
.unwrap()
}
// fetching boards create within the past 24 hours
// AND whose name contains some string
// by composing predicate expressions
fn recent_boards_and_name_contains_2(conn: &PgConnection, contains: &str) -> Vec<Board> {
let contains = format!("%{}%", contains);
let past_day = chrono::Utc::now() - chrono::Duration::days(1);
let predicate = boards::name.ilike(contains).and(boards::created_at.ge(past_day));
boards::table
.filter(predicate)
.load(conn)
.unwrap()
}
// fetching boards create within the past 24 hours
// OR whose name contains some string
// by chaining filter methods
fn recent_boards_or_name_contains(conn: &PgConnection, contains: &str) -> Vec<Board> {
let contains = format!("%{}%", contains);
let past_day = chrono::Utc::now() - chrono::Duration::days(1);
boards::table
.filter(boards::name.ilike(contains))
.or_filter(boards::created_at.ge(past_day))
.load(conn)
.unwrap()
}
// fetching boards create within the past 24 hours
// OR whose name contains some string
// by composing predicate expressions
fn recent_boards_or_name_contains_2(conn: &PgConnection, contains: &str) -> Vec<Board> {
let contains = format!("%{}%", contains);
let past_day = chrono::Utc::now() - chrono::Duration::days(1);
let predicate = boards::name.ilike(contains).or(boards::created_at.ge(past_day));
boards::table
.filter(predicate)
.load(conn)
.unwrap()
}
```
Okay, let's also derive `diesel::Queryable` for cards:
```diff
-#[derive(serde::Serialize)]
+#[derive(serde::Serialize, diesel::Queryable)]
#[serde(rename_all = "camelCase")]
pub struct Card {
pub id: i64,
pub board_id: i64,
pub description: String,
pub status: Status,
pub created_at: chrono::DateTime<chrono::Utc>,
}
```
We went over a bunch of query examples above but here's a few more:
```rust
use diesel::prelude::*;
use diesel::PgConnection;
// generated by diesel-cli from DB schema
use crate::schema::cards;
// handwritten by us
use crate::models::{Card, Status};
// fetch all cards
fn all_cards(conn: &PgConnection) -> Vec<Card> {
cards::table
.load(conn)
.unwrap()
}
// fetch cards by board
fn cards_by_board(conn: &PgConnection, board_id: i64) -> Vec<Card> {
cards::table
.filter(cards::board_id.eq(board_id))
.load(conn)
.unwrap()
}
// fetch cards by status
fn cards_by_status(conn: &PgConnection, status: Status) -> Vec<Card> {
cards::table
.filter(cards::status.eq(status))
.load(conn)
.unwrap()
}
```
So it seems like we can craft almost every query our server needs to support using Diesel's DSL. _Almost_. One of the queries we'd like to support is returning a "board summary" which is the count of all the cards within a board grouped by their status. This is how we'd write the SQL query:
```sql
SELECT count(*), status
FROM cards
WHERE cards.board_id = $1
GROUP BY status;
```
Unfortunately we cannot use Diesel's generated DSL to craft this query. Like all ORMs though, Diesel provides a method to run SQL directly, and that method is `diesel::sql_query` so let's talk about how we'd use that to get our board summary.
First we need to create a new model which will be the result of our query and derive `diesel::QueryableByName` for it as well as decorate its members with the diesel types they map to:
```rust
#[derive(Default, serde::Serialize)]
pub struct BoardSummary {
pub todo: i64,
pub doing: i64,
pub done: i64,
}
// this will be the result of our diesel::sql_query query
#[derive(diesel::QueryableByName)]
pub struct StatusCount {
#[sql_type = "diesel::sql_types::BigInt"]
pub count: i64,
#[sql_type = "Status_enum"]
pub status: Status,
}
// converting from a list of StatusCounts to a BoardSummary
impl From<Vec<StatusCount>> for BoardSummary {
fn from(counts: Vec<StatusCount>) -> BoardSummary {
let mut summary = BoardSummary::default();
for StatusCount { count, status } in counts {
match status {
Status::Todo => summary.todo += count,
Status::Doing => summary.doing += count,
Status::Done => summary.done += count,
}
}
summary
}
}
```
Why do structs which are the result of "regular" Diesel queries have to derive `diesel::Queryable` but structs which are the result of `diesel::sql_query` queries have to derive `diesel::QueryableByName`? According to the Diesel docs the former deserializes DB row columns by index and the latter deserializes DB row columns by name, and the first one is more performant but the latter is a bit more foolproof, hence its use for arbitrary SQL queries.
Anyway, after jumping through all of the hoops above we can finally implement a function to fetch board summaries:
```rust
// fetching summary of cards on a board by id
fn board_summary(conn: &PgConnection, board_id: i64) -> BoardSummary {
diesel::sql_query(format!(
"SELECT count(*), status FROM cards WHERE cards.board_id = {} GROUP BY status",
board_id
))
.load::<StatusCount>(conn)
.unwrap()
.into()
}
```
#### Inserting Data
We don't want to just fetch boards, we'd like to create them as well! As a reminder, the `CreateBoard` model only has a `name` member because the `id` and `created_at` members will be set by the DB. And this is how we'd update our `CreateBoard` model to use it with Diesel's DSL:
```diff
// boards has to be in scope to be used in table_name attribute
use crate::schema::boards;
-#[derive(serde::Deserialize)]
+#[derive(serde::Deserialize, diesel::Insertable)]
+#[table_name = "boards"]
pub struct CreateBoard {
pub name: String,
}
```
After deriving an `diesel::Insertable` impl for `CreateBoard` we can use it directly in Diesel insert queries:
```rust
use diesel::prelude::*;
use diesel::PgConnection;
use crate::schema::boards;
use crate::models::{Board, CreateBoard};
// create and return new board from CreateBoard model
fn create_board(conn: &PgConnection, create_board: CreateBoard) -> Board {
diesel::insert_into(boards::table)
.values(&create_board)
.get_result(conn) // return inserted board result
.unwrap()
}
// same as above except without using the CreateBoard model
fn create_board_2(conn: &PgConnection, board_name: String) -> Board {
diesel::insert_into(boards::table)
.values(boards::name.eq(board_name))
.get_result(conn) // return inserted board result
.unwrap()
}
```
We follow the same steps for the `CreateCard` struct:
```diff
// cards has to be in scope to be used in table_name attribute
use crate::schema::cards;
- #[derive(serde::Deserialize)]
+ #[derive(serde::Deserialize, diesel::Insertable)]
#[serde(rename_all = "camelCase")]
+ #[table_name = "cards"]
pub struct CreateCard {
pub board_id: i64,
pub description: String,
}
```
Example create functions:
```rust
use diesel::prelude::*;
use diesel::PgConnection;
use crate::schema::cards;
use crate::models::{Card, CreateCard};
// create and return new card using CreateCard model
fn create_card(conn: &PgConnection, create_card: CreateCard) -> Card {
diesel::insert_into(cards::table)
.values(&create_card)
.get_result(conn) // return inserted card result
.unwrap()
}
// same as above except without using CreateCard model
fn create_card_2(conn: &PgConnection, board_id: i64, description: String) -> Card {
diesel::insert_into(cards::table)
.values((cards::board_id.eq(board_id), cards::description.eq(description)))
.get_result(conn) // returns inserted card result
.unwrap()
}
```
#### Updating Data
If we derive `diesel::AsChangeSet` for the `UpdateCard` model:
```diff
// cards has to be in scope to be used in table_name attribute
use crate::schema::cards;
-#[derive(serde::Deserialize)]
+#[derive(serde::Deserialize, diesel::AsChangeset)]
+#[table_name = "cards"]
pub struct UpdateCard {
pub description: String,
pub status: Status,
}
```
We can now pass `UpdateCard` directly to the `set` method of Diesel update queries:
```rust
use diesel::prelude::*;
use diesel::PgConnection;
use crate::schema::cards;
use crate::models::{Card, UpdateCard};
// update card in DB using UpdateCard model
fn update_card(conn: &PgConnection, card_id: i64, update_card: UpdateCard) -> Card {
diesel::update(cards::table.filter(cards::id.eq(card_id)))
.set(update_card)
.get_result(conn) // return updated card result
.unwrap()
}
// same as above except without using model
fn update_card_2(conn: &PgConnection, card_id: i64, description: String, status: Status) -> Card {
diesel::update(cards::table.filter(cards::id.eq(card_id)))
.set((cards::description.eq(description), cards::status.eq(status)))
.get_result(conn) // return updated card result
.unwrap()
}
```
#### Deleting Data
Some delete query examples:
```rust
use
gitextract_5feznwjv/ ├── .gitignore ├── assets/ │ └── main.rs ├── license-apache ├── license-mit ├── posts/ │ ├── chat-server.md │ ├── common-rust-lifetime-misconceptions.md │ ├── learning-rust-in-2020.md │ ├── learning-rust-in-2024.md │ ├── restful-api-in-sync-and-async-rust.md │ ├── rust-in-non-rust-servers.md │ ├── sizedness-in-rust.md │ ├── too-many-brainfuck-compilers.md │ ├── tour-of-rusts-standard-library-traits.md │ ├── translations/ │ │ ├── jp/ │ │ │ └── common-rust-lifetime-misconceptions.md │ │ ├── rus/ │ │ │ └── common-rust-lifetime-misconceptions.md │ │ ├── tr/ │ │ │ └── why-blog.md │ │ └── zh-hans/ │ │ ├── chat-server.md │ │ ├── common-rust-lifetime-misconceptions.md │ │ ├── learning-rust-in-2020.md │ │ ├── learning-rust-in-2024.md │ │ ├── rust-in-non-rust-servers.md │ │ ├── sizedness-in-rust.md │ │ ├── tour-of-rusts-standard-library-traits.md │ │ └── why-blog.md │ └── why-blog.md └── readme.md
SYMBOL INDEX (1 symbols across 1 files)
FILE: assets/main.rs
function main (line 1) | fn main() {
Condensed preview — 26 files, each showing path, character count, and a content snippet. Download the .json file or copy for the full structured content (1,022K chars).
[
{
"path": ".gitignore",
"chars": 13,
"preview": ".DS_Store\nwip"
},
{
"path": "assets/main.rs",
"chars": 105,
"preview": "fn main() {\n println!(\"hopefully github detects this file and categorizes my repo as a Rust repo\");\n}\n"
},
{
"path": "license-apache",
"chars": 9723,
"preview": " Apache License\n Version 2.0, January 2004\n http"
},
{
"path": "license-mit",
"chars": 1023,
"preview": "Permission is hereby granted, free of charge, to any\nperson obtaining a copy of this software and associated\ndocumentati"
},
{
"path": "posts/chat-server.md",
"chars": 64450,
"preview": "# Beginner's Guide to Concurrent Programming: Coding a Multithreaded Chat Server using Tokio\n\n_04 May 2024 · #rust · #as"
},
{
"path": "posts/common-rust-lifetime-misconceptions.md",
"chars": 41801,
"preview": "# Common Rust Lifetime Misconceptions\n\n_19 May 2020 · #rust · #lifetimes_\n\n**Table of Contents**\n- [Intro](#intro)\n- [Th"
},
{
"path": "posts/learning-rust-in-2020.md",
"chars": 20317,
"preview": "# Learning Rust in 2020\n\n_09 May 2020 · #rust · #programming · #exercises_\n\n**Table of Contents**\n- [Intro](#intro)\n- [T"
},
{
"path": "posts/learning-rust-in-2024.md",
"chars": 26388,
"preview": "# Learning Rust in 2024\n\n_21 September 2024 · #rust · #coding · #exercises_\n\n**Table of contents**\n- [TL;DR](#tldr)\n- [0"
},
{
"path": "posts/restful-api-in-sync-and-async-rust.md",
"chars": 127350,
"preview": "\n# RESTful API in Sync & Async Rust\n\n_11 May 2021 · #rust · #diesel · #rocket · #sqlx · #actix-web_\n\n**Table of Contents"
},
{
"path": "posts/rust-in-non-rust-servers.md",
"chars": 25206,
"preview": "\n\n# Using Rust in Non-Rust Servers to Improve Performance\n_22 October 2024 · #rust · #wasm · #ffi · #performance_\n\n\n\n**T"
},
{
"path": "posts/sizedness-in-rust.md",
"chars": 44573,
"preview": "# Sizedness in Rust\n\n_22 July 2020 · #rust · #sizedness_\n\n**Table of Contents**\n\n- [Intro](#intro)\n- [Sizedness](#sizedn"
},
{
"path": "posts/too-many-brainfuck-compilers.md",
"chars": 77254,
"preview": "# Learn Assembly by Writing Entirely Too Many Brainfuck Compilers\n\n_01 November 2020 · #assembly · #compilers_\n\n**Table "
},
{
"path": "posts/tour-of-rusts-standard-library-traits.md",
"chars": 143876,
"preview": "# Tour of Rust's Standard Library Traits\n\n_31 March 2021 · #rust · #traits_\n\n**Table of Contents**\n- [Intro](#intro)\n- ["
},
{
"path": "posts/translations/jp/common-rust-lifetime-misconceptions.md",
"chars": 30942,
"preview": "# Rustのライフタイムについてのよくある誤解\n\n2020年5月19日 · 読了時間 30分 · #rust · #lifetimes\n\nTranslated by [Yusuke Kominami](https://github.com"
},
{
"path": "posts/translations/rus/common-rust-lifetime-misconceptions.md",
"chars": 47674,
"preview": "# Распространенные заблуждения о временах жизни в Rust\n\n*19 мая 2020 г. · 37 минут · #rust · #lifetimes*\n\n**Оглавление**"
},
{
"path": "posts/translations/tr/why-blog.md",
"chars": 2881,
"preview": "# Neden Blog?\n\n_02 May 2020 · #blogging_\n\nRust'ı öğrenme sürecinde kavramları anlamlandırmaya ve düzene sokmaya çalışıyo"
},
{
"path": "posts/translations/zh-hans/chat-server.md",
"chars": 46005,
"preview": "# 并发编程新手指南: 使用Tokio实现多线程聊天服务器\n\n_2024年5月4日 · #rust · #async · #concurrency · #tokio_\n\n![聊天服务器演示](../../../assets/chat-ser"
},
{
"path": "posts/translations/zh-hans/common-rust-lifetime-misconceptions.md",
"chars": 29299,
"preview": "# Rust 中常见的有关生命周期的误解\n\n_2020 年 5 月 19 日 · 预计阅读 30 分钟 · #rust · #生命周期_\n\n译者：[SHJ](https://github.com/Foreverhighness)\n\n**目录"
},
{
"path": "posts/translations/zh-hans/learning-rust-in-2020.md",
"chars": 7439,
"preview": "# 2020 年的 Rust 学习指南\n\n_2020年 5月 9日 · #rust · #programming · #exercises_\n\n**目录**\n- [前言](#前言)\n- [省流](#省流)\n- [Rust 练习平台比较](#"
},
{
"path": "posts/translations/zh-hans/learning-rust-in-2024.md",
"chars": 14478,
"preview": "# 2024 年的 Rust 学习指南\n\n_2024 年 9 月 21 日 · #rust · #coding · #exercises_\n\n**目录**\n\n- [TL; DR](#tl-dr)\n- [0）参考资料](#0参考资料)\n- ["
},
{
"path": "posts/translations/zh-hans/rust-in-non-rust-servers.md",
"chars": 18629,
"preview": "# 在非 Rust 服务器中逐步引入 Rust 以提高性能\n\n_2024 年 10 月 22 日 · #rust · #wasm · #ffi · #performance_\n\n**目录**\n\n- [介绍](#介绍)\n- [策略](#策略)"
},
{
"path": "posts/translations/zh-hans/sizedness-in-rust.md",
"chars": 30721,
"preview": "# Rust中的Sizedness\n\n_22 July 2020 · #rust · #sizedness_\n\n**目录**\n\n- [介绍](#介绍)\n- [Sizedness(大小确定性)](#sizedness大小确定性)\n- [`Si"
},
{
"path": "posts/translations/zh-hans/tour-of-rusts-standard-library-traits.md",
"chars": 168377,
"preview": "# Rust 标准库特性指南\n\n_2021年 3月 31日 · #rust · #traits_\n\n**目录**\n\n- [引入 Intro](#引入-intro)\n- [特性的基础知识 Trait Basics](#特性的基础知识-trai"
},
{
"path": "posts/translations/zh-hans/why-blog.md",
"chars": 1467,
"preview": "# 为什么写博客？\n\n_2020年 5月 2日 · #blogging_\n\n在学习 Rust 时我一直被几个概念所困扰，所以我尝试通过写 Markdown 组织我的想法。\n我很快便有了好几个 Markdown 文件。我想， _也许可以开始写"
},
{
"path": "posts/why-blog.md",
"chars": 3470,
"preview": "# Why blog?\n\n_02 May 2020 · #blogging_\n\nWhile learning Rust I was struggling with certain concepts so I tried to organiz"
},
{
"path": "readme.md",
"chars": 3109,
"preview": "# pretzelhammer's Rust blog 🦀\n\nI write educational content for Rust beginners and Rust advanced beginners.\n\nHere are my "
}
]
About this extraction
This page contains the full source code of the pretzelhammer/rust-blog GitHub repository, extracted and formatted as plain text for AI agents and large language models (LLMs). The extraction includes 26 files (963.4 KB), approximately 301.0k tokens, and a symbol index with 1 extracted functions, classes, methods, constants, and types. Use this with OpenClaw, Claude, ChatGPT, Cursor, Windsurf, or any other AI tool that accepts text input. You can copy the full output to your clipboard or download it as a .txt file.
Extracted by GitExtract — free GitHub repo to text converter for AI. Built by Nikandr Surkov.