Rust on Yun Sheng's Site

Rust Learning: Modules

Sat, 27 Jun 2026 10:48:21 -0400

Overview

Rust modules is one of those things where I thought I learned it but it keeps giving me surprises.

I was recently contributing to an inner source rust project and got tripped by modules and realized that I don’t actually understand them.

So here I am trying to learn again and write things down.

I’ll go over the basics which explains the concept and syntax. Then I’ll show some idioms people use.

Might not be exhaustive but this is aligning with my understanding.

tldr

Rust’s module tree is explicit, you have to write it yourself.
mod is the keyword for you to build the module tree.
pub mod abc declares module abc and makes it public.
use is the keyword to “import” other modules, it creates shortcuts.
pub use abc creates shortcuts and shares that with users of your module.
To refer something in a sibling module, you need to use relative path super::sibling_name or absolute paths crate::path_to_sibling.

The Basics

Module Tree

I believe this is a rust concept, it means what module structure(tree) does your program has.

This isn’t a thing in languages like Python because in those languages the module tree maps to the file system.

Say in python I want to create my math package with four operators in it, I would do the following in mymath.py

mymath.py

# mymath.py

def add(a, b):
    return a + b

def minus(a, b):
    return a - b

def multiply(a, b):
    return a * b

def divide(a, b):
    return a / b

And in other places they could simply import mymath and use mymath.add() or from mymath import add then just use add()

In rust having mymath.rs is not enough, you need to explicitly tell the compiler you have a module.

`mod` keyword - to declare or define modules in the module tree

mod keyword is the keyword to declare or define modules in rust. Note that declare and define a module are slightly different.

When using the inline syntax mod defines the module, when using the file+folder way mod declares the module.

To define a module, there are three ways.

(1) Inline

src/main.rs

// src/main.rs
mod my_module {
    pub fn add(a: i32, b: i32) -> i32 {
        a + b
    }
}
fn main() {
    let three = my_module::add(1, 2);
    println!("Hello, world! {}", three);
}

In this way we defined and declared the module at the same time.

(2) File and Directory

The above feels like good tool to do ad-hoc things. Almost feels like in C++ where you open a namespace anywhere and starts adding things in it. A more natural way IMO would be using files and directories.

In rust there exists two ways, one of them being “legacy”, and the other one is the recommended way.

(2.1) [legacy] mod.rs + module_name/

You simply create a folder and name the folder name to be the module name.

Then you would need to have a mod.rs in the folder. You would use mod.rs to declare which modules are available to the outside world.

It could also re-export some internal modules to a different path. We’ll go over re-export in the use section.

folder structure

# folder structure
.
├── Cargo.lock
├── Cargo.toml
└── src
    ├── main.rs
    └── my_module
        ├── mod.rs
        └── operations.rs

3 directories, 5 files

src/my_module/mod.rs

// src/my_module/mod.rs

// this matches the file name operations.rs
pub mod operations;

src/my_module/operations.rs

// src/my_module/operations.rs

// pub here is needed otherwise this would be a private function
pub fn add(a: i32, b: i32) -> i32 {
    a + b
}

src/main.rs

// src/main.rs

// this is to declare the module
// module name is the directory name
mod my_module;

fn main() {
    let three = my_module::operations::add(1, 2);
    println!("{}", three);
}

(2.2) [recommended] module_name.rs + module_name/

The above is nice but there is one catch. In a big project there would be a lot of mod.rs files. Later a new way was introduced, essentially pulling mod.rs out of the folder and name it same as the folder (module name).

folder structure

# folder structure

.
├── Cargo.lock
├── Cargo.toml
└── src
    ├── main.rs
    ├── my_module
    │   └── operations.rs
    └── my_module.rs

3 directories, 5 files

I essentially pulled mod.rs out and renamed it. The rest is the same but for completeness sake I’ll put them below.

src/my_module.rs

// src/my_module.rs

// this matches the file name operations.rs
pub mod operations;

src/my_module/operations.rs

// src/my_module/operations.rs

// pub here is needed otherwise this would be a private function
pub fn add(a: i32, b: i32) -> i32 {
    a + b
}

src/main.rs

// src/main.rs

// this is to declare the module
// module name is the directory name
mod my_module;

fn main() {
    let three = my_module::operations::add(1, 2);
    println!("{}", three);
}

Visibility: public or private

You might have noticed that in the code example above there are some pub keywords here and there.

Everything in rust is defaulted to be private. Adding pub simply makes the declared module public and can be accessed outside.

Every module can have child modules, and in order to use something from the child module, the child module must make it visible for the parent.

Let’s look at a concrete example.

Say we have my_module and inside my_module we have operations, let’s say we need some utility functions that are only available for operations but we don’t want it to be exposed to users of my_module.

We would (1) create a sub module under operations, mark the functions as pub and then (2) declare the module in places we need to use. and (3) do not expose it in operations.rs

.
├── Cargo.lock
├── Cargo.toml
└── src
    ├── main.rs
    ├── my_module
    │   ├── operations
    │   │   └── utils.rs
    │   └── operations.rs
    └── my_module.rs

4 directories, 6 files

I’ll only pick the interesting files here

src/my_module/operations.rs

// src/my_module/operations.rs

// modules_playground::my_module::operations::utils
// this is to declare `utils` in the module tree.
// `utils` is a child of `operations`
mod utils;

pub fn add(a: i32, b: i32) -> i32 {
    utils::utility_func();
    a + b
}

src/my_module/operations/utils.rs

// src/my_module/operations/utils.rs
pub fn utility_func() {
    println!("I'm a utility function in my_module::operations::utils module");
}

The pub on utility_func in utils.rs is required, otherwise the function won’t be accessible in operations.rs.

The mod utils; in operations.rs is important, because it brought utils into the module tree. Mounted utils as a child of operations.

Note that in main.rs there is no access to utils::utility_func because it was not made public in operations.rs.

To make utils::utility_func available in main.rs, we could write pub mod utils instead of mod utils in operations.rs. The line pub mod utils does two things.

It mounts the module to the module tree.
It makes the module public for whoever is mounting the current module. (operations) in this case.

It can be confusing because sometimes when slapping on the pub keyword, a single line does two things.

There are also other fine grained control of visibility like pub(crate), pub(super) but we’ll go over them later.

Using modules from sibling

There is a tool called cargo modules, using that tool you can check the module tree.

Running cargo modules structure at the root of the project shows the following output

crate modules_playground
├── fn main: pub(crate)
└── mod my_module: pub(crate)
    └── mod operations: pub
        ├── fn add: pub
        └── mod utils: pub(self)
            └── fn utility_func: pub

The utils module we created is under operations, what if I want to have a utils module under my_module and share it across my_module?

In order to do this, let’s define the utils module under my_module by creating a utils.rs file inside my_module/ directory. Then in my_module.rs we need to declare it with mod utils;

.
├── Cargo.lock
├── Cargo.toml
└── src
    ├── main.rs
    ├── my_module
    │   ├── operations.rs
    │   └── utils.rs
    └── my_module.rs

3 directories, 6 files

src/my_module/utils.rs

// src/my_module/utils.rs

// this needs to be `pub` because it's used with other modules inside `my_module`
pub fn utility_func() {
    println!("I'm a utility function in my_module::operations::utils module");
}

src/my_module.rs

// src/my_module.rs
pub mod operations;

// we do not need to have `pub` here because this `utils` is private for my_module
mod utils;

And in order to use it from operations.rs, you need to either call it with a relative path, or call it with an absolute path.

To call it with a relative path, operations needs to go up to my_module then go down to utils.

To call it with an absolute path, operations can just specify the path from the root.

Let’s see both in action src/my_module/operations.rs

// src/my_module/operations.rs

pub fn add(a: i32, b: i32) -> i32 {
    // relative path
    super::utils::utility_func();

    // absolute path
    crate::my_module::utils::utility_func();
    a + b
}

Visibility: all pub keywords

So now we know that the mod keyword is to declare/define modules. And the pub keywork is to tweak the visibility. There are some variants of pub, I’ll list them below

pub, public everywhere
pub(crate), public in your current crate
pub(super), public to your parent(no grandparents) and sibling and kids of siblings(nephews/nieces)
pub(in path::to::module), public to the specific module path
pub(self) or ignored, public to yourself, essentially private

pub(super) granting access to nephews/nieces seems strange but it’s because of the following rule.

When you grant visibility to a module, you automatically grant visibility to every single module nested inside of it.

`mod` Summary

So we know that mod keyword declares/defines a module in the module tree. And pub keyword can change the visibility of the module. If you write them in the same line it does two things with one line.

pub mod abc; means declaring abc in the current module and make it public to whoever uses the current module.

`use` Keyword

In the last example above (calling sibling modules), we saw how to call the function with different paths

src/my_module/operations.rs

// src/my_module/operations.rs

pub fn add(a: i32, b: i32) -> i32 {
    // relative path
    super::utils::utility_func();

    // absolute path
    crate::my_module::utils::utility_func();
    a + b
}

However it would be tedious to keep writing the path.

use keyword could be used to shorten the path. It feels similar like import in Python, or use in C++.

It’s easier to think that use creates shortcuts.

We could use till the module and call the function like utils::utility_func(). Similar to import utils in Python.

src/my_module/operations.rs

// src/my_module/operations.rs
use super::utils;

pub fn add(a: i32, b: i32) -> i32 {
    utils::utility_func();
    a + b
}

Or we could use the full path and call utility_func() without utils::. Similar to from utils import utility_func in Python.

src/my_module/operations.rs

// src/my_module/operations.rs
use super::utils::utility_func;

pub fn add(a: i32, b: i32) -> i32 {
    utility_func();
    a + b
}

pub use, aka the Facade Pattern

pub use is a way to share the shortcut use created. It’s known as re-export.

Let’s look at a deeply nested module tree example.

.
├── Cargo.lock
├── Cargo.toml
└── src
    ├── main.rs
    ├── my_module
    │   ├── aaa
    │   │   └── bbb.rs
    │   └── aaa.rs
    └── my_module.rs

4 directories, 6 files

crate modules_playground
├── fn main: pub(crate)
└── mod my_module: pub(crate)
    └── mod aaa: pub
        └── mod bbb: pub
            └── fn bbb_func: pub

Use this as an exercise to see if you can fill out what is needed in each file to create the module tree like this.

src/my_module.rs

1
2

// src/my_module.rs
pub mod aaa;

src/my_module/aaa.rs

1
2

// src/my_module/aaa.rs
pub mod bbb;

src/my_module/aaa/bbb.rs

1
2

// src/my_module/aaa/bbb.rs
pub fn bbb_func() {}

src/main.rs

// src/main.rs
mod my_module;

fn main() {
    my_module::aaa::bbb::bbb_func();
}

In main.rs, it is clear that bbb_func() is deeply nested, we could use pub use inside my_module.rs to create a shortcut and make that shortcut available, so clients (main.rs) don’t really need to know about this aaa::bbb:: module tree structure.

src/my_module.rs

1
2
3

// src/my_module.rs
mod aaa; // still need to declare `aaa`, but no need to make it public
pub use aaa::bbb; // creating the shortcut and sharing it

src/main.rs

// src/main.rs
mod my_module;

fn main() {
    // using the shortcut created in `my_module.rs`
    my_module::bbb::bbb_func();
}

If we wanted we could go one step further to create a shortcut of bbb_func on my_module.

src/my_module.rs

1
2
3

// src/my_module.rs
mod aaa; // still need to declare `aaa`, but no need to make it public
pub use aaa::bbb::bbb_func; // creating the shortcut and sharing it

src/main.rs

// src/main.rs
mod my_module;

fn main() {
    // using the shortcut created in `my_module.rs`
    my_module::bbb_func();
}

The Idioms

The pub use above is one of the commonly used idiom/pattern. Let’s go over a few that I saw. This might not be exhaustive but you gotta start somewhere.

using type vs using functions

In my understanding, the convention for functions is to have the module name, the convention for types is to import the full path.

use super::utils;
use super::utils::MyType;

fn foo() {
    utils::utility_func();

    let x: MyType = 0;
}

Prelude

Some packages have stuff that most people would like to use anyways, so they export them in a prelude module and users would use *. Examples like use rayon::prelude::*;, use bevy::prelude::*;, use std::io::prelude::*;

Facade Pattern

We already covered this when we go over pub use. This is essentially creating shortcuts and share them with the users of your module.

The pattern I saw

mod my_module {
    pub struct Type1{}
    pub struct Type2{}

    pub mod my_module {
        pub use super::Type1;
        pub use super::Type2;
    }
}

pub use self::my_module::*; // without `self` there would be ambiguity, the `my_module` in this line is referring to the outer one

I saw this code when working on the inner source project.

I was super confused then I learned that the reason the modules are structured like this, is to give clients the ability to either refer Type1 or refer it with my_module::Type1.

The gotcha I got was the last line didn’t have self::, so it had ambiguity where rust didn’t know which my_module it is referring to. Because the outer sibling module and the nested module share the same name, omitting self:: causes ambiguity for the compiler over which my_module is being targeted. Adding self:: explicitly tells the compiler to look at the local module in the current scope (the outer one), fixing the ambiguity.

Rust Modules are Completely Closed

Rust modules feels like C++’s namespaces (probably because of the use keyword). C++ namespaces could be “opened” anywhere and have things added to it. It then begs the question, can rust modules be “opened” anywhere and have things added to it?

No, modules are completely closed in rust.

Rust Polars Parquet

Wed, 17 Jun 2026 07:08:39 -0400

Overview

First time I heard about Parquet is from an old friend. I recently moved to a new team that seems to work with parquet files. Hence I thought it might be a good idea to do some drills provided by LLMs. Plus I found that Polars’s original implementation is Rust, just another reason for me to practice Rust. Here is the repo of the LLM guided parquet practice.

I did some other drills like the iterator drill, the country-cli drill. But I got lazy and didn’t do the write up and now my memory isn’t that fresh.

tldr

Do write ups after finishing practicing for best return.
Use LazyFrame whenever possible
group_by destroys the rows, over keeps them
DataFrame transforms rows

What is a Parquet file?

I won’t (can’t) go into the details of what a Parquet file is. In order for me to practice the drills I needed to understand what a Parquet is at a very high level. And that is the level I’m going to explain now.

Parquet, or more specifically Apache Parquet is a file format. Files encoded in this format usually have .parquet as the suffix, the examples I used to do the drill is here.

This file format is designed for efficient data storage and retrieval. There are a few features/traits of this format

it is a binary format
it is column-oriented
it carries each column’s type information
it is compressed well (2024 Jan NY cities taxi rides is like 50mb)

Binary Format

Again, I won’t (can’t) go into the binary format itself, I think it is fun to drill down to the binary format and understand it. But for practical reasons, learning how to use it has a higher priority now. Maybe sometime later :)

What is Column Oriented?

I hear this term a lot and I don’t think I 100% understand it. I’ll try to write down my current understanding.

Before column-oriented became a buzz word every web scale kid talks about, most formats/tools we used are row-oriented.

What that means is each record has to have data of the full row. Say we have a csv file that stores basic student information and their test scores for different subjects

id,name,email,english_score,math_score,art_score
1,aaa,aaa@bb.com,84,90,NULL
2,bbb,bbb@bb.com,NULL,94,NULL
3,ccc,ccc@bb.com,NULL,NULL,98

In the example above, not all students have test scores for all 3 subjects, but we still needed to have NULL in their record. This is because we view the data as rows.

One issue with having NULLs in rows is wasted space.

In the csv above, 5 NULLs were there just as placeholders to indicate that the student didn’t take that test(yeah yeah yeah I know you could leave it empty but that’s missing the point).

You might have realized that this csv could be modeled with a relational database like Postgres and apply some normalization techniques.

Student Table
- id(pk)
- name
- email

English Test Table
- id(pk)
- student_id(fk)
- score
- questions
- answers

Math Test Table
- id(pk)
- student_id(fk)
- score
- questions
- answers

Art Test Table
- id(pk)
- student_id(fk)
- score
- questions
- result_s3_url

Aside of the detour, let’s come back to column oriented.

Had the csv example above be stored in a parquet file, it would be something like

// schema
id: Int32
name: String
email: String
english_score: Int32
math_score: Int32
art_score: Int32

// data
id:[1,2,3]
name:[aaa,bbb,ccc]
email: [aaa@bb.com, bbb@bb.com, ccc@bb.com]
english_score: [84] (Definition Levels [1, 0, 0])
math_score: [90,94]       (Definition Levels [1, 1, 0])
art_score: [98]        (Definition Levels [0, 0, 1])

// Definition Levels is at a high level how parquet knows which field is NULL

Some other benefits that come with column-oriented storage

When performing SELECT name, math_score on a row on a row-oriented format/database, you need to load the entire row in memory then throw away the columns you don’t need, whereas SELECT name, math_score on parquet could only pull out the columns you needed.
Because columns are of the same type, compression algorithms could perform very well, resulting in a very small file size.

Apache Arrow?

When doing research Apache Arrow keeps coming up. This stackoverflow post answer is awesome.

After reading it my understanding is

Apache Arrow is an in-memory format specification for columnar data
Arrow Files is a serialization protocol that allows you to serialize the data structures created by Apache Arrow. (no compression)
The Arrow Protocol is designed for you to just map the file in memory(mmap) without deserialization to use it
Whereas parquet does compression and is more for archiving purposes

Preparing the repo and printing schema

The first drill is to simply load the parquet file and print the schema.

I initially came up with

1
2
3

let f = File::open("./data/yellow_tripdata_2024-01.parquet")?;
let mut pr = ParquetReader::new(f);
println!("{:?}", pr.schema()?);

The above would return

shengy@Mac ~/repos/ruststuff/one-more-time/polars-practice (main|dirty:1) [07:11:07]
> cargo run
   Compiling polars-practice v0.1.0 (/Users/shengy/repos/ruststuff/one-more-time/polars-practice)
    Finished `dev` profile [unoptimized + debuginfo] target(s) in 0.87s
     Running `target/debug/polars-practice`
Schema { fields: {"VendorID": Field { name: "VendorID", dtype: Int32, is_nullable: true, metadata: None }, "tpep_pickup_datetime": Field { name: "tpep_pickup_datetime", dtype: Timestamp(Nanosecond, None), is_nullable: true, metadata: None }, "tpep_dropoff_datetime": Field { name: "tpep_dropoff_datetime", dtype: Timestamp(Nanosecond, None), is_nullable: true, metadata: None }, "passenger_count": Field { name: "passenger_count", dtype: Int64, is_nullable: true, metadata: None }, "trip_distance": Field { name: "trip_distance", dtype: Float64, is_nullable: true, metadata: None }, "RatecodeID": Field { name: "RatecodeID", dtype: Int64, is_nullable: true, metadata: None }, "store_and_fwd_flag": Field { name: "store_and_fwd_flag", dtype: Utf8View, is_nullable: true, metadata: None }, "PULocationID": Field { name: "PULocationID", dtype: Int32, is_nullable: true, metadata: None }, "DOLocationID": Field { name: "DOLocationID", dtype: Int32, is_nullable: true, metadata: None }, "payment_type": Field { name: "payment_type", dtype: Int64, is_nullable: true, metadata: None }, "fare_amount": Field { name: "fare_amount", dtype: Float64, is_nullable: true, metadata: None }, "extra": Field { name: "extra", dtype: Float64, is_nullable: true, metadata: None }, "mta_tax": Field { name: "mta_tax", dtype: Float64, is_nullable: true, metadata: None }, "tip_amount": Field { name: "tip_amount", dtype: Float64, is_nullable: true, metadata: None }, "tolls_amount": Field { name: "tolls_amount", dtype: Float64, is_nullable: true, metadata: None }, "improvement_surcharge": Field { name: "improvement_surcharge", dtype: Float64, is_nullable: true, metadata: None }, "total_amount": Field { name: "total_amount", dtype: Float64, is_nullable: true, metadata: None }, "congestion_surcharge": Field { name: "congestion_surcharge", dtype: Float64, is_nullable: true, metadata: None }, "Airport_fee": Field { name: "Airport_fee", dtype: Float64, is_nullable: true, metadata: None }}, metadata: {} }

Later during some refactoring, instead of getting schema from ParquetReader, I got the DataFrame from ParquetReader and called the schema() method on it.

// in main
let f = File::open("./data/yellow_tripdata_2024-01.parquet")?;
let pr = ParquetReader::new(f);
let df_jan = pr.finish()?;

// basic::drill_1_print_schema
pub fn drill_1_print_schema(df: &DataFrame) {
    println!("{:?}", df.schema());
}

shengy@Mac ~/repos/ruststuff/one-more-time/polars-practice (main|clean) [07:08:21]
> cargo run
   Compiling polars-practice v0.1.0 (/Users/shengy/repos/ruststuff/one-more-time/polars-practice)
    Finished `dev` profile [unoptimized + debuginfo] target(s) in 2.53s
     Running `target/debug/polars-practice`
Schema { fields: {"VendorID": Int32, "tpep_pickup_datetime": Datetime('ns'), "tpep_dropoff_datetime": Datetime('ns'), "passenger_count": Int64, "trip_distance": Float64, "RatecodeID": Int64, "store_and_fwd_flag": String, "PULocationID": Int32, "DOLocationID": Int32, "payment_type": Int64, "fare_amount": Float64, "extra": Float64, "mta_tax": Float64, "tip_amount": Float64, "tolls_amount": Float64, "improvement_surcharge": Float64, "total_amount": Float64, "congestion_surcharge": Float64, "Airport_fee": Float64}, metadata: () }

I just noticed that the schema printed by DataFrame has less information than the schema printed by ParquetReader though.

agy answer for this

The reason is that ParquetReader is reading the physical Parquet/Arrow schema. This physical schema includes a lot of low-level file metadata, like whether a specific column is_nullable, dictionary encodings, and parquet-specific metadata. On the other hand, DataFrame.schema() prints the logical Polars schema. Polars simplifies this representation in memory to just be a mapping of Column Name -> DataType because that’s all the logical query engine needs to know to plan operations.

DataFrame and LazyFrame

Starting from drill 5 I realized that the best practice is to take in LazyFrame if possible. LazyFrame is similar to Rust’s iterators, you can write all the operations/expressions/transformations, but it won’t be actually executed until you call collect(). And when it is executed Polars would help optimize it so it’s as fast as possible.

On the other hand operations on DataFrames are executed immediately, so there is no opportunity for Polars to help optimize it based on a new transformation down the line.

I first learned about lazy vs eager in programming languages. This, this and this are some example links.

rolling_window feature requires strings

In extra drill 1 I needed to use a method called rolling_mean. It was not available and later I found that it is under the crate feature rolling_window.

After installing this feature it was still failing with mysterious messages. Turns out the fix was to have strings also included in the features. I believe rolling_window feature depends on strings feature.

agy says

The underlying reason here isn’t necessarily that rolling_window always depends on strings. It’s because in your specific drill, you are performing a rolling window over a datetime column (tpep_pickup_datetime). Polars heavily relies on the strings feature when formatting, parsing, or interacting with datetime objects under the hood. Polars is incredibly modular, so hitting these weird feature-flag combinations is a very common rite of passage in Rust Polars!

How to read the types, Expr, PlSmallStr

When working through the basic drills I learned to read the type hints from Polars. I got very confused about when should I pass in a string like stuff vs when should I pass in col("stuff")

Examples are

let x = df
    .lazy()
    .group_by([col("tpep_pickup_datetime").dt().hour().alias("hour")])
    .agg([col("total_amount").mean(), col("trip_distance").mean()])
    .sort(
        ["hour"],
        SortMultipleOptions::new().with_order_descending(false),
    );

Why in this case group_by takes col("tpep_pickup_datetime") and why does sort take "hour"?

What I learned is to follow the types.

Expr

If we look at group_by’s type. It is the following

`1`	`pub fn group_by<E: AsRef<[IE]>, IE: Into<Expr> + Clone>(self, by: E) -> LazyGroupBy`

This means group_by takes in a slice of IE as reference, where IE is Into<Expr>. Meaning anything that implemented the Into<Expr> trait could be passed in and the result would be Expr.

So all the babbling above means group_by takes in a slice of Exprs

And in Polars, col("stuff") is an Expr.

Note that since it takes Into<Expr>, the following also works

let x = df
    .lazy()
    .group_by(["payment_type"]) // here
    .agg([col("tip_amount").mean()])
    .sort(
        ["payment_type"],
        SortMultipleOptions::new().with_order_descending(false),
    );

I didn’t pass in [col("payment_type")], instead I passed in ["payment_type"].

Here an Into happened and converted "payment_type" to an Expr

PlSmallStr

The sort method in the example above takes in ["payment_type"], but this is not converting it into an Expr.

See sort’s type

`1`	`pub fn sort(self, by: impl IntoVec<PlSmallStr>, sort_options: SortMultipleOptions) -> Self`

sort’s by parameter takes IntoVec<PlSmallStr>. Honestly I don’t know the exact details of how PlSmallStr works, but the way I read this is: it takes something that could be converted into a vector, and that something is a small string.

So when I see PlSmallStr, I know it doesn’t take Expr, so I wouldn’t pass col("stuff") to it.

DataFrame, LazyFrame

As I work on more drills, specifically extra drill 3 when I was working on over.

I noticed that group_by is a method on the Frame, whereas over is a method on the Expr. We’ll go deeper to group_by and over later.

The DataFrame methods(DataFrame, LazyFrame) often return the DataFrame. Learned from extra drill 2, most DataFrame methods transform the rows, they don’t filter. More of the below as well.

group_by vs over

Although I have been using GROUP BY in sql a lot I think I didn’t fully master it in my mind. Now with the drill I think I got a better mental model.

I think it is very important to visualize things in your mind before you go and write code. This applies to algorithms as well, it is always easier for me to “see” how the graph is being traversed and how the code is handling that.

Back to group by, let me describe how I visualize it in my mind.

Assume that we have 5 rows in the DataFrame

ID	Color	Shape	Weight
1	red	ball	10
2	green	cone	12
3	blue	cone	13
4	red	cone	14
5	green	ball	15

I want to get the average weight by color.

I would group_by color. And in my mind, after group by color, the table is broken into rows And the rows are grouped by color scattering into groups/heaps (搓堆儿 in Beijing Dialect)

It looks something like the following in my mind

Now I want to calculate the average weight of each of these groups. So I apply the calculation I have in agg for each group. In the current case it’s simply calculating the average of weight. So each group would have one number, which is the average weight of that group.

Then the final output would be two columns, the group_by field, in this case the color. And the aggregated result.

Color	Avg Weight
red	17
green	13.5
blue	13

The above is the simple example, there are a few more things to note.

group by destroys the original DataFrame

If you haven’t noticed yet, group_by destroys the original DataFrame and splits it into groups. It might not be apparent in other languages but in rust you can see it clearer because group_by takes in self, it actually consumes it.

group by multiple columns

The other thing is you could group by multiple columns, the main difference is how many groups(堆儿) would you have. In the example above if we group by (color, shape), we would have the following groups

(red, ball)
(green, cone)
(blue, cone)
(red, cone)
(green, ball)

Which doesn’t make much sense since each group has one row, but you get the idea :)

agg takes in a slice

In our example above we only calculated the average of weight of each group. But agg can take in a slice. What does it mean to pass in a slice with two elements?

This means for each group you could calculate multiple things and each of them would be a new column in the final result.

For instance we could not only calculate the average weight but also calculate the number of rows in the group, and end up with a result like

Color	Avg Weight	Count
red	17	2
green	13.5	2
blue	13	1

Now group_by is all g, let’s talk about over

over

I believe over is actually called window functions. In the drill it first came up at extra drill 3.

In some sense group_by and over is very similar, they all perform calculation on a subset of data.

But from this drill I figured out a way to visualize over.

First things first, over does not destroy the DataFrame. So when I visualize over, instead of breaking the DataFrame, I visualize it as adding a new column to the entire table.

Using the color ball example we have, say we want to calculate average weight with over, the way I would visualize it is to

Add a column to the DataFrame
Express the calculations
Specify the calculations over what
Give the column a name

(1) Add a new column

`1`	`lf.with_column(...);`

ID	Color	Shape	Weight	???
1	red	ball	10	???
2	green	cone	12	???
3	blue	cone	13	???
4	red	cone	14	???
5	green	ball	15	???

(2) Specify the calculation

1
2
3

lf.with_column(
        col("Weight").mean() // not finished yet
);

(3) Specify the calculation over what

1
2
3

lf.with_column(
        col("Weight").mean().over([col("Color")])? // question mark is because over returns PolarsResult
);

(4) Give the column a name

1
2
3

lf.with_column(
        col("Weight").mean().over([col("Color")])?.alias("Avg Weight By Color")
);

So after all these the DataFrame should look like

ID	Color	Shape	Weight	Avg Weight By Color
1	red	ball	10	17
2	green	cone	12	13.5
3	blue	cone	13	13
4	red	cone	14	17
5	green	ball	15	13.5

Notice that the original DataFrame is still there, we just added a column.

The transforms rows mindset

While doing extra drill 2 I came across learning the idea that most DataFrame transforms the rows.

The drill wanted me to filter by payment_type == 1 or payment_type == 2.

Following is the filter on LazyFrame, but something interesting is happening.

lf.filter(
    col("payment_type")
    .eq(lit(1))
    .or(col("payment_type").eq(lit(2))));

If we ignore filter, ignore the payment_type == 2 part, the code above is simplified as

`1`	`col("payment_type").eq(lit(1))`

As we were going through the different types above, col returns Expr, eq consumes Expr and returns Expr.

I initially thought that the eq would do the filtering for me, but it turns out that it only transforms the DataFrame.

The example is explained in the comments. But let me try to write it down one more time.

Say our column of payment_type is now [1, 2, 1, 1, 2].

col("payment_type") would give you [1, 2, 1, 1, 2]

Then col("payment_type").eq(lit(1)) would transform [1, 2, 1, 1, 2] to [true, false, true, true, false] by applying eq.

If we want to actually filter here we would slap on a filter like

`1`	`lf.filter(col("payment_type").eq(lit(1)));`

In the drill we wanted to calculate the percentage of people paying by credit cards(payment_type == 1).

Say col("payment_type") is [1, 2, 1, 1, 2].

payment_type == 1 would result in [true, false, true, true, false].

So the percentage is 3 / 5 (3 true out of 5 total).

There is a trick applied here, where after we got [true, false, true, true, false], we did cast(DataType::Float64). This casts true to 1.0, false to 0.0.

So [true, false, true, true, false] transformed into [1.0, 0.0, 1.0, 1.0, 0.0]. And if you run the mean on this you actually get the percentage of trues. Which is the percentage of people paid by credit cards.

I assume I already confused you, the code below probably explains it better.

Let’s look at how this calculates the percentage. Assuming the rows that fall into one specific PULocationID group have the payment types [1, 2, 1, 1, 2], here is how the expression evaluates for that group:

let x = lf
    .filter(
        col("payment_type")
            .eq(lit(1))
            .or(col("payment_type").eq(lit(2))),
    )
    .group_by([col("PULocationID")])
    .agg([
        col("payment_type")          // [1, 2, 1, 1, 2]
            .eq(lit(1))              // [true, false, true, true, false]
            .cast(DataType::Float64) // [1.0, 0.0, 1.0, 1.0, 0.0]
            .mean()                  // 3.0 / 5.0
            .alias("credit_pct"),
        col("payment_type")          // [1, 2, 1, 1, 2]
            .eq(lit(2))              // [false, true, false, false, true]
            .cast(DataType::Float64) // [0.0, 1.0, 0.0, 0.0, 1.0]
            .mean()                  // 2.0 / 5.0
            .alias("cash_pct"),
    ]);

or on lit

If we go back to the filtering part, it asks: filter by payment_type == 1 or payment_type == 2.

In the first try I actually wrote

`1`	`lf.filter(col("payment_type").eq(lit(1).or(lit(2)))); // WRONG`

This actually passed compile, but this is wrong, because or is applied on lit(1), not on the result of eq

`1`	`lf.filter(col("payment_type").eq(lit(1)).or(lit(2))); // ALSO WRONG`

This is wrong as well, because or(lit(2)) would always be true

`1`	`lf.filter(col("payment_type").eq(lit(1)).or(col("payment_type").eq(lit(2)))); // CORRECT`

The above is the correct one. Not gonna lie the need to repeat col("payment_type") is a bit annoying.

If using LazyFrame construct it with scan_parquet

The drills started by loading a DataFrame via ParquetReader. Later, agy told me that doing this is actually bad because the entire DataFrame is read into memory.

In order to only read the columns needed and leave the rest out of memory, agy suggested LazyFrame::scan_parquet.

I tried it out and it actually worked :)

Some other small things

The following are things that I noticed during the practice but I don’t think it’s that much to go over, hence grouping them in a h2 here.

with_column vs select

with_column adds new columns, where select would slice other columns and only keep the ones you selected.

columns could do math

You could do calculations for entire columns. basic drill 8 simply divided two columns.

#[allow(unused)]
pub fn drill_8_new_trip_percentage(lf: LazyFrame) -> Result<(), Box<dyn std::error::Error>> {
    // 8. Add a new column tip_percentage calculated as tip_amount / fare_amount * 100. Print the first 5 rows showing only fare_amount, tip_amount, and tip_percentage.

    let x = lf
        .filter(col("fare_amount").gt(2.5))
        .with_columns([(col("tip_amount") / col("fare_amount") * lit(100.0)).alias("tip_percentage")])
        .select([col("fare_amount"), col("tip_amount"), col("tip_percentage")])
        .sort(
            ["tip_percentage"],
            SortMultipleOptions::new().with_order_descending(true),
        )
        .limit(20);
    println!("{}", x.collect()?);
    Ok(())
}

POLARS_FMT_MAX_ROWS to show more rows

When printing the results, I noticed that some of the rows are omitted. A quick Google search shows that setting the POLARS_FMT_MAX_ROWS environment variable would allow more rows to be displayed.

Rust Learning: Deref

Fri, 29 May 2026 21:50:26 -0400

tldr

deref() returns reference because
- Rust desugars *a to *a.deref()
- Rust needs the size at compile time
- We don’t want to consume by calling deref()
deref coercion can happen
- on function parameters
- on method call expressions
- on assignments with type annotations
as_ref() and as_deref() always return references (or containers of references), which always implement Copy

Overview

I was practicing rust by doing this and this.

While chatting with Claude and trying to learn, I saw that there is this as_deref() call in the iterator chain.

This triggered some memory (trauma) about Deref and I decided to give it another shot with the help of Claude.

The following is (as usual), what I learned from teacher Claude.

The Confusion

When I first saw Deref trait, my instinct/past experience is telling me this is an “interface” that you could implement on your custom type so when you do *my_instance it triggers the specific “dereference” logic and gives you the underlying value.

Then if we look at Deref’s definition here, you can tell the required method is returning a reference instead of a concrete.

pub trait Deref {
    type Target: ?Sized;

    // Required method
    fn deref(&self) -> &Self::Target; // <----- here, returning a reference instead of Self::Target
}

So to summarize my confusion: Why is deref() in Deref trait returning a reference when it’s “dereferencing”?

The Answer

Apparently in Rust, when you call * on a type that implemented the Deref trait it would call deref() then still slap on the * operator

let boxed_value = Box::new(7);

println!("{}", *boxed_value);
// *boxed_value would be converted to *boxed_value.deref()
// where the desugar is *&i32, which gives i32

But Why?

The above seems to make sense but why can’t the deref() function return Target instead of &Target?

Claude explained that it’s because

Target might not be Sized, so its size might not be known at compile time. In Rust, types must be Sized to be passed or returned by value because the compiler needs to know how much stack space to allocate. For types like str or [T], they must live behind a pointer (like &str).
Returning Target would move it out from the containing struct, causing issues.

Deref Coercion

Now since we are reading/learning about Deref, the term Deref Coercion would be shown and we would be confused.

Deref Coercion in function parameters

Let’s look at an example below.

String implements Deref and its Target is str, its deref() returns &str.

yo() takes &str but passed &String, Rust would do a implicit conversion (coercion) from &String to &str.

fn yo(s: &str) {
    println!("{}", s);
}

let name = String::from("hi");
yo(&name); // here &String is coerced to &str

Note that Rust would do this as many times as needed. So if we have a Box that has a String in it, this would work the same.

fn yo(s: &str) {
    println!("{}", s);
}
let boxed_name = Box::new(String::from("hi"));
yo(&boxed_name); // Box<String>.deref() gets &String then String.deref() gets &str

Deref Coercion in method call expressions

Deref Coercion not only happens when passing function arguments, but also happens when we call methods.

let name = String::from("hi");

name.contains("h");
// .contains(&self, pat: P) exists on `str` but not `String`

The method call coercion doesn’t require you to add & yourself because in Method-call expressions the derefs are being considered.

The first step is to build a list of candidate receiver types. Obtain these by repeatedly dereferencing the receiver expression’s type, adding each type encountered to the list, then finally attempting an array unsized coercion at the end, and adding the result type if that is successful.

Deref Coercion in assignments with type annotations

Deref Coercion could also happen when doing an assignment with type annotations. This is kinda self explanatory so I’ll just skip.

1
2

let name = String::from("hi");
let s: &str = &name; // &String coerced to &str on assignment

1
2

let boxed_name = Box::new(String::from("hi"));
let s: &str = &boxed_name; // &Box<String> coerced to &String coerced to &str on assignment

as_deref() on Option

If you still remember, the reason I started this post is I saw this as_deref call in the iterator chain and got me thinking.

But all the discussions above don’t really help, do they?

Although the place where I was confused is on Vec, later practice shows that it might be more beneficial to understand as_deref on Options.

What does as_deref() on Option do?

Let’s look at as_ref() first

To speak about as_deref() on Options let’s have a look at as_ref() first.

Option’s as_ref says it

Converts from &Option to Option<&T>.

So given &Option<String> I can get an Option<&String>.

Yeah yeah sure sure, but what exactly does this mean?

Think about the following, if I have an Option<String>, and I have another function that takes in a &String. Is there a way for me to pass the String in the Option as a borrow to the function without consuming the original Option?

as_ref() is the answer, because it converts &Option<String> to Option<&String>, this gives you the ability to look inside the option without consuming it.

Let’s have a look at an example

fn yo(s: &String) {
    println!("{}", s);
}

let value = Some("stuff".to_owned());
if let Some(inner) = value {
    yo(&inner);
    // value is consumed here, no longer available
    // inner's type is actually String, hence the need to borrow before passing to `yo`
}

let another = Some("other stuff".to_owned());
if let Some(inner) = another.as_ref() { // as_ref() takes &self, so another is borrowed here instead of consumed
    yo(inner);
    // here `inner`'s type is actually &String
}
// `another` is still available

Now deref() should feel natural

What if the function above requires &str instead of &String?

here is where as_deref() comes into play, it not only does what as_ref() does, it also calls deref() as it sees fit to do the conversions.

fn yo(s: &str) {
    println!("{}", s);
}

let another = Some("other stuff".to_owned());
if let Some(inner) = another.as_deref() {
    // inner's type is actually `&str`
    yo(inner);
}
// another is still available here

Granted the as_deref() is not strictly needed in this specific case because &String would be coerced to &str anyways.

Then the natural question is, what is the case where I absolutely needs as_deref() over as_ref()?

The answer is when yo() requires Option<&str>

fn yo(x: Option<&str>) {
    println!("{:?}", x);
}

let another = Some("other stuff".to_owned());
yo(another.as_deref());
// another is still available
// side note: yo does consume the option but as_deref() provided a copy
// as_ref() and as_deref() always return references (or containers of references), which always implement Copy

But How are these implemented?

Now we know why does as_ref() and as_deref() exist, when and why to use them. The next question is, how are they implemented?

How are as_ref() implemented?

Let’s have a look at Option’s as_ref()

// &self is a short hand of self: &Self
pub const fn as_ref(&self) -> Option<&T> {
    match *self {
        Some(ref x) => Some(x),
        None => None,
    }
}

1
2

let value = Some("stuff".to_owned());
let x = value.as_ref(); // x's type is Option<&String>

value.as_ref() is being called
as_ref(&self) takes &self, which is a short hand of as_ref(self: &Self)
Which means Self is Option<String>
Which means self’s type is &Option<String>

Now we are clear about the types of self and Self. Let’s go to the match

We are matching *self (note that this could be written better with matching ergonomics)
As we said, self’s type is &Option<String> so matching *self means matching Option<String>
Now we are looking at Some(ref x) matching on Option<String>
- ref x in a pattern means “instead of moving the matched value, give me a reference to it”.
- So we got a reference of String which is &String
- And we returned it as Some(x) which is of type Option<&String>

// with matching ergonomics as_ref() could be written as
pub const fn as_ref(&self) -> Option<&T> {
    match self {
        Some(x) => Some(x), // because `self` is &Option<String>, `x` is automatically bound as &String
        None => None,
    }
}

How are as_deref() implemented?

Let’s see the code first.

pub const fn as_deref(&self) -> Option<&T::Target>
where
    T: [const] Deref,
{
    self.as_ref().map(Deref::deref)
}

And let’s use our example

1
2

let value = Some("stuff".to_owned());
let x = value.as_deref(); // x's type is Option<&str>

value.as_deref() being called
as_deref(&self) takes &self, which means self: &Self which means self’s type is &Option<String>
self.as_ref() is called, we get Option<&String> back
We call map with Deref::deref, so Option<&String> becomes Option<&str>
- This is because String’s Deref Target is str
- and deref() returns &Target so deref() returns &str

Bonus: DerefMut

DerefMut is the mut version of deref. If we have a &mut String, Rust can coerce it to &mut str if DerefMut is implemented.

Rust Learning: Lifetime

Wed, 20 May 2026 07:36:50 -0400

Disclaimer

I’m in the process of learning and figuring out my confusions, there might be mistakes. What I’m writing here is more of me trying to figure things out and some of my current understandings.

tldr

Can be used in functions, in structs, in traits
lifetime subtyping
'long:'short means 'long outlives 'short, also 'long is a subtype of 'short
Three syntaxes
- Right after & means annotating the lifetime of the reference
- As generic lifetime parameters
- In trait bounds
'static
- Right after & means this reference needs to be available for the processes lifetime
- T: 'static means
  - For reference fields in T, the lifetime has to be valid for the entire program
  - Or T owns the fields
- F: 'static where F is closure means
  - F should own the captured data (move closure)
  - Or F only capture 'static data.
Type outlives lifetime
- T: 'a
  - For reference fields in T, the lifetime has to outlive 'a

Overview

I recently came across learning Rust (again, like the 100th time). I thought maybe if I start writing things down, it helps solidifying my understanding a bit.

This post is about some of my confusions and thoughts that occurred when I was reading the Rust book.

My first confusion

In Chapter 10 Section 3, the example function was trying to explain how to write a function that returns the longer string without taking ownership.

1
2
3

fn longest<'a>(x: &'a str, y: &'a str) -> &'a str {
    if x.len() > y.len() { x } else { y }
}

I’ve seen this many times, but this time I wanted to dig a bit deeper.

The book says:

The function signature now tells Rust that for some lifetime ‘a, the function takes two parameters, both of which are string slices that live at least as long as lifetime ‘a. The function signature also tells Rust that the string slice returned from the function will live at least as long as lifetime ‘a.

In practice, it means that the lifetime of the reference returned by the longest function is the same as the smaller of the lifetimes of the values referred to by the function arguments. These relationships are what we want Rust to use when analyzing this code.

The part that got me thinking was both of which are string slices that live at least as long as lifetime ‘a.

Looking at the function signature, I couldn’t see where it expressed at least as long as. Say if I have a function that takes two ints, I know I need exactly two ints.

Again, long conversations with Gemini, the reasoning seems to be called lifetime subtyping. And it’s similar to the OOP concept that I am familiar with.

In the following code, we are passing in a Cat to a function that takes an Animal, and it seems normal.


class Animal {
public:
    virtual string name() = 0;
    virtual int weight() = 0;
};

class Cat : public Animal {
public:
    string name() { return "Cat"; }
    int weight() { return 20; }
};

void logAnimal(Animal *a) {
    std::cout << a.name() << " " << a.weight() << std::endl;
}

Cat *c = new Cat; // let's ignore the memory leak
logAnimal(c);

This is polymorphism 101: a Cat inherits from Animal. Wherever I need an Animal, I could pass in a Cat because Cat inherits from Animal; in other words, Cat is an Animal.

What I didn’t know was, there is another “in other words”: Cat is a subtype of Animal.

With the detour above out of the way, let’s have a look at the function again:

1
2
3

fn longest<'a>(x: &'a str, y: &'a str) -> &'a str {
    if x.len() > y.len() { x } else { y }
}

There are 4 'as, they are not all the same, the first one longest<'a> is where we are declaring a generic type 'a.

The later 3 are using 'a declared by us

If you look at lifetime 'a as a type, then combine what we said about polymorphism and subtypes, things start to make sense.

The parameter x: &'a str is saying if the passed in argument’s lifetime “satisfies” 'a then it’s good. “satisfying” here in lifetime means the lifetime passed in can not be gone before 'a. In other words, the lifetime passed in has to live at least as long as 'a.

So say the lifetime of the argument is 'b, what we are saying above is: 'b has to be at least as long as 'a.

I happen to know that there is a term in Rust called outlive. If we have two lifetimes 'long and 'short, then the syntax 'long:'short in trait bounds means 'long outlives 'short. We’ll come back to this when we talk about lifetime in trait bounds.

So now it is clear to me why did the book say both of which are string slices that live at least as long as lifetime ‘a. It is because the lifetime parameter we declared in our function signature is like a “Base Class”, you could pass in an argument with a lifetime which has a longer lifetime and it would satisfy the parameter.

The other thing is when you pass in two different lifetimes to longest, rust would find out the overlap and bind it to 'a

Lifetime on structs

With above it is clear to me that lifetimes in Rust are special generic types, and fn longest<'a>(x: &'a str, y: &'a str) -> &'a str is a generic function.

Naturally after a generic function you would think about a generic class.

This is where I got confused yet another time by the syntax. But to be fair I think the confusion comes from the generic syntax not from lifetimes(they are all generics though)

struct User<'a> {
    name: &'a str,
}

fn main() {
    let name = "my name";
    let u = User { name };
    println!("{}", u.name);
}

So far things seems pretty normal, struct User<'a> defines a struct which has a generic type parameter (in this case it’s a lifetime) 'a.

And the lifetime is used to say that the string slice you refer to that represents the name needs a lifetime, and that lifetime needs to outlive 'a

I started to feel confusing when I wanted to create a constructor.

In Rust methods are defined outside of the struct using impl.

The following is what I did at first try.

// WRONG

impl User<'a> {
    fn new(name: &'a str) -> Self {
        Self { name }
    }
}

The compile error is “undeclared lifetime”, and the fix is to add <'a> after impl

error[E0261]: use of undeclared lifetime name `'a`
 --> src/main.rs:5:11
  |
5 | impl User<'a> {
  |           ^^ undeclared lifetime
  |
help: consider introducing lifetime `'a` here
  |
5 | impl<'a> User<'a> {
  |     ++++

impl<'a> User<'a> {
    fn new(name: &'a str) -> Self {
        Self { name }
    }
}

This got me thinking, why do I need to have <'a> after impl, isn’t User<'a> enough to say that my type is taking a generic lifetime parameter?

impl<'a> is where 'a is declared as a new generic parameter. Without it, Rust sees 'a in impl User<'a> as an undeclared name — the same way using a variable before declaring it would be an error.

So now with the syntax issue resolved lifetime on structs are clear. Let’s recap

When defining a struct you can take in lifetimes
lifetimes are special generic lifetime parameters hence they follow the same generic syntax
The generic lifetime parameter is instantiated when we create an instance, it binds to the lifetime it get passed to (So our struct “passes through” the lifetime to it’s own fields)
The lifetime took in could annotate reference fields in the struct (which the struct does not own)
Rust would make sure that the lifetime we took in outlives the instance of our class

See Gemini generated example below (well Gemini just took it from here)

// 1. We declare the generic lifetime `<'a>` right after the struct name.
// 2. We apply it to the `part` field.
struct ImportantExcerpt<'a> {
    part: &'a str,
}

fn main() {
    let novel = String::from("Call me Ishmael. Some years ago...");
    let first_sentence = novel.split('.').next().unwrap();

    // The struct instance `i` now holds a reference to the data inside `novel`.
    let i = ImportantExcerpt {
        part: first_sentence,
    };

    println!("Excerpt: {}", i.part);
}

This struct has the single field part that holds a string slice, which is a reference. As with generic data types, we declare the name of the generic lifetime parameter inside angle brackets after the name of the struct so that we can use the lifetime parameter in the body of the struct definition. This annotation means an instance of ImportantExcerpt can’t outlive the reference it holds in its part field.

When we created an instance of ImportantExcerpt, it also took in a lifetime parameter, the lifetime i took in is the lifetime of novel because ImportantExcerpt took in first_sentence which is a &str which refers to novel. The lifetime tracks where the data lives(novel), not what variable you used to access it(first_sentence).

So as long as novel outlives i Rust would not complain.

Side note: Gemini actually hallucinated pretty hard here, in one version it keeps saying that 'a is the lifetime of i. I’m actually considering subscribing to Claude

Lifetime on Traits

Self Review Note:

After learning lifetimes on structs, I was looking at lifetime on Traits, the following is an example Gemini gave me. It seems like the trait bound T: 'a is redundant and is implied by Rust.

But let’s go through the process together

The Lifetime Bound

use std::fmt::Display;

// We declare a lifetime 'a and a generic type T
struct Logger<'a, T>
{
    borrowed_data: &'a T,
}

impl<'a, T> Logger<'a, T>
where
    T: Display + 'a, // This means T must implement Display and outlive 'a
{
    fn info(&self) {
        println!("INFO: {}", self.borrowed_data);
    }

    fn error(&self) {
        println!("ERROR: {}", self.borrowed_data);
    }
}

fn main() {
    let novel = String::from("Call me Ishmael. Some years ago...");

    let l = Logger {
        borrowed_data: &novel,
    };

    l.info();
    l.error();
}

And here T outlive 'a means if T contains any references, those references need to outlive 'a (those references needs to live longer than 'a)

Side topic: I learned that in rust different impl blocks can have different trait bounds, also the struct definition can have trait bounds. Seems like the idiomatic way is to keep the struct clean for bounds and have the bounds on the impl blocks

When staring at the code snippet above I just feel like there is something still not making sense. Logger takes in a T and references it borrowed_data: &'a T, 'a is the lifetime that is passed in, T is also the type that is passed in, aren’t they the same?

fn main() {
    let novel = String::from("Call me Ishmael. Some years ago...");

    let l = Logger {
        borrowed_data: &novel,
    };

    l.info();
    l.error();
}

In the above T is String, and 'a' is the lifetime of novel, feels like they are the same thing to some extent?

Claude says

1
2
3

struct Logger<'a, T> {
    borrowed_data: &'a T,  // <-- this implies T: 'a, always
}

So two more questions here

Why/how does it imply T: 'a?
In what cases do I actually need to put T: 'a in the trait bound?

For (1), the reason why rust can imply T:'a from &'a T is because

&'a T means a reference to T for a duration of 'a
Which means T needs to live at least as long as 'a
Which means T outlives 'a
Which is T: 'a

For (2) I don’t have a good answer now, but the most common case seems to be traits slapping on the 'static lifetime

The static Lifetime Bound

Here is an example of having 'static in a trait bound.

pub fn spawn<F, T>(f: F) -> JoinHandle<T>
where
    F: FnOnce() -> T + Send + 'static,
    T: Send + 'static,

Having 'static here on T means if T has any reference fields, they have to outlive 'static (which means they have to live as long as the program), or T has to own the fields.

The 'static on F is saying F should own the captured data (move closure) or only capture 'static data

Summary

The deeper I dig the more questions I have. I did find a few links that seems worth checking out

I did have a simliar feeling of leaning C++ again, where you just keep having questions over questions over questions and at some point you don’t remember why you are figuring these things out. And after you think you are crystal clear, how are you going to apply this in your own code? I’m still not sure what is the answer of that. Maybe I need to write a bit more Rust to internalize these concepts?

Rust on Yun Sheng's Site

Rust Learning: Modules

Overview

tldr

The Basics

Module Tree

mod keyword - to declare or define modules in the module tree

(1) Inline

(2) File and Directory

(2.1) [legacy] mod.rs + module_name/

(2.2) [recommended] module_name.rs + module_name/

Visibility: public or private

Using modules from sibling

Visibility: all pub keywords

mod Summary

use Keyword

pub use, aka the Facade Pattern

The Idioms

using type vs using functions

Prelude

Facade Pattern

The pattern I saw

Rust Modules are Completely Closed

Rust Polars Parquet

Overview

tldr

What is a Parquet file?

Binary Format

What is Column Oriented?

Apache Arrow?

Preparing the repo and printing schema

DataFrame and LazyFrame

rolling_window feature requires strings

How to read the types, Expr, PlSmallStr

Expr

PlSmallStr

DataFrame, LazyFrame

group_by vs over

group by destroys the original DataFrame

group by multiple columns

agg takes in a slice

over

The transforms rows mindset

or on lit

If using LazyFrame construct it with scan_parquet

Some other small things

with_column vs select

columns could do math

POLARS_FMT_MAX_ROWS to show more rows

Rust Learning: Deref

tldr

Overview

The Confusion

The Answer

But Why?

Deref Coercion

Deref Coercion in function parameters

Deref Coercion in method call expressions

Deref Coercion in assignments with type annotations

as_deref() on Option

What does as_deref() on Option do?

Let’s look at as_ref() first

Now deref() should feel natural

But How are these implemented?

How are as_ref() implemented?

How are as_deref() implemented?

Bonus: DerefMut

Rust Learning: Lifetime

Disclaimer

tldr

Overview

My first confusion

Lifetime on structs

Lifetime on Traits

The Lifetime Bound

The static Lifetime Bound

Summary

`mod` keyword - to declare or define modules in the module tree

`mod` Summary

`use` Keyword