Haixuan Xavier Tao

Reading

Reading in Polars

Reading in Polars is pretty straightforward:

use polars::prelude::*;

//...

    let mut df = CsvReader::from_path(path)?
        .with_n_threads(Some(1)) // comment for multithreading
        .with_encoding(CsvEncoding::LossyUtf8)
        .has_header(true)
        .finish()?;

Reading in Native Rust

Reading in Rust using csv and serde requires that you already have a struct, in my case my struct is utils::NativeDataFrame

    let file = File::open(path)?;

    let mut rdr = csv::ReaderBuilder::new().delimiter(b',').from_reader(file);
    let mut records: Vec<utils::NativeDataFrame> = rdr
        .deserialize()
        .into_iter()
        .filter_map(|result| match result {
            Ok(rec) => rec,
            Err(e) => None,
        })
        .collect();

Performance

For reading, Polars is faster than Pandas and Native Rust, being able to do it in multithreading.