Massive memory usage when type annotations needed about rust HOT 19 OPEN

Behold!

trait Expression {}

trait ValidGrouping {
    type IsAggregate;
}

trait MixedAggregates<Other> {}

trait NonAggregate {}

impl<T> NonAggregate for T where T: ValidGrouping {}

impl<F, Predicate> FilterDsl<Predicate> for SelectStatement<F> where (): Expression + NonAggregate {}

struct SelectStatement<From> {}

trait FilterDsl<Predicate> {}

impl<A, B> ValidGrouping for (A, B)
where
    (A, B): ValidGrouping,
    (): MixedAggregates<<(A, B) as ValidGrouping>::IsAggregate>,
{
    type IsAggregate = ();
}

from rust.

Managed to reduce down to a single 9Mb rs file with no external dependencies, chewing on that for a bit.

from rust.

that's really cool! what tools are you using?

• treereduce-rust to automatically reduce cases
• cargo-expand to flatten crates into a single file and expand macro
• systemd-run to run rustc on reduction attempts generated by treereduce-rust but limit memory usage to a certain amount.

Interestingness criteria for treereduce-rust is that rustc gets killed by the memory usage.

Tempted to make something like cargo-flatten that would behave like cargo-expand, but change expanded stdlib derives back to #[derive(Clone, Copy, ...)] form they are usually not interesting and expanded versions require to enable a bunch of nightly features.

Under 20kb now, rerunning it again with "rustc uses over 5Gb" criteria, previous attempt lead to something that uses over 500Mb, but succeeds.

from rust.

and this issue, well,

timeout 1m cargo check or such will do the trick. Or running this in a docker container with a low memory like 4 GB should make the offending rustc process exit due to a SIGKILL promptly.

from rust.

30ish seconds per check, 6 checks in parallel, 3Mb now. Was down to 700kb at one point but messed up the check so went back to slower/more reliable. My 🥔 PC needs more memory.

Making progress. Some result tomorrow hopefully.

from rust.

does it also reproduce on the latest nightly?

from rust.

Sorry for being unclear. Yes, this also happens in latest nightly rustc 1.80.0-nightly (84b40fc90 2024-05-27).

from rust.

This also happens in nightly (not sure how to check the version).

rustc +nightly --version --verbose

from rust.

This has a reproducer, it's just the entire .tar.gz of a project with, yk, diesel and sqlx, which means it is not minimal by far.

from rust.

rustc +nightly --version --verbose

rustc 1.80.0-nightly (da159eb33 2024-05-28)
binary: rustc
commit-hash: da159eb331b27df528185c616b394bb0e1d2a4bd
commit-date: 2024-05-28
host: x86_64-unknown-linux-gnu
release: 1.80.0-nightly
LLVM version: 18.1.6

This version still reproduces the issue.

I'm happy to help to try to find a more minimal reproducer, are there any tips on how to do that?

from rust.

the problem is that all the automated ways of doing so require a script that you can run to completion, and this issue, well,

from rust.

Taking a shot in the dark here, but would it be possible to make the script interrupt the compilation process if it ever uses more than some amount of ram?

from rust.

Update: This still happens with the following versions:

rustc 1.81.0-nightly (8337ba918 2024-06-12)
binary: rustc
commit-hash: 8337ba9189de188e2ed417018af2bf17a57d51ac
commit-date: 2024-06-12
host: x86_64-unknown-linux-gnu
release: 1.81.0-nightly
LLVM version: 18.1.7

rustc 1.79.0 (129f3b996 2024-06-10)
binary: rustc
commit-hash: 129f3b9964af4d4a709d1383930ade12dfe7c081
commit-date: 2024-06-10
host: x86_64-unknown-linux-gnu
release: 1.79.0
LLVM version: 18.1.7

Is there any way I can help debug this issue?

from rust.

Is there any way I can help debug this issue?

Yes. Produce a smaller program that reproduces the rampant memory usage.

from rust.

I've minimized by hand to this Cargo.toml:

[package]
name = "ttrpgs"
version = "0.1.0"
edition = "2021"

[dependencies]
diesel = { version = "2.1.6", features = ["postgres", "chrono"] }
rocket_sync_db_pools = { version = "0.1.0", features = ["diesel_postgres_pool"] }

With this src/main.rs:

diesel::table! {
    language (id) {
        id -> Int4,
        name -> Varchar,
        created_at -> Timestamp,
        updated_at -> Timestamp,
    }
}

pub struct GameEntry {
    pub id: i32,
}

pub struct GameEntryCard {
    pub game_entry: GameEntry,
    pub languages: Vec<Language>,
}

pub type GameEntryCardDBValue = ();

impl GameEntryCard {
    pub fn from_db_result(value: GameEntryCardDBValue) -> Vec<Self> {
        loop {}
    }
}

pub struct Language {
    pub id: i32,
}

use diesel::PgExpressionMethods;
use diesel::QueryDsl;
use diesel::RunQueryDsl;

use rocket_sync_db_pools::{database, diesel};

#[database("ttrpgs")]
pub struct Connection(diesel::PgConnection);

pub async fn details(game: &GameEntryCard, conn: Connection) {
    let language_ids = game.languages.iter().map(|x| x.id).collect();
    let other_languages = conn
        .run(|c| {
            language::table
                .filter(language::columns::id.is_distinct_from(language_ids))
                .load(c)
        })
        .await.unwrap();
}

fn main() {}

Taking a break for now. This is still very far from minimal because all the type system complexity is in the diesel APIs being used here.

from rust.

A bit smaller - no macro and all traits are explicitly imported

This Cargo.toml

[package]
name = "ttrpgs"
version = "0.1.0"
edition = "2021"

[dependencies]

diesel = { version = "2.1.6", features = ["32-column-tables", "postgres_backend"], default-features = false }

with this body

use diesel::expression::Expression;
use diesel::expression::ValidGrouping;
use diesel::internal::table_macro::FromClause;
use diesel::internal::table_macro::SelectStatement;
use diesel::internal::table_macro::StaticQueryFragmentInstance;
use diesel::query_builder::AsQuery;
use diesel::sql_types::Int4;
use diesel::PgExpressionMethods;
use diesel::QueryDsl;
use diesel::QuerySource;
use diesel::Table;

pub struct table;

impl QuerySource for table {
    type FromClause = StaticQueryFragmentInstance<()>;
    type DefaultSelection = <Self as Table>::AllColumns;
}

impl AsQuery for table {
    type Query = SelectStatement<FromClause<Self>>;
}
impl Table for table {
    type AllColumns = (id,);
}

pub struct id;
impl Expression for id {
    type SqlType = Int4;
}

impl ValidGrouping<()> for id {}

pub fn run<F, R>(__f: F) -> R {}

pub fn details(languages: &[i32]) {
    let language_ids = languages.iter().map(|x| x + 1).collect();
    run(|c| table.filter(id.is_distinct_from(language_ids)));
}

from rust.

Night was not very productive due to me messing up some arguments, but it's been running since morning and now is sitting at 900kb as of right now, going down by about 1.5kb every 30 seconds or so.

1000LOC, 40 traits and 24 structs. At this point it can be done by hand I guess but should be done tomorrow either way.

from rust.

that's really cool! what tools are you using?

from rust.

From what I can see it gets stuck inside rustc_trait_selection going recursively deep and wide

from rust.