Incorrect matched token slice and span when involving a dash.

[WindSoilder]

Given the following code:

use logos::Logos;

#[derive(Logos, Debug, Clone, Copy, PartialEq)]
#[logos(skip r"[ \t]+")]
#[logos(source = [u8])]
pub enum Token {
    #[regex("-?(0[xob])?[0-9][0-9_]*")]
    Int,
    #[token("-")]
    Dash,
}

fn main() {
    let mut lex = Token::lexer(b"-0x");
    assert_eq!(lex.next(), Some(Ok(Token::Dash)));
    assert_eq!(lex.slice(), b"-");   // the slice is `-0`
    assert_eq!(lex.span(), 0..1);    // the span is `0..2`
}

And test failed with the following message:

thread 'main' panicked at src/main.rs:16:5:
assertion `left == right` failed
  left: [45, 48]
 right: [45]
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace

I think the result is not right in the case, since the lexer correctly say the token is a Dash. It's impossible for the slice containing one more byte.

[jeertmans]

Hi @WindSoilder, what is the faulty line? I don't see any equality test for [45, 48].

Also, can you include the output of debugged graph? See: https://logos.maciej.codes/debugging.html.

[WindSoilder]

Hi, the faulty line is assert_eq!(lex.slice(), b"-");, b"-" is just a vec![45].

Sure here is the graph:

{
    1: ::<skip> (<skip>),
    2: {
        [09] ⇒ 2,
          ⇒ 2,
        _ ⇒ 1,
    },
    4: ::Int,
    5: {
        [0-9] ⇒ 5,
        _ ⇒ 5,
        _ ⇒ 4,
    },
    6: [0-9] ⇒ 5,
    10: ::Dash,
    11: {
        0 ⇒ 13,
        [1-9] ⇒ 5,
        _ ⇒ 10,
    },
    13: {
        [0-9] ⇒ 5,
        _ ⇒ 5,
        b ⇒ 6,
        o ⇒ 6,
        x ⇒ 6,
        _ ⇒ 4,
    },
    14: {
        [09] ⇒ 2,
          ⇒ 2,
        - ⇒ 11,
        0 ⇒ 13,
        [1-9] ⇒ 5,
    },
}

[jeertmans]

My bad, didn't see it was bytes. Interesting bug, definitely to add to the long list of bugs I have no idea how to solve ^^'.

Jokes aside, I have time to review PRs and answer questions, but solving bugs like that takes too much time, which I don't have for this project, unfortunately.