Invalid sequence set when listing envelopes #518

edwardzcn-decade · 2024-12-10T09:48:00Z

Hi, I use IMAP to test one email account. After setting the configuration and solving the tls problem. Finally, envelope list only work for 1 mail which send/receieve by my self.

you can see from the output: 1

2024-12-10T09:44:07.320899Z DEBUG email::email::envelope::list::imap: found 1 imap envelopes

| ID               | FLAGS             | SUBJECT            | FROM                      | DATE                              |
|------------------|-------------------|--------------------|---------------------------|-----------------------------------|
| 1242             |                   | Hihi               | Chuannan Zhang            | 2024-12-10 17:26+08:00            |

Bug

Next I send a test email (with only 'test' body) from gmail/output. And himalaya return error. Seems it successfully links to the INBOX and finds exists(2) with the incoming mail. But it cant resolve the IMAP task.

2024-12-10T09:39:22.663292Z DEBUG keyring::service: define global service name name="himalaya-cli"
2024-12-10T09:39:22.670467Z  INFO himalaya::email::envelope::command::list: executing list envelopes command
2024-12-10T09:39:22.672094Z DEBUG email::imap: building 1 IMAP clients
2024-12-10T09:39:22.680882Z DEBUG client::build: rustls::client::hs: No cached session for DnsName("imap.qq.com")    
2024-12-10T09:39:22.681651Z DEBUG client::build: rustls::client::hs: Not resuming any session    
2024-12-10T09:39:22.736403Z DEBUG client::build: rustls::client::hs: ALPN protocol is None    
2024-12-10T09:39:22.736420Z DEBUG client::build: rustls::client::hs: Using ciphersuite TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256    
2024-12-10T09:39:22.736720Z DEBUG client::build: rustls::client::tls12: ECDHE curve is EcParameters { curve_type: NamedCurve, named_group: secp256r1 }    
2024-12-10T09:39:22.736729Z DEBUG client::build: rustls::client::tls12: Server DNS name is DnsName("imap.qq.com")    
2024-12-10T09:39:23.102235Z DEBUG client::build: email::imap: using password authentication
2024-12-10T09:39:23.102304Z  INFO client::build: process::command: run shell command cmd="pass show qqmail"
2024-12-10T09:39:23.102314Z DEBUG client::build: process::command: inherit stdin from parent
2024-12-10T09:39:23.102320Z DEBUG client::build: process::command: stdout piped
2024-12-10T09:39:23.102324Z DEBUG client::build: process::command: stderr piped
2024-12-10T09:39:23.342150Z DEBUG client::build: process::command: shell command gracefully exited code=0
2024-12-10T09:39:23.342386Z DEBUG client::build: email::imap: supported auth mechanisms mechanisms=[Plain, Login, XOAuth2]
2024-12-10T09:39:23.342401Z DEBUG client::build: email::imap: trying auth mechanism… mechanism=Plain
2024-12-10T09:39:23.589632Z DEBUG client::build: email::imap: authentication succeeded! mechanism=Plain
2024-12-10T09:39:23.591261Z  INFO email::email::envelope::list::imap: listing IMAP envelopes from mailbox INBOX
2024-12-10T09:39:23.591714Z DEBUG email::imap: client 1/1 is free, locking it
2024-12-10T09:39:23.592300Z DEBUG email::email::envelope::list::imap: UTF7-encoded mailbox name="INBOX"
2024-12-10T09:39:23.713906Z  WARN select_mailbox{client=1}: imap_codec::response: Rectified missing `text` to "..."    
2024-12-10T09:39:23.713993Z DEBUG email::email::envelope::list::imap: mailbox selected name="INBOX" data=SelectDataUnvalidated { flags: Some([Answered, Flagged, Deleted, Draft, Seen]), exists: Some(2), recent: Some(0), unseen: Some(1), permanent_flags: Some([Asterisk, Flag(Answered), Flag(Flagged), Flag(Deleted), Flag(Draft), Flag(Seen)]), uid_next: Some(1247), uid_validity: Some(1517461405) }
Error: 
   0: cannot fetch IMAP messages
   1: cannot resolve IMAP task
   2: unexpected BAD response: Sequence set is inavlid!

Location:
   /Users/edwardzcn-mini/.cargo/registry/src/index.crates.io-6f17d22bba15001f/pimalaya-tui-0.2.1/src/himalaya/backend.rs:513

The text was updated successfully, but these errors were encountered:

soywod · 2024-12-10T10:29:32Z

Hi, First of all, thank you for your interest in Himalaya CLI! Can you try to list envelopes without pagination? himalaya envelope list --page-size 0 I do not have a single clue about what is going on. I tried to reproduce your issue, without success. I suspect the issue to come from your email provider, qq.com. Would it be possible for you to create us a fake account there, so I could debug? You can send me credentials either as a private message on the Matrix chat, or by email at: clement.douin at posteo.net

edwardzcn-decade · 2024-12-10T11:00:59Z

It does work with --page-size 0 param. Here is the result:

But why I need this param. I find the explanation in the sample config that customize the number of envelopes to show by page.

Would it be possible for you to create us a fake account there, so I could debug? You can send me credentials either as a private message on the Matrix chat, or by email at: clement.douin at posteo.net

Sry that I could not provide you a fake account to test. In short, the qq.com mail service asks for phone number registered with ID card :( However, is it possible to hypothesize the path (e.g., whether it lies in the client or the core dependency) based on the trace information in the first paragraph? I will continue conducting some tests in this area.

BTW, I use Gmail in my daily life through proxy tools. I mentioned in the core repository issue, this requires proxy feature support in advance. ~~I just want to quick start wihout oauth2 setting up and proxy so I choose qq mail service~~

soywod · 2024-12-10T12:35:06Z

But why I need this param. I find the explanation in the sample config that customize the number of envelopes to show by page.

When using --page-size 0 you disable the pagination, therefore the IMAP sequence sent is not the same (1:*, versus 1,2,3,4…). I don't get why the server sends an error on the sequence set received. Unfortunately the sequence is not logged. I will add a log soon, then you can either wait for the next patch release or build from sources (using Cargo directly).

DreamMaoMao · 2024-12-28T23:51:43Z

I also have this problem, but I can use imap.qq.com mailbox in aerc.

DreamMaoMao · 2024-12-29T00:02:42Z

2024-12-29T00:00:29.554522Z DEBUG list_envelopes{folder="INBOX" opts=ListEnvelopesOptions { page_size: 10, page: 0, query: None }}: email::email::envelope::list::imap: UTF7-encoded mailbox name="INBOX"
2024-12-29T00:00:29.554631Z TRACE list_envelopes{folder="INBOX" opts=ListEnvelopesOptions { page_size: 10, page: 0, query: None }}:select_mailbox{client=1}: imap_client::stream: wrote 27/27 bytes
2024-12-29T00:00:29.625562Z TRACE list_envelopes{folder="INBOX" opts=ListEnvelopesOptions { page_size: 10, page: 0, query: None }}:select_mailbox{client=1}: imap_client::stream: read 298/1024 bytes
2024-12-29T00:00:29.625686Z  WARN list_envelopes{folder="INBOX" opts=ListEnvelopesOptions { page_size: 10, page: 0, query: None }}:select_mailbox{client=1}: imap_client::tasks::tasks::select: missing required UNSEEN OK untagged response
2024-12-29T00:00:29.625719Z DEBUG list_envelopes{folder="INBOX" opts=ListEnvelopesOptions { page_size: 10, page: 0, query: None }}: email::email::envelope::list::imap: mailbox selected name="INBOX" data=SelectDataUnvalidated { flags: Some([Answered, Flagged, Deleted, Draft, Seen]), exists: Some(7), recent: Some(0), unseen: None, permanent_flags: Some([Asterisk, Flag(Answered), Flag(Flagged), Flag(Deleted), Flag(Draft), Flag(Seen)]), uid_next: Some(3693), uid_validity: Some(1415888772) }
2024-12-29T00:00:29.625882Z TRACE list_envelopes{folder="INBOX" opts=ListEnvelopesOptions { page_size: 10, page: 0, query: None }}:fetch_envelopes_by_sequence{client=1}: imap_client::stream: wrote 59/59 bytes
2024-12-29T00:00:29.685929Z TRACE list_envelopes{folder="INBOX" opts=ListEnvelopesOptions { page_size: 10, page: 0, query: None }}:fetch_envelopes_by_sequence{client=1}: imap_client::stream: read 43/1024 bytes
Error: 
   0: cannot fetch IMAP messages
   1: cannot resolve IMAP task
   2: unexpected BAD response: Sequence set is inavlid!

soywod · 2025-01-01T10:58:07Z

I will add the log for the next patch release. From there we will be able to debug properly this issue.

Refs: <pimalaya/himalaya#518>

Refs: #518

soywod · 2025-01-10T10:37:41Z

trace log added on master, should be part of the next patch release. It will look like this:

2025-01-10T10:52:06.012370Z TRACE list_envelopes{folder="INBOX" opts=ListEnvelopesOptions { page_size: 10, page: 0, query: None }}:fetch_envelopes_by_sequence{client=1}: imap_client::tasks: enqueue task command=Command { tag: Tag("0.2"), body: Fetch { sequence_set: SequenceSet([Range(Value(1), Value(1))]+), macro_or_item_names: MessageDataItemNames([Uid, Flags, Envelope, BodyStructure]), uid: false } }

It should be enough to debug the faulty sequence set.

Refs: <pimalaya/himalaya#518>

Refs: <#518>

soywod · 2025-01-11T16:22:21Z

New release v1.1.0 out, could you share the new trace logs?

tiejunhu · 2025-01-15T08:55:03Z

New release v1.1.0 out, could you share the new trace logs?

2025-01-15T08:53:54.240281Z  INFO list_envelopes{folder="INBOX" opts=ListEnvelopesOptions { page_size: 10, page: 0, query: None }}: email::email::envelope::list::imap: listing IMAP envelopes from mailbox INBOX
2025-01-15T08:53:54.240425Z DEBUG list_envelopes{folder="INBOX" opts=ListEnvelopesOptions { page_size: 10, page: 0, query: None }}: email::imap: client 1/1 is free, locking it
2025-01-15T08:53:54.241373Z DEBUG list_envelopes{folder="INBOX" opts=ListEnvelopesOptions { page_size: 10, page: 0, query: None }}: email::email::envelope::list::imap: UTF7-encoded mailbox name="INBOX"
2025-01-15T08:53:54.242104Z TRACE list_envelopes{folder="INBOX" opts=ListEnvelopesOptions { page_size: 10, page: 0, query: None }}:select_mailbox{client=1}: imap_client::tasks: enqueue task command=Command { tag: Tag("0.2.0DGcYZHj"), body: Select { mailbox: Inbox } }
2025-01-15T08:53:54.242452Z TRACE list_envelopes{folder="INBOX" opts=ListEnvelopesOptions { page_size: 10, page: 0, query: None }}:select_mailbox{client=1}: imap_client::stream: wrote 27/27 bytes
2025-01-15T08:53:54.330874Z TRACE list_envelopes{folder="INBOX" opts=ListEnvelopesOptions { page_size: 10, page: 0, query: None }}:select_mailbox{client=1}: imap_client::stream: read 319/1024 bytes
2025-01-15T08:53:54.330966Z  WARN list_envelopes{folder="INBOX" opts=ListEnvelopesOptions { page_size: 10, page: 0, query: None }}:select_mailbox{client=1}: imap_codec::response: Rectified missing `text` to "..."
2025-01-15T08:53:54.332470Z DEBUG list_envelopes{folder="INBOX" opts=ListEnvelopesOptions { page_size: 10, page: 0, query: None }}: email::email::envelope::list::imap: mailbox selected name="INBOX" data=SelectDataUnvalidated { flags: Some([Answered, Flagged, Deleted, Draft, Seen]), exists: Some(175), recent: Some(0), unseen: Some(19), permanent_flags: Some([Asterisk, Flag(Answered), Flag(Flagged), Flag(Deleted), Flag(Draft), Flag(Seen)]), uid_next: Some(18046), uid_validity: Some(1396879027) }
2025-01-15T08:53:54.332938Z TRACE list_envelopes{folder="INBOX" opts=ListEnvelopesOptions { page_size: 10, page: 0, query: None }}:fetch_envelopes_by_sequence{client=1}: imap_client::tasks: enqueue task command=Command { tag: Tag("0.3.0QybN93U"), body: Fetch { sequence_set: SequenceSet([Range(Value(175), Value(166))]+), macro_or_item_names: MessageDataItemNames([Uid, Flags, Envelope, BodyStructure]), uid: false } }
2025-01-15T08:53:54.333056Z TRACE list_envelopes{folder="INBOX" opts=ListEnvelopesOptions { page_size: 10, page: 0, query: None }}:fetch_envelopes_by_sequence{client=1}: imap_client::stream: wrote 63/63 bytes
2025-01-15T08:53:54.398376Z TRACE list_envelopes{folder="INBOX" opts=ListEnvelopesOptions { page_size: 10, page: 0, query: None }}:fetch_envelopes_by_sequence{client=1}: imap_client::stream: read 43/1024 bytes
Error:
   0: cannot fetch IMAP messages
   1: cannot resolve IMAP task
   2: unexpected BAD response: Sequence set is inavlid!

Location:
   /Users/brew/Library/Caches/Homebrew/cargo_cache/registry/src/index.crates.io-6f17d22bba15001f/pimalaya-tui-0.2.2/src/himalaya/backend.rs:513

  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ BACKTRACE ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
                                ⋮ 5 frames hidden ⋮
   6: himalaya::email::envelope::command::list::EnvelopeListCommand::execute::{{closure}}::h1a6d2659f7484457
      at <unknown source file>:<unknown line>
   7: tokio::runtime::park::CachedParkThread::block_on::hd49b157562e7ba7e
      at <unknown source file>:<unknown line>
   8: tokio::runtime::context::runtime::enter_runtime::h0320535dccedfcb8
      at <unknown source file>:<unknown line>
   9: himalaya::main::h83523fee0f5b28d9
      at <unknown source file>:<unknown line>
  10: std::sys::backtrace::__rust_begin_short_backtrace::hf58157a173c41404
      at <unknown source file>:<unknown line>
  11: std::rt::lang_start::{{closure}}::h6ec574c0f97256e7
      at <unknown source file>:<unknown line>
  12: std::rt::lang_start_internal::hd3a596400139984d
      at <unknown source file>:<unknown line>
  13: _main<unknown>
      at <unknown source file>:<unknown line>

soywod · 2025-01-17T12:57:53Z

SequenceSet([Range(Value(175), Value(166))]+)

I bet the server does not appreciate the range order. As far as I remember, the range should be commutative. 175:166 should be the same as 166:175. But I cannot find back this information in RFCs, any idea @duesee?

soywod · 2025-01-17T12:59:49Z

Just found back in RFC 3501:

seq-range       = seq-number ":" seq-number
                    ; two seq-number values and all values between
                    ; these two regardless of order.
                    ; Example: 2:4 and 4:2 are equivalent and indicate
                    ; values 2, 3, and 4.
                    ; Example: a unique identifier sequence range of
                    ; 3291:* includes the UID of the last message in
                    ; the mailbox, even if that value is less than 3291.

duesee · 2025-01-17T16:38:46Z

You were faster, hehe :-) The order -- when resolved to a sorted set -- should not be important. AFAIR, imap-types already reorders accordingly when, e.g., iterating over a sequence.

Do you think it makes sense to always reorder and take away the possibility to send descending sequences in imap-types? This could be a measure to protect ourselves against faulty servers... I'm not entirely sure about any downsides, though.

Hmm... maybe some implementation wants to get messages in reversed order? Say... you want to fetch 100:1 starting with 100. When we exclude this possibility, people would need to write 100,99,98,... manually. Not sure...

duesee · 2025-01-17T16:50:17Z

Context: duesee/imap-codec#411 We had the same bug but resolved it.

soywod · 2025-01-18T04:53:43Z

Hmm... maybe some implementation wants to get messages in reversed order? Say... you want to fetch 100:1 starting with 100. When we exclude this possibility, people would need to write 100,99,98,... manually. Not sure...

Well, according to the RFC we should not be able to do that, since the sequence is re-ordered at request level:

Example: 2:4 and 4:2 are equivalent and indicate values 2, 3, and 4.

It seems just not possible to ask for 4, 3 then 2 using sequence set.

Context: duesee/imap-codec#411 We had the same bug but resolved it.

Awesome, I checked and indeed it should fix this particular issue as well. Let's wait for the v2 then 🙂

duesee · 2025-01-18T15:32:56Z

Awesome, I checked and indeed it should fix this particular issue as well. Let's wait for the v2 then 🙂

Oh, no. What I meant was that when you iterate through a sequence set, it resolves now correctly. But this is rather only done when imap-types is used for a server.

As a client you can still send 5:1, which, to my understanding, would break the server. In this case the user of imap-types needs to ensure to use 1:5 instead of 5:1. But maybe it's a good idea to transparently change 5:1 to 1:5 in imap-types then indeed...

soywod · 2025-01-18T19:55:05Z

But maybe it's a good idea to transparently change 5:1 to 1:5 in imap-types then indeed...

Isn't it a perfect use case for introducing a new quirk? Something like quirk_enforce_ascending_sequence.

soywod self-assigned this Jan 1, 2025

soywod added this to Pimalaya Jan 1, 2025

soywod moved this to Todo in Pimalaya Jan 1, 2025

echo-bravo-yahoo mentioned this issue Jan 6, 2025

Empty envelope search result leads to page 1 out of bounds error #535

Closed

soywod moved this from Todo to In Progress in Pimalaya Jan 10, 2025

soywod added a commit to pimalaya/core that referenced this issue Jan 10, 2025

fix(email): return empty envelopes when empty query response

3167e48

Refs: <pimalaya/himalaya#518>

soywod added a commit that referenced this issue Jan 10, 2025

fix(envelope): prevent out of bound error on empty search result

50b8d36

Refs: #518

soywod closed this as completed Jan 10, 2025

github-project-automation bot moved this from In Progress to Done in Pimalaya Jan 10, 2025

soywod reopened this Jan 10, 2025

soywod moved this from Done to In Progress in Pimalaya Jan 10, 2025

soywod added a commit to pimalaya/imap-client that referenced this issue Jan 10, 2025

feat(task): add trace log of enqueued tasks

cfd385c

Refs: <pimalaya/himalaya#518>

soywod moved this from In Progress to Pending in Pimalaya Jan 10, 2025

soywod added a commit that referenced this issue Jan 10, 2025

refactor(envelope): add trace log to debug imap sequence set

e3cbbbc

Refs: <#518>

soywod changed the title ~~[Bug] Cannot fetch IMAP messages~~ Invalid sequence set when listing envelopes Jan 10, 2025

duesee mentioned this issue Jan 18, 2025

Descending sequence leads to BAD response modern-email/defects#40

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Invalid sequence set when listing envelopes #518

Invalid sequence set when listing envelopes #518

edwardzcn-decade commented Dec 10, 2024

soywod commented Dec 10, 2024 via email •

edited

Loading

edwardzcn-decade commented Dec 10, 2024 •

edited

Loading

soywod commented Dec 10, 2024 •

edited

Loading

DreamMaoMao commented Dec 28, 2024

DreamMaoMao commented Dec 29, 2024

soywod commented Jan 1, 2025

soywod commented Jan 10, 2025 •

edited

Loading

soywod commented Jan 11, 2025

tiejunhu commented Jan 15, 2025 •

edited

Loading

soywod commented Jan 17, 2025

soywod commented Jan 17, 2025 •

edited

Loading

duesee commented Jan 17, 2025 •

edited

Loading

duesee commented Jan 17, 2025

soywod commented Jan 18, 2025

duesee commented Jan 18, 2025

soywod commented Jan 18, 2025

Invalid sequence set when listing envelopes #518

Invalid sequence set when listing envelopes #518

Comments

edwardzcn-decade commented Dec 10, 2024

Bug

soywod commented Dec 10, 2024 via email • edited Loading

edwardzcn-decade commented Dec 10, 2024 • edited Loading

soywod commented Dec 10, 2024 • edited Loading

DreamMaoMao commented Dec 28, 2024

DreamMaoMao commented Dec 29, 2024

soywod commented Jan 1, 2025

soywod commented Jan 10, 2025 • edited Loading

soywod commented Jan 11, 2025

tiejunhu commented Jan 15, 2025 • edited Loading

soywod commented Jan 17, 2025

soywod commented Jan 17, 2025 • edited Loading

duesee commented Jan 17, 2025 • edited Loading

duesee commented Jan 17, 2025

soywod commented Jan 18, 2025

duesee commented Jan 18, 2025

soywod commented Jan 18, 2025

soywod commented Dec 10, 2024 via email •

edited

Loading

edwardzcn-decade commented Dec 10, 2024 •

edited

Loading

soywod commented Dec 10, 2024 •

edited

Loading

soywod commented Jan 10, 2025 •

edited

Loading

tiejunhu commented Jan 15, 2025 •

edited

Loading

soywod commented Jan 17, 2025 •

edited

Loading

duesee commented Jan 17, 2025 •

edited

Loading