feat(nns): Add pagination to list_neurons API

[max-dfinity]

No description provided.

[github-actions]

If this pull request affects the behavior of any canister owned by
the Governance team, remember to update the corresponding
unreleased_changes.md file(s).

To acknowldge this reminder (and unblock the PR), dismiss this
code review by going to the bottom of the pull request page, and
supply one of the following reasons:

1. Done.

2. No canister behavior changes.

[mbjorkqvist]

Thanks for adding this, @max-dfinity!

If a client requests multiple pages of list neurons, I suppose it is possible that the number of neurons matching the query may change in the meantime. Is this something that should be taken into account, or is the current "best effort" approach deemed good enough? I could imagine that if a new neuron is added during the requests, it would have a high ID and end up at the end of the list (in the last request)? I tried to look for more details of the approach, but neither this PR, nor the related JIRA ticket has any description, and the design linked in the epic also doesn't seem to cover this work.

Done

[mbjorkqvist]

Thanks a lot for adjusting the API to take the Rosetta cold storage wallet use case into account, @max-dfinity! From what I can tell this latest approach would solve the issue with potentially having to go back to cold storage for each subsequent page. I left some comments mainly related to documentation.

[mbjorkqvist]

IIUC, this is the only place in the code where there is documentation on how to use the list_neurons API? I could imagine that the common use case is to pass in None for both page_number and page_size in the initial call, and if the number of neurons returned in the response is smaller than total_neurons_found, call list_neurons again with page_number: Some(1) and page_size: None, and continue with increasing values of page_number until the total number of neurons received (sum of all neurons in all responses) matches total_neurons_found, but also only as long as total_neurons_found doesn't change (increase) between responses. What about if someone passes in a page_size that is larger than MAX_LIST_NEURONS_RESULTS, in the hope of being able to avoid pagination? IIUC, they would, from the number of neurons returned in the response, and the value of total_neurons_found, have to figure out that the page_size they specified is too large? All this to say that while this design change certainly helps for the cold storage wallet case (thanks for taking this into account!), there do seem to be some edge cases that could benefit from more documentation/examples.

The documentation in the portal should also be updated.

[max-dfinity]

@mbjorkqvist Do the doc changes and design changes address your concern?

Instead of total_neurons_found, it is now returning 'total_pages_available', which avoids specifying whether or not each neuron they put in the list actually exists. It avoids some edge cases there. I also explained how that works in the documentation. Let me know if you think it can be improved more, and also thank you for the thoughtful feedback!