Threading and parallelization

[CarlinLiao]

I actually slightly disagree with this. I think you can use Python threads effectively to some extent. For example, you can do some I/O related things in the initialization in parallel (look into asyncio). Also, from what I've read, I think threads are beneficial if you are a lot of light-weight tasks at the same time. Just don't expect any more than this.

If you want to run multiple chunks in parallel then the multiprocessing library would be useful, but like Rishabh said, it might not actually be that useful for the results that you want. If performance ever does become something you care about, consider switching to C++ or Rust.

Originally posted by @pyrito in #2 (comment)