Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

batch queue is empty (1 warnings) #58

Open
Huang-Jin opened this issue Oct 24, 2018 · 1 comment
Open

batch queue is empty (1 warnings) #58

Huang-Jin opened this issue Oct 24, 2018 · 1 comment

Comments

@Huang-Jin
Copy link

Hello,
Thank you for your excellent work and codes. When I'm training using your code, I find that it outputs warnings every iteration, like this:

# 2018-10-24 15:19:12     3800 | total_loss:      91.93  learning_rate:    0.00025  
batch queue is empty (101 warnings)
# 2018-10-24 15:21:03     3900      0.90 ips            18:14:20 rem | total_loss:      137.2  learning_rate:    0.00025  
batch queue is empty (201 warnings)

I find that this warning comes from https://github.com/lmb-freiburg/demon/blob/devel/multivih5datareaderop/multivih5datareader.cpp.

Batch_sptr getBatch()
    {
      bool print_warning = !param.test_phase;
      Batch_sptr result = nullptr;
      while( !result )
      {
        std::lock_guard<std::mutex> lock(batches_mutex);
        if( batches.size() )
        {
          result = batches.front();
          batches.pop();
        }
        else
        {
          if(print_warning)
          {
            static int empty_count = 0;
            ++empty_count;
            if( !((empty_count-1)%100) )
              std::cerr << "batch queue is empty (" << empty_count << " warnings)\n";
          }
          print_warning = false;
        }
      }
      return result;
    }

So, does it means that there are no data input and the training is useless?
Waiting for your apply!

@benjaminum
Copy link
Collaborator

It means that the training has to wait for data and could be faster.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants