Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

ERROR: Could not find a valid gem 'logstash-devutils' (= 0.0.15) in any repository #68

Open
michaelhidalgo opened this issue May 9, 2017 · 25 comments
Labels

Comments

@michaelhidalgo
Copy link

michaelhidalgo commented May 9, 2017

Hi guys, I'm newbie to ruby as you will noticed but I wanted to install the logstash-devutils gem (https://rubygems.org/gems/logstash-devutils/versions/0.0.15-java) on the Mac following this command :

gem install --local logstash-devutils -v 0.0.15

And i get below error message:
:
ERROR: Could not find a valid gem 'logstash-devutils' (= 0.0.15) in any repository

Any thoughts?

I also tried latest version of the gem
gem install logstash-devutils
ERROR: Could not find a valid gem 'logstash-devutils' (>= 0), here is why:
Found logstash-devutils (1.3.1), but was for platform java

@michaelhidalgo
Copy link
Author

Update: I downloaded the gem manually and when I do try to install it manually, i get a dependency errors

Unable to resolve dependencies: logstash-devutils requires logstash-core-plugin-api (<= 2.99, >= 2.0)

@jordansissel
Copy link
Contributor

This is failing to install because the ruby you are using is not JRuby.

I have a broader question, though, what makes you try to install logstash-devutils manually? Our intention is that this library provides support for logstash tests and logstash plugin tests. You generally do not need to install it manually.

What are you trying to do that makes you want to install logstash-devutils manually?

@michaelhidalgo
Copy link
Author

michaelhidalgo commented May 10, 2017

Thanks for the feedback, I do have JRuby but the issue seems to be a dependency hell.

In regards to your question: I'm trying to find a way to create a logstash configuration (including customized grok patterns) and be able to provide a unit test of the configuration file. This is to ensure quality when providing a custom configuration file.

In your case, you have some grok patterns that does not seem to have a test associated to it, so I want to implement a mechanism to provide a test scenario that can be used to find issues on the patterns and so on.

To give you a concrete example, please check this out this cool hack https://github.com/gquintana/gquintana.github.io/blob/master/sources/2016-09-07-Testing-Logstash-configuration/spec/02_logstash_nginx_filter_spec.rb#L2

As you can see on line number 2, the guy is using require "logstash/devutils/rspec/spec_helper"
and I wanted to replace that import with the gem (i.e be able to reuse your spec helper without needing the full project)

Any ideas how to address that issue?

@jordansissel
Copy link
Contributor

I do have JRuby but the issue seems to be a dependency hell.

I don't know what you mean by dependency hell.

I also tried latest version of the gem
gem install logstash-devutils
ERROR: Could not find a valid gem 'logstash-devutils' (>= 0), here is why:
Found logstash-devutils (1.3.1), but was for platform java

This error message is an indication that you are running gem under MRI (regular ruby, not jruby). The clue for me is that it says "Found logstash-devutils (1.3.1), but was for platform java" which means your gem is not running under jruby.

In general, you do not need to install logstash-devutils manually, ever. I personally haven't installed this gem manually in a very long time. Our dependency listing includes it (in plugins and in logstash itself) and you will get it automatically.

I wanted to replace that import with the gem

(Pardon the detailed explanation below if you already know these things, I'm trying to get you a good answer)

In ruby, require is a function that loads a ruby file. A gem provides these ruby files and instructs the ruby runtime about paths where ruby files live.

I looked at what your link, and I think I understand a little bit more now.

What you linked has a test.sh that uses a file we no longer ship with main logstash releases (bin/rspec). In order to run this, you may be best getting logstash setup from source:

  • Clone the logstash repo, and run the Building Logstash steps
  • Then run the test.sh from that source you linked.

I do not believe there are shortcuts to this. If you want a shortcut, can you help me understand what you want to avoid?

@michaelhidalgo
Copy link
Author

michaelhidalgo commented May 11, 2017

Hi @jordansissel , thanks for the further explanation, it is very useful specially since I'm a newcomer to ruby itself.

What I meant by dependency hell, is if you look at the gem published here https://rubygems.org/gems/logstash-devutils/versions/1.3.1-java, it says it has many dependencies and even when I tried to install the gem including all the dependencies (gem install --include-dependencies) I got some warnings that dependencies were not found. Anyway I'm pretty sure I was doing something wrong when installing the gem and need to understand better the switch from ruby to jruby.

I was expecting to run the unit testing isolated without needing logstash repo, that is for convenience. Basically the use case is that we can write logstash configurations for several purposes and for parsing log files but provide also a unit test with a sample log, something similar to what you saw in the example.

Since the gem was published, I was expecting that you could use it in your project without relying totally on the logstash project. I certainly can test just the grok pattern, but I'm more concern on testing the output of the logstash file after doing several mutations and applying filters.

As you can see basically the use case is to provide a Logstash configuration file for any specific purpose but also provide a unit test with a sample log that can be used for troubleshooting purposes as versions of the log file can be different.

So in order to answer your question, I'm looking for a way to distribute our configurations and unit testing without needing logstash code although it might not be possible as I can see.

I will follow your recommendations about cloning the Logstash repo and building it in the way you have explained.

Thanks for all your help.

@jslirola
Copy link

jslirola commented Jun 7, 2018

@michaelhidalgo Did you fix the issue? I'm trying to follow the basic tutorial to write a logstash filter plugin but I am not able to install my example:

$ bundler install
Fetching gem metadata from https://rubygems.org/.........
Could not find gem 'logstash-devutils' in any of the gem sources listed in your Gemfile.

Thank you.

@pedropereiractw
Copy link

pedropereiractw commented Dec 11, 2018

Hi anyone found a solution for this problem? I'm using JRuby and even after I install manualy the 'logstash-devutils' I can't run the bundle install:

Fetching gem metadata from https://rubygems.org/......... Could not find gem 'logstash-devutils' in any of the gem sources listed in your Gemfile.

@jsvd
Copy link
Member

jsvd commented Dec 11, 2018

I'm having trouble replicating this, what jruby are you using? can you post ruby -v ?

@pedropereiractw
Copy link

$ jruby -v jruby 9.2.5.0 (2.5.0) 2018-12-06 6d5a228 Java HotSpot(TM) 64-Bit Server VM 25.191-b12 on 1.8.0_191-b12 +jit [mswin32-x86_64]

@jsvd
Copy link
Member

jsvd commented Dec 11, 2018

can you do which bundler? are you sure the bundler you're running is the one installed in jruby?

@pedropereiractw
Copy link

pedropereiractw commented Dec 11, 2018

I'm running it like jruby -S bundle install and if I runn jruby -S gem list I can find this entry: logstash-devutils (1.3.1 java)

@jsvd
Copy link
Member

jsvd commented Dec 11, 2018

is bundle installed locally in that jruby installation? -S will make jruby look for the "script" (bundle/gem) in bin or the PATH. so it's not guaranteed that using -S will make it use the jruby's bundle.

I'm having trouble finding any other reason other than jruby/bundler/gem weirdness since I'm able to run that command locally and it's a normal gem installation like any other

@pedropereiractw
Copy link

So I uninstall every ruby version of my machine, reinstall only JRuby and I'm using only bundle install and kept bump into the same error, here is my gem env:

`RubyGems Environment:

  • RUBYGEMS VERSION: 2.7.6
  • RUBY VERSION: 2.5.0 (2018-12-06 patchlevel 0) [java]
  • INSTALLATION DIRECTORY: C:/jruby-9.2.5.0/lib/ruby/gems/shared
  • USER INSTALLATION DIRECTORY: C:/Users/ctw00220/.gem/jruby/2.5.0
  • RUBY EXECUTABLE: C:/jruby-9.2.5.0/bin/jruby.exe
  • EXECUTABLE DIRECTORY: C:/jruby-9.2.5.0/bin
  • SPEC CACHE DIRECTORY: C:/Users/ctw00220/.gem/specs
  • SYSTEM CONFIGURATION DIRECTORY: C:/jruby-9.2.5.0/etc
  • RUBYGEMS PLATFORMS:
    • ruby
    • universal-java-1.8
  • GEM PATHS:
    • C:/jruby-9.2.5.0/lib/ruby/gems/shared
    • C:/Users/ctw00220/.gem/jruby/2.5.0
  • GEM CONFIGURATION:
    • :update_sources => true
    • :verbose => true
    • :backtrace => false
    • :bulk_threshold => 1000
    • "install" => "--no-rdoc --no-ri --env-shebang"
    • "update" => "--no-rdoc --no-ri --env-shebang"
  • REMOTE SOURCES:
  • SHELL PATH:
    • C:\Program Files\ConEmu\ConEmu\Scripts
    • C:\Program Files\ConEmu
    • C:\Program Files\ConEmu\ConEmu
    • C:\ProgramData\DockerDesktop\version-bin
    • C:\Program Files\Docker\Docker\Resources\bin
    • C:\Program Files (x86)\Common Files\Oracle\Java\javapath
    • C:\Windows\system32
    • C:\Windows
    • C:\Windows\System32\Wbem
    • C:\Windows\System32\WindowsPowerShell\v1.0\
    • C:\Windows\System32\OpenSSH\
    • C:\Program Files\Intel\WiFi\bin\
    • C:\Program Files\Common Files\Intel\WirelessCommon\
    • C:\apache-maven-3.6.0\bin
    • C:\Program Files\Java\jdk1.8.0_191\bin
    • C:\Program Files\nodejs\
    • C:\ProgramData\chocolatey\bin
    • C:\HashiCorp\Vagrant\bin
    • C:\Program Files (x86)\Yarn\bin\
    • c:\jruby-9.2.5.0\bin`

@pedropereiractw
Copy link

I switch to an Ubuntu machine and install JRuby with RVM, when I run:

ruby -v

I get:

jruby 9.2.0.0 (2.5.0) 2018-05-24 81156a8 Java HotSpot(TM) 64-Bit Server VM 25.191-b12 on 1.8.0_191-b12 +jit [linux-x86_64]

When I try to run bundle install I kept getting the same error:

Fetching gem metadata from https://rubygems.org/........ Could not find gem 'logstash-devutils unknown' in any of the gem sources listed in your Gemfile.

My Gem ENV:

`RubyGems Environment:

  • RUBYGEMS VERSION: 2.7.6
  • RUBY VERSION: 2.5.0 (2018-05-24 patchlevel 0) [java]
  • INSTALLATION DIRECTORY: /home/vagrant/.rvm/gems/jruby-9.2.0.0
  • USER INSTALLATION DIRECTORY: /home/vagrant/.gem/jruby/2.5.0
  • RUBY EXECUTABLE: /usr/share/rvm/rubies/jruby-9.2.0.0/bin/jruby
  • EXECUTABLE DIRECTORY: /home/vagrant/.rvm/gems/jruby-9.2.0.0/bin
  • SPEC CACHE DIRECTORY: /home/vagrant/.gem/specs
  • SYSTEM CONFIGURATION DIRECTORY: /usr/share/rvm/rubies/jruby-9.2.0.0/etc
  • RUBYGEMS PLATFORMS:
    • ruby
    • universal-java-1.8
  • GEM PATHS:
    • /home/vagrant/.rvm/gems/jruby-9.2.0.0
    • /home/vagrant/.rvm/gems/jruby-9.2.0.0@global
  • GEM CONFIGURATION:
    • :update_sources => true
    • :verbose => true
    • :backtrace => false
    • :bulk_threshold => 1000
    • "install" => "--no-rdoc --no-ri"
    • "update" => "--no-rdoc --no-ri"
  • REMOTE SOURCES:
  • SHELL PATH:
    • /home/vagrant/.rvm/gems/jruby-9.2.0.0/bin
    • /home/vagrant/.rvm/gems/jruby-9.2.0.0@global/bin
    • /usr/share/rvm/rubies/jruby-9.2.0.0/bin
    • /usr/share/rvm/bin
    • /home/vagrant/bin
    • /home/vagrant/.local/bin
    • /usr/local/sbin
    • /usr/local/bin
    • /usr/sbin
    • /usr/bin
    • /sbin
    • /bin
    • /usr/games
    • /usr/local/games
    • /snap/bin
    • /usr/lib/jvm/java-8-oracle/bin
    • /usr/lib/jvm/java-8-oracle/db/bin
    • /usr/lib/jvm/java-8-oracle/jre/bin`

Any lights about this?

@pedropereiractw
Copy link

Solved it adding to my gemspecs the follow:

s.platform = Gem::Platform::JAVA if defined?(RUBY_ENGINE) && RUBY_ENGINE == 'jruby'

@hacksics
Copy link

hacksics commented Oct 16, 2019

"jruby -S bundle install" will solve this problem

@vinayakg
Copy link

I am still getting this error, wanted to run tests for a PR.
Please help

bundle install
Fetching gem metadata from https://rubygems.org/.........
Could not find gem 'logstash-devutils' in any of the gem sources listed in your
Gemfile.

@michaelhidalgo
Copy link
Author

michaelhidalgo commented Apr 11, 2020

Hi folks, I'm sorry for not being a good citizen here, I was working on a company that closed their operations so I could not continue researching this, that's mainly why I have not helped out with more feedback.

I will keep this ticket open so maybe someone can help.

@molenzwiebel
Copy link

@michaelhidalgo Did you fix the issue? I'm trying to follow the basic tutorial to write a logstash filter plugin but I am not able to install my example:

$ bundler install
Fetching gem metadata from https://rubygems.org/.........
Could not find gem 'logstash-devutils' in any of the gem sources listed in your Gemfile.

Thank you.

I managed to solve this by using the jruby instance bundled with logstash instead. In my case, the following worked:

$ /usr/share/logstash/bin/ruby -S bundle install

(substitute for other commands in the README as appropriate)

@vinayakg
Copy link

vinayakg commented May 18, 2020 via email

@singhksandeep25
Copy link

I also faced the same issue. For me installing using jruby did the trick as logstash is using jruby. You can read more here https://gist.github.com/jordansissel/978956

@cameronkerrnz
Copy link

cameronkerrnz commented Aug 19, 2020

The problem appears due to a PATH ordering issue. I'm not sure what the issue is (and the plugin development documentation is not clear about this). I'm setting up a Docker image that I shall use for logstash plugin development, and using a clone of the logstash-filter-example, I can observe the following with regard to the ordering of PATH

jruby/bin BEFORE logstash/bin --> SUCCESS using 'bundle install'
logstash/bin BEFORE jruby/bin --> FAILURE using 'bundle install'

EDIT: Actually, in retrospect, jruby should be before logstash. There is a notion within logstash of an 'outer/infra' version of JRuby (which is the version you get asked to provide) and the 'runtime' version of JRuby which logstash vendors..... this makes for a very confusing experience. In then interests of making something reproducible, I've build a Docker image for plugin development: https://github.com/cameronkerrnz/logstash-plugin-dev

@bluebob
Copy link

bluebob commented Jan 22, 2021

Useful thread, but I too got stuck in jRuby/dependency hell. Luckily for me I needed a Docker container as the endpoint, so I just did a multistage build. With the Docker container obviously you've got full control of the environment, and the jRuby:9.2 base image seems to build nicely for my simple filter plugin. Thought I'd mention it as a possibility for some out there.

@awanganddong
Copy link

logstash-devutils 需要依托于jruby 而不是ruby

这个需要特别注意。

@kares kares added the defunct label Mar 14, 2022
@fransf-wtax
Copy link

I don't know if it's related to how I installed jruby (using Homebrew), but I don't have a bin/ruby or bin/bundler or bin/gem in my JRuby install (in /opt/homebrew/opt/jruby/bin).
So I had to explicitly use jbundler install instead of bundler install to install dependencies.
And I had to use jgem install rspec to install rspec instead of gem install rspec.
Is this normal (I'm entirely new to Ruby)? If so it might make sense to update the READMEs in all the projects, which say to run bundler install.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

No branches or pull requests