When deployed to Tomcat, response is always null

[plusparth]

When restd is deployed to tomcat, it fails to give any type of response on any type of query. It does not provide any errors in errors.log, or in catalina.out. These files are found in /var/log/tomcat7. This was tested on the drmf2016 server. In addition, when restd is run manually, it works perfectly fine.

[plusparth]

@physikerwelt Do you have any thoughts on this? I thought originally that maybe it wasn't loading in the properties file correctly, but it did not output any errors to the log. In addition, after some googling, I found that stdout should be redirected to catalina.out, but it is not happening here. Also, I tried to put in log statements but it did not work.

[physikerwelt]

Did you look into /var/log in the vagrant instance or at the host machine?

[physikerwelt]

You could also search for catalina.out on the entire system, or test another tomact war file that is known to work first.

[plusparth]

@physikerwelt I checked in the vagrant instance. There are no other catalina.out files other than the one in /var/log/tomcat7/catalina.out

[physikerwelt]

it seems that the log file is named app.log

2016-08-05 16:56:30,208 [http-bio-8081-exec-45] [admin     ] INFO  restx.StdRestxMainRouter - << [RESTX REQUEST] GET /texquery ? query=\sin >> OK - 5.252 s
2016-08-05 18:03:45,578 [localhost-startStop-2] [          ] INFO  restx.stats.RestxStatsCollector - saving stats to /usr/share/tomcat7/.restx/stats/restx-stats-5b39c8b553c821e7cddc6da64b5bd2ee--a4ebcd2e-d9ed-41d7-9c78-28a74ca051ca--0--prod.json failed. Exception: /usr/share/tomcat7/.restx/stats/restx-stats-5b39c8b553c821e7cddc6da64b5bd2ee--a4ebcd2e-d9ed-41d7-9c78-28a74ca051ca--0--prod.json (No such file or directory)

[plusparth]

Yes, I saw that, but there don't seem to be errors or my logs there either.

[physikerwelt]

Can you describe which problems you have with tomcat.
Unfortunately we currently can not make port 8081 accessible from outside the wmf test cluster, but if you set up local portforwarding 8081->localhost:8081 you can try
http://localhost:8081/restd/api/@/ui/api-docs/#/operation/base-x/GET/___texquery
which gives you the same result as the basex instance on port 10043 (as far as I can see).

[plusparth]

@physikerwelt When I do the same thing you described, I get a response null (the same error we got before with the restd interface locally), but when running the restd interface directly, it works.

When I just tried this again, it worked. I tried the same thing earlier (same interface, I tried undeploying and redeploying the war file, etc) and it never worked. I am not closing this issue since I changed nothing and it just started working. I even tried doing a HTTP GET request directly on the url it was testing and it still gave me response null.

[plusparth]

In fact if you look at the app.log you can see that the first queries it recorded in the log are the ones from today just after I saw it magically start working again :/

[plusparth]

@physikerwelt It's broken again. When I redeployed it after changing the properties file to the math00000.xml file directly instead of the directory (it was always giving a response with no results, and I wanted to see if this would fix it), it again gives null responses.

[physikerwelt]

I do not understand your question. Is the tomcat service running? Does baseX work?

[physikerwelt]

You can check that with service XXX status and jps respectively.

[plusparth]

@physikerwelt It is the same issue as before where deploying restd through tomcat results in the response being null for all queries. When I try it running restd directly, it gives me some response. It restarted this issue when I undeployed and redeployed it to change the path of the data, because I wanted to give it just a file instead of the data directory to see what happened.

I was trying to fix the issue that no matter the query, even with the restd interface running locally and the file directly instead of a directory, it seems to always return "<?xml version=\"1.0\" encoding=\"utf-8\"?>\n<results xmlns=\"http://ntcir-math.nii.ac.jp/\"/>" as the response for any query.

[physikerwelt]

@plato2000 can you try to fix this issue, or did you arrive at the end of your capabilities.

[plusparth]

I don't think I will be able to fix this issue. It seems to produce no traceable errors and simply returns null. The oddest thing is that it started working randomly in the middle as you noticed earlier. For now, I am continuing with the boolean search operators for searches while running the search engine directly instead of through tomcat.

However, in the process of debugging this, I discovered that it seemed that both when the server was running directly and working through tomcat for that brief period of time, any request seemed to return an empty set (not null, but just <?xml version=\"1.0\" encoding=\"utf-8\"?>\n<results xmlns=\"http://ntcir-math.nii.ac.jp/\"> without any results. This persisted even after I recreated the indices. Then, I transferred this file to my local machine and tried using the same index file there and it was able to produce results. This issue turned out to be an issue with the system.properties file overwriting the path I set for the data directory when running the server.

[physikerwelt]

Did you see any problems with running the engine directly?

[plusparth]

@physikerwelt I did not see any problems while running the engine directly. However, I did notice that when Apache was running at the same time as running the engine directly, after about 1-2 minutes, the engine would be killed (the same issue where it seems to run out of memory and be killed). Could this be the issue with running it through Tomcat? (the basex process gets killed, but tomcat keeps the restd service running)?

Also, how would this be fixed? Is it possible to assign the vagrant instance more memory?

[plusparth]

@physikerwelt Is this possible?

[HowardCohl]

@plato2000 Why don't you ask this question to the wikimedia-labs irc http://webchat.freenode.net/?channels=wikimedia-labs

[HowardCohl]

@plato2000 Have you used this irc before?

[plusparth]

@physikerwelt After talking to the people on the Mediawiki IRC, I was told that we should create a bigger instance (large instead of medium). Is this a good option, and if so, how would I do this? Configuration wise, what would be necessary to transfer everything over?

For how, are there things that I can disable on the drmf2016 instance to free up some more RAM temporarily?

[HowardCohl]

It's wikimedia-labs irc, not Mediawiki irc.

[plusparth]

@HowardCohl Yes, that's the IRC I used.

[plusparth]

@physikerwelt I may have found the issue - it seems that when deployed through tomcat, the BaseX server doesn't run. When run directly, netstat shows something running on port 1984 (or something) which is apparently the basex server as shown in the command line output. However, when deployed to Tomcat, this process doesn't run. This seems to be why requests return null - they don't go anywhere. I will look into possible fixes for this.

[physikerwelt]

I did discuss with @HowardCohl that it might be unreasonable, if you spend too much time for the tomcat deployment. Does everything work as expected if you run the search engine via mvn: execute?

[plusparth]

Yes - I will move on from this. I hope that finding out that the BaseX server doesn't run is helpful in debugging this for whoever looks at this later. Should I set up puppet and cron to just run the process using mvn execute?