TCK: WebSocket NegDep Challenge

[volosied]

Challenged tests
Main Package: https://github.com/jakartaee/platform-tck/tree/main/websocket/spec-tests/src/main/java/com/sun/ts/tests/websocket/negdep

wsc_negdep_invalidpathparamtype_srv_onclose_web
wsc_negdep_invalidpathparamtype_srv_onerror_web
wsc_negdep_invalidpathparamtype_srv_onmessage_web
wsc_negdep_invalidpathparamtype_srv_onopen_web
wsc_negdep_malformedpath_web
wsc_negdep_multiplepaths_web
wsc_negdep_onclose_srv_duplicate_web
wsc_negdep_onclose_srv_toomanyarguments_web
wsc_negdep_onerror_srv_duplicate_web
wsc_negdep_onerror_srv_toomanyarguments_web
wsc_negdep_onmessage_pasrv_nomoreendpoints_web
wsc_negdep_onmessage_ppsrv_nomoreendpoints_web
wsc_negdep_onmessage_srv_binarybytebufferint_web
wsc_negdep_onmessage_srv_binaryinputstreamboolean_web
wsc_negdep_onmessage_srv_binaryinputstreamboolean_web
wsc_negdep_onmessage_srv_binarynodecoder_web
wsc_negdep_onmessage_srv_pongboolean_web
wsc_negdep_onmessage_srv_pongduplicate_web
wsc_negdep_onmessage_srv_textbigdecimal_web
wsc_negdep_onmessage_srv_textduplicate_web
wsc_negdep_onmessage_srv_textnodecoder_web
wsc_negdep_onmessage_srv_textreaderboolean_web
wsc_negdep_onmessage_srv_textstringint_web
wsc_negdep_onopen_srv_duplicate_web
wsc_negdep_onopen_srv_toomanyarguments_web

TCK Version
Main Branch of platform-tck/websocket

Description
This is partly a challenge and partly a request for improvement. These web socket tests listed above have moved to use the Arquillian framework. They invoke the application and then expect an exception to occur, as these are negative tests (some misconfiguration exists in the app).


While testing on Liberty, we have encountered various errors when trying to get these tests to run. By design, these apps trigger DeploymentException, which causes them to not start in the server. As a result, Arquillian and any related plugins we use cannot handle the app (since it doesn’t exist for us). After making a few fixes on our side, the last error we’ve encountered is related to the URL: 



java.lang.RuntimeException: Could not lookup value for field protected java.net.URL com.sun.ts.tests.websocket.common.client.WebSocketCommonClient.url
        at org.jboss.arquillian.test.impl.enricher.resource.ArquillianResourceTestEnricher.enrich(ArquillianResourceTestEnricher.java:68)
        at org.jboss.arquillian.test.impl.TestInstanceEnricher.enrich(TestInstanceEnricher.java:51)
        at org.jboss.arquillian.container.test.impl.ClientTestInstanceEnricher.enrich(ClientTestInstanceEnricher.java:48)
         …

Caused by: java.lang.RuntimeException: All Providers for type class java.net.URL returned a null value: [org.jboss.arquillian.container.test.impl.enricher.resource.URLResourceProvider@573d07e6]
        at org.jboss.arquillian.test.impl.enricher.resource.ArquillianResourceTestEnricher.lookup(ArquillianResourceTestEnricher.java:126)
        at org.jboss.arquillian.test.impl.enricher.resource.ArquillianResourceTestEnricher.enrich(ArquillianResourceTestEnricher.java:66)

We’re unsure if any other implementations have been tested yet, but we would expect similar issues to occur. Arquillian is trying to do some Test enrichment, but due to the deployment exception, there is no available URL for these tests.

We propose to remove the invoke all in these tests: 




try {
  invoke(false);
} catch (Exception tfe) {
  // DeploymentException
}



and instead rely on the Arquillian’s ShouldThrowException annotation. This has the added bonus of being able to explicitly check for DeploymentException, instead of just catching Exception as is currently done. This is similar to the CDI TCK’s approach, as linked here and here.

Some extra handling would also be needed to verify that any previously deployed endpoints are removed from service. A draft PR can be provided.

Of if anyone has a workaround for the URL issue above, we’d be happy to accept other suggestions.

Additional context
None

[pnicolucci]

Tagged as a challenge. @markt-asf @joakime can you take a look?

[markt-asf]

The tests all pass with Tomcat and Eclipse Tyrus. I'm not adverse to improving the tests but this looks more like an issue with the Liberty / Arquillian integration at this point.

[brideck]

this looks more like an issue with the Liberty / Arquillian integration at this point.

Which part do you consider to be the issue here? That Liberty fails to start these applications? Or that the Liberty Arquillian plugin considers this to be an Arquillian deployment exception? Since the application doesn't start, there is no valid URL that could be supplied. The plugin could certainly fake one up, but it already knows at this point that no endpoints will exist so why bother?

I'm surprised that other plugin implementations wouldn't treat this sort of case the same way.

[jansupol]

@pnicolucci What JDK do you use to run the TCK? Can you try with 21?

[brideck]

What JDK do you use to run the TCK? Can you try with 21?

17 & 21. Things work the same in both cases.

[jansupol]

Here is Tyrus script with some old glassfish that works with JDK 21.

[brideck]

Right, so the fundamental difference here is that Glassfish seems to go ahead and deploy the application anyway:

[2024-12-19T15:28:16.829040-08:00] [GF 8.0.0-M3] [INFO] [] [jakarta.enterprise.system.core] [tid: _ThreadID=51 _ThreadName=admin-listener(3)] [levelValue: 800] [[
  wsc_negdep_invalidpathparamtype_srv_onclose_web was successfully deployed in 186 milliseconds.]]

The WebSocket spec (https://jakarta.ee/specifications/websocket/2.2/jakarta-websocket-spec-2.2#deployment-errors) states:

Other errors arise as a result of a malformed WebSocket application. Chapter 4 provides several examples of WebSocket endpoints that are malformed. In such cases, the container must provide an informative error message to the deployer during the deployment process [WSC-5.2.1-2].

In both cases, a deployment error raised during the deployment process must halt the deployment of the application, any well formed endpoints deployed prior to the error being raised must be removed from service and no more WebSocket endpoints from that application may be deployed by the container, even if they are valid [WSC-5.2.1-3].

Liberty has chosen to fulfill "must halt the deployment of the application" as meaning that the application does not end up deployed at all.

[volosied]

Is there any chance these tests could be excluded from this release? This would allow more time to investigate between the implementations and the Arquillian plugins.

The application starting in Glassfish looks a bit concerning since should not start due to a deployment exception. As mentioned, Liberty fails to start the app, and this leads no URL provided error later on in Arqullian.

Additionally, my original suggestion was to use @ShouldThrowException, but, after looking at Tomcat's plugin, I'm not so sure it would handle ShouldThrowException? Glassfish may not either.

[volosied]

Following up on my request to exclude the tests. Thanks!

[markt-asf]

Looking at this is on my TODO list. I want to understand why the test passes with Tomcat before taking any action on this.

[brideck]

The test currently expects the app to deploy, but for none of the endpoints to exist, which I believe is exactly how Tomcat currently performs (I know @volosied did a little investigation of Tomcat awhile back, so perhaps he can recall). That's definitely the case for Glassfish.

Open Liberty doesn't deploy the app (therefore none of the endpoints exist), but that doesn't play nicely with the test, which does not expect a deployment failure.

We actually worked around the app deployment failure in the old TCK by special casing these specific tests in our TCK porting package to ignore the exception and proceed, which is pretty broken. We don't want to do the same thing here because our Arquillian plugin is open source, available for anyone to use, and not specific to TCK testing. The old porting package is proprietary and specific to TCK testing, so that was a marginally acceptable solution there.

[volosied]

I was pushing for excluding these tests more due to Glassfish's behavior (deploying the app even with an exception).

As for Tomcat, here's the output below. It does report startup failed due to previous errors. But I don't know how it interacts with Arquillian to determine that startup failed. I would expect no URLs to be available, and thus a similar error as in Liberty.

09-Dec-2024 15:31:32.910 INFO [main] org.apache.catalina.startup.Catalina.start Server startup in [280] milliseconds
09-Dec-2024 15:32:12.944 INFO [Catalina-utility-1] org.apache.catalina.startup.HostConfig.deployWAR Deploying web application archive [/Users/volosied/Downloads/apache-tomcat-11.0.2/webapps/wsc_negdep_invalidpathparamtype_srv_onclose_web.war]
09-Dec-2024 15:32:12.992 SEVERE [Catalina-utility-1] org.apache.catalina.core.StandardContext.startInternal Error during ServletContainerInitializer processing
	jakarta.servlet.ServletException: jakarta.websocket.DeploymentException: The type [com.sun.ts.tests.websocket.negdep.StringHolder] is not permitted as a path parameter. Parameters annotated with @PathParam may only be Strings, Java primitives or a boxed version thereof.
		at org.apache.tomcat.websocket.server.WsSci.onStartup(WsSci.java:116)
		at org.apache.catalina.core.StandardContext.startInternal(StandardContext.java:4393)
		at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:164)
		at org.apache.catalina.core.ContainerBase.addChild(ContainerBase.java:566)
		at org.apache.catalina.core.StandardHost.addChild(StandardHost.java:653)
		at org.apache.catalina.startup.HostConfig.deployWAR(HostConfig.java:933)
		at org.apache.catalina.startup.HostConfig$DeployWar.run(HostConfig.java:1875)
		at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539)
		at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
		at org.apache.tomcat.util.threads.InlineExecutorService.execute(InlineExecutorService.java:75)
		at java.base/java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:123)
		at org.apache.catalina.startup.HostConfig.deployWARs(HostConfig.java:735)
		at org.apache.catalina.startup.HostConfig.deployApps(HostConfig.java:387)
		at org.apache.catalina.startup.HostConfig.check(HostConfig.java:1642)
		at org.apache.catalina.startup.HostConfig.lifecycleEvent(HostConfig.java:263)
		at org.apache.catalina.util.LifecycleBase.fireLifecycleEvent(LifecycleBase.java:109)
		at org.apache.catalina.core.ContainerBase.backgroundProcess(ContainerBase.java:940)
		at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1139)
		at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1143)
		at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.run(ContainerBase.java:1121)
		at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539)
		at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305)
		at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305)
		at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
		at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
		at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:63)
		at java.base/java.lang.Thread.run(Thread.java:833)
	Caused by: jakarta.websocket.DeploymentException: The type [com.sun.ts.tests.websocket.negdep.StringHolder] is not permitted as a path parameter. Parameters annotated with @PathParam may only be Strings, Java primitives or a boxed version thereof.
		at org.apache.tomcat.websocket.pojo.PojoPathParam.validateType(PojoPathParam.java:65)
		at org.apache.tomcat.websocket.pojo.PojoPathParam.<init>(PojoPathParam.java:41)
		at org.apache.tomcat.websocket.pojo.PojoMethodMapping.getPathParams(PojoMethodMapping.java:298)
		at org.apache.tomcat.websocket.pojo.PojoMethodMapping.<init>(PojoMethodMapping.java:193)
		at org.apache.tomcat.websocket.server.WsServerContainer.addEndpoint(WsServerContainer.java:132)
		at org.apache.tomcat.websocket.server.WsServerContainer.addEndpoint(WsServerContainer.java:242)
		at org.apache.tomcat.websocket.server.WsSci.onStartup(WsSci.java:113)
		... 26 more
09-Dec-2024 15:32:12.993 SEVERE [Catalina-utility-1] org.apache.catalina.core.StandardContext.startInternal Context [/wsc_negdep_invalidpathparamtype_srv_onclose_web] startup failed due to previous errors
09-Dec-2024 15:32:12.997 INFO [Catalina-utility-1] org.apache.catalina.startup.HostConfig.deployWAR Deployment of web application archive [/Users/volosied/Downloads/apache-tomcat-11.0.2/webapps/wsc_negdep_invalidpathparamtype_srv_onclose_web.war] has finished in [53] ms

[brideck]

I'm pretty sure that Deployment of web application archive [/Users/volosied/Downloads/apache-tomcat-11.0.2/webapps/wsc_negdep_invalidpathparamtype_srv_onclose_web.war] has finished in [53] ms means that from an Arquillian perspective there would be no errors encountered, but that's for a Tomcat expert to know for sure.