Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[query] avoid hanging the JVM in Dataproc #13916

Merged
merged 5 commits into from
Oct 26, 2023
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 2 additions & 1 deletion hail/python/hail/docs/change_log.md
Original file line number Diff line number Diff line change
Expand Up @@ -54,7 +54,7 @@ critically depend on experimental functionality.**

## Version 0.2.125

Released 2023-10-25
Released 2023-10-26

### New Features

Expand Down Expand Up @@ -100,6 +100,7 @@ Released 2023-10-25

- (hail#13894) Fix #13837 in which Hail could break a Spark installation if the Hail JAR appears on the classpath before the Scala JARs.

- (hail#13919) Fix #13915 which prevented using a glob pattern in `hl.import_vcf`.

## Version 0.2.124

Expand Down
3 changes: 1 addition & 2 deletions hail/python/hailtop/utils/utils.py
Original file line number Diff line number Diff line change
Expand Up @@ -638,8 +638,7 @@ def is_transient_error(e: BaseException) -> bool:
and e.args[0] == "Response payload is not completed"):
return True
if (isinstance(e, aiohttp.ClientOSError)
and len(e.args) >= 2
and 'sslv3 alert bad record mac' in e.args[1]):
and 'sslv3 alert bad record mac' in e.strerror):
# aiohttp.client_exceptions.ClientOSError: [Errno 1] [SSL: SSLV3_ALERT_BAD_RECORD_MAC] sslv3 alert bad record mac (_ssl.c:2548)
#
# This appears to be a symptom of Google rate-limiting as of 2023-10-15
Expand Down
1 change: 1 addition & 0 deletions hail/python/test/hail/conftest.py
Original file line number Diff line number Diff line change
Expand Up @@ -9,6 +9,7 @@

from hail import current_backend, init, reset_global_randomness
from hail.backend.service_backend import ServiceBackend
from hailtop.hail_event_loop import hail_event_loop
from hailtop.utils import secret_alnum_string
from .helpers import hl_init_for_test, hl_stop_for_test

Expand Down
13 changes: 12 additions & 1 deletion hail/src/main/scala/is/hail/backend/BackendServer.scala
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,7 @@ package is.hail.backend

import java.net.InetSocketAddress
import java.nio.charset.StandardCharsets
import java.util.concurrent._
import com.sun.net.httpserver.{HttpContext, HttpExchange, HttpHandler, HttpServer}

import org.json4s._
Expand All @@ -26,12 +27,22 @@ class BackendServer(backend: Backend) {
// 0 => let the OS pick an available port
private[this] val httpServer = HttpServer.create(new InetSocketAddress(0), 10)
private[this] val handler = new BackendHttpHandler(backend)
private[this] val executor = Executors.newFixedThreadPool(1,
new ThreadFactory() {
private[this] val childFactory = Executors.defaultThreadFactory()

def newThread(r: Runnable): Thread = {
val t = childFactory.newThread(r)
t.setDaemon(true)
t
}
})

def port = httpServer.getAddress.getPort

def start(): Unit = {
httpServer.createContext("/", handler)
httpServer.setExecutor(null)
httpServer.setExecutor(executor)
httpServer.start()
}

Expand Down