DWI HYLAND^TERENCE^EDWARD^MR

[23:41:15] Resolving study UID from AHI image set…
[23:41:15]   StudyInstanceUID: 1.2.840.113704.7.1.0.32116203198196250.1688428796.1
[23:41:15]   Batch prefix: batches/20260520T233306Z_1_2_840_113704_7_1_0_32116203198196250_1688428796_1/
[23:41:15] Setting inference status to PENDING and triggering infer tick…
[23:41:15] Waiting for DWI inference (polling every 10s, timeout 900s)…
[23:41:15]   Inference status: PENDING
[23:41:25]   Inference status: INFERENCING
[23:41:35]   Inference status: INFERENCING
[23:41:45]   Inference status: INFERENCING
[23:41:55]   Inference status: INFERENCING
[23:42:05]   Inference status: INFERENCING
[23:42:15]   Inference status: FAILED

[FAILED]
Traceback (most recent call last):
  File "/home/ec2-user/ahir-solution/research_dashboard/pipelines/dwi.py", line 207, in run
    state = _poll_until_done(study_uid, log)
  File "/home/ec2-user/ahir-solution/research_dashboard/pipelines/dwi.py", line 122, in _poll_until_done
    raise RuntimeError(f"Inference {status}: {err}")
RuntimeError: Inference FAILED: infer subprocess failed rc=1
<frozen importlib._bootstrap_external>:1328: FutureWarning: The cuda.cudart module is deprecated and will be removed in a future release, please switch to use the cuda.bindings.runtime module instead.
[s3] downloaded 993 objects from s3://ahi-incoming/batches/20260520T233306Z_1_2_840_113704_7_1_0_32116203198196250_1688428796_1/
[select] DWI selected desc='ep2d_diff_4scan_trace_p2_s2_3mm_TRACEW' b=1000.0 files=42 slices=42
[select] Selected SeriesInstanceUID=1.3.12.2.1107.5.2.50.177636.30000023070311564673400000093 SeriesNumber=9 instances=42
[select] route=dwi model=dwi_segmentation
[select] DWI b-value filtering kept 42/42 slices at b~1000.0 (tol=120.0)
[convert] DICOM -> NIfTI: /srv/infer-work/tmp1zzw6wh0/output/input_image.nii.gz
Traceback (most recent call last):
  File "/home/ec2-user/miniconda3/lib/python3.13/site-packages/urllib3/connection.py", line 198, in _new_conn
    sock = connection.create_connection(
        (self._dns_host, self.port),
    ...<2 lines>...
        socket_options=self.socket_options,
    )
  File "/home/ec2-user/miniconda3/lib/python3.13/site-packages/urllib3/util/connection.py", line 85, in create_connection
    raise err
  File "/home/ec2-user/miniconda3/lib/python3.13/site-packages/urllib3/util/connection.py", line 73, in create_connection
    sock.connect(sa)
    ~~~~~~~~~~~~^^^^
ConnectionRefusedError: [Errno 111] Connection refused

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/ec2-user/miniconda3/lib/python3.13/site-packages/urllib3/connectionpool.py", line 787, in urlopen
    response = self._make_request(
        conn,
    ...<10 lines>...
        **response_kw,
    )
  File "/home/ec2-user/miniconda3/lib/python3.13/site-packages/urllib3/connectionpool.py", line 493, in _make_request
    conn.request(
    ~~~~~~~~~~~~^
        method,
        ^^^^^^^
    ...<6 lines>...
        enforce_content_length=enforce_content_length,
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    )
    ^
  File "/home/ec2-user/miniconda3/lib/python3.13/site-packages/urllib3/connection.py", line 445, in request
    self.endheaders()
    ~~~~~~~~~~~~~~~^^
  File "/home/ec2-user/miniconda3/lib/python3.13/http/client.py", line 1333, in endheaders
    self._send_output(message_body, encode_chunked=encode_chunked)
    ~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/ec2-user/miniconda3/lib/python3.13/http/client.py", line 1093, in _send_output
    self.send(msg)
    ~~~~~~~~~^^^^^
  File "/home/ec2-user/miniconda3/lib/python3.13/http/client.py", line 1037, in send
    self.connect()
    ~~~~~~~~~~~~^^
  File "/home/ec2-user/miniconda3/lib/python3.13/site-packages/urllib3/connection.py", line 276, in connect
    self.sock = self._new_conn()
                ~~~~~~~~~~~~~~^^
  File "/home/ec2-user/miniconda3/lib/python3.13/site-packages/urllib3/connection.py", line 213, in _new_conn
    raise NewConnectionError(
        self, f"Failed to establish a new connection: {e}"
    ) from e
urllib3.exceptions.NewConnectionError: <urllib3.connection.HTTPConnection object at 0x7fc64a59ba10>: Failed to establish a new connection: [Errno 111] Connection refused

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/ec2-user/miniconda3/lib/python3.13/site-packages/requests/adapters.py", line 667, in send
    resp = conn.urlopen(
        method=request.method,
    ...<9 lines>...
        chunked=chunked,
    )
  File "/home/ec2-user/miniconda3/lib/python3.13/site-packages/urllib3/connectionpool.py", line 841, in urlopen
    retries = retries.increment(
        method, url, error=new_e, _pool=self, _stacktrace=sys.exc_info()[2]
    )
  File "/home/ec2-user/miniconda3/lib/python3.13/site-packages/urllib3/util/retry.py", line 519, in increment
    raise MaxRetryError(_pool, url, reason) from reason  # type: ignore[arg-type]
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
urllib3.exceptions.MaxRetryError: HTTPConnectionPool(host='127.0.0.1', port=8000): Max retries exceeded with url: /infer/dwi_segmentation?output=image (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7fc64a59ba10>: Failed to establish a new connection: [Errno 111] Connection refused'))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/opt/AHIR/s3_to_monai_infer.py", line 1671, in <module>
    main()
    ~~~~^^
  File "/opt/AHIR/s3_to_monai_infer.py", line 1479, in main
    infer_txt = run_monai_inference(dict(infer_params))
  File "/opt/AHIR/s3_to_monai_infer.py", line 1457, in run_monai_inference
    monailabel_infer_file(
    ~~~~~~~~~~~~~~~~~~~~~^
        monai_url=monai_url,
        ^^^^^^^^^^^^^^^^^^^^
    ...<5 lines>...
        infer_params=params,
        ^^^^^^^^^^^^^^^^^^^^
    )
    ^
  File "/opt/AHIR/s3_to_monai_infer.py", line 513, in monailabel_infer_file
    r = requests.post(f"{monai_url}/infer/{model}", params=params, data=data, files=files, timeout=1800)
  File "/home/ec2-user/miniconda3/lib/python3.13/site-packages/requests/api.py", line 115, in post
    return request("post", url, data=data, json=json, **kwargs)
  File "/home/ec2-user/miniconda3/lib/python3.13/site-packages/requests/api.py", line 59, in request
    return session.request(method=method, url=url, **kwargs)
           ~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/ec2-user/miniconda3/lib/python3.13/site-packages/requests/sessions.py", line 589, in request
    resp = self.send(prep, **send_kwargs)
  File "/home/ec2-user/miniconda3/lib/python3.13/site-packages/requests/sessions.py", line 703, in send
    r = adapter.send(request, **kwargs)
  File "/home/ec2-user/miniconda3/lib/python3.13/site-packages/requests/adapters.py", line 700, in send
    raise ConnectionError(e, request=request)
requests.exceptions.ConnectionError: HTTPConnectionPool(host='127.0.0.1', port=8000): Max retries exceeded with url: /infer/dwi_segmentation?output=image (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7fc64a59ba10>: Failed to establish a new connection: [Errno 111] Connection refused'))

[06:56:46] Resolving study UID from AHI image set…
[06:56:47]   StudyInstanceUID: 1.2.840.113704.7.1.0.32116203198196250.1688428796.1
[06:56:47]   Batch prefix: batches/20260520T233306Z_1_2_840_113704_7_1_0_32116203198196250_1688428796_1/
[06:56:47] Setting inference status to PENDING and triggering infer tick…
[06:56:47] Waiting for DWI inference (polling every 10s, timeout 900s)…
[06:56:47]   Inference status: PENDING
[06:56:57]   Inference status: INFERENCING
[06:56:58] Resolving study UID from AHI image set…
[06:56:58]   StudyInstanceUID: 1.2.840.113704.7.1.0.32116203198196250.1688428796.1
[06:56:58]   Batch prefix: batches/20260520T233306Z_1_2_840_113704_7_1_0_32116203198196250_1688428796_1/
[06:56:58]   Inference already INFERENCING — joining poll (not resetting)…
[06:56:58] Waiting for DWI inference (polling every 10s, timeout 900s)…
[06:56:58]   Inference status: INFERENCING
[06:57:07]   Inference status: INFERENCING
[06:57:08]   Inference status: INFERENCING
[06:57:17]   Inference status: INFERENCING
[06:57:18]   Inference status: INFERENCING
[06:57:27]   Inference status: INFERENCING
[06:57:28]   Inference status: INFERENCING
[06:57:37]   Inference status: INFERENCING
[06:57:38]   Inference status: INFERENCING
[06:57:47]   Inference status: INFERENCING
[06:57:48]   Inference status: INFERENCING
[06:57:57]   Inference status: INFERENCING
[06:57:58]   Inference status: INFERENCING
[06:58:07]   Inference status: INFERENCING
[06:58:08]   Inference status: INFERENCING
[06:58:17]   Inference status: INFERENCING
[06:58:18]   Inference status: INFERENCING
[06:58:27]   Inference status: INFERENCING
[06:58:28]   Inference status: INFERENCING
[06:58:37]   Inference status: INFERENCING
[06:58:38]   Inference status: INFERENCING
[06:58:47]   Inference status: INFERENCING
[06:58:48]   Inference status: INFERENCING
[06:58:57]   Inference status: COMPLETED
[06:58:57]   Inference COMPLETED. Output: inference-results/1_2_840_113704_7_1_0_32116203198196250_1688428796_1/20260522T065647Z/
[06:58:57] Downloading inference outputs from S3…
[06:58:57]   label downloaded
[06:58:57] Computing ischemic core volume…
[06:58:58]   Ischemic core: 10.3 mL  (voxel counts: {1: 8757})
[06:58:58] Uploading research outputs to S3…
[06:58:58]   Uploaded 3 files
[06:58:58] DWI pipeline complete. Core: 10.3 mL
[06:58:58]   Inference status: COMPLETED
[06:58:58]   Inference COMPLETED. Output: inference-results/1_2_840_113704_7_1_0_32116203198196250_1688428796_1/20260522T065647Z/
[06:58:58] Downloading inference outputs from S3…
[06:58:58]   label downloaded
[06:58:59] Computing ischemic core volume…
[06:58:59]   Ischemic core: 10.3 mL  (voxel counts: {1: 8757})
[06:58:59] Uploading research outputs to S3…
[06:58:59]   Uploaded 3 files
[06:58:59] DWI pipeline complete. Core: 10.3 mL