I am testing a python script with ftputil to transfer a handful of
files to Box using FTPS (ftp and sftp not available at my
institution).
-rw-r--r-- 1 mlewis staff 27560175 Oct 26 22:59 402971 - 40297
Combined.dicom.zip
-rw-r--r-- 1 mlewis staff 48234848 Oct 26 22:59
402971_40297_Combined_AP_1_Abdomen_20210801121318_40297.nii
-rw-r--r-- 1 mlewis staff 20533503 Oct 26 22:59 402972 - 40297
Compton.dicom.zip
-rw-r--r-- 1 mlewis staff 17787690 Oct 26 22:59 402973 - 40297
PhotoElectric.dicom.zip
-rw-r--r-- 1 mlewis staff 14186985 Oct 26 22:59 402974 - 40297
Noise.dicom.zip
-rw-r--r-- 1 mlewis staff 354179 Oct 26 22:59 402976 - 40297
MADplot global.dicom
The Nifti file is the largest of the bunch and the only one to throw
an exception. I tried FTPHost(timeout=None and a large number) as well
as using callback=FTPHost.keep_alive()
and nothing worked. Any suggestions on how to make this transfer more
robust. I get the same exception with the same file on extremely
divergent platforms:
Traceback (most recent call last):
File "/Users/mlewis/opt/miniconda3/envs/box/lib/python3.9/site-packages/ftputil/file.py",
line 174, in close
self._session.voidresp()
File "/Users/mlewis/opt/miniconda3/envs/box/lib/python3.9/ftplib.py",
line 259, in voidresp
resp = self.getresp()
File "/Users/mlewis/opt/miniconda3/envs/box/lib/python3.9/ftplib.py",
line 244, in getresp
resp = self.getmultiline()
File "/Users/mlewis/opt/miniconda3/envs/box/lib/python3.9/ftplib.py",
line 230, in getmultiline
line = self.getline()
File "/Users/mlewis/opt/miniconda3/envs/box/lib/python3.9/ftplib.py",
line 212, in getline
line = self.file.readline(self.maxline + 1)
File "/Users/mlewis/opt/miniconda3/envs/box/lib/python3.9/socket.py",
line 704, in readinto
return self._sock.recv_into(b)
File "/Users/mlewis/opt/miniconda3/envs/box/lib/python3.9/ssl.py",
line 1275, in recv_into
return self.read(nbytes, buffer)
File "/Users/mlewis/opt/miniconda3/envs/box/lib/python3.9/ssl.py",
line 1133, in read
return self._sslobj.read(len, buffer)
socket.timeout: The read operation timed out
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/Users/mlewis/opt/miniconda3/envs/box/lib/python3.9/site-packages/ftputil/host.py",
line 526, in upload
ftputil.file_transfer.copy_file(
File "/Users/mlewis/opt/miniconda3/envs/box/lib/python3.9/site-packages/ftputil/file_transfer.py",
line 200, in copy_file
target_fobj.close()
File "/Users/mlewis/opt/miniconda3/envs/box/lib/python3.9/site-packages/ftputil/file.py",
line 174, in close
self._session.voidresp()
File "/Users/mlewis/opt/miniconda3/envs/box/lib/python3.9/site-packages/ftputil/error.py",
line 226, in __exit__
raise FTPIOError(*exc_value.args, original_error=exc_value) from exc_value
ftputil.error.FTPIOError: The read operation timed out
Debugging info: ftputil 5.0.4, Python 3.9.18 (darwin)
Hi Matthew,
Thanks for the interesting problem. ;-)
From reading the traceback(s), I don't see any obvious reason why
the program might fail in this way.
On 2023-10-27 07:17, Matthew Lewis wrote:
> I am testing a python script with ftputil to transfer a handful of
> files to Box using FTPS (ftp and sftp not available at my
> institution).
Just to check we're on the same page: You _upload_ files to
ftp.box.com ?
> -rw-r--r-- 1 mlewis staff 27560175 Oct 26 22:59 402971 - 40297
> Combined.dicom.zip
>
> -rw-r--r-- 1 mlewis staff 48234848 Oct 26 22:59
> 402971_40297_Combined_AP_1_Abdomen_20210801121318_40297.nii
And this is the only file that fails, giving you the socket
timeout? Does that always happen or were there cases where the
upload finished without raising an exception?
> The Nifti file is the largest of the bunch and the only one to throw
> an exception. I tried FTPHost(timeout=None and a large number) as well
> as using callback=FTPHost.keep_alive()
Would you please send a minimal self-contained code example that
reproduces the problem?
Generally, there are timeouts on the FTP connection level and
timeouts on the socket level. It seems more common to run into
timeouts on the FTP level, so your case is more interesting. ;-)
`FTPHost.keep_alive` works only on the FTP connection level
whereas the `timeout` argument, as I understand the `ftplib`
documentation, works on the socket level. So only the `timeout`
argument (measured in seconds) should have an observable effect
on your problem (if at all).
What was the "large number" you tried for the `timeout` argument?
It could also be that not all involved socket objects got the
timeout applied. For an extra check, you could add a print
statement - including the argument `flush=True` - in
`FTPFile.close` before the `voidresp` call in line 174 and check
how long it takes from then on for the exception to occur. If the
time is much shorter than the timeout you used, this would
suggest that the socket object that was affected by the timeout
exception didn't get your `timeout` argument applied.
> and nothing worked. Any suggestions on how to make this transfer more
> robust. I get the same exception with the same file on extremely
> divergent platforms:
What's the newest Python version you used? Does "extremely
divergent" mean you also tried the code on Linux and Windows? If
you haven't already and it's not too much work, could you try the
code with Python 3.12 or 3.11?
> Traceback (most recent call last):
>
> File "/Users/mlewis/opt/miniconda3/envs/box/lib/python3.9/site-packages/ftputil/file.py",
> line 174, in close
>
> self._session.voidresp()
>
> File "/Users/mlewis/opt/miniconda3/envs/box/lib/python3.9/ftplib.py",
> line 259, in voidresp
>
> resp = self.getresp()
>
> File "/Users/mlewis/opt/miniconda3/envs/box/lib/python3.9/ftplib.py",
> line 244, in getresp
>
> resp = self.getmultiline()
>
> File "/Users/mlewis/opt/miniconda3/envs/box/lib/python3.9/ftplib.py",
> line 230, in getmultiline
>
> line = self.getline()
>
> File "/Users/mlewis/opt/miniconda3/envs/box/lib/python3.9/ftplib.py",
> line 212, in getline
>
> line = self.file.readline(self.maxline + 1)
>
> File "/Users/mlewis/opt/miniconda3/envs/box/lib/python3.9/socket.py",
> line 704, in readinto
>
> return self._sock.recv_into(b)
>
> File "/Users/mlewis/opt/miniconda3/envs/box/lib/python3.9/ssl.py",
> line 1275, in recv_into
>
> return self.read(nbytes, buffer)
>
> File "/Users/mlewis/opt/miniconda3/envs/box/lib/python3.9/ssl.py",
> line 1133, in read
>
> return self._sslobj.read(len, buffer)
>
> socket.timeout: The read operation timed out
It seems this exception happens at the end of the transfer (in
`FTPFile.close`). Moreover, it seems to me that timeout happens
during reading FTP status messages from the server, which happens
on a different connection than the actual file transfer.
Can you check if the file was transferred completely despite the
exception? Granted, even if the file is complete we should still
investigate the problem more and find a way to prevent the
exception.
Best regards,
Stefan