Secure FTP

[pohmelie]

After #36 I read about FTPS, SFTP and FTP over SSH.

• SFTP is SSH extension and we should ignore it

• FTPS is extension of FTP (https://tools.ietf.org/html/rfc2228)

• FTP over SSH is (as I realized) just tunneling and have problems(?) with data connections

• implicit ftps mode (Add implicit FTPS support #81)

• explicit ftps mode (add client support for explicit tls #183)

[rsichnyi]

FTP over SSH

yes, it's just tunneling...

i was thinking of FTPS - actually this might be quite simple, but as usually the main problem is finding free time for doing things (especially with a full-time job)

[creatorrr]

👍 Any updates on this? Happy to help out.

[pohmelie]

@creatorrr, I tried out ssl module:

import asyncio
import ssl


req = b"GET / HTTP/1.1\r\nHost: www.python.org\r\n\r\n"


async def flush(outgoing, writer):
    if outgoing.pending:
        writer.write(outgoing.read())
        await writer.drain()


async def foo():
    BLOCK_SIZE = 8192
    # ssl_context = ssl.SSLContext(ssl.PROTOCOL_SSLv23)
    ssl_context = ssl.create_default_context(purpose=ssl.Purpose.SERVER_AUTH)
    ssl_context.check_hostname = False

    incoming = ssl.MemoryBIO()
    outgoing = ssl.MemoryBIO()
    ssl_object = ssl_context.wrap_bio(incoming, outgoing)

    reader, writer = await asyncio.open_connection("python.org", 443)
    while True:
        try:
            ssl_object.do_handshake()
            break
        except ssl.SSLWantReadError:
            await flush(outgoing, writer)
            incoming.write(await reader.read(BLOCK_SIZE))

    ssl_object.write(req)
    await flush(outgoing, writer)
    data = None
    while True:
        try:
            data = ssl_object.read(BLOCK_SIZE)
            print(data)
        except ssl.SSLWantReadError:
            await flush(outgoing, writer)

        await flush(outgoing, writer)
        data = await reader.read(BLOCK_SIZE)
        # print("->", data)
        if not data:
            break
        incoming.write(data)

    writer.close()


if __name__ == "__main__":
    loop = asyncio.get_event_loop()
    loop.run_until_complete(foo())

But it's pretty annoying… also, I was thinking about who will need this, since I did not find any FTPS server online. Feel free to implement this functionality is you have enough courage and time 😉

[creatorrr]

Haha, thanks for looking into this, @pohmelie! I will try and see if I can get things to work and create a patch.

[oleksandr-kuzmenko]

I did not find any FTPS server online

#81 — works with my private FTPS.

cc @pohmelie

[pohmelie]

Unfortunately, there is an asyncio bug, which will come out when your path io is slower than network io. Solution for this (except wait for fix) is own wrapper, like present above.

[markshhsu]

Unfortunately, there is an asyncio bug, which will come out when your path io is slower than network io. Solution for this (except wait for fix) is own wrapper, like present above.

@pohmelie, for the example code you provided above, I import uvloop and the example code can run without any errors, seems uvloop can fix this issue?

[pohmelie]

@markshhsu, it looks like yes.

[Midnighter]

Just to note that I'm failing to connect to an SFTP server.

Trying to form a connection like this:

    async with aioftp.Client.context(
        host=settings.host,
        port=settings.port,
        user=settings.username,
        password=settings.password.get_secret_value(),
        ssl=True,
        socket_timeout=10,
        path_timeout=10,
    ) as client:
        logger.info("Connected.")
        logger.info(str(await client.list()))

which results in this error:

    async with aioftp.Client.context(
  File "~/miniconda3/envs/fetch/lib/python3.8/contextlib.py", line 171, in __aenter__
    return await self.gen.__anext__()
  File "~/miniconda3/envs/fetch/lib/python3.8/site-packages/aioftp/client.py", line 1199, in context
    await client.connect(host, port)
  File "~/miniconda3/envs/fetch/lib/python3.8/site-packages/aioftp/client.py", line 604, in connect
    await super().connect(host, port)
  File "~/miniconda3/envs/fetch/lib/python3.8/site-packages/aioftp/client.py", line 131, in connect
    reader, writer = await self._open_connection(host, port)
  File "~/miniconda3/envs/fetch/lib/python3.8/asyncio/streams.py", line 52, in open_connection
    transport, _ = await loop.create_connection(
  File "~/miniconda3/envs/fetch/lib/python3.8/asyncio/base_events.py", line 1050, in create_connection
    transport, protocol = await self._create_connection_transport(
  File "~/miniconda3/envs/fetch/lib/python3.8/asyncio/base_events.py", line 1080, in _create_connection_transport
    await waiter
  File "~/miniconda3/envs/fetch/lib/python3.8/asyncio/sslproto.py", line 529, in data_received
    ssldata, appdata = self._sslpipe.feed_ssldata(data)
  File "~/miniconda3/envs/fetch/lib/python3.8/asyncio/sslproto.py", line 189, in feed_ssldata
    self._sslobj.do_handshake()
  File "~/miniconda3/envs/fetch/lib/python3.8/ssl.py", line 944, in do_handshake
    self._sslobj.do_handshake()
ssl.SSLError: [SSL: WRONG_VERSION_NUMBER] wrong version number (_ssl.c:1124)

If I set the ssl argument to ssl.SSLContext(ssl.PROTOCOL_TLS) then I see a message

Using selector: EpollSelector

but nothing happens and I have to interrupt.

I can connect to the server with the same information when using pysftp and disabling the SSH known hosts check.

    cnx_opts = pysftp.CnOpts()
    cnx_opts.hostkeys = None
    with pysftp.Connection(
        host=settings.host,
        port=settings.port,
        username=settings.username,
        password=settings.password.get_secret_value(),
        cnopts=cnx_opts,
    ) as sftp:
        logger.info("Connected.")
        logger.info(sftp.pwd)
        logger.info(sftp.listdir())

Do you have any advice on what I'm doing wrong?

[pohmelie]

@Midnighter, since you have success with pysftp lib, I think you are trying to do SFTP, but it is unrelated to aioftp and ftp protocol. aioftp have partial FTPS support though. This information is in first post of this issue.

[Midnighter]

🤔 indeed the distinction between SFTP and FTP over SSH (which you demonstrated in your comment) was not clear to me. Thank you for the heads up.