所有这种类似情况基本都和DNS有关

from: http://mediakey.dk/~cc/openssh-disabled-reverse-dns-lookup/

The post is mostly to myself, but hopefully useful for others too.

When connecting to an OpenSSH server, one of the first things that the server does is to perform a reverse DNS loopuk on the client’s IP, which is in general a slow operation and causing for an unnecessary delay during authentication. However this reverse DNS lookup can be disabled.

Do the following to disable it:

# In /etc/ssh/sshd_config:
UseDNS no