You can gain up to 10 % of raw speed if you disable the tcpd wrapper. This might make your system more vulnerable -- check out what the TCP wrapper does before disabling it.
Just edit /etc/inetd.conf and change the line that looks like this:
imap stream tcp nowait.1000 root /usr/sbin/tcpd imapd
imap stream tcp nowait.1000 root /usr/sbin/imapd imapd
Change the /usr/sbin/imap to the right path, just in case it isn't installed there. (While you are there, check out ConnectionsPerMinute.)
You also need to restart inetd. Do this:
killall -HUP inetd