Help hurricane victims by donating to the American Red Cross.

From Ollie's Web Site

Sysadmin Topics for Administering sulfurous

Confirming vital processes are running

1. "ps aux --cols 120 | less" is an excellent tool to see a static list of all running processes. You should see spamd (the spamassassin daemon) and several qmail processes.
2. "qmailctl stat" shows up time for the 4 key qmail processes. All should show positive uptime.

Starting / stopping qmail

• Stop qmail - "qmailctl stop" Note: this is not instantaneous. qmail-send completes its queue before shutting down. Check status with "qmailctl stat."
• Start qmail - "qmailctl start." Check that all 4 processes are running (have non-zero uptime) with "qmailctl stat."
• Restart qmail - "qmailctl restart." Check with "qmailctl stat."

Turn qmail-scanner-queue + spamassassin on / off

• Start scanning - "/root/tcp.smtp-scan" is a script that starts spamassassin, resets /etc/tcp.smtp.cdb to use qmail-scanner-queue and restarts qmail. Following this, accepted mail passes through spamassassin.
• Stop scanning - "/root/tcp.smtp-noscan" is a script that resets /etc/tcp.smtp.cdb to stop using qmail-scanner-queue, restarts qmail and stops spamassassin. Following this, qmail on sulfurous is a pass-through server, forwarding accepted mail to exchange.

Spamassassin control

• /etc/init.d/spamassassin - stops, starts, restarts spamd, the spamassassin daemon. Run it without any command to get the list of commands.

Logs

Here are some of the more interesting log files:

• /var/log/qmail/ - contains several days of logs from qmail-send, i.e., what qmail sends out, principally to exchange. Use /var/log/qmail/logreader (or logreader-current) to view the log. Use /var/log/qmail/logreader-tail to get a live update log.
• /var/log/qmail/smtpd/ - contains several days of logs from qmail-smtpd, i.e., what qmail receives from the outside. Bad users are logged here. Use /var/log/qmail/smtpd/logreader-tail to get a live update log.
• /var/log/mail-info - qmail-scanner drops its information here. Use tail -f /var/log/mail.info for live update, or cat /var/log/mail.info | less for static view.
• /var/spool/qscan - contains very detailed qmail-scanner logs. Look here for details of how qmail-scanner and spamassassin are working.

top

Top is a good live-update summary of running processes, including memory useage.

Listing files

• midnight commander - use F3 to list file in scrollable window.
• cat filename - lists contents of filename. Best combined with less to get pager capability, e.g., cat mail.info | less.
• tail filename - lists last 10 lines of filename.
• tail -f filename - continuously updated listing of latest lines in filename.

server bogs down

Although elimination of bad rcptto plus added memory seem to have cured the problem, if the number of processes climbs too high and everything is bogging down, try:

• run /root/tcp.smtp-noscan - this may take a while, just waiting for the server to get around to running the script. Once it runs, unprocessed emails should rather speedily be sent on to exchange. Check the numbers with "qmailctl stat."

To add a good user email address

• 1. To add a Good User go to /var/qmail/control and add user to end of moregoodrcptto
• 2. To update user database go to /var/qmail/bin and run qmail-newmgrt
• 3. To verify check /var/qmail/control and check the time stamp on moregoodrcpt.cdb

To check bad user list for mistakes

• Go to /var/log/qmial/smtpd and read baduserout
• Update the list by running /root/badusers.

HQM Broken into 2/16/07

On Saturday morning, 2/17/07, I noted that the logwatch file for HQM for the previous day had not shown up in my mail. I logged on and ran 'qmailctl stat' to find everything running, but over 69,000 messages in the queue. Further, the qmail logs showed vast number of entries of the form:

delivery 984997: deferral: Unable_to_chdir_to_maildir._(#4.2.1)/

On investigating the queue, I found most of the messages were bounce messages back to qscand@hqm.hamonusa.com, the ostensible sender of the original messages. Since qscand is treated as a local by qmail, qmail tried to deliver them to that user, but there was no maildir structure. As a resulted, qmail posted the above deferral message.

I then reviewed the home directory for qscand and found a new subdirectory named "," which had been created at 2/16 12:52 pm. Within it was a subdirectory called "perlsend" which contained a perl mailing script, a mail message to be mailed and a file containing 113,673 email addresses. It looks like the crook gained access, uploaded a tar containing these files, unpacked them and ran the script. Over half of the messages bounced, creating the big mess in the queue (I could see most of the trail in the file ".bash_history").

I then investigated how the crook had gotten in. It appears that the problem was that the user qscand was created with a real shell (/bin/bash) instead of a fake shell (/bin/false) per qscand installation recommendations. Additionally, the password was weak and easily guessed. I speculate that an automated login script found the weakness and exploited it.

Running logwatch manually showed the following sshd activity yesterday:

Users logging in through sshd:
    mlies:
       64.8.212.2 (64-8-212-2.client.dsl.net): 2 times
    qscand:
       172.167.95.82 (ACA75F52.ipt.aol.com): 3 times
       172.165.188.144 (ACA5BC90.ipt.aol.com): 1 time
       200.69.243.81 (customer243-81.iplannetworks.net): 1 time

 SFTP subsystem requests: 2 Time(s)

 **Unmatched Entries**
 reverse mapping checking getaddrinfo for customer243-81.iplannetworks.net failed - POSSIBLE BREAK-IN ATTEMPT! : 3 time(s)

Actions:

1. Shut down qmail while investigating.
2. Changed entry in /etc/passwd to /bin/false. Prevents logins.
3. Renamed the "," subdirectory and made it readable only by root.
4. Used maildirmake to give qscand a maildir. qmail immediately began delivering the queued bounce messages. The intention is to get all the bounces delivered and then spot check them. Assuming they are all bounces from the above, then I will delete them.
5. I also found the same vulnerability in user spamd. I performed the same fix.
6. I checked sulfurous and found no comparable vulnerability.
7. I am still checking what I did not get the logwatch report. It has not been obviously disabled.
8. (2/20/07) Restricted ssh logins to two known accounts via AllowUsers in /etc/ssh/sshd_config.