Unix Made Easy: Domainkeys,DKIM and SPF with Postfix

Unix Made Easy: Domainkeys,DKIM and SPF with Postfix: "Domainkeys,DKIM and SPF with Postfix SPAM and Phishing has been a growing problem for a long time and more recently the battle to stamp i..."

Domainkeys,DKIM and SPF with Postfix

Domainkeys,DKIM and SPF with Postfix

SPAM and Phishing has been a growing problem for a long time and more recently the battle to stamp it out has been getting more aggressive resulting in a lot of legitimate mail starting to get discarded as SPAM/Phishing.

For less technical users and all but the best system administrators, it is often near impossible to jump through all the hurdles to ensure all mail always gets where it's intended. In many cases misconfigurations or at lest sub-optimum configurations (eg. reverse DNS mismatches) play a part. Adding to the problem is that many anti-SPAM mechanisims discard (or quarantine and then expire) mail after it has been accepted by the server, thus defeating the design of SMTP that no mail should be lost (it would either delivered or bounced back to the sender). In many cases the mail losses go unnoticed, or are just accepted as normal and people have to re-send mail that does not get through.

Almost all SPAM is forged, often using legitimate addresses or domains as the fake source addresses. Domainkeys (originally proposed by Yahoo!) provides a means of verifying the mail has in fact come from where it claims which all sounds good. If widely implemented this could largely stamp out many Phishing mails and much more.

Additionally, SPF (Sender Policy Framework) can be added to verify the source of the email is legitimate.

These all add credibility to your mail and reduce the risks of having your domain blacklisted or your mail silently discarded by others systems.

There is however plenty to consider....

DomainKeys Warts and all

The first snag we hit is that there are in fact two standards which are confusingly named: DomainKeys (DK) and DomainKeys Identified Mail (DKIM). Although plain DK is historic and should not be used for verification (a point missed by many), there are still systems (including reports of some major web mail providers) out there who run verification on the legacy standard. That wouldn't be so bad if it wasn't that the same DNS records are used by both standards so if you don't sign mail with legacy standard on sending, it looks to systems that do verify against the legacy standard like the mail is faked. This means that (for now anyway) we have to support both standards on sending to ensure that mail gets through.

Secondly, almost all the information I have found on setting up DK/DKIM seems to say how to configure things in testing mode (where mail gets treated the same as unsigned mail) and then stops there. Even many of the worlds leading tech companies are running their DK/DKIM in testing mode, and I'm sure the ones reading this are thinking that it can't be them! That's fine if they are testing, but few seem to be brave enough to bite the bullet and switch off test mode. This effectively means that although they have DK/DKIM, they are requesting that everyone ignores the DK/DKIM signatures and treats their mail as unsigned.

And finally, it's not without it's vulnerabilities. The public key (for verification) is distributed by DNS in a TXT record. This is back to the classic crypto key exchange problem. If your DNS can be compromised or just faked to the receiving server or upstream DNS caches, then anyone can pretend to be sending mail from you, or even DoS your mail (eg. corrupt your public key so verification always fails and your mail gets discarded). This DoS could in fact be used even if the sender doesn't support DK/DKIM as fake DNS records would tell everyone that they do. Nothing to be alarmed by - it is still possible to disrupt mail flow without DK/DKIM if DNS gets compromised.

Not really completely bullet proof, but provides more integrity than no verification at all. Personally I think the problem is that SMTP was designed at a time where there was no reason to worry about what would come through email - times have changed and the underlying SMTP protocol isn't hardened against the abuse that happens now. All the add-ons will only have impact if they are widely deployed, but how many admins out there even have a clue they exist? So long as basic SMTP is alive and kicking the problems will continue. The only certain way to stop it is to replace SMTP with a modern protocol where verification is mandatory at every stage, but that's not going to happen any time soon.

DK/DKIM only validates part of the mail and to get the full benefit of authenticating mail all the way really needs to be combined with other technologies like SPF (see later) and ADSP (see later).

Should you use it?

There are a number of aspects to DK/DKIM and SPF that undermine their value:

• Almost everyone operating their DKIM in test mode, effectively requesting peers to treat mail as unsigned
• There is no formal requirement for setting a domain policy so mail can easily be forged
• DKIM ADSP (previously known as ASP) provides the beginnings of a policy mechanism for DKIM, but at this time is not formalised, and recommendations include running it in a mode where it is acceptable not to sign messages. This again defeats the effectiveness of the system.
• DKIM doesn't authenticate the envelope but rather selected aspects of the mail. This means that if these aspects are then replicated exactly other aspects of the mail (including the envelope and hence recipient) may be changed.
• Neither DK/DKIM nor SPF is widely used beyond a few major mail providers. SPF does seem to be more widely deployed in organisations that are heavily phished. The many small providers, corporates etc. aren't paying any attention, and I doubt even know anything about it.
• None of them protect against account breakins (eg. via a trojaned machine) and other mail that would appear to authenticate properly.
• If misconfigured (either on the sending or receiving side), it could make a real disaster of your mail

The big thing in the favour of DKIM and SPF is that they add credibility to mail from your domain. If the mail checks out then odds are it's legit, and in making it more difficult for spammers and fraudsters to use your domain, you reduce the chance of it being blaclisted.

If you are also running a highly phished domain then it can be useful to discourage abuse of your domain. That said, a quick check of a few high street banks which I see an enormous amount of phishing of, only one I checked had SPF configured - that's all! It's such an easy way to protect their customers from phishing, yet few can be bothered.

How DK/DKIM it works

What DK/DKIM does is relatively simple, though not without it's warts. The concept is that key parts of the mail get cryptographically signed with a private key on the sending server, and then verified on the receiving server.

Each server doing signing can have a unique "selector" with a matching key making it easier to have multiple independent machines without having to keep keys in sync across them all. It also means provides a degree of isolation if a key or server gets compromised.

It's unclear just how effective it currently is with almost everyone running in test mode, or if some systems are even ignoring that and using DK/DKIM for spam filtering anyway.

A note on chroot in Postfix

Postfix is often run with at least some services being chroot (default in Debian Lenny), but some older installs do not do this. There are security benefits of chroot, though it does make setup a bit more tricky as sockets for the milters have to be placed within the chroot rather than the /var/run as they would normally be.

Typically Postfix will chroot to it's spool directory of /var/spool/postfix.

There are three approaches to dealing with chroot - either create a directory in the chroot area and configure the milters to put their sockets there, create directories and bind mount the relevant directories from /var/run into the chroot, or run the milters networked so that there are no sockets at all.

Personally I favour the first option and create the directory /var/spool/postfix/milter where I configure all the milter sockets to be. This means that in the Postfix config it will see all the sockets under /milter where all the milter configs will have them under /var/spool/postfix/milter

# mkdir /var/spool/postfix/milter
# mkdir /var/spool/postfix/milter/dk-filter
# chown dk-filter.dk-filter /var/spool/postfix/milter/dk-filter
# chmod 2755 /var/spool/postfix/milter/dk-filter
# mkdir /var/spool/postfix/milter/dkim-filter
# chown dkim-filter.dkim-filter /var/spool/postfix/milter/dkim-filter
# chmod 0755 /var/spool/postfix/milter/dkim-filter

If you are concearned about the possible risks of having a world writable directory then you could just make subdirectories with appropriate permissions for each milter.

The advantage of sticking to Unix sockets is that the permissions can be controlled making them more secure.

Keep this in mind with the config that follows as you may need to adapt it to the location and configuration of your Postfix.

DKIM preparation

Start off by installing dkim-filter. This is a milter which can be used in Postfix to do signing and verification.

Next, we need to generate keys. I am going to base this on handling multiple domains (eg. virtual hosted) on the same box so we are going to create a key per domain on each server.

# mkdir -p /etc/mail/dkim/keys/domain1
# cd /etc/mail/dkim/keys/domain1
# dkim-genkey -r -d domain1

At this point we have our key pair for domain1. The selector (identifier that says what key we are using) be the filename that dkim-filter pulls the key from. We can either rename the key, or I prefer to just symlink it. So for example, if we are on a server mail2.domain1, we probably just want to call the selector mail2 to keep things simple:

# ln -s default.private mail2

Likewise, you can do the same for domain2, domain3, and so on for all the domains that your server handles.

Next, we are going to tell dkim-filter what key to use for what mail. Create a file /etc/dkim-keys.conf and put the following in it:

*@domain1:domain1:/etc/mail/dkim/keys/domain1/mail2
*@domain2:domain2:/etc/mail/dkim/keys/domain2/mail2
*@domain3:domain3:/etc/mail/dkim/keys/domain3/mail2

Now, you need to take some time to look at how your network is configured. In many cases machines may be allowed to use the server as an outbound relay. Any machines that do this need to be explicitly defined else dkim-filter will not sign mail from them. If you do need to tell dkim-filter about these then create a file /etc/dkim-internalhosts.conf and put the machines that can use this server as a relay in:

127.0.0.1
::1
localhost
server2.domain1
server1.domain2

All that needs doing is to do the final config in /etc/dkim-filter.conf. From the default, the only things you will probably need to do is uncomment the line:

KeyList        /etc/dkim-keys.conf

And, add the InternalHosts (if needed):

InternalHosts    /etc/dkim-internalhosts.conf

You may want to take more control over how dkim-filter behaves under different circumstances. See the man page and look at On-* options which may also be added to the config telling it how to handle mail.

If you are running Postfix chroot (see above) then add/change the line in /etc/default/dkim-filter to put the socket within the chroot:

SOCKET="local:/var/spool/postfix/milter/dkim-filter/dkim-filter.sock"

Now restart dkim-filter and hopefully everything will work as expected:

# /etc/init.d/dkim-filter restart
Restarting DKIM Filter: dkim-filter.

The only other thing you need to do is add postfix into the dkim-filter group else it will not be able to connect to the socket to talk to dkim-filter.

We will handle the Postfix side of things later once all the parts are in place.

DK preparation

This is the historic method and is much less tidy than dkim-filter. Install dk-filter.

We will be using the same keys, but DK normally puts them in a different location. For convenience I just symlink them:

# mkdir /etc/mail/domainkeys
# cd /etc/mail/domainkeys
# ln -s /etc/mail/dkim/keys

The only thing to watch is permissions as dk-filter tries to read the keys as it's uer rather than root. To solve this I suggest changing permissions on the keys:

# cd keys/domain1
# chgrp dk-filter *
# chmod g+r *

And repeat this for all the other domains.

Next we need to create some lists of domains, keys etc. for dk-filter. These are similar to what we did for dkim-filter, but beware, are not all the same.

Easy one first - internal hosts is the same so I just symlink it:

# cd /etc/mail/domainkeys
# ln -s /etc/dkim-internalhosts.conf internalhosts

We also need a list of domains that we shoud sign. Create /etc/mail/domainkeys/domains containing:

domain1
domain2
domain3

... as needed.

The list of keys to use is also a different format. Create /etc/mail/domainkeys/keylist containing:

*@domain1:/etc/mail/domainkeys/keys/domain1/mail2
*@domain2:/etc/mail/domainkeys/keys/domain2/mail2
*@domain3:/etc/mail/domainkeys/keys/domain3/mail2

... and more as needed.

The config for dk-filter is all done with command line arguments. Typically these would be added in to /etc/default/dk-filter. I have added the following to the bottom of the file:

DAEMON_OPTS="$DAEMON_OPTS -i /etc/mail/domainkeys/internalhosts"
DAEMON_OPTS="$DAEMON_OPTS -d /etc/mail/domainkeys/domains"
DAEMON_OPTS="$DAEMON_OPTS -k -s /etc/mail/domainkeys/keylist"
DAEMON_OPTS="$DAEMON_OPTS -b s"

The last line is important because it causes dk-filter to sign only. If we do verification on DK then we just become part of the problem of legacy systems still running.

If you are running Postfix chroot (see above) then also add/change the line in /etc/default/dk-filter to put the socket within the chroot:

SOCKET="/var/spool/postfix/milter/dk-filter/dk-filter.sock"

Now restart dk-filter and hopefully everything will work as expected:

# /etc/init.d/dk-filter restart
Restarting DomainKeys Filter: dk-filter.

The only other thing you need to do is add postfix into the dk-filter group else it will not be able to connect to the socket to talk to dk-filter.

Now that we have the filters working we can get Postfix hooked up.

DK/DKIM Postfix configuration

Edit your /etc/postfix/main.cf file and add the lines (or add to them if you already have milters:

smtpd_milters =
    unix:/var/run/dkim-filter/dkim-filter.sock
    unix:/var/run/dk-filter/dk-filter.sock
non_smtpd_milters =
    unix:/var/run/dkim-filter/dkim-filter.sock
    unix:/var/run/dk-filter/dk-filter.sock
milter_default_action = accept

Or for a chroot configuration of Postfix (see above):

smtpd_milters =
    unix:/milter/dkim-filter/dkim-filter.sock
    unix:/milter/dk-filter/dk-filter.sock
non_smtpd_milters =
    unix:/milter/dkim-filter/dkim-filter.sock
    unix:/milter/dk-filter/dk-filter.sock
milter_default_action = accept

These tell Postfix where to find the sockets for talking to the filters. Restart Postfix and hopefully now mail will be getting signed:

# /etc/init.d/postfix restart

Now we need to publish the public keys to make sure that people can verify the mail.

DK/DKIM DNS configuration

One catch here is that you will need to be able to add TXT records with underscores in them and some DNS providers have problems with this.

In the directories that you created the keys for each domain there will be a default.txt file which contains the DNS record that has to be added to that domain. For now I also suggest you add a t=y flag to it to indicate that it should be in test mode (don't treat mail any different to unsigned mail even if it fails verification):

default._domainkey IN TXT "v=DKIM1; g=*; k=rsa; t=y; p=MIGf........."

In your DNS record change default to whatever the selector is for this server (ie. mail2 in our example):

mail2._domainkey IN TXT "v=DKIM1; g=*; k=rsa; t=y; p=MIGf........."

This is what goes in your DNS zone. Beware that with many web interfaces you will have to put in mail2._domainkey as the name and the part in quotes (not including the quotes) as the value, ensuring that you are creating a TXT record.

For DK, a default policy for a domain is probably worth setting to discourage mail from being rejected by legacy systems verifying DK:

_domainkey IN TXT "t=y;o=~"

This says that it is in testing mode (ie. treat it the same as unsigned mail), and that not all mail will be signed. This should give DK verifies no reason to reject any mail any more than an unsigned mail.

You can test your DNS config with the policy tester at http://domainkeys.sourceforge.net/policycheck.html and the selector tester at http://domainkeys.sourceforge.net/selectorcheck.html

Testing

It can take some time for DNS to propagate, so ensure that the DNS records you added have become available before trying to test.

It is worth having access to some mail accounts with major mail providers who use DK/DKIM so that you can test. Yahoo! and Google are good places to start, but other providers are also worth testing.

Send mails to your test accounts from all your domains, and from all your test accounts to all your domains, and examine the message headers at the other side.

You should see that signatures are being added:

You should also see a line indicating that verification has succeeded:

If you run into trouble then check that the correct fields are making it into the headers:

Also check the DNS:

$ host -t txt _domainkey.domain1
$ host -t txt mail2._domainkey.domain1

It's worth checking against other DNS servers. Google's public DNS is useful for this:

$ host -t txt _domainkey.domain1 8.8.8.8
$ host -t txt _domainkey.domain1 8.8.8.8
$ host -t txt mail2._domainkey.domain1 8.8.4.4
$ host -t txt mail2._domainkey.domain1 8.8.4.4

There are also test reflectors listed at: http://testing.dkim.org/reflector.html

Going Live

Once you have tested sufficiently and run the system in test mode for long enough to be confident that everything is working then you may like to switch off test mode.

To turn off test mode remove the t=y fields from selector DNS records to indicate that all mail for the domain is signed. At this point other systems should start rejecting mail from your domain that does not verify.

This means that spammers / fraudsters attempting to fake mail from your domain will hit problems and ultimately people using DKIM should have less reason to block mail from your domain due to it being used as a fake source of SPAM / phishing.

Keep in mind however that if something goes wrong (eg. someone mangles the DNS records, messes up the dkim-filter configuration or something) then this could also end up disrupting your mail.

I would reccomend some form of monitoring in place to ensure that everything is working as designed and to be able to detect breakages quickly.

ADSP (was ASP)

DKIM ADSP is at this time not a formal standard, but none the less it takes care of the policy for your domain, and hence you may like to put some thought into using it at this stage. It is once again a TXT DNS record and for now a good place to start is:

_adsp._domainkey IN TXT "dkim=unknown"

This simply states that not all mail from the domain will be signed, hence servers should still accept unsigned mail. This is the recomended state, but if you want to start enforcing it (eg. your domain is being faked by phishers) then you can tell the world that all mail should be signed, and it's worth verifying that all your mail is actually being signed first:

_adsp._domainkey IN TXT "dkim=all"

You can go a step further and explicitly tell the world to discard mail that is unsigned:

_adsp._domainkey IN TXT "dkim=discardable"

There is still much debate about this, and if discarding mail (rather than rejecting it on the edge servers) is actually a good idea at all. My opinion is that so far as possible, mail should never be discarded as if there is a fault upstream the sender doesn't know about it and can't rectify the problem. If mail is rejected then an increase in failures will be noticed on well run systems which are being monitored and the admins can investigate and correct the problems.

The other problem with discarding mail is that it appears to spammers that they are being successful. Really, I would like to demonstrate as clearly as possible to spammers that they are failing and discourage them by rejecting the mail. If it is clearly a waste of time spamming then less people will try it.

There are arguments about the backscatter / blowback problem with rejecting mail, but again, if systems reject mail then it's a problem for those running systems that relay mail and they should harden their systems. If they are creating backscatter then they deserve to have their servers blacklisted.

Adding SPF

You will often see SPF (Sender Policy Framework) related lines in the headers of mail verified with with DKIM, and they work nicely together. SPF is simply a way of publishing policies about what sources of mail should be trusted - kind of like a MX record for sending servers for a domain.

With Postfix I use postfix-policyd-spf-perl to validate SPF. The man page gives you most of what you need to know.

The first thing you need to be aware of is that if you have any secondary servers that forward mail to this one, SPF checks will have to be skipped for them as they will be seen as a source of forged mail. To fix this you will need to edit the actual Perl and add the source addresses of these relays - line 86 in the version in Debian/Lenny:

use constant relay_addresses => map(
    NetAddr::IP->new($_),
    qw( 1.2.3.4  5.6.7.8 )
); # add addresses to qw (  ) above separated by spaces using CIDR notation.

Be aware that if the package is upgraded then these will be overwritten.

Add postfix-policyd-spf-perl to the /etc/postfix/master.cf so that it is started when needed:

spfcheck  unix  -       n       n       -       0       spawn
    user=policyd-spf argv=/usr/sbin/postfix-policyd-spf-perl

Put in your smtpd_recipient_restrictions the policy check:

smtpd_recipient_restrictions =
    reject_invalid_hostname,
    reject_non_fqdn_sender,
    reject_non_fqdn_recipient,
    reject_unknown_sender_domain,
    reject_unknown_recipient_domain,
    permit_mynetworks,
    reject_non_fqdn_hostname,
    permit_sasl_authenticated,
    reject_unauth_destination,
    check_recipient_access pcre:/etc/postfix/toaccess_pcre,
    check_recipient_access hash:/etc/postfix/toaccess,
    check_policy_service unix:private/spfcheck
    check_policy_service inet:127.0.0.1:60000,
    reject_rbl_client bl.spamcop.net,
    reject_rbl_client dnsbl.sorbs.net,
    reject_rbl_client zen.spamhaus.org,
    permit

Make sure that the line is added after reject_unauth_destination else you could end up approving mail to any destination (open relay). At that point you should be ready to go - restart Postfix and see what happens.

All going to plan you should see things like this logged occasionally:

postfix/policy-spf[*****]: : SPF Pass (Mechanism 'ip4:***.***.***.***/**' matched): Envelope-from: someone@somedomain
postfix/policy-spf[*****]: handler sender_policy_framework: is decisive.

Lastly, you need to setup your DNS so that others can verify your mail sources. Although there is a specific SPF record in DNS, for now you will almost certainly have to use a TXT record:

@ IN TXT "v=spf1 mx a:mail.domain include:senders.domain ~all"

@ means default for the domain (ie. when you lookup the base domain), but you can as easily specify the record for subdomains.

v=spf1 identifies it as an SPF record and gives the version.

mx says that mail could come from a machine matching the MX records for your domain. For smaller domains this is often all that is needed.

a specifies an A or AAAA record where mail may come from. This may be an outbound-only mail relay, a security applicance, a webserver that mails customers directly or perhaps a marketing company's systems who sends out mail blasts on your behalf.

include specifies another TXT record to include which is useful if you run a large outfit need to break up your records into managable chunks.

There are various other mechanisms (eg. ipv4, ipv6 which specify address ranges) that can be added but most will probably only be use to people with large amounts of mail infrastructure to worry about and they can be easily looked up.

Lastly, ~all says that all other sources should soft fail (retryable failure, useful for testing). This can also be -all meaning to fail (reject/bounce) other sources, ?all meaning to ignore the policy (again usful for testing), and +all meaning to accept all others which is probably not a good idea. With the a, mx, etc. the + is implied - ie. saying mx really means +mx

You can find much more on this syntax at: http://www.openspf.org/SPF_Record_Syntax

Like with DKIM, this needs testing and accounts at major web mail providers will often have a verification header in them. Test throughly before setting to -all where other mail sources will not be able to send mail as your domain. If you have forgotten to include one of your legitimate outbound mail sources then this too will be blocked from sending mail.

Record keeping

When deploying technologies like this it is very easy to loose track of all the places where configuration is hiding that need to be changed if for example you add another server or just change the address of an existing one.

With small setups it's generally all left in the head of whoever set it up. As they are not administrating it on a continous basis, they often forget and then mistakes happen. Likewise if they leave, their replacement will have no familiarity with what configuration is where.

In larger organisations there is far more infrastructure and it can be hard work keeping track of it all. Administration is done by many people and unless they communicate effectively it is a recipe for disaster.

In any size organsiation, keeping good records of your configuration, work notes of who did what configuration and checklists/work instructions (eg. for deploying new servers) are vital to ensuring that everything remains under control.

Cacti

I am updating my Postfix templates for Cacti for moitoring DKIM and SPF and these will be available shortly.

Monitoring is vital to smooth running of mail as well as long term planning so get yours configured.

Unix Made Easy: HACMP

Unix Made Easy: HACMP: "Cluster planning: The area of cluster planning is a large one. Not only does it include planning for the types of hardware (CPUs, networks..."

HACMP

Cluster planning:

The area of cluster planning is a large one. Not only does it include planning for the types of hardware (CPUs, networks, and disks) to be used in the cluster, but it also includes other aspects. These include resource planning, that is, planning the desired behavior of the cluster in failure situations. Resource planning must take application loads and characteristics into account, as well as priorities.

High availability:

A high availability solution will ensure that any failure of any component of the solution, be it hardware, software, or system management, will not cause the application and its data to be inaccessible to the user community.

High availability is:

The masking or elimination of both planned and unplanned downtime.
The elimination of single points of failure (SPOFs)
Fault resilience, but not fault tolerance

High availability systems are an excellent solution for applications that can withstand a short interruption should a failure occur, but which must be restored quickly.

The difference between fault tolerance and high availability is:
A fault tolerant environment has no service interruption, while a highly available environment has a minimal service interruption.

System downtime is either planned or unplanned, with planned downtime accounting for the vast majority of the total.

HACMP allows you to minimize or eliminate this planned downtime from your operation, by allowing you to maintain a service to your customers while performing hardware upgrades, software upgrades, or other maintenance activity at the same time.

Services may be moved from one cluster node to another at will, allowing the original node to undergo maintenance without affecting the availability of the service. When the maintenance activity is completed, the service may be moved back to the node which was originally running it.

Unplanned downtime has one of two causes: periods caused by hardware failures, and those periods caused by software failures.

Hardware has been getting more and more reliable over time and will continue to do so, but hardware remains a cause of failures. Together with facilities provided by AIX, HACMP can protect your operation from a hardware failure, by automatically moving the services provided by the failing node to other nodes within the cluster.

Cluster nodes

One of HACMP's key design strengths is its ability to provide support across the entire range of RISC System/6000 products. Because of this built-in flexibility and the facility to mix and match RISC System/6000 products, the effort required to design a highly available cluster is significantly reduced.

The designated hardware should only be used on an appropriate IBM ^ pSeries, RS/6000 Platform, or 9076 Scalable POWERParallel Platform (SP).

The following sections will deal with the various:

Operating system levels

CPU options
Disk storages for CRM
Cluster node considerations available to you when you are planning your HACMP cluster.


Operating system level

Before installation of HACMP, make sure to have the proper version of the

operating system. Here is a list of required operating system levels for HACMP

versions (see Table2-1) and Parallel System Support Program versions (see

Table2-2).

Table 2-1 Required OS level for HACMP

Table 2-2 PSSP versions for SP installation

AIX OS levelHACMP Version

4.3.1

HACMP Version

4.4.0

HACMP Version

4.4.1

4.3.2 yesnono

4.3.3 yesyesyes

5.1 noyes*yes

* Note:

The following restrictions apply to HACMP for AIX Version 4.4.0 support of IBM AIX 5L for Power Version 5.1. IBM intends to remove these restrictions through further APARs on HACMP.


Enhanced Journaled File Systems are not supported on shared volume groups.


Fencing is not supported for concurrent mode volume groups created on 9333 disks.


HACMP can only run on 32-bit AIX kernels. Even if the hardware is capable of supporting 64-bit kernels, it must be booted using the the bosboot command with a 32-bit kernel.


The VSM-based xhacmpm configuration utility is not supported.

HACMP versionPrerequisite PSSP version

HACMP Version 4.3.1 for AIXPSSP Version 3.1

HACMP Version 4.4.0 for AIXPSSP Version 3.2

HACMP Version 4.4.1 for AIXPSSP Version 3.2




CPU options

HACMP is designed to execute with RS/6000 uniprocessors, SMP servers, and SP systems in a no single point of failure server configuration. HACMP supports the IBM ^ pSeries and the RS/6000 models that are designed for server application and meet the minimum requirements for internal memory, internal disk, and I/O slots.










Cluster node considerations:

Your major goal throughout the planning process is to eliminate single points of failure. A single point of failure exists when a critical cluster function is provided by a single component. If that component fails, the cluster has no other way of providing that function, and the service depending on that component becomes unavailable.

HACMP for AIX is designed to recover from a single hardware or software failure. It may not be able to handle multiple failures, depending on the sequence of failures. For example, the default event scripts cannot do an adapter swap after an IP address takeover (IPAT) has occurred if only one standby adapter exists for that network.

How to eliminate the single point of failure

Table2-6 summarizes potential single points of failure within an HACMP cluster and describes how to eliminate them.



Cluster networks

HACMP differentiates between two major types of networks: TCP/IP networks and non-TCP/IP networks. HACMP utilizes both of them for exchanging heartbeats. HACMP uses these heartbeats to diagnose failures in the cluster. Non-TCP/IP networks are used to distinguish an actual hardware failure from the failure of the TCP/IP software. If there were only TCP/IP networks being used, and the TCP/IP software failed, causing heartbeats to stop, HACMP could falsely diagnose a node failure when the node was really still functioning. Since a non-TCP/IP network would continue working in this event, the correct diagnosis could be made by HACMP. In general, all networks are also used for verification, synchronization, communication, and triggering events between nodes. Of course, TCP/IP networks are used for communication with client machines as well.

TCP/IP networks:

The following sections describe supported TCP/IP network types and network considerations.

Supported TCP/IP network types

Basically every adapter that is capable of running the TCP/IP protocol is a supported HACMP network type. There are some special considerations for certain types of adapters, however. The following gives a brief overview on the supported adapters and their special considerations.

Below is a list of TCP/IP network types as you will find them at the configuration time of an adapter for HACMP.


Generic IP


ATM


Ethernet


FCS


FDDI


SP Switch


SLIP


SOCC


Token-Ring




Heartbeating in HACMP

The primary task of HACMP™ is to recognize and respond to failures. HACMP uses heartbeating to monitor the activity of its network interfaces, devices and IP labels.

Heartbeating connections between cluster nodes are necessary because they enable HACMP to recognize the difference between a network failure and a node failure. For instance, if connectivity on the HACMP network (this network's IP labels are used in a resource group) is lost, and you have another TCP/IP based network and a non-IP network configured between the nodes, HACMP recognizes the failure of its cluster network and takes recovery actions that prevent the cluster from becoming partitioned.

To avoid cluster partitioning, we highly recommend configuring redundant networks in the HACMP cluster and using both IP and non-IP networks. Out of these networks, some networks will be used for heartbeating purposes.

In general, heartbeats in HACMP can be sent over:


TCP/IP networks.


Serial (non-IP) networks (RS232, TMSCSI, TMSSA and disk heartbeating).

The Topology Services component of RSCT carries out the heartbeating function in HACMP.

Topology services and heartbeat rings

HACMP uses the Topology Services component of RSCT for monitoring networks and network interfaces. Topology Services organizes all the interfaces in the topology into different heartbeat rings. The current version of RSCT Topology services has a limit of 48 heartbeat rings, which is usually sufficient to monitor networks and network interfaces.

Heartbeat rings are dynamically created and used internally by RSCT. They do not have a direct, one-to-one correlation to HACMP networks or number of network interfaces. The algorithm for allocating interfaces and networks to heartbeat rings is complex, but generally follows these rules:


In an HACMP network, there is one heartbeat ring to monitor the service interfaces, and one for each set of non-service interfaces that are on the same subnet. The number of non-service heartbeat rings is determined by the number of non-service interfaces in the node with the largest number of interfaces.


The number of heartbeat rings is approximately equal to the largest number of interfaces found on any one node in the cluster.

Note that during cluster verification, HACMP calls the RSCT verification API. This API performs a series of verifications, including a check for the heartbeat ring calculations, and issues an error if the limit is exceeded.

Heartbeating over IP aliases

This section contains information about heartbeating over IP aliases.


Overview
In general, HACMP subnetting requirements can be complicated to understand and may require that you reconfigure networks in AIX® to avoid features such as multiple subnet routes, which can lead to a single point of failure for network traffic









Overview

In general, HACMP™ subnetting requirements can be complicated to understand and may require that you reconfigure networks in AIX® to avoid features such as multiple subnet routes, which can lead to a single point of failure for network traffic.

When planning your cluster networks, you may need to:


Reconfigure IP addresses of HACMP interfaces that will be used at boot time or


Update/etc/hosts with the new boot time IP addresses.

Heartbeating over IP Aliases is useful because it:


Uses automatically generated IP aliases for heartbeating


Heartbeating over IP Aliasing provides an option where the addresses used for heartbeating can be automatically configured by HACMP in a subnet range that is outside of the range used for the base NIC or any service addresses.

Although Heartbeating over IP Aliasing automatically configures proper aliases for heartbeating, you must still be aware of the implications of subnet routing for all boot and service IP addresses. That is, failure to plan subnets properly can lead to application failures that are not detectable by HACMP. Reliable HACMP cluster communication still requires that the interfaces on a single network can communicate with the other nodes on that network.


Enables you to avoid reconfiguration of boot time addresses and /etc/hosts.

RSCT sets up the heartbeat rings to go over a separate range of IP aliases. This lets you use a specified subnet in a non-routable range for a heartbeat ring, preserving your other subnets for routable traffic. This also allows you to avoid reconfiguring boot time addresses and entries in /etc/hosts.


Makes HACMP topology configuration easier to understand.


Does not require that you obtain additional routable subnets from the network administrator.

For instance, you can use heartbeating over aliases in HACMP, if due to the network system administration restrictions, the IP addresses that your system can use at boot time must reside on the same subnet. (In general, if there are no system administration restrictions, the IP addresses that your system can use at boot time can reside on either the same or different subnets).





Heartbeating over disk

You can configure a non-IP point-to-point heartbeating network, called a disk heartbeating network, over any shared disk in an enhanced concurrent mode volume group. Heartbeating over disk provides another type of non-IP point-to-point network for failure detection.

Disk heartbeating networks provide an alternative to other point-to-point networks such as RS232 that have cable length restrictions, or TMSSA which require special disk adapter hardware and cabling. Heartbeating over disk does not require additional or specialized hardware, cabling or microcode; it can use any disk that is also used for data and for which volume groups and file systems are included in an HACMP™ resource group.

In a disk heartbeating network, two nodes connected to the disk periodically write heartbeat messages and read heartbeat messages (written by the other node) on a small, non-data portion of the disk. A disk heartbeating network, like the other non-IP heartbeating networks, connects only two nodes. In clusters with more than two nodes, use multiple disks for heartbeating. Each node should have a non-IP heartbeat path to at least one other node. If the disk heartbeating path is severed, at least one node cannot access the shared disk.

You have two different ways for configuring a disk heartbeating network in a cluster:


You can create an enhanced concurrent volume group shared by multiple nodes in your cluster. Then you use the HACMP Extended Configuration SMIT path to configure a point-to-point pair of discovered communication devices.

or


You can start by creating a cluster disk heartbeating network, and then add devices to it using the Add Pre-Defined Communication Interfaces and Devices panel in SMIT.

The HACMP cluster verification utility verifies that the disk heartbeating networks are properly configured.

Heartbeating over disk and fast method for node failure detection

With HACMP, you can reduce the time it takes for node failure to be realized throughout the cluster. If you have a disk heartbeating network configured, and specify a parameter for a disk heartbeating NIM, then when a node fails, HACMP uses a disk heartbeating network to place a departing message on the shared disk so neighboring nodes are aware of the node failure within one heartbeat period.

Common Linux log files name and usage

Common Linux log files name and usage

• /var/log/message: General message and system related stuff
• /var/log/auth.log: Authenication logs
• /var/log/kern.log: Kernel logs
• /var/log/cron.log: Crond logs (cron job)
• /var/log/maillog: Mail server logs
• /var/log/qmail/ : Qmail log directory (more files inside this directory)
• /var/log/httpd/: Apache access and error logs directory
• /var/log/lighttpd: Lighttpd access and error logs directory
• /var/log/boot.log : System boot log
• /var/log/mysqld.log: MySQL database server log file
• /var/log/secure: Authentication log
• /var/log/utmp or /var/log/wtmp : Login records file
• /var/log/yum.log: Yum log files

In short /var/log is the location where you should find all Linux logs file. However some applications such as httpd have a directory within /var/log/ for their own log files. You can rotate log file using logrotate software and monitor logs files using logwatch software

Red Hat Top Tools (priority order?)

CPU Tools	Memory Tools	Process Tools	Disk Tools
1 . top	1 . top	1 . top	1 . iostat -x
2 . vmstat	2 . vmstat s	2 . ps -o pmem	2 . vmstat -D
3 . mpstat P	3 . ipcs		3 . sar -DEV #
all	4 . ps -o vss,rss	3 . gprof
4 . ps ef		4 . strace,ltrace	4 . nsfstat
5 . sar u	5 . sar -r -B -W	5 . sar	5 . NEED MORE!
6 . procinfo
7 . iostat
8 . gnomesystemmonitor	6 . meminfo
9 . KDEmonitor	7 . free
10 . oprofile	8 . gnome-system-monitor
	9 . KDE-monitor
	10 . oprofile

Linux Server Hardening

chmod 711 /
chmod 711 /home
chmod 711 /etc
chmod 711 /var
chmod 711 /usr/etc
chmod 711 /usr/local/etc
chmod 711 /var/log
chmod 711 /sbin
chmod 711 /usr/sbin
chmod 711 /usr/local/sbin

chmod 644 /etc/motd

groupadd deva
chmod 750 /usr/bin/wget
chown root:deva /usr/bin/wget
chmod 750 /usr/bin/perlcc
chown root:deva /usr/bin/perlcc
chmod 750 /usr/bin/byacc
chown root:deva /usr/bin/byacc
chmod 750 /usr/bin/yacc
chown root:deva /usr/bin/yacc
chmod 750 /usr/bin/cc
chown root:deva /usr/bin/cc
chmod 750 /usr/bin/gcc
chown root:deva /usr/bin/gcc

chmod 700 /bin/dmesg
chmod 700 /bin/mount
chmod 700 /bin/rpm
chmod 700 /usr/bin/write
chmod 700 /usr/bin/talk
chmod 700 /usr/bin/ipcrm
chmod 700 /usr/bin/ipcs
chmod 700 /usr/bin/free
chmod 700 /usr/bin/locate
chmod 700 /usr/bin/wall
chmod 700 /usr/bin/finger
chmod 700 /sbin/arp
chmod 700 /sbin/ifconfig
chmod 700 /usr/sbin/repquota
chmod 700 /usr/sbin/tcpdump
chmod 700 /usr/bin/nmap
chmod 700 /usr/bin/wget
chmod 700 /usr/bin/perlcc
chmod 700 /usr/bin/byacc
chmod 700 /usr/bin/yacc
chmod 700 /usr/bin/cc
chmod 700 /usr/bin/gcc
chmod 700 /usr/bin/who
chmod 700 /usr/bin/w
chmod 700 /usr/bin/nc

chmod 1733 /tmp/.ICE-unix
chmod 1733 /tmp/.X11-unix
chmod 660 /var/run/utmp

chmod 000 /usr/bin/rcp
chmod 000 /usr/bin/links
chmod 000 /usr/bin/scp
chmod 000 /usr/bin/elinks
chmod 700 /usr/bin/lwp-*
chmod 000 /usr/bin/GET
chmod 700 /usr/bin/curl
chmod 700 /usr/bin/*++*
chmod 700 /usr/bin/*cc*
chmod 700 /usr/bin/yum
chmod 700 /usr/bin/up2date
chmod 700 /usr/sbin/up2date


chmod u-s /usr/bin/at
chmod u-s /usr/bin/chage
chmod u-s /usr/bin/chfn
chmod u-s /usr/bin/chsh
chmod u-s /usr/bin/crontab
chmod u-s /usr/bin/expiry
chmod u-s /usr/bin/gpasswd
chmod u-s /usr/bin/lppasswd
chmod u-s /usr/bin/newgrp
chmod u-s /usr/bin/rcp
chmod u-s /usr/bin/rlogin
chmod u-s /usr/bin/rsh
chmod u-s /usr/libexec/ssh-keysign