Postfix architecture

Postfix architecture

  • Description

  • Postfix architecture overview

  • Presentation of other configuration

    • Setting up postfix as relay of email to the internal.

      • Definition of a user list

      • Internal email redirection

      • A flat file or an interrogation of the Ldap

Description

Now that we’ve seen the basics of using postfix with a simple relay server and local delivery, let’s take a look at the architecture and some more “special” configurations. For information when I gave postfix training for companies the training was 4 days complete. You will understand that in 7 hours x 4 (days) we can cover a lot more material. In other words if you are looking to do something that was not covered in the training must googler it may exist: P.

Postfix architecture overview

During the first session I mentioned that postfix consists of several applications that are defined in the file /etc/postfix/master.cf, this file contains the list as well as parameters available specifically for the process.

Consult the file to give us a glimpse:

 

cat /etc/postfix/master.cf | grep -v "^#" # ========================================================================== # service type private unpriv chroot wakeup maxproc command + args # (yes) (yes) (no) (never) (100) # ========================================================================== smtp inet n - y - - smtpd pickup unix n - y 60 1 pickup cleanup unix n - y - 0 cleanup qmgr unix n - n 300 1 qmgr tlsmgr unix - - y 1000? 1 tlsmgr rewrite unix - - y - - trivial-rewrite bounce unix - - y - 0 bounce defer unix - - y - 0 bounce trace unix - - y - 0 bounce verify unix - - y - 1 verify flush unix n - y 1000? 0 flush proxymap unix - - n - - proxymap proxywrite unix - - n - 1 proxymap smtp unix - - y - - smtp relay unix - - y - - smtp showq unix n - y - - showq error unix - - y - - error retry unix - - y - - error discard unix - - y - - discard local unix - n n - - local virtual unix - n n - - virtual lmtp unix - - y - - lmtp anvil unix - - y - 1 anvil scache unix - - y - 1 scache maildrop unix - n n - - pipe flags=DRhu user=vmail argv=/usr/bin/maildrop -d ${recipient} uucp unix - n n - - pipe flags=Fqhu user=uucp argv=uux -r -n -z -a$sender - $nexthop!rmail ($recipient) ifmail unix - n n - - pipe flags=F user=ftn argv=/usr/lib/ifmail/ifmail -r $nexthop ($recipient) bsmtp unix - n n - - pipe flags=Fq. user=bsmtp argv=/usr/lib/bsmtp/bsmtp -t$nexthop -f$sender $recipient scalemail-backend unix - n n - 2 pipe flags=R user=scalemail argv=/usr/lib/scalemail/bin/scalemail-store ${nexthop} ${user} ${extension} mailman unix - n n - - pipe flags=FR user=list argv=/usr/lib/mailman/bin/postfix-to-mailman.py ${nexthop} ${user}

It’s a big process that postfix can use, every process of course a utility (GNU / Linux is bloated but not so much: P)

An image often facilitates the understanding of the interaction, so we will start with a “simple” version:

Using processes when the postfix server is used as a relay server:

 

p>Description of processes:

 

  • smtpd: The smtpd service is the service that supports the SMTP protocol that receives the connections on port 25 and transmits them to the other process. It is also he who will send the communication to the other SMTP server.

 

  • cleanup: The cleanup service will perform a validation of the email (header and content), if it detects a problem it will inform the customer, moreover the service will carry out operations on the emails:

    • If there are missing headers (From :, To :, Message-Id :, and Date 🙂

    • Deleting Duplicate Destination Addresses

    • Deleting the bcc header (to create a second email)

    • In addition it is possible to have rewrite address, adding domain name, … (see the manual)

  • incoming: incoming is not a process but a queue, all newly received email is placed in this queue, in the next diagram processes and queues are better represented. Postfix uses several queues, when you have a postfix performance issue there is a command to see which queues the emails are in, this allows you to better identify the source of the problem.

  • active: active is also a queue, the latter is the queue currently being processed, a limited number of emails are allowed in this queue.

  • qmgr: The qmgr service is the system that manages the queues the latter will take the emails in the active queue and do the processing if the email can not be delivered it will return it in the queue deferred as we had seen during the error of the relay server. This process will also remove double-bounce emails in other words if an email is rejected and the rejection message is also forwarded to an invalid address. This process can send the email to another TRIVIAL-REWRITE process to rewrite the content.

    • qmgr will use the correct method for delivery, smtpd, local, depending on the situation.

Here is the process when receiving an email.

 

Take note that I have not taken a course of computer graphics but the 2 images above are taken on the site http://wiki.ncad.fr/index.php?title=Postfix. Under the same license as this formation: D. (CC by nc sa)

There is another representation of the more detailed processing process like this one found on wikipedia:

 
p>In addition to this configuration in place it is possible to define other for example when setting up an anti-spam system add an email processing service before delivery.

Let’s take the master.cf file:

 

 

# ========================================================================== # service type private unpriv chroot wakeup maxproc command + args # (yes) (yes) (no) (never) (100) # ========================================================================== smtp inet n - y - - smtpd pickup unix n - y 60 1 pickup cleanup unix n - y - 0 cleanup qmgr unix n - n 300 1 qmgr [... OUPUT COUPÉ ...]

Of course there is the no of the service, its type, I would like to pay attention to the chroot column, as you can see some process sound chroot in other words they are limited in a directory for processing. This is mainly for security reasons, you will find that the majority of services that are chroot are externally exposed services or processing data that has not yet been validated. This allows that if there is for example a buffer overflow in the smtpd service the attacker will only have access to the directory where the service evolves.

I’ll let you look with pleasure: P, canonical options, trivial-rewrite, …

Presentation of other configuration

Without going into a big detail we will see other type of postfix configuration, as said I unfortunately can not take as much time as I would like but hey … It’s for you motivate to look more: D.

Setting up postfix as relay of email to the internal.

We saw the configuration for receiving email with local delivery as well as configuring a postfix as a relay server for sending email to the external. Make a mix now, here is a diagram of a possible configuration:

 

A little text, so we have:

  • The coco.com domain with DNS configuration (mx coco.com = 66.32.12.33)

  • By choice the exchange service is NOT exposed on the Internet.

  • Postfix therefore receives emails, ideally performs processing on it (anti-spam, address rewrite if required …) thereafter it sends the email to exchange.

We have seen how to accept emails, and send it back … But because there is one but otherwise I will not write this section: P. We will have to address some point:

  1. As the users are not locally defined how will we achieve the validation of addresses and aliases?

  2. The postfix service must relay emails regardless of the source on the internet, so no limitation on the IP source as we did previously. We must now limit the relay not on the IP but on the destination domain name. We do not want to open a public smtp service on the internet that allows sending email for any domain.

  3. When passing the email to the exchange.com server, unfortunately the DNS resolution may provide the MX information as the postfix server that already processes the email: – /. It will be necessary to see how the redirection of the email can be realized so that the IP is that of the internal exchange. Good when it is our domain we can do a pass with the DNS but we will see a purely postfix method, because unfortunately we do not always control the DNS.

Here are the 3 points to address to be able to do it.

Definition of a user list

As a reminder, we saw that the limitation of users is defined by the instruction:

$ sudo postconf | grep local_recipient_maps local_recipient_maps = proxy:unix:passwd.byname $alias_maps

 

Let’s look at the quick documentation: local_recipient_maps and LOCAL_RECIPIENT_README, well this is for you if you want to go further or validate that I have not said a big lie further ;-). It is possible that sometimes I also misunderstood: D, if it is the case a small email that will make me pleasure: D.

So if we look at the system currently uses the file / etc / passwd and the value of $ alias_maps. So the proxy statement and the result of alias_map is what is already the value for the aliases

So if we look at the system currently uses the file / etc / passwd and the value of $ alias_maps. So the proxy statement and the result of alias_map is what is already the value for the aliases

$ postconf | grep ^alias_maps alias_maps = hash:/etc/aliases

Woww it looks like a system of lookup table types, if you click you will see other type available !!

As you can see it is possible to use several type, for the moment we will set up a flat user definition file, but technically it is also possible to define an LDAP connection to have the list of users directly from AD . I will come back I have a warning to communicate to you before you choose this solution: D. (A bit like a radio show must I keep my audience: P)

Let’s start by well demonstrated limitation, I try to send an email to me@mailtraining.shibarecords.com

$ telnet 172.17.0.2 25 Trying 172.17.0.2... Connected to 172.17.0.2. Escape character is '^]'. ehtlo 220 mail01.mood.shibarecords.com ESMTP Postfix 502 5.5.2 Error: command not recognized ehlo toto 250-mail01.mood.shibarecords.com 250-PIPELINING 250-SIZE 10240000 250-VRFY 250-ETRN 250-ENHANCEDSTATUSCODES 250-8BITMIME 250 DSN mail from: toto@shibarecords.com 250 2.1.0 Ok rcpt to:moi@mailtraining.shibarecords.com 550 5.1.1 <moi@mailtraining.shibarecords.com>: Recipient address rejected: User unknown in local recipient table quit 221 2.0.0 Bye Connection closed by foreign host.

The email is rejected because not in the list, COOL it works NOT as expected, now add the list of authorized users via a text file. We will take a table lookup format say dbm it’s part !!

Editing /etc/postfix/main.cf:

local_recipient_maps = proxy:unix:passwd.byname $alias_maps dbm:/etc/postfix/lst-user

Creating the file, I will add only the user me for the domain:

moi@mailtraining.shibarecords.com OK

Generation of the binary file:

$ sudo postmap /etc/postfix/lst-user

We restart postfix and we validate:

$ telnet 172.17.0.2 25 Trying 172.17.0.2... Connected to 172.17.0.2. Escape character is '^]'. 220 mail01.mood.shibarecords.com ESMTP Postfix ehlo toto 250-mail01.mood.shibarecords.com 250-PIPELINING 250-SIZE 10240000 250-VRFY 250-ETRN 250-ENHANCEDSTATUSCODES 250-8BITMIME 250 DSN mail from:toto@shibarecords.com 250 2.1.0 Ok rcpt to:moi@mailtraining.shibarecords.com 451 4.3.0 <moi@mailtraining.shibarecords.com>: Temporary lookup failure ^] telnet> q Connection closed.

oops, the logs say what? /var/log/mail.log

Feb 2 13:47:12 mail01 postfix/master[307]: daemon started -- version 3.1.0, configuration /etc/postfix Feb 2 13:47:15 mail01 postfix/smtpd[312]: error: unsupported dictionary type: dbm Feb 2 13:47:15 mail01 postfix/smtpd[312]: connect from unknown[172.17.0.1] Feb 2 13:47:27 mail01 postfix/smtpd[312]: warning: dbm:/etc/postfix/lst-user is unavailable. unsupported dictionary type: dbm Feb 2 13:47:27 mail01 postfix/smtpd[312]: warning: dbm:/etc/postfix/lst-user lookup error for "moi@mailtraining.shibarecords.com" Feb 2 13:47:27 mail01 postfix/smtpd[312]: NOQUEUE: reject: RCPT from unknown[172.17.0.1]: 451 4.3.0 <moi@mailtraining.shibarecords.com>: Temporary lookup failure; from=<toto@shibarecords.com> to=<moi@mailtraining.shibarecords.com> proto=ESMTP helo=

Good if you think I did it on purpose not even: P, I just did not know what type to use I chose any: P, here’s the result: P. So we will resume the configuration and see the modules postfix currently available we will not make mistakes try to find a supported! We can but it’s less beautiful: P.

$ postconf -m btree cidr environ fail hash inline internal memcache nis pipemap proxy randmap regexp socketmap sqlite static tcp texthash unionmap unix

Now we can see all the type to support. I will choose hash

local_recipient_maps = proxy:unix:passwd.byname $alias_maps hash:/etc/postfix/lst-user

I regenerate the file with portmap so that the format is the right one: D. We restart postfix and test!

$ telnet 172.17.0.2 25 Trying 172.17.0.2... Connected to 172.17.0.2. Escape character is '^]'. 220 mail01.mood.shibarecords.com ESMTP Postfix ehlo toto 250-mail01.mood.shibarecords.com 250-PIPELINING 250-SIZE 10240000 250-VRFY 250-ETRN 250-ENHANCEDSTATUSCODES 250-8BITMIME 250 DSN mail from:toto@xiuehfeu.com 250 2.1.0 Ok rcpt to: moi@mailtraining.shibarecords.com 250 2.1.5 Ok data 354 End data with . un courriel Woww . 250 2.0.0 Ok: queued as 15190A04008 quit 221 2.0.0 Bye Connection closed by foreign host.

Youppii it works, the first time ;-). Consult the logs to learn more.

Feb 3 13:16:56 mail01 postfix/smtpd[189]: 15190A04008: client=unknown[172.17.0.1] Feb 3 13:17:01 mail01 postfix/cleanup[192]: 15190A04008: message-id=<> Feb 3 13:17:01 mail01 postfix/qmgr[188]: 15190A04008: from=<toto@xiuehfeu.com>, size=201, nrcpt=1 (queue active) Feb 3 13:17:01 mail01 postfix/local[193]: 15190A04008: to=<moi@mailtraining.shibarecords.com>, relay=local, delay=18, delays=17/0.01/0/0.02, dsn=5.1.1, status=bounced (unknown user: "moi") Feb 3 13:17:01 mail01 postfix/cleanup[192]: 6E0BBA0400D: message-id=<20170203131701.6E0BBA0400D@mail01.mood.shibarecords.com> Feb 3 13:17:01 mail01 postfix/bounce[194]: 15190A04008: sender non-delivery notification: 6E0BBA0400D Feb 3 13:17:01 mail01 postfix/qmgr[188]: 6E0BBA0400D: from=<>, size=2040, nrcpt=1 (queue active) Feb 3 13:17:01 mail01 postfix/qmgr[188]: 15190A04008: removed Feb 3 13:17:01 mail01 postfix/smtp[195]: warning: relayhost configuration problem Feb 3 13:17:01 mail01 postfix/smtp[195]: 6E0BBA0400D: to=<toto@xiuehfeu.com>, relay=none, delay=0.35, delays=0.01/0.01/0.33/0, dsn=4.3.5, status=deferred (Host or domain name not found. Name service error for name=mail_relay.fai.com type=A: Host not found) Feb 3 13:17:03 mail01 postfix/smtpd[189]: disconnect from unknown[172.17.0.1] ehlo=1 mail=1 rcpt=1 data=1 quit=1 commands=5

So the email is accepted by the postfix system of course during the delivery it is not able to perform the operation because there is no associated user. The system not being able to deliver the email tries to inform the sender to indicate that there was a problem during the delivery.

Before moving to the internal email redirection stage, I would like to share an experience.

Internal email redirection

Now that we have seen how to set up an email redirection to an internal server!

We will allow the redirection of e-mails for the domain coco.com, in order to identify it as domain permit we will update the relay_domains entry in the file /etc/postfix/main.cf:

$ postconf | grep "relay_domains =" relay_domains = ${{$compatibility_level} < {2} ? {$mydestination} : {}}

Here we go :

mydestination = mailtraining.shibarecords.com mail01.mood.shibarecords.com relay_domains = coco.com mynetworks = 127.0.0.0/8 [::ffff:127.0.0.0]/104 [::1]/128 172.17.0.0/16

Good if we had already covered the formation of the creation of a container for the reception of email, I could have just put the order for the start … In fact I already have it, I have in advance on the training we will see him at the next class: P.

$ docker run --hostname mail.coco.com -e SMTP_HOSTNAME=mail.coco.com -e ACCEPT_DOMAIN=coco.com srvsmtp $ docker ps | grep smt 1f1bda628647 srvsmtp "/root/run.sh" About a minute ago Up About a minute 25/tcp mad_jepsen $ docker inspect mad_jepsen | grep IPA "SecondaryIPAddresses": null, "IPAddress": "172.17.0.3", "IPAMConfig": null, "IPAddress": "172.17.0.3",

Well ideally we would do a validation, but I’m super confident of my container: P, but there is no user other than root so root@coco.com.

We will add this email address in the list of accepted addresses. So in the file previously create.

$ echo ""root@coco.com OK" >> lst-user $ cat lst-user moi@mailtraining.shibarecords.com OK root@coco.com OK

We regenerate the file (DO NOT FORGET IT):

$ postmap /etc/postfix/lst-user

We restart postfix or reload because of the change to the relay_domains parameter. It’s part of a test that will not work: P

$ telnet 172.17.0.2 25 Trying 172.17.0.2... Connected to 172.17.0.2. Escape character is '^]'. 220 mail01.mood.shibarecords.com ESMTP Postfix ehlo toto 250-mail01.mood.shibarecords.com 250-PIPELINING 250-SIZE 10240000 250-VRFY 250-ETRN 250-ENHANCEDSTATUSCODES 250-8BITMIME 250 DSN mail from: toto@ofjifj.com 250 2.1.0 Ok rcpt to:root@coco.com 250 2.1.5 Ok data 354 End data with . un courriel . 250 2.0.0 Ok: queued as 6835AA04008 quit 221 2.0.0 Bye Connection closed by foreign host.

Consult the logs:

Feb 3 13:42:26 mail01 postfix/smtpd[237]: connect from unknown[172.17.0.1] Feb 3 13:42:40 mail01 postfix/smtpd[237]: 6835AA04008: client=unknown[172.17.0.1] Feb 3 13:42:45 mail01 postfix/cleanup[240]: 6835AA04008: message-id=<> Feb 3 13:42:45 mail01 postfix/qmgr[234]: 6835AA04008: from=<toto@ofjifj.com>, size=183, nrcpt=1 (queue active) Feb 3 13:42:46 mail01 postfix/smtpd[237]: disconnect from unknown[172.17.0.1] ehlo=1 mail=1 rcpt=1 data=1 quit=1 commands=5 Feb 3 13:43:05 mail01 postfix/smtp[241]: 6835AA04008: to=<root@coco.com>, relay=none, delay=31, delays=11/0.01/20/0, dsn=4.4.3, status=deferred (Host or domain name not found. Name service error for name=mail_relay.fai.com type=MX: Host not found, try again)

The system tries to send the emails externally through the relay server configuration, but we want it to be the machine internally (aka our container next door).

How to do ?? We will take advantage of having covered the architecture to have a look at the process of treatment, a small zoom on the section that I want to highlight:

 

As you can see it is possible when managing the queue to perform a rewrite with as reference the transport system. Let’s try it !! We will define the transport_map entry in the /etc/postfix/main.cf file

transport_maps = hash:/etc/postfix/transport

File creation and generation of the binary format:

$ cat /etc/postfix/transport coco.com smtp:172.17.0.3:25 .coco.com smtp:172.17.0.3:25 $ sudo postmap /etc/postfix/transport

We reload the configuration and validate the configuration: D and BOOM

Feb 3 22:24:37 mail01 postfix/qmgr[232]: 2F0AAA04008: from=<toot@cejf.com>, size=183, nrcpt=1 (queue active) Feb 3 22:24:37 mail01 postfix/smtp[243]: 2F0AAA04008: to=<root@coco.com>, relay=172.17.0.3[172.17.0.3]:25, delay=11, delays=11/0.01/0.16/0.03, dsn=2.0.0, status=sent (250 2.0.0 Ok: queued as 76AE960758E) Feb 3 22:24:37 mail01 postfix/qmgr[232]: 2F0AAA04008: removed Feb 3 22:24:38 mail01 postfix/smtpd[239]: disconnect from unknown[172.17.0.1] ehlo=1 mail=1 rcpt=1 data=1 quit=1 commands=5

As we are never too sure let’s validate that the email was REALLY received by the server:

$ docker exec mad_jepsen ls -R /root/Maildir /root/Maildir: cur new tmp /root/Maildir/cur: /root/Maildir/new: 1486160677.Vfe02Ie0699bM508368.mail.coco.com /root/Maildir/tmp: $ docker exec mad_jepsen cat /root/Maildir/new/1486160677.Vfe02Ie0699bM508368.mail.coco.com Return-Path: <toot@cejf.com> X-Original-To: root@coco.com Delivered-To: root@coco.com Received: from mail01.mood.shibarecords.com (unknown [172.17.0.2]) by mail.coco.com (Postfix) with ESMTP id 76AE960758E for <root@coco.com>; Fri, 3 Feb 2017 22:24:37 +0000 (UTC) Received: from toto (unknown [172.17.0.1]) by mail01.mood.shibarecords.com (Postfix) with ESMTP id 2F0AAA04008 for <root@coco.com>; Fri, 3 Feb 2017 22:24:26 +0000 (UTC) un courriel

These magnificent: D.

Well, it’s time to talk philosophy a bit, you do what you want from the next section: P

A flat file or an interrogation of the Ldap

As we saw in the LookUp Table list, it is also possible to have ldap or mysql connectors. Personally I am of the automatically generated flat text file type and here is why, not just because a text file is easy to read!

If you have a service exchange so with all the hardware Active directory, kerberos, LDAP, … You will probably be tempted to tell you cool, Fuck the flat file me direct LDAP to have the list of addresses. In itself I say yes good idea because:

 

  1. When adding an email address, this will be instantly available on the relay postfix server, no manual addition to the flat file

  2. No flat file management in place of the exchange.

However, I would like to highlight some points to consider:

  • Unavailability of LDAP: During my first test of using the dbm format when receiving the email, as postfix was not able to perform a lookup of the file it refused the email. This behavior will be equivalent if your LDAP service is unavailable, result if you made an update of your AD emails will not be received. Well you could tell yourself anyway the AD / LDAP service is on the same machine as exchange (honestly I do not know if it’s possible ;-)) so the emails will not be delivered anyway. Maybe but the SMTP service when it communicates to another server smtp respect the RFC so will keep in line with you the email and you can force the treatment of the tail because under your control. We are talking about ldap but it is just as valid with mysql or other. Originally even emails bouncé which was very problematic for the image of the company, it seems that it is no longer the case … But I admit that I do not want to make a test: P.
  • Autonomy of the postfix service: My goal is to make sure that my email service is independent of the services of the infrastructure to allow interruptions for updating and unavailability of the service. All while keeping the emails in accordance with the RFC locally while waiting for the distant service to fall on these legs!

  • Generating the file: As I want to achieve with the validation of emails, during the implementation I made a script at the client who was extracting e-mail addresses from LDAP and generated a flat file with the list of users and aliases This generation was realized at the time so a shift of maximum 1 hour between the creation and the availability on the Internet. A little trick if you perform this operation, this is risk management but I consider it a good assurance, I improved the script by setting up a validation. In addition to making sure I had the connection I had set up a validation on the number of addresses collected, if the number of emails with more than 20 entry deleted I blocked the generation of the new file. This mainly to make sure that the collection is honest, I put an option to bypass this validation. So if the AD server has an internal problem emails remain valid and requires the assistance of an administrator to do the validation, this may also be due to intensive housekeeping!