EZFAQ 0.40 - ezmlm-idx and ezmlm FAQ Fred Lindberg, lindberg@id.wustl.edu & Fred B. Ringel, fredr@rivertown.net 22-NOV-1999 This document is a collection of frequently asked questions about ezmlm-idx. Where applicable, ezmlm itself is also covered. This FAQ presumes familiarity with Unix, and with the basic concepts of E-mail and mailing lists. This FAQ is updated for ezmlm-0.53 and ezmlm- idx-0.40. ______________________________________________________________________ Table of Contents 1. General Information 1.1 Acknowledgements 1.2 What is this document? 1.3 Terminology 1.4 What is the difference between ezmlm and ezmlm-idx? 1.5 Where can I get all of the ezmlm-related programs? 1.6 Where can I find documentation for ezmlm and patches? 1.7 Where do I send comments on this document? 1.8 How to experiment with new versions of ezmlm-idx. 2. Quick start 3. Overview of mailing list management and mailing list managers 4. Overview of ezmlm function 4.1 The basic setup. 4.2 Inventions in ezmlm. 4.3 The qmail delivery mechanism. 4.4 What the different programs do. 4.5 What the different files in the list directory do. 4.6 The paper path for posts. 4.7 The ezmlm path for moderation messages. 4.8 The ezmlm path for administrative messages. 4.9 The ezmlm path for bounces. 4.10 Messages to list-owner and list-digest-owner. 4.11 Structure of subscriber databases. 4.12 Local case in E-mail addresses. 4.13 Testing SENDER to allow posts only from list subscribers. 4.14 How cookies work. 4.15 How moderator E-mail addresses are stored. 4.16 How subscription moderation works. 4.17 How remote administration works. 4.18 How message moderation works. 4.19 How QMQP support works 4.20 How messages are stored in the archive. 4.21 How the message index works. 4.22 How threading works. 4.23 How digests work. 4.24 How WWW archive access works. 4.25 How ezmlm-tstdig works. 4.26 How sublists work. 4.27 How sublisting can be made transparent to the user. 4.28 How to service commands in the subject line. 4.29 How to support alternative command names. 4.30 How to add your own commands. 4.31 How remote administrators can retrieve a subscriber list 4.32 How remote administrators can determine the number of subscribers 4.33 How remote admins can see if an address is a subscriber or not 4.34 How remote administrators can search the subscription log 4.35 How text file editing works. 4.36 How subject line prefixes work. 4.37 How bounces are handled. 4.38 How the info and faq commands work. 4.39 How the global ezmlm list address works. 4.40 How ezmlm-cron works. 4.41 How ezmlm-make works. 4.42 What names can I use for my lists? 4.43 Lists in virtual domains 4.44 How do I make customization simple for me/my users? 5. ezmlm support for SQL databases. 5.1 Why use an SQL database with ezmlm? 5.2 Why not to use an SQL database with ezmlm. 5.3 Tables used for (My)SQL support. 5.3.1 Address tables. 5.3.2 Subscriber log tables. 5.3.3 Message logging tables. 5.4 How to set up a simple list with SQL support. 5.4.1 Helper programs for SQL-enabled lists. 5.5 Manually manipulating the subscribers of a SQL-enabled list. 5.6 Converting to and from and SQL database. 5.7 Optimizing MySQL for ezmlm. 5.7.1 Address SELECTs, additions, removals. 5.8 Maintenance of the MySQL database. 6. Possible error conditions in ezmlm lists. 6.1 What do I do if ezmlm doesn't work? 6.2 How do I report ezmlm bugs? 6.3 Where do I send suggestions for ezmlm-idx improvements? 6.4 Using ezmlm-test to check the ezmlm(-idx) programs. 6.5 Using ezmlm-check to find setup errors. 6.6 Posts are rejected: Sorry, no mailbox here by that name (#5.1.1). 6.7 Post are not sent to subscribers. 6.8 ezmlm-make fails: usage: ezmlm-make ... 6.9 ezmlm-make fails: Unable to create ... 6.10 ezmlm-make fails: ... ezmlmrc does not exist 6.11 Index/get/thread requests fail quietly or with errors from ezmlm-manage. 6.12 Digest triggering requests fail. 6.13 Remote administration (un)subscribe confirm requests go to the user, not the moderator. 6.14 (Un)subscribers does not receive a (un)subscribe acknowledgement 6.15 Messages posted to a moderated list are sent out without moderation. 6.16 Messages posted to a moderated list do not result in moderation requests. 6.17 Moderation request replies do not result in the appropriate action. 6.18 Moderator comments with moderation request replies are not added to the post/sent to the poster. 6.19 Some headers are missing from messages in the digest. 6.20 Some Received: headers are missing from messages. 6.21 My Mutt users cannot thread their digest messages. 6.22 Posts fail: Message already has Mailing-List (#5.7.2). 6.23 The last line of a 6.24 No CONFIRM requests are sent to moderators. 6.25 Deliveries fail ``temporary qmail-queue error'' 6.26 How to deal with corrupted subscriber lists 6.27 Vacation program replies are treated as bounces by ezmlm. 6.28 Digests do not come at regular hours. 6.29 Preventing loops from misconfigured subscriber addresses. 6.30 A user can subscribe and receives warning and probe messages, but no messages from the list. 7. Customizing ezmlm-make operation via ezmlmrc 7.1 Using ezmlm-make to edit existing lists. 7.2 What is ezmlmrc? 7.3 Changing defaults for 7.4 Changing default moderator directories. 7.5 Adapting ezmlm-make for virtual domains. 7.6 Setting up ezmlm-make for special situations. 8. Restricting message posting to the list. 8.1 Requiring the list address in To:/Cc: headers. 8.2 Rejecting messages sent from other mailing lists. 8.3 Restricting posts based on the Subject line. 8.4 Restricting the size of posts. 8.5 Restricting posts based on MIME content-type. 8.6 Restricting posts to list subscribers. 8.7 Restricting posts to an arbitrary set of E-mail addresses (higher security option). 8.8 Completely restricting posts. 8.9 A general solution to restricting posts based on SENDER. 9. Customizing outgoing messages. 9.1 Adding a trailer to outgoing messages. 9.2 Adding a subject prefix to outgoing messages. 9.3 Adding a header to outgoing messages. 9.4 Adding a message number header. 9.5 Removing headers from outgoing messages. 9.6 Removing MIME parts from messages. 9.7 Limiting ``Received:'' headers in outgoing messages. 9.8 Setting ``Reply-To: list@host''. 9.9 Configuring the list so posts are not copied to the original sender. 9.10 Customizing ezmlm notification messages. 9.11 Specifying character set and content-transfer-encoding for outgoing ezmlm messages. 10. Customizing archive retrieval. 10.1 Specifying the format for retrieved messages. 10.2 Specifying the default format for digests and archive retrieval. 10.3 Limiting the number of messages per -get/-index request. 11. Restricting archive retrieval. 11.1 Restricting archive access to subscribers. 11.2 Restricting available archive retrieval commands. 11.3 Restricting archive retrieval to moderators. 11.4 Allowing archive retrieval from a non-public list. 12. Customizing digests. 12.1 Setting up a digest list. 12.2 Generating daily digests. 12.3 Generating the first digest. 12.4 Adding standard administrative information to digests. 12.5 Controlling the digest format. 12.6 Customizing bounce handling. 13. Remote administration. 13.1 How can I remotely add moderators, subscriber aliases, etc? 13.2 Moderating posts from a secondary account. 13.3 Moderating subscription from a secondary account. 13.4 Automatically approving posts or subscriptions. 13.5 Allowing remote administrators to get a subscriber list. 13.6 Allowing remote administrators to retrieve or search a subscription log. 13.7 Allowing users to get a subscriber list. 13.8 Changing the timeout for messages in the moderation queue. 13.9 Finding out how many messages are waiting for moderation. 13.10 Using the same moderators for multiple lists. 13.11 Using different moderators for message and subscription moderation. 13.12 Setting up moderated lists with the list owner as the ``super moderator'' able to add/remove moderators remotely. 13.13 Customizing ezmlm administrative messages. 13.14 Manually approving a message awaiting moderation. 13.15 Manually rejecting a message awaiting moderation. 14. Sublists. 14.1 Sublists of ezmlm lists. 14.2 Sublists of non-ezmlm lists. 14.3 How to set up a cluster of list and sublists with standard databases. 15. Migration to Ezmlm from other Mailing List Managers. 15.1 Basic Concepts. 15.2 Setting up ezmlm to respond to host-centric commands. 15.3 Commands of other mailinglist managers recognized by ezmlm. 15.3.1 Listproc/Listserv. 15.3.2 Majordomo. 15.3.3 Smartlist. 16. Optimizing list performance. 16.1 Crond-generated digests for better performance. 16.2 Optimizing execution of ezmlm-warn(1). 16.3 Decreasing ezmlm-warn time out to increase performance. 16.4 Use ezmlm without ezmlm-idx for maximum performance. 16.5 Not archiving to maximize performance. 16.6 Sublists to maximize performance. 17. Miscellaneous. 17.1 How do I quickly change the properties of my list? 17.2 Open archived list with daily digests. 17.3 Variations in moderation 17.4 Lists that allow remote admin, but not user initiated subscription or archive retrieval. 17.5 Lists that allow remote admin, user archive retrieval, but not user-initiated subscription. 17.6 Lists that restrict archive retrieval to subscribers. 17.7 Lists that do not allow archive retrieval at all. 17.8 Lists that do not allow archive retrieval and do not allow digest triggering per mail. 17.9 Lists that allow archive retrieval only to moderators, but allow user-initiated subscription. 17.10 Lists that do not require user confirmation for (un)subscription. 17.11 Announcement lists for a small set of trusted posters 17.12 Announcement lists allowing moderated posts from anyone. 17.13 Announcement lists with less security and more convenience. 18. Ezmlm-idx compile time options. 18.1 Location of binaries. 18.2 Location of man pages. 18.3 Base directory of qmail-installation. 18.4 Short header texts, etc. 18.5 Arbitrary limits. 18.6 Command names. 18.7 Error messages. 18.8 Paths and other odd configuration items. 19. Multiple language support. 19.1 Command names. 19.2 Text files. 19.3 Multi-byte character code support. 20. Subscriber notification of moderation events. 20.1 General opinions. 20.2 Users should know that the list is subscription moderated. 20.3 Subscribers should know that posts are moderated. 20.4 Senders of posts should be notified of rejections. 21. Ezmlm-idx security. 21.1 General assumptions. 21.2 SENDER manipulation. 21.3 ezmlm cookies. 21.4 Lists without remote admin/subscription moderation. 21.5 Message moderation. 21.6 Subscription moderation. 21.7 Remote administration. 21.8 Remote editing of ezmlm text files. 21.9 Digest generation and archive retrieval. 21.10 Convenience for security: the ezmlm-manage ``-S'' and ``-U'' switches. 21.11 Denial of service. 21.12 Moderator anonymity. 21.13 Confidentiality of subscriber E-mail addresses. 21.14 Help message for moderators. 21.15 Sublists. 21.16 SQL databases. 21.17 Reporting security problems. ______________________________________________________________________ 11.. GGeenneerraall IInnffoorrmmaattiioonn 11..11.. AAcckknnoowwlleeddggeemmeennttss Many ezmlm users have contributed to improvements in ezmlm-idx. These are listed in the RREEAADDMMEE..iiddxx file in the ezmlm-idx distribution. Others have through questions and suggestions inspired parts in this FAQ, or pointed out errors or omissions. Thanks! Direct contributions are attributed to the respective authors in the text. Thanks again! 11..22.. WWhhaatt iiss tthhiiss ddooccuummeenntt?? This FAQ contains answers to many questions that arise while installing ezmlm, ezmlm-idx, and while setting up and managing ezmlm mailing lists. See ``'' for a brief summary of what is ezmlm and what is ezmlm-idx. Many aspects of ezmlm are covered in several places in this FAQ. The early sections explain how ezmlm works. Later sections discuss how to deal with possible errors/problems. Subsequent sections discuss details of customization and list setup in a _H_O_W_T_O form. Finally, there are sections on information philosophy for moderated lists and on security aspects on ezmlm lists. This is an evolving document. If you find any errors, or wish to comment, please do so to the authors. This FAQ is currently aimed at system administrators and knowledgeable users, and heavily weighted towards questions specific to the ezmlm-idx add-on. If you have problems with the ezmlm-idx package, please start by reading the ``man'' pages which come with each program, then this document and other ezmlm documentation which is identified here. If you have exhausted these resources, try the ezmlm and qmail mailing lists and their respective mailing list archives. If you have solved a problem not in the documentation, write it up as a proposed section of a FAQ and send it to the authors. This way, it can be added to the next version of this FAQ. 11..33.. TTeerrmmiinnoollooggyy This document uses a number of terms. Here are the meanings ascribed to them by the authors. DDIIRR The base directory of the list. SSEENNDDEERR The envelope sender of the message, as passed to ezmlm by qmail via the $SENDER environment variable. LLOOCCAALL The local part of the envelope recipient. For list-get-1@host, it is usually _l_i_s_t_-_g_e_t_-_1. If host is a virtual domain, controlled by _u_s_e_r_-_s_u_b, then local would be _u_s_e_r_-_s_u_b_-_l_i_s_t_-_g_e_t_-_1. mmooddddiirr Base directory for moderators. Moderator E-mail addresses are stored in a hashed database in mmooddddiirr//ssuubbssccrriibbeerrss//. By default, ``moddir'' is DDIIRR//mmoodd//. To add or remove moderators: % ezmlm-sub DIR/moddir moderator@host.domain % ezmlm-unsub DIR/moddir moderator@host.domain ddoottddiirr The second argument of ezmlm-make is the main .qmail file for the list. dotdir is the directory in which this ``dot file'' resides, i.e. the directory part of the ``dot'' argument. This is usually the home directory of the user controlling the list (but NOT necessarily of the one creating the list). Thus, _d_o_t_d_i_r is ~~aalliiaass// if ``root'' creates a list: # ezmlm-make ~alias/list ~alias/.qmail-list ... _d_o_t_d_i_r is where the ..eezzmmllmmrrcc file is expected when the ezmlm- make(1) ``-c'' switch is used (see ``Customizing ezmlm-make opera- tion''). eezzmmllmm bbiinnaarryy ddiirreeccttoorryy The directory where the ezmlm-binaries are normally stored, as defined at compile time in ccoonnff--bbiinn. This is compiled into the programs and does not change just because you have moved the program. eezzmmllmm--ggeett((11)) This is a reference to the ezmlm-get.1 man page. Access it with one of the following: % man ezmlm-get % man 1 ezmlm-get or if you have not yet installed ezmlm-idx (replace ``xxx'' with the version number): % cd ezmlm-idx-0.xxx % man ./ezmlm-get.1 bbaasseeddiirr The list directory when referencing the list subscriber address database. For E-mail addresses stored in a set of files within DDIIRR//ssuubbssccrriibbeerrss//, the ``basedir'' is ``DIR''. aaddddrreessss ddaattaabbaassee A collection of E-mail addresses stored in a set of files within the ``subscribers'' subdirectory of the basedir, DDIIRR//ssuubbssccrriibbeerrss//. mmeessssaaggee mmooddeerraattoorr An address to which moderation requests for posts to the list are sent. The moderation requests are formatted with ``From:''-``reject'' and a ``To:''-``accept'' default headers for moderator replies. A reply to the ``reject'' address leads to the rejection of the post. A reply to the ``accept'' address leads to the acceptance of the post. Any E-mail address can be a moderator E-mail address. Any number of moderator E-mail addresses can be used. If a post is sent from a moderator E-mail address, the moderation request is sent to that E-mail address only. If a post is sent from an E-mail address that is not a moderator, a moderation request is sent to all moderators. The first reply to the moderation request determines the fate of the message. Further requests for the action already taken are silently ignored, while a request for the contrary action results in an error message stating the actual fate of the message. Thus, if you want to ``accept'' the message and it has already been accepted, you receive no reply, but if you attempt to ``reject'' it, you will receive an error message stating that the message already has been accepted. Most lists are not message moderated. If they are, the owner is usually a ``message moderator'', sometimes together with a few other trusted users. For an announcement list, it is common to make all the ``official announcers'' ``message moderators''. This way, they can post securely and ``accept'' their own posts, while posts from other users will be sent to this set of ``official announcers'' for approval. ssuubbssccrriippttiioonn mmooddeerraattoorr An E-mail address where subscription moderation requests are sent. A subscription moderation request is sent after a user has confirmed her intention to subscribe. The subscription moderation request is sent to all moderators. As soon as a reply to this message is received, the user is subscribed and notified. Any E-mail address can be a subscription moderator and any number of subscription moderators can be used. Unsubscribe requests are never moderated (except when the ezmlm- manage(1) ``-U'' flag is used and the sender attempts to remove an address other than the one s/he is sending from). It is hard to imagine a legitimate mailing list that would want to prevent unsubscriptions. rreemmoottee aaddmmiinniissttrraattoorr When a remote administrator subscribes or unsubscribes a list member, the ``confirm'' request is sent back to the remote administrator, rather than to the subscriber's E-mail address. This allows the remote administrator to (un)subscribe any list member without the cooperation of the subscriber at that address. Any E-mail address can be a remote administrator and any number of E-mail addresses can be remote administrators. The set of E-mail addresses that are ``remote administrators'' and ``subscription moderators'' are always the same. This set of E-mail addresses can be ``remote administrators'', ``subscription moderators'' or both. For most lists, the owner would be the ``remote administrator'', if s/he wishes to moderate messages, the owner would be the ``message moderator'' and if s/he wishes to moderate subscriptions the owner would also be the ``subscription moderator''. The list's ``message moderator(s)'' can be the same, but can also be set up to be completely different. CChhaannggiinngg lliisstt ````oowwnneerrsshhiipp'''' Within this FAQ there are references to the need to check or change the list ``ownership.'' This is not a reference to the individual user who is the ``list-owner'', but a reference to the ownership of the files by your operating system which make up the list and reside in DDIIRR//. To change the ownership of DDIIRR// and everything within: % chown -R user DIR % chgrp -R group DIR Depending on your system/shell, it may be possible to combine these commands into either: % chown -R user.group DIR % chown -R user:group DIR 11..44.. WWhhaatt iiss tthhee ddiiffffeerreennccee bbeettwweeeenn eezzmmllmm aanndd eezzmmllmm--iiddxx?? ezmlm-0.53 is a qmail-based mailing list manager written by Dan J. Bernstein. It has all the basic functionality of a mailing list manager, such as subscriber address management including automated bounce handling as well as message distribution and archiving. ezmlm-idx is an add-on to ezmlm. It adds multi-message threaded message retrieval from the archive, digests, message and subscription moderation, and a number of remote administration function. It modifies the configuration program ezmlm-make(1) so that it uses a text file template rather than compiled-in texts in list creation. In this manner, ezmlm-idx allows easy setup of lists in different languages and customization of default list setup. ezmlm-idx also adds MIME handling, and other support to streamline use with languages other than English. As an ezmlm add-on, ezmlm-idx does not work without ezmlm and tries to be compatible with ezmlm as much as possible. ezmlm-idx also modifies the ezmlm subscriber database to be case insensitive to avoid many unsubscribe problems. New in ezmlm-idx-0.40 are better support for announcement lists, support for QMQP to offload message distribution onto external hosts, simplified optional SQL database use (MySQL or PostgreSQL), more flexibility in determining which messages should be moderated, a WWW interface to the list archives, and many small improvements. ezmlm-idx-0.32 adds improved handling of very large lists with optimized bounce handling, ezmlm-split(1) for forwarding (un)subscribe requests to sublists to allow sublisting transparent to the subscriber, and SQL support to allow sublisting with improved message authentication and monitoring of list function, as well as dynamic addition/removal/reconfiguration of sublists. Also, subscriber ``From:'' lines are logged with support for finding a subscription address from a name. The qmail DEFAULT variable is used, if present. Together, these additions eliminate the most common problems making ezmlm use and administration even easier. This document is a FAQ for ezmlm-idx. However, many of the basic items that are discussed also apply to ezmlm per se. Referring to the two paragraphs above, it should be relatively easy to figure out which features require ezmlm-idx. 11..55.. WWhheerree ccaann II ggeett aallll ooff tthhee eezzmmllmm--rreellaatteedd pprrooggrraammss?? We have now registered ezmlm.org to make access to ezmlm-idx and related programs/documentation easier. www.ezmlm.org is currently an alias for Fred B. Ringel's www.rivertown.net/~ezmlm/ and ftp.ezmlm.org an alias for Fred Lindberg's ftp.id.wustl.edu. DDaann JJ.. BBeerrnnsstteeiinn''ss eezzmmllmm--00..5533 +o +o +o +o +o +o +o +o TThhee llaatteesstt vveerrssiioonn ooff eezzmmllmm--iiddxx ezmlm-idx releases are numbered ``ezmlm-idx-0.xy[z]''. Versions with the same ``x'' are backwards compatible. A change in ``x'' signifies major changes, some of which _m_a_y require list changes (see UPGRADE.idx). However, backwards compatibility with ezmlm-0.53 list will be maintained. Thus, this is an issue only if you are already using an older version of ezmlm-idx. Addition of ``z'' are bug fixes only. Thus, ezmlm-idx-0.301 is ezmlm-idx-0.30 with known bugs fixed (but no other significant changes). When available, patches are named ``filename-0.xy[z].diff'', where ``0.xy[z]'' corresponds to the release to which they apply. When a number of bugs (or a significant bug) are found a bug-fix release is made incorporating all the patches for the previous version. To get the latest features, look for the highest number (``e.g. ezmlm-idx-0.40''). Any bugs in versions with new features are expected to be limited to the new features. To get the most solid version, get the highest 3-digit number, i.e. a bug fix. If you already run a version in that series and a new bug fix is released, see CHANGES.idx to determine if it is worthwhile to upgrade. Most bugs so far have been relevant only when using lists in very unusual ways or with rarely used options. +o +o ftp mirror in Austria. +o http access to the same mirror. +o ftp mirror in Japan. eezzmmllmmrrcc((55)) ffiilleess ffoorr ddiiffffeerreenntt llaanngguuaaggeess The latest versions at the time of release of a package are included in that package. Thus, this directory will have a file labeled with the current ezmlm-idx version number only if it has been updated later than the package. ezmlmrc(5) files are updated and new ones are added all the time, also with bug fix releases. Therefore, always look at the latest package. Please note that ezmlmrc may change significantly between versions. Thus, do not expect the ezmlm-idx-0.324 ezmlmrc.es to work with ezmlm-idx-0.40. ezmlmrc(5) files contain some release-specific configurations. Do not use a later file (other than from bug fix releases) with an earlier version of the programs. It is usually OK to use a version from an earlier package (see UPGRADE.idx), but some new functionality may nor be available. To contribute an ezmlmrc(5) file in a new language, start with the en_US version from the latest package, and send the gzipped file to lindberg@id.wustl.edu. Please leave comments intact and in English and do not change the order of items in the file. This will facilitate maintenance. +o +o +o +o eezzmmllmm--iissssuubb--00..0055 +o . Use ezmlm-issub only if you do not use ezmlm-idx. The same functionality is available in ezmlm-idx and the packages are not compatible. +o Also via mirrors mentioned above. RRPPMMss aanndd SSRRPPMMSS ooff qqmmaaiill,, eezzmmllmm aanndd eezzmmllmm--iiddxx +o +o 11..66.. WWhheerree ccaann II ffiinndd ddooccuummeennttaattiioonn ffoorr eezzmmllmm aanndd ppaattcchheess?? mmaann ppaaggeess All ezmlm component programs come with their own man pages. Thus, for info on _e_z_m_l_m_-_s_e_n_d, type: % man ezmlm-send or if you have unpacked ezmlm, but not made it or installed it: % cd ezmlm-0.53 % man ./ezmlm-send.1 eezzmmllmm((55)) General info on ezmlm and list directories is in eezzmmllmm..55: % man ezmlm or % cd ezmlm-0.53 % man ./ezmlm.5 _N_O_T_E_: Installation of the ezmlm-idx package updates some existing man pages to reflect changes made by the patch (e.g. ezmlm- send(1), ezmlm(5)). TTeexxtt ffiilleess iinn tthhee ddiissttrriibbuuttiioonn ezmlm comes with a RREEAADDMMEE file with general instructions, an IINNSSTTAALLLL file with installation instructions, an UUPPGGRRAADDEE file for upgrading from a previous version and a CCHHAANNGGEESS file with information on changes from previous versions. ezmlm-idx comes with similar files suffixed with ``..iiddxx''. Most other patches or add-ons contain similar files and man pages and should contain identifying suffixes (.iss for ezmlm-issub, for example). For a discussion of the authors' understanding of ezmlm security, see ``Ezmlm-idx security''. ````EEzzmmaann'''',, aann eezzmmllmm//iiddxx mmaannuuaall The ezmlm manual is a brief manual that is meant for list subscribers, list moderators and remote administrators, and as an introduction for list owners. It is useful even if you do not use ezmlm-idx. Features requiring ezmlm-idx are marked as such. The manual is available as a set of html files, as a text file, and in a ``letter'' and ``A4'' postscript version: +o ezman for download +o An on-line html version TThhiiss FFAAQQ This FAQ is built from a sgml source. It is available in the following formats: +o A text file +o An on-line html version +o Html for download +o A postscript (letter) version +o A postscript (A4) version +o Via mirrors mentioned for the ezmlm-idx package. +o An up-to-date text version,FFAAQQ..iiddxx, included with the ezmlm-idx package. WWWWWW rreessoouurrcceess AAnn oonn--lliinnee vveerrssiioonn ooff tthhiiss FFAAQQ The main site with an up-to-date mirror list. German mirror. Polish mirror. Japanese mirror. Portuguese mirror. Austrian mirror. Canadian mirror. GGeenneerraall qqmmaaiill aanndd eezzmmllmm iinnffoo +o Dan J. Bernstein's qmail page +o Dan J. Bernstein's ezmlm page +o Russell Nelson's qmail page +o Mirrors of www.qmail.org . Substitute your two-letter country abbreviation for ``ISO''. TThhee qqmmaaiill mmaaiilliinngg lliisstt aarrcchhiivvee +o TThhee eezzmmllmm mmaaiilliinngg lliisstt aarrcchhiivvee +o This archive of the ezmlm list is searchable from 11/97-present. ezmlm-cgi(1) is used to allow direct access to the sublist archive. MMaaiilliinngg lliissttss Please read other documentation and mailing list archives before posting questions to the lists. It's also useful to ``lurk'' on the list for a few days, (i.e. to subscribe and read without posting) before asking your questions on the list. To subscribe, send mail to the E-mail addresses listed: +o Dan Bernstein's ezmlm list: ezmlm-subscribe@list.cr.yp.to +o A digest version of the ezmlm list fredr-ezmlm-digest- subscribe@rivertown.net +o Dan Bernstein's qmail list: qmail-subscribe@list.cr.yp.to +o The Japanese ezmlm list: ezmlm-subscribe@jp.qmail.org +o The Japanese qmail list: qmail-subscribe@jp.qmail.org +o A ezmlm/idx digest list of djb-qmail: qmail-digest- subscribe@id.wustl.edu +o A ezmlm/idx sublist of djb-qmail (you can test ezmlm-idx commands): qmail-index@id.wustl.edu 11..77.. WWhheerree ddoo II sseenndd ccoommmmeennttss oonn tthhiiss ddooccuummeenntt?? To the authors via E-mail: +o Fred Lindberg, lindberg@id.wustl.edu +o Fred B. Ringel, fredr@rivertown.net 11..88.. HHooww ttoo eexxppeerriimmeenntt wwiitthh nneeww vveerrssiioonnss ooff eezzmmllmm--iiddxx.. ezmlm-idx>=0.23 writes DDIIRR//ccoonnffiigg in a standard format. If ezmlm- make(1) is invoked with the ``-e'' or ``-+'' switch and the ``DIR'' argument only, ezmlm-make(1) will read other arguments from this file. The difference between the switches is that with ``-e'' the options used are the ones specified on the command line, whereas with ``-+'' they are the ones currently active for the list, as overridden by any command line options. Thus, with just: % ezmlm-make -+ DIR you can rebuild the list, without affecting any archives, list state variables, etc. You will _l_o_s_e _m_a_n_u_a_l _c_u_s_t_o_m_i_z_a_t_i_o_n_s _t_o _s_o_m_e _o_f _y_o_u_r _f_i_l_e_s. However, text files and DDIIRR//hheeaaddeerraadddd are protected against being overwritten, so that your manual customizations of these files are retained. To override this protection, simply specify the used edit switch twice, e.g. ``-ee'' and ``-++'', respectively. This is a feature introduced in ezmlm-idx-0.40. To test a new version of ezmlm-idx or to run several version, make the new version as per IINNSSTTAALLLL..iiddxx (if you haven't used ezmlm-idx before) or UUPPGGRRAADDEE..iiddxx (if you've got a previous version of ezmlm-idx installed), setting ccoonnff--bbiinn to a new directory. You can use either the current directory or any other directory. If not using the current dir, you also have to: % make setup If you now edit the list using the new ezmlm-make program, the list will automatically be configured to use the new binaries. To change back to the ``default'' installation, just edit the list again, this time with the old ezmlm-make(1). If your system has an //eettcc//eezzmmllmmrrcc file, you may need to temporarily place the eezzmmllmmrrcc((55)) file for the ezmlm version you want to test in ddoottddiirr of the list and use the ezmlm-make(1) ``-c'' switch (see ``Terminology: dotdir''). ezmlm-idx>=0.314 comes with ezmlm-test(1), a program that tests most functions of ezmlm+idx and can be used before installation. 22.. QQuuiicckk ssttaarrtt 1. Create a use ``eztest'' for testing. If you use another name, add the switch ``-u another_name'' to the ezmlm-test(1) line below. (The space between the switch and the argument is required.) 2. Unpack the ezmlm-0.53 distribution. 3. Unpack the ezmlm-idx distribution. 4. Move the ezmlm-idx files to the ezmlm-0.53 directory. 5. Edit ccoonnff--bbiinn and ccoonnff--mmaann to reflect the target directories. 6. build and install: % cd ezmlm-0.53 % patch < idx.patch % make; make man % su # su eztest % ./ezmlm-test % exit # make setup # exit 7. Make a list and digest list % ezmlm-make -rdugm -5 me@host ~/list ~/.qmail-list me-list host % ezmlm-sub ~/list me@host % ezmlm-sub ~/list/digest me@host % ezmlm-sub ~/list/mod me@host where ``me'' is your user name and ``host'' the host your list is on. Now, you are the owner, remote administrator, and subscriber of both list@host and the accompanying digest list list-digest@host. Only subscribers are allowed to access the archive and to post. To post to the list, mail to list@host. For a user to subscribe, s/he should mail to list-subscribe@host and for help to list-help@host. When a non-subscriber posts, you will be asked to approve, reject, or ignore the request. If you want to subscriber joe@joehost.dom, mail list-subscribe-joe=joehost.dom@host. Digests are generated about every two days, when 30 messages have arrived since the last digest, or when more than 64 kbytes of message body has arrived. To manage the digest list, use the same commands as the main list, but replace ``list'' with ``list-digest''. The sender restriction on posting used in this setup works, but is not secure. For more info, read the man pages (start with ezmlm(5) and ezmlm-make(1)), this FAQ (FFAAQQ..iiddxx in the distribution), RREEAADDMMEE//RREEAADDMMEE..iiddxx, IINNSSTTAALLLL//IINNSSTTAALLLL..iiddxx, and UUPPGGRRAADDEE..iiddxx. 33.. OOvveerrvviieeww ooff mmaaiilliinngg lliisstt mmaannaaggeemmeenntt aanndd mmaaiilliinngg lliisstt mmaannaaggeerrss (To be written. Until then, please consult the manual for ezmlm and ezmlm-idx related material.) 44.. OOvveerrvviieeww ooff eezzmmllmm ffuunnccttiioonn 44..11.. TThhee bbaassiicc sseettuupp.. In designing ezmlm, _D_a_n _J_. _B_e_r_n_s_t_e_i_n has used the unix philosophy of small component programs with limited and well defined functions. Requests for specific functions can then be met by the addition of new programs. Thanks to the program execution mechanism Dan built into qmail, it is easy to execute several small programs per delivery in a defined sequence. It is also very easy to add shell scripts for further customization. 44..22.. IInnvveennttiioonnss iinn eezzmmllmm.. Dan J. Bernstein has written ezmlm in C. It is written for speed and reliability even in the face of power loss and NFS. These features are augmented to a large extent by the ruggedness of the qmail (also by Dan) delivery mechanism (see qmail-command(8)). ezmlm uses some routines and techniques that still are not frequently seen in many mailing list managers. For example, subscriber E-mail addresses are stored in a hash so that searches require reading only, at most, 2% of the E-mail addresses. ezmlm has a optional message archive, where messages are stored 100 per directory, again to allow more efficient storage and retrieval. Important files are written under a new name and, only when safely written, moved in place, to assure that crashes do not leave the list in an undefined state. In addition, ezmlm has a number of new inventions. One of these is bounce detection, which generates an automatic warning containing information identifying the messages which have bounced, followed by a probe message to the E-mail addresses for which mail has bounced. If the probe bounces, the address is unsubscribed. Thus, the system won't remove E-mail addresses due to temporary bounces: it takes 12 days after the first bounce before a warning is sent, and another 12 days of bounces after the warning bounce before the probe message is set. Another Dan J. Bernstein invention is the use of cryptographic cookies based on a timestamp, address, and action. These are used to assure that the user sending a request to subscribe or unsubscribe really controls the target address. It is also used to prevent forgery of warning or probe messages to make it exceedingly difficult to subvert the bounce detection mechanism to unsubscribe another user. 44..33.. TThhee qqmmaaiill ddeelliivveerryy mmeecchhaanniissmm.. See qmail(7), qmail-local(8), qmail-command(8), envelopes(5), and dot- qmail(5). Briefly, qmail having resolved the delivery address delivers it via the ..qqmmaaiill file that most completely matches the address. This file may be a link to another file, as is the case in ezmlm lists. qmail then delivers the message according to successive lines in this file forwarding it to an address, storing it, or piping it to a program. In the latter case, the program is expected to exit 0 leading delivery to proceed to the next line in the ..qqmmaaiill file, or 99 leading to success without delivery to succeeding lines. An exit code of 100 is a permanent error leading to an error message to the SENDER. An exit code of 111 is used for temporary errors, leading to re- delivery until successful or until the queue lifetime of the message has been exceeded. Delivery granularity is the ..qqmmaaiill file and re-deliveries start at the top. Thus, if the message fails temporarily at a later line, the delivery according to an earlier line will be repeated. Similarly, qmail may have made deliveries successfully according to most of the ..qqmmaaiill file and then fail permanently. The SENDER is informed that the delivery failed, but not about at which point. ezmlm takes advantage of these basic mechanisms to build a fast, efficient, and very configurable mailing list manager from a set of small independent programs. 44..44.. WWhhaatt tthhee ddiiffffeerreenntt pprrooggrraammss ddoo.. See ezmlm(5) and the man pages for the different programs (listed in ezmlm(5)). 44..55.. WWhhaatt tthhee ddiiffffeerreenntt ffiilleess iinn tthhee lliisstt ddiirreeccttoorryy ddoo.. See ezmlm(5). 44..66.. TThhee ppaappeerr ppaatthh ffoorr ppoossttss.. Messages to the list are delivered to a ..qqmmaaiill file, usually ~~//..qqmmaaiill-- lliissttnnaammee which is linked to DDIIRR//eeddiittoorr. Here, the message is first delivered to ezmlm-reject(1) which can reject messages based on subject line contents, MIME content-type, and message body length. It also by default rejects all messages that do not have the list address in the ``To:'' or ``Cc:'' header. This eliminates most bulk spam. If the list is set up for restrictions based on envelope SENDER, the next delivery is to one or more instances of ezmlm-issubn(1). If the messages passed this check, it is usually delivered to ezmlm-send(1) for distribution. If the list is message moderated, it is instead delivered to ezmlm-store(1) which queues the message and sends out a moderation request. ezmlm-gate(1) is used by some other setups. It will for message moderated lists invoke ezmlm-send(1) directly if the message is from a specific set of SENDERs, and in other cases ezmlm- store(1) to send the message out for moderation. You can specify a separate ..qqmmaaiill-like file for ezmlm-gate(1). The lines will be executed and the return codes determine if the message is rejected, sent to the list, or sent to the moderator. See man page for details. If the list is configured for digests, DDIIRR//eeddiittoorr also contains an ezmlm-tstdig(1) line followed by an ezmlm-get(1) line. If ezmlm- tstdig(1) determines that the criteria are met for digest generation, it exits with an exit code of 0, causing the ezmlm-get(1) line to be executed leading to a digest mailing. Otherwise, ezmlm-tstdig(1) exits 99, resulting in the remainder of the DDIIRR//eeddiittoorr file to be ignored too long. The digest is not related to the message being delivered, but the delivery is used to trigger execution of the relevant programs. In addition, DDIIRR//eeddiittoorr contains a number of house-keeping functions. These are invocations of ezmlm-warn(1) to send out bounce warnings and and (if the list is moderated) ezmlm-clean(1) to clean the moderation queue of messages that have been ignored. Again, these functions are not related to the specific message delivered, but the delivery itself is used as a convenient ``trigger'' for processing. 44..77.. TThhee eezzmmllmm ppaatthh ffoorr mmooddeerraattiioonn mmeessssaaggeess.. Replies to moderation requests are channeled to DDIIRR//mmooddeerraattoorr. This file contains an invocation of ezmlm-moderate(1) which invokes ezmlm- send(1) for accepted messages and sends out a rejection notice for rejected messages. It also sends error messages if the message is not found or already accepted/rejected _c_o_n_t_r_a_r_y to the moderation message. Thus, if you accept a message already accepted, no error message is sent. ezmlm-clean(1) is also invoked from DDIIRR//mmooddeerraattoorr for house keeping. 44..88.. TThhee eezzmmllmm ppaatthh ffoorr aaddmmiinniissttrraattiivvee mmeessssaaggeess.. Administrative requests for both list and digest lists are captured by ~~//..qqmmaaiill--lliissttnnaammee--ddeeffaauulltt linked to DDIIRR//mmaannaaggeerr. Here they are delivered first to ezmlm-get(1) which processed archive retrieval requests, exiting 99 after successful completion which causes the rest of the delivery lines to be ignored. If the request is not for ezmlm- get(1) it rapidly exits 0. This leads to invocation of ezmlm-manage(1) which handles subscriber database functions, help messages, and (if configured) editing of DDIIRR//tteexxtt// files. Again, ezmlm-warn(1) lines are included for bounce directory processing. If configured, an ezmlm-request(1) line is present. This program constructs valid ezmlm requests from command in the subject lines of messages sent to listname-request@host and exits 99. These requests are mailed and will then return to be processed by one of the other programs. 44..99.. TThhee eezzmmllmm ppaatthh ffoorr bboouunncceess.. Bounces to the list are handled by DDIIRR//bboouunncceerr. For the digest list this is DDIIRR//ddiiggeesstt//bboouunncceerr. The two were combined in previous versions, which is still supported. As this leads to problems with list names ending in ``digest'', the functions are separate with lists set up or edited with ezmlm-idx>=0.32. The bounce is first delivery is to ezmlm-weed(1) which removes delivery delay notification and other junk. The second to ezmlm-return(1) which analyzes valid bounces storing the information in DDIIRR//bboouunnccee// for the list and DDIIRR//ddiiggeesstt//bboouunnccee// for the digest. This is the information that ezmlm-warn(1) (invoked from DDIIRR//eeddiittoorr and DDIIRR//mmaannaaggeerr) uses and processes for automatic bounce handling. ezmlm-return(1) will also unsubscribe a subscriber from whom a probe message has bounced. 44..1100.. MMeessssaaggeess ttoo lliisstt--oowwnneerr aanndd lliisstt--ddiiggeesstt--oowwnneerr.. These are processed by DDIIRR//oowwnneerr and delivered to DDIIRR//mmaaiillbbooxx by default. It is better to put the real owner address in this location. This can be done manually, via editing of eezzmmllmmrrcc((55)), or via the ezmlm-make(1) -5 switch. Again, some house-keeping functions are also executed. 44..1111.. SSttrruuccttuurree ooff ssuubbssccrriibbeerr ddaattaabbaasseess.. ezmlm subscriber E-mail addresses are stored within DDIIRR//ssuubbssccrriibbeerrss// as a hashed set of 53 files. The hash calculated from the address determines which of the 53 files and address is stored in. Thus, to find out if an address is a subscriber, ezmlm has to read at most about 2% of the E-mail addresses. The hash function insures that E- mail addresses are reasonably evenly distributed among the 53 files. Addresses in the files in DDIIRR//ssuubbssccrriibbeerrss// are stored as strings starting with ``T'', followed by the address, followed by a zero byte. This is the same format as taken by qmail-queue(8) on file descriptor 1. Thus, subscriber lists can be directly copied to qmail without any further processing. With ezmlm-idx>=0.32 you can use an SQL server for the subscriber databases. Please see the SQL section (``ezmlm support for SQL datbases''). 44..1122.. LLooccaall ccaassee iinn EE--mmaaiill aaddddrreesssseess.. rfc822 states that the host part of an address is case insensitive, but that case of the local part should be respected and the interpretation of it is the prerogative of the machine where the mailbox exists. Thus, ezmlm preserves the case of the local part, but converts the host part to lower case. ezmlm proper also bases the hash on the case of the local part, so that USER@host and user@host are not (usually) stored in the same file. Locally, deliveries are most often case insensitive, i.e. mail to USER@host and user@host are delivered to the same mail box. A consequence of this is that many users use E-mail addresses with different case interchangeably. The problem is that when USER@host is subscribed, ezmlm will not find that address in response to an unsubscribe request from user@host. This is even more problematic when E-mail addresses have been added by hand to e.g. moderator lists. ezmlm-idx>=0.22 changes address storage to make comparisons case insensitive and store E-mail addresses based on the hash of the all lower case address. Case is maintained for the local part. Thus, if USER@host is subscribed, mail is set to USER@host, but user@host is recognized as a subscriber and an unsubscribe request from user@host will remove USER@host from the subscriber list. To maintain backwards compatibility with old subscriber lists, a second lookup is made for partially upper case E-mail addresses in some cases. This will find USER@host subscribed with a case sensitive hash as well. If may be useful to move all old mixed case E-mail addresses to the ``new'' positions. Without this, USER@host subscribed with the old system will be able to unsubscribe as USER@host, but not as user@host. After the repositioning, s/he will be successfully able to use any case in an unsubscribe request, e.g. UsEr@host. To do this: % ezmlm-list DIR | grep -G '[A-Z]' > tmp.tmp % xargs ezmlm-sub DIR < tmp.tmp This works, because subscribing an address, even if it already exists, will assure that it is stored with a case insensitive hash. On some systems, the grep ``-G'' switch need/should not be used. 44..1133.. TTeessttiinngg SSEENNDDEERR ttoo aallllooww ppoossttss oonnllyy ffrroomm lliisstt ssuubbssccrriibbeerrss.. This mode of operation is automatically set up if you specify the ezmlm-make(1) ``-u'' switch. Since there may be some addresses that should be allowed to post, but are not subscribers of list or list- digest, ezmlm-make(1) sets up an additional address database in DDIIRR//aallllooww//. Use ezmlm-sub(1), ezmlm-unsub(1), and ezmlm-list(1) to manipulate these addresses. If the list is configured for remote administration (see ``How remote administration works''), you can add/remove addresses from the DDIIRR//aallllooww// database by mailing list- allow-subscribe@listhost and list-allow-unsubscribe@listhost, respectively. Other commands that access subscriber databases work in the same manner. To similarly restrict archive access, use the ezmlm-make(1) ``-g'' switch. Since SENDER is under the control of a potential attacker, it is not secure to use tests of SENDER for anything important. However, when replies are always sent to SENDER (such as for archive access), a check of SENDER can prevent the sending of information to E-mail addresses not in the database. To test sender, use the program ezmlm-issubn(1). It will return 0 (true for the shell, success for qmail deliveries) if SENDER is in at least one of a set of subscriber databases. If not, it will return 99 (false for the shell: success, but skip remainder of ..qqmmaaiill file for qmail deliveries). The basedirs of the subscriber lists (i.e. the directories in which the ``subscriber'' dirs are located) are given as arguments. ezmlm-issubn(1) can take any number of arguments. Thus, to permit an action if SENDER is a subscriber to the list in any of DDIIRR//, DDIIRR//ddiiggeesstt//, or DDIIRR//aallllooww// and exit silently, put the following into the relevant ..qqmmaaiill file: |/usr/local/bin/ezmlm/ezmlm-issubn DIR DIR/digest DIR/allow [...] |/path/action_program Restricting your list to posts from your subscribers is as easy as that. If your ezmlm binaries are in a different directory, you may have to modify the ezmlm-issubn(1) path. ezmlm-issubn(1) has a ``-n'' switch which ``negates/reverses'' the exit code. To do an action if SENDER is _N_O_T a subscriber of any of the lists: |/usr/local/bin/ezmlm/ezmlm-issubn -n DIR/deny [dir2 ...] |/path/other_program To automatically configure the list with a blacklist address database in DDIIRR//ddeennyy, use the ezmlm-make(1) ``-k'' switch. If the list is configured for remote administration (see ``How remote administration works'') and if you are a remote administrator, you can manipulate the ``deny'' database remotely by sending mail to list-deny-subscribe- user=userhost@listhost, etc. 44..1144.. HHooww ccooookkiieess wwoorrkk.. Each ezmlm list has it's own ``key'' created by ezmlm-make at setup time. This key is stored in DDIIRR//kkeeyy, and you can improve it by adding garbage of your own to it. However, changing the key will make all outstanding cookies invalid, so this should be done when the list is established. When ezmlm receives an action request, such as ``subscribe'', it constructs a cookie as a function of: +o the request, +o the time, +o and the target address. The cookie and these items are then assembled into a address that is sent out as the ``Reply-To:'' address in the confirmation request sent to the subscriber. When the subscriber replies, ezmlm first checks if the timestamp is more than 1,000,000 seconds old (approx 11.6 days) and rejects the request if it is. Next, ezmlm recalculates the cookie from the items. If the cookies match, the request is valid and will be completed. Depending on the circumstances, ezmlm generates an error message or a new cookie based on the current time and sends the target a new confirmation request. Dan has based these cookies on cryptographic functions that make it very unlikely that a change in any part of the cookie or the items will result in a valid combination. Thus, it is virtually impossible to forge a request even for someone who has a number of valid requests to analyze. Since the algorithm ezmlm uses is available, the security rests on the key (and the correctness of the algorithm). Anyone who knows the key for your lists can easily construct valid requests. As ezmlm-make(1) doesn't use a truly random process to generate the key, it is theoretically possible that someone with sufficient knowledge about your system can guess your key. In practice, this is very unlikely, and the safety of the system is orders of magnitude higher than that of other mechanisms that you may rely on in your list management and mail transport (exclusive of strong encryption, such as _P_G_P). 44..1155.. HHooww mmooddeerraattoorr EE--mmaaiill aaddddrreesssseess aarree ssttoorreedd.. Moderator E-mail addresses are stored just like ezmlm subscriber addresses, in a set of up to 53 files within the ssuubbssccrriibbeerrss subdirectory of the list's bbaasseeddiirr//. For subscribers, the bbaasseeddiirr// is the list directory itself, i.e. DDIIRR//. For moderators, the default is DDIIRR//mmoodd//, which can be overridden by placing a bbaasseeddiirr name (starting with a ``/'') in DDIIRR//mmooddssuubb, DDIIRR//rreemmoottee, or DDIIRR//mmooddppoosstt for subscription moderation, remote administration, and message moderation, respectively. This permits the use of one moderator database for multiple lists. _N_o_t_e_: _S_u_b_s_c_r_i_p_t_i_o_n _m_o_d_e_r_a_t_o_r_s _a_n_d _r_e_m_o_t_e _a_d_m_i_n_i_s_t_r_a_t_o_r_s _a_r_e _a_l_w_a_y_s _t_h_e _s_a_m_e _a_d_d_r_e_s_s_e_s_. _I_f _b_o_t_h DDIIRR//mmooddssuubb and DDIIRR//rreemmoottee contain paths, only the DDIIRR//mmooddssuubb path is used. 44..1166.. HHooww ssuubbssccrriippttiioonn mmooddeerraattiioonn wwoorrkkss.. Subscription moderation is a simple extension of the ezmlm subscribe mechanism. Once the user has confirmed the subscribe request, a new request is constructed with a _d_i_f_f_e_r_e_n_t _a_c_t_i_o_n _c_o_d_e. This is sent out to the moderator(s). When a moderator replies with a valid request and cookie combination, the user is subscribed. The user is then also welcomed to the list. Other moderators won't know that the request has already been approved. If other moderators reply to the request, no notification of the duplicate action is sent to the subscriber of the duplicate action. Ezmlm knows that this is a repeat request since the target address is already a subscriber. The moderators are not informed about the result, unless there was an error (subscribing a target that is already a subscriber is not considered an error). This cuts down the number of messages a moderator receives. Any list moderator knows (or _s_h_o_u_l_d know) the qmail/ezmlm/unix paradigm: _i_f _y_o_u_'_r_e _n_o_t _t_o_l_d _o_t_h_e_r_w_i_s_e_, _y_o_u_r _c_o_m_m_a_n_d _w_a_s _c_a_r_r_i_e_d _o_u_t _s_u_c_c_e_s_s_f_u_l_l_y. This may be counterintuitive to those used to some other operating systems, but in our experience it doesn't take long to get used to the reliability and efficiency of U*ix/qmail/ezmlm. Subscription moderation is enabled by creating DDIIRR//mmooddssuubb and adding the subscription moderator to DDIIRR//mmoodd//: % ezmlm-sub DIR/mod moderator@host To use an alternative basedir for subscription moderators, place that directory name with a leading ``/'' in DDIIRR//mmooddssuubb. 44..1177.. HHooww rreemmoottee aaddmmiinniissttrraattiioonn wwoorrkkss.. The term ``remote administration'' is used to denote the ability of a list administrator by E-mail to add or remove any E-mail address from the subscriber list without the cooperation of the user. Normally, when user@userhost sends a message to list-subscribe- other=otherhost@listhost to subscribe other@otherhost, the confirmation request goes to other@otherhost. However, if remote administration is enabled and user@userhost is a moderator, a confirmation request (with a different action code) is sent back to user@userhost instead. The reply from the administrator is suppressed in the welcome message sent to the new subscriber (other@otherhost). This protects the identity of the remote administrator. Remote administration is enabled by creating DDIIRR//rreemmoottee and adding the remote administrator E-mail address(es) to DDIIRR//mmoodd//: % ezmlm-sub DIR/mod remoteadm@host To use an alternative basedir for remote administrators, place that directory name with a leading ``/'' in DDIIRR//mmooddssuubb. Remote administra- tors and subscription moderators databases always consist of the same E-mail addresses. If both are enabled and one of DDIIRR//mmooddssuubb and DDIIRR//rreemmoottee contains an alternative basedir name, this basedir is used for both functions. If both DDIIRR//mmooddssuubb and DDIIRR//rreemmoottee contain direc- tory names, the one in DDIIRR//mmooddssuubb is used for both functions. Remote administrators can add and remove addresses to the digest list, the ``allow'' list (user aliases for lists using SENDER restrictions on posting and archive access), and if used the ``deny'' list containing addresses that are denied posting rights to the list. The latter is easy to circumvent and intended to block errant mail robots, rather than human users. 44..1188.. HHooww mmeessssaaggee mmooddeerraattiioonn wwoorrkkss.. ezmlm-store(1), invoked in DDIIRR//eeddiittoorr, receives messages for message moderated lists. If DDIIRR//mmooddppoosstt does not exist, ezmlm-store(1) just calls ezmlm-send(1) and the message is posted to the list as if it were not moderated. If DDIIRR//mmooddppoosstt exists, ezmlm-store(1) places the message in DDIIRR//mmoodd//ppeennddiinngg//. It also sends a moderation request to all the moderators. Included with this request is a copy of the message. The ``From:'' and ``Reply-To:'' E-mail addresses contain codes for ``reject'' and ``accept'', together with a unique message name (derived from the message timestamp and process id) and a cookie based on these items. When a moderator replies, ezmlm-moderate(1) is invoked via DDIIRR//mmooddeerraattoorr. ezmlm-moderate(1) validates the request, and if the request is valid and the message is found in DDIIRR//mmoodd//ppeennddiinngg//, it carries out the requested action. If the request is ``reject'' the post is returned to SENDER with an explanation and an optional moderator comment. If the request is ``accept'' the message is posted to the list via ezmlm-send(1). As the request is processed, a stub for the message is created in DDIIRR//mmoodd//rreejjeecctteedd// or DDIIRR//mmoodd//aacccceepptteedd// for ``reject'' and ``accept'' requests, respectively. If a valid reply is received but the message is no longer in DDIIRR//mmoodd//ppeennddiinngg//, ezmlm-moderate(1) looks for the corresponding stub in DDIIRR//mmoodd//rreejjeecctteedd// and DDIIRR//mmoodd//aacccceepptteedd//. If the stub is found and the fate of the message was the one dictated by the new request, no further action is taken. If, however, no stub is found or the request and the actual message fate do not match, a notification is sent to the moderator. This scheme was chosen to impart a maximum of information with a minimum of messages. Also, it is the least demoralizing setup for multiple moderator lists, where it is important not to notify subsequent moderators that their work was in vain since the action of the first responding moderator has already resulted in processing of the message. If a message is not ``rejected'' or ``accepted'' it remains in DDIIRR//mmoodd//ppeennddiinngg// until it times out. Cleanup of both messages and stubs is accomplished by ezmlm-clean(1) which is invoked through both DDIIRR//eeddiittoorr and DDIIRR//mmooddeerraattoorr for message moderated lists. ezmlm- clean(1) looks at the timestamp used to generate the message/stub name. If it is older than 120 hours (configurable in a range of 24-240 hours, by placing the value in DDIIRR//mmooddttiimmee) it is removed. Unless suppressed with the ezmlm-clean(1) ``-R'' switch, the SENDER of the message is notified. By default, the E-mail addresses of message moderators are stored as a subscriber list with a basedir of DDIIRR//mmoodd//. This can be changed to any other bbaasseeddiirr by placing the name of that directory with a leading ``/'' in DDIIRR//mmooddppoosstt. Although the default basedirs for message moderation and subscription moderation/remote administration are the same, both the functions and actors are entirely independent. 44..1199.. HHooww QQMMQQPP ssuuppppoorrtt wwoorrkkss qmail processes messages on a first-come-first-served basis. This means that when it receives a post to 100,000 subscribers, it will try all the recipients before processing the next message. Often, it is desirable to offload this work to an external host so that the main list host remains responsive to e.g. ``subscribe'' and archive access commands, as well as to other mail is it is not a dedicated mail host. ezmlm-idx allows the main distribution work to be offloaded to an external server via the QMQP protocol. Configure qmail-qmqpc(1) on the list host, and qmail-qmqpd(1) on the mail host (see qmail docs for details), then create the file DDIIRR//qqmmqqppsseerrvveerrss//00. The list housed in DDIIRR will now use the QMQP server for posts, by the local qmail for other messages. If you apply the qmail-qmqpc.tar.gz patch (included in the ezmlm-idx distribution), you can specify the QMQP server IP addresses, one per line, in DDIIRR//qqmmqqppsseerrvveerrss//00, just as you normally would in //vvaarr//qqmmaaiill//ccoonnttrrooll//qqmmqqppsseerrvveerrss. If the first server cannot be contacted, the installation will try the second, and so on. The advantage of controlling the servers locally is that you can specify different servers for different lists. A good idea is to set up also the list host as a QMQP server and use that as the last IP address. This way, the list host will be used if the main QMQP server cannot be contacted. Of course, ezmlm does not loose messages, but rather lets qmail redeliver the post if no QMQP server is available. 44..2200.. HHooww mmeessssaaggeess aarree ssttoorreedd iinn tthhee aarrcchhiivvee.. The structure of the ezmlm list archive is described in the ezmlm(5) manual page. Basically, the message is stored in DDIIRR//aarrcchhiivvee//nn//mm, where ``n'' is the message number divided by 100 and ``m'' the remainder (2 digits). The first message is stored in DDIIRR//aarrcchhiivvee//00//0011. 44..2211.. HHooww tthhee mmeessssaaggee iinnddeexx wwoorrkkss.. The ezmlm-idx(1) adds the option (default) of a message index to ezmlm. The ``From:'' line, the subject, the author's E-mail address and name and the time of receipt are logged for each message as it is received. The subject is ``normalized'' by concatenating split lines and removing reply-indicators such as ``Re:''. A hash of the normalized subject with all white space removed is also stored. The hash for any message within a thread is almost always the same and is used together with the order of receipt to connect a set of messages into a ``thread''. A hash is needed due to the inconsistent handling by MUAs of white space in rfc2047-encoded subject headers. The message index is stored as DDIIRR//aarrcchhiivvee//nn//iinnddeexx, where ``n'' is the message number mod 100. Thus, the directory DDIIRR//aarrcchhiivvee//5522// stores messages 5200 through 5299 and the file ``index'' which contains the index for those messages. The message index can be retrieved with the -index command (see ezmlm- get(1)). You can also retrieve a range of messages, a specific thread, or generate a message digest (see ezmlm-get(1)). Each of these commands can be disabled or restricted as desired by the list owner. The ezmlm-idx(1) can be used at any time to either reconstruct an existing index or create one an index for an existing message archive. without one. 44..2222.. HHooww tthhrreeaaddiinngg wwoorrkkss.. A ezmlm thread is just a message number-ordered set of messages with identical ``normalized'' subject entries. This is a very reliable method for threading messages. It does not rely on any variably present ``In-Reply-To:'' or ``References:'' headers. If the subject changes, the continuation becomes a separate thread very close to the original thread in a digest. ezmlm uses this mechanism to return message sets threaded and with a thread and author index, unless specifically told not to do so with the ``n'' format specifier. Naturally, lists set up without a message index (using the ezmlm-make ``-I'' switch) do not maintain thread information. 44..2233.. HHooww ddiiggeessttss wwoorrkk.. A ``digest'' is just an ordered collection of messages from a list, usually sent out regularly depending on the time and traffic volume since the last digest. Digest subscribers thus can read messages as ``threads'' once daily, rather than receiving a constant trickle of messages. As a major change in ezmlm-idx-0.30, the digest is no longer a totally separate ezmlm-list, but a part of the main list. This has security advantages, makes setup and administration easier, saves space, and allows a consistent way for subscribers of both ``list'' and ``list- digest'' to retrieve missed messages from a single archive. The digest of the list ``list'' is always called ``list-digest''. To set up a list with a digest, simply use the ezmlm-make(1) ``-d'' switch. You subscribe to and unsubscribe from a digest the same way as for the main list, except that the request is sent to e.g. list- digest-subscribe@host rather than to list-subscribe@host. Any option such as remote admin or subscription moderation that is active for the list applies also to the digest list. Any restrictions in posts or archive retrieval set up for the list, automatically accept both subscribers of the main list and of the digest list. The changes in ezmlm-idx>=0.30 allow all programs to service both list and list-digest functions. All digest-specific files are stored in DDIIRR//ddiiggeesstt//. Digest list subscriber addresses in DDIIRR//ddiiggeesstt//ssuubbssccrriibbeerrss// and digest list bounce information in DDIIRR//ddiiggeesstt//bboouunnccee//. Text files are shared between list and digest. To get the local part of the list or list-digest name in a context sensitive manner, use ``<#l#>'' (lower case ``L'') in the text file. In order to generate digest, the list needs to be archived and indexed (both default). You can retrieve sets of messages from the message archive. Such sets are always returned to the SENDER of the request. ``Digests'' are a special form of such a set/request. First, there are no restrictions on the number of messages that can be in a digest (which is balanced by the requirement for a ``digest code'' that needs to be specified in order to create a digest based on a mailed request). Second, special files (DDIIRR//ddiiggiissssuuee and DDIIRR//ddiiggnnuumm) keep track of the digest issue and the message number, amount, and time when the last digest was created. Thus, the system is adapted to make it easy to create the regular collections of messages commonly referred to as ``digests''. Digest can be generated in several different ways: CCoommmmaanndd lliinnee ezmlm-get can be invoked on the command line, or via a script from e.g. crond(8): % ezmlm-get DIR If for some reason the digest should be disseminated via a separate list, the digest can be redirected to a ``target address'' with the ezmlm-get(1) ``-t'' switch. This may be useful if a non-standard digest list name is required. In this case, the list disseminating the digest must be set up as a sublist of the main list (see ``How sublists work''). ffrroomm DDIIRR//eeddiittoorr This is the default and does not require and additional setup. It works well with most lists. The only possible advantage is for very low traffic lists and for lists where it is important that a digest be sent out at a specific time (as DDIIRR//eeddiittoorr digests are triggered only when messages are received). In DDIIRR//eeddiittoorr, ezmlm-get(1) needs to be combined with ezmlm- tstdig(1) so that digests are generated only if certain criteria are met (in this case, more than 30 messages, 64 kbytes of message body or 48 hours since the latest digest). Add these lines after the ezmlm-send line in DDIIRR//eeddiittoorr: |/usr/local/bin/ezmlm/ezmlm-tstdig -t48 -m30 -k64 DIR || exit 99 |/usr/local/bin/ezmlm/ezmlm-get diglist@host DIR || exit 0 To set this up automatically when you create the list: % ezmlm-make -d DIR dot local host [code] Again, the ezmlm-get(1) ``-t'' switch can be used for non-standard arrangements to redirect the digest. The ezmlm-make(1) ``-4'' switch can be used to specify alternative ezmlm-tstdig(1) parame- ters. ffrroomm DDIIRR//mmaannaaggeerr This is useful only if you want digests at specific times, and you do not have access to crond(8) on the list host. ezmlm- get(1) is in it's normal place in DDIIRR//mmaannaaggeerr before ezmlm- manage(1), but a digest code is specified in the ezmlm-get(1) command line. To trigger digests requires a regular trigger messages generated from e.g. crond(8) (see below), but this can be done from _any_ host, not only the list host. ezmlm-make(1) sets up ezmlm-get(1) this way if a digest ``code'' is given as the 5th ezmlm-make(1) command line argument. However, you need to set up the trigger messages separately (see below): % ezmlm-make DIR dot local host code To also test for message volume with this setup, generate trigger messages with the granularity you'd like, and add a ezmlm-tstdig(1) line to DDIIRR//mmaannaaggeerr. E.g., use a trigger message every 3 hours and the following ezmlm-tstdig(1) line before ezmlm-get(1): |/usr/local/bin/ezmlm/ezmlm-tstdig -t24 -m30 -k64 DIR || exit 99 In general, a cron-triggered digest is preferred for very large lists and for lists with very low traffic. Again, the ezmlm-get(1) ``-t'' switch can be used for non-standard arrangements to redirect the digest. For most lists, the digesting from DDIIRR//eeddiittoorr works very well, and does not require any extra setup work. CCoommbbiinnaattiioonn sseettuuppss The default setup in the ezmlmrc(5) file included in the distribution is the DDIIRR//eeddiittoorr triggered setup described above. If you in addition use ezmlm-cron(1) or crond(8) directly to generate trigger messages to list-dig.code@host, you can get regular digests (via the trigger messages and DDIIRR//mmaannaaggeerr), with extra digest sent when traffic is unusually high (via the ezmlm- tstdig/ezmlm-get limits set in DDIIRR//eeddiittoorr). This works best when the time argument on the ezmlm-tstdig(1) command line is the same as the trigger message interval, and the other ezmlm- tstdig(1) parameters are set so that they are only rarely exceeded within the normal digest interval. 44..2244.. HHooww WWWWWW aarrcchhiivvee aacccceessss wwoorrkkss.. If the list is set up with ezmlm-make -i, ezmlm-archive(1) will be invoked from DDIIRR//eeddiittoorr. This program creates indices for threads, subjects, and authors under DDIIRR//aarrcchhiivvee from the iinnddeexx files. ezmlm- cgi(1) is set up per user or globally (see man page) and told about different lists via the //eettcc//eezzmmllmm//eezzccggiirrcc file. ezmlm-cgi(1) presents and used the index created by ezmlm-archive(1) and converts these and the messages to html on-the-fly. To be as efficient as possible, ezmlm-cgi(1) outputs only basic html. However, style sheets are supported and can be used to customize formatting without modification of ezmlm-cgi(1). Extra buttons can be added via the config file. See man page for details. 44..2255.. HHooww eezzmmllmm--ttssttddiigg wwoorrkkss.. ezmlm-tstdig(1) looks at DDIIRR//nnuumm and DDIIRR//ddiiggnnuumm to determine how many messages and how much traffic (in terms of bytes of message body) has arrived to the list since the latest digest. It also determines how much time has passed since the last digest was generated. If any of the criteria specified by command line switches exists, ezmlm- tstdig(1) exits 0, causing the invocation of the next line in the .qmail file. If not, ezmlm-tstdig(1) exits 99 causing qmail to skip the rest of the .qmail file. ezmlm-tstdig(1) looks at LOCAL to determine if it is invoked in the command line, in DDIIRR//eeddiittoorr, or in DDIIRR//mmaannaaggeerr. In the latter two cases, ezmlm-tstdig(1) verifies that the list local address is correct. If invoked in DDIIRR//mmaannaaggeerr, ezmlm- tstdig(1) exits 0 for all action requests except list-dig, so that is does not interfere with the normal functions of ezmlm-get(1) and ezmlm-manage(1). ezmlm-tstdig(1) uses DDIIRR//ttssttddiigg as a flag to avoid problems caused by starting the program when another copy is already running. ezmlm-make(1) automatically configures ezmlm-tstdig(1) with the parameters ``-t48 -m30 -k64'', which can be overridden with the ``-3'' switch. 44..2266.. HHooww ssuubblliissttss wwoorrkk.. ezmlm uses the concept of sublists. Sublists are regular ezmlm lists, except that they only accept messages from their parent list, which is placed in the file DDIIRR//ssuubblliisstt. sublists are used to split the load of a large mailing list among several hosts. All you need to do to set up a local sublist of e.g. the qmail@list.cr.yp.to list is to create a ezmlm list, and put ``qmail@list.cr.yp.to'' into DDIIRR//ssuubblliisstt of you list, and subscribe the sublist to the main qmail list. Now anyone can subscribe to your local list which handles its own bounces, subscribe requests, etc. The load on the main list is only the single message to your local list. Sublists will not add their own mailing list header and they will not add a subject prefix. Normally, sublists will use their own message number, rather than that used by the main list. With ezmlm-idx>=0.23, sublists that are not archived and not indexed, will instead use the main list message number. This way, bounce messages from the sublist can refer the subscriber to the main list archive. This is not done for indexed/archived sublists for security reasons (an attacker could overwrite messages in the sublist archive). With ezmlm-idx>=0.31, there is support for using ezmlm as a sublist of a mailing list run by another mailing list manager. To set this up, set up a normal ezmlm sublist, then edit DDIIRR//eeddiittoorr so that the _e_z_m_l_m_- _s_e_n_d line contains the command line option ``--hh _X_-_L_i_s_t_p_r_o_c_e_s_s_o_r_- _V_e_r_s_i_o_n_:'' (before DDIIRR). As the header text, you need to use a header that the main list manager adds to messages. Now your sublist will accept only messages from the main list requiring that they come from that list _a_n_d contain the header specified. ezmlm-idx>=0.313 also has added protection against the malicious subscription of the ezmlm list to mailing lists run by other list managers. If the ezmlm-reject(1) line in DDIIRR//eeddiittoorr has ``-h'' and ``DDIIRR'' on it, ezmlm-reject(1) will read DDIIRR//hheeaaddeerrrreejjeecctt and reject messages that have any header specified in that file. See the ezmlm- reject(1) man page for suitable headers. 44..2277.. HHooww ssuubblliissttiinngg ccaann bbee mmaaddee ttrraannssppaarreenntt ttoo tthhee uusseerr.. Often you create a local sublist of a list that you do not control. Local users know to subscribe to your local list. However, occasionally, you want to run your own list as a main list and a series of sublists per geographic site, or split onto several hosts if the list is too large to be handled by a single computer. You may also want to split the load of a ``well known'' list host that is getting overwhelmed with traffic. ezmlm supports sublists, but here the fact that the user has to interact with the correct sublist is a problem. What if the user doesn't remember which sublist s/he is subscribed to? What if you change the name of a sublist host or move a sublist to a different host? ezmlm-idx&-0.32 adds ezmlm-split(1), which allows sublisting transparent to the user. This program is invoked before ezmlm- manage(1) in DDIIRR//mmaannaaggeerr. If it detects a subscribe or unsubscribe command, it will forward the command to the appropriate sublist based on a ``split file'' DDIIRR//sspplliitt. This file contains entries, one per line, of the format: domain:lo:hi:sublistname@sublisthost edu:::othersub@otherhost :1:26:third@thirdhost For each address, a hash in the range 0-52 is calculated. The ``domain'' is the last two parts of the host name, reversed. Thus, for id.wustl.edu it would be ``edu.wustl''. The domain is considered to match if the characters in the split file match. It is advisable to use only the last part of the domain for compatibility with the SQL version version (see section ``ezmlm support for SQL datbases''). Thus, any address *@*.domain with a hash between ``lo'' and ``hi'' inclusive would match the first line and be forwarded to sublistname@sublisthost. *@*.edu (independent of hash) would match the second line and be forwarded to othersub@otherhost. Of remaining requests, a request for any target address with a hash between 1 and 26 would be forwarded to the sublist third@thirdhost. Remaining requests would be passed on to the local list. The domain is useful for ``geographic'' splitting, and the hash for load splitting (within a domain). The user interacts only with the main list, and does not need to know from which sublist s/he is serviced. ezmlm-idx sublists use the message number of the main list message if they are not indexed. This allows sublists to in bounce messages refer the subscriber to the main list archive. Use ezmlm-make(1) in conjunction with ezmlmsubrc(5) to set up the sublists. See man pages for further details. Since the addresses are stored locally, the system is very fast and robust, but it is difficult to add new sublists. ezmlm-split(1) -D supports parsing addresses on stdin and splitting them to stdout (see man page). Thus, if you divide the domain of some sublist(s) onto a net set of sublists, you can use ezmlm-list(1) to collect the addresses, ezmlm-split -D with the new split file to split them, then after clearing the local subscriber databases use ezmlm-sub(1) to add the correct addresses to each new sublist. The section on SQL support describes an alternative way of managing sublists (see section ``ezmlm support for SQL datbases''). 44..2288.. HHooww ttoo sseerrvviiccee ccoommmmaannddss iinn tthhee ssuubbjjeecctt lliinnee.. Rfc2142 (standards track) says that for each mailing list list@host, there MUST be an administrative address list-request@host. This is not the default for ezmlm, but can be added with ezmlm-make(1) ``-q'', which adds a ezmlm-request(1) line before the ezmlm-manage(1) line in DDIIRR//mmaannaaggeerr. This address is used to manage commands in the ``Subject:'' line, by translating them into appropriate ezmlm command messages. When migrating from other mailing list managers which use this method to issue list commands, configuring ezmlm to respond to such commands may be useful. In addition, some software manufacturers sell MUAs and mail gateways that are unable to correctly transport rfc822-compliant Internet mail with certain characters in the local part of the address. ezmlm-request(1) services the list-request@host address per rfc2142 (standards track). It is usually invoked in DDIIRR//mmaannaaggeerr before ezmlm- get(1) and ezmlm-manage(1). It ignores all requests that are not for the list-request address. For requests to the list-request@host address, ezmlm-request(1) parses the ``Subject:'' line. If a ezmlm command address starting with the contents of DDIIRR//oouuttllooccaall (e.g. list- get45) is on the command line, ezmlm-request(1) generates the corresponding full ezmlm request message. If the subject does not start with the contents of DDIIRR//oouuttllooccaall, ezmlm-request(1) prefixes the line with the contents of DDIIRR//oouuttllooccaall, thereby building a complete ezmlm command. If a host name is specified, it must match the contents of DDIIRR//oouutthhoosstt, i.e. ezmlm-request(1) in this function will only generate command messages for the local list. Thus, a subject of ``subscribe'' to list-request@host will be auto- magically rewritten as a message to list-subscribe- userlocal=userhost@host. Similarly, any ezmlm command or ``Reply- To:'' address can be pasted into the subject field and sent to list- request@host. ezmlm-request(1) does not validate the command name, but invalid commands result in a ``help'' message in reply via ezmlm- manage(1). This allows ezmlm-request(1) to also service custom commands, like list-faq@host that you may have created for your list. If the ``Subject:'' is empty or does not start with a letter, ezmlm- request(1) will attempt to interpret the first message body line that starts with a letter in the first position. When ezmlm-request(1) has successfully processed a ''request'' command, it exits 99 to skip the rest of DDIIRR//mmaannaaggeerr. To set up a list to include ezmlm-request processing, use the ezmlm- make(1) ``-q'' switch. The default is to not do this. 44..2299.. HHooww ttoo ssuuppppoorrtt aalltteerrnnaattiivvee ccoommmmaanndd nnaammeess.. ezmlm-idx>=0.23 allows alternate names for all user commands. This can be used to e.g. make a message to list-remove@host to result in an ``unsubscribe'' action. This may help migration from other mailing list managers and in non-English environments. The use of aliases allows ezmlm to respond to new command names, while always responding correctly to the standard commands. If ezmlm-request(1) is used it will automatically be able to deal with any commands you set up for the list, within ezmlm or as separate programs. See ``Multiple language support'' on how to set up command aliases. 44..3300.. HHooww ttoo aadddd yyoouurr oowwnn ccoommmmaannddss.. The qmail/ezmlm mechanism makes it very easy to add your own commands. You can add them to DDIIRR//mmaannaaggeerr, but this requires great care in terms of ordering and exit codes. Easier is to set them up separately with a ..qqmmaaiill--lliisstt--ccoommmmaanndd file. Let's assume you want to allow anyone to determine how many subscribers are subscribed to your list with the command list- count@host. Just create a program to do the work: #!/bin/sh DTLINE='Delivered-To: list-count@host processor' grep "$DTLINE" > /dev/null && { echo "This message is looping"; exit 100; } { echo "$DTLINE" cat <>1000) on a busy list, since by default all bounces are stored in a single directory and ezmlm-warn(1) examines all of them with each invocation. ezmlm-idx->=0.32 changes bounce handling to improve performance for large lists. Bounces are stored in subdirectories of DDIIRR//bboouunnccee//dd//, one per 10,000 seconds. The corresponding address hashes are stored in 16 subdirectories of DDIIRR//bboouunnccee//hh//. Instead of looking at all bounces, ezmlm-warn(1) processes only the bounces in DDIIRR//bboouunnccee//dd// subdirectories that are ``due''. In addition, ezmlm- warn(1) uses DDIIRR//bboouunnccee//llaassttdd as a simple lockout, to assure that it will do work only at most once every 5.5 hours. (Times are scaled to the ezmlm-warn(1) ``-t'' argument if used.) Together, these changes assure that bounce handling will scale well in the default configuration, even for very large lists. 44..3388.. HHooww tthhee iinnffoo aanndd ffaaqq ccoommmmaannddss wwoorrkk.. The _-_i_n_f_o and _-_f_a_q commands simply reply with the contents of the DDIIRR//tteexxtt//iinnffoo and DDIIRR//tteexxtt//ffaaqq files. Edit these files directly or remotely (see ``How to remotely edit dir/text files''). The DDIIRR//tteexxtt//iinnffoo file should start with a single line that is meaningful as is and describes the list. This will be used in later versions to allow automatic assembly of the global ``list-of-lists'' (see ``How to set up a global list address like majordomo@host or listserv@host''). 44..3399.. HHooww tthhee gglloobbaall eezzmmllmm lliisstt aaddddrreessss wwoorrkkss.. Sometimes, it is desirable to have a host- or user-wide address that can list available mailing lists. ezmlm-request(1) can be used to set up a global address, such as ezmlm@host which allows the user to see and interact with a number of different mailing lists. This is especially useful when your users are used to other mailing list managers, such as ``majordomo'' or ``listproc''. ezmlm-request(1) is set up to answer requests to the address (see ``How to set up a global list address like majordomo@host or listserv@host''). There, it interprets the first line of the message body as a command. It will reply directly to ``lists'' and ``which'' commands. All other commands will be used to construct messages to the respective lists. Where other mailing list managers use synonyms of ezmlm commands, ezmlm-request(1) recognizes these and translates them to the corresponding ezmlm commands. ezmlm-request(1) will build commands also of unrecognized commands. Thus, if you create new commands for a list, ezmlm-request(1) will automatically support them. If the user does not specify the complete list address, ezmlm- request(1) will attempt to complete the name. See the ezmlm-reject(1) man page for more info. 44..4400.. HHooww eezzmmllmm--ccrroonn wwoorrkkss.. If you are a user and have crond(8) access, if you do not need to get digests at specific times, or if you are a system administrator setting up lists, there is no reason for you to use ezmlm-cron(1). If you are a system administrator not allowing users crond(8) access or a user that needs digests at specific times, but without crond(8) access, read on. ezmlm-cron(1) is a very restrictive interface to crond(8). ezmlm- cron(1) can be used to create digest trigger messages. If a list is set up with a digest code (see ezmlm-make(1) and ezmlm-get(1)) ezmlm will generate a digest from the list joe-sos@host sent to to subscribers of joe-sos-digest@dighost when receiving a message to joe- sos-dig-code@host where ``code'' is the digest code. ezmlm-cron(1) can be used to generate such messages at regular intervals. The file eezzccrroonnrrcc is set up by the sysadmin and controls what trigger messages specific users may set up via ezmlm-cron(1). Usually, the ezcronrc of that use will have an entry like ``user:user-:host:10'' allowing ``user'' to create trigger messages for up to 10 lists with names starting with ``user-'' and on the host ``host''. To list the ezcronrc line controlling your use of ezmlm-cron(1): % ezmlm-cron -c To list all entries that you've created: % ezmlm-cron -l To add an entry to trigger digests from list@host every morning at 0230: % ezmlm-cron -t 02:30 -i24 list@host code A new entry for the same list overwrites an old entry. To delete the entry above: % ezmlm-cron -d list@host or use ezmlm-cron to trigger messages at a different time: % ezmlm-cron -t 16:16 -i24 list@host code 44..4411.. HHooww eezzmmllmm--mmaakkee wwoorrkkss.. ezmlm lists allow almost infinite customization. The component build, together with the qmail delivery mechanism makes it possible to create any variant of list function imaginable. However, this complexity makes it somewhat daunting to the average user wanting to set up a mailing list. ezmlm-make(1) allows automated list setup, while permitting a large amount of configurability. At first glance, ezmlm-make(1) has many complicated options. However, these can be applied iteratively through the ezmlm-make(1) edit mechanism. Also, they are intended to be relatively complete so that execution of ezmlm-make(1) by e.g. a GUI can be used to safely set up and edit any list. ezmlm-make(1) reads its command line arguments and switches, then creates the list directory. If the ``-e'' edit or ``-+'' sticky edit switches are not specified, ezmlm-make(1) will fail if the directory already exists. The directory argument must be an absolute path starting with a slash. The dot-qmail file argument, if specified, must also be absolute. ezmlm-make(1) next reads ezmlmrc(5) located in the //eettcc// directory with a default install. If not found, the file in the ezmlm binary directory will be used. The second ezmlm-make command line argument specify the root name of the .qmail files. If the ezmlm-make(1) ``-c'' switch is used, ezmlm-make(1) will look in that directory for a ..eezzmmllmmrrcc file and use it instead. If this file does not exist, ezmlm- make(1) will print a warning and use the previously discussed ezmlmrc(5) files in the same order. You can also use ``-C _e_z_m_l_m_r_c_._a_l_t'' to use _e_z_m_l_m_r_c_._a_l_t as the ezmlmrc(5) file. Again, ezmlm- make(1) will fall back to the others with a warning, if the specified ezmlmrc(5) file is not found. When not run in ``-e edit'' or ``-+'' sticky edit modes, ezmlm-make(1) first creates the list directory. It also as the last step of its action creates DDIIRR//kkeeyy containing the key used for cookie generation. The ezmlmrc(5) file consists of a number of file names relative to the list directory, followed by conditional flags (see ezmlm-make(1) and ezmlmrc(5) for details). If all the conditional flags (controlled by the corresponding command line switches) are true, the lines that follow are entered into the named file. There are also tags to erase files. Tags in the format <#X#> (where ``X'' is any number, except ``1'' and ``2'') are replaced by the corresponding ezmlm-make(1) switch argument. The ezmlm-make(1) command line arguments and the ezmlm binary path can be similarly substituted into the text. Thus, ezmlmrc(5) controls (within reason) the entire operation of ezmlm- make(1). ezmlmrc(5) is also set up so that no messages or file containing list state information are lost. Therefore, ezmlm-make(1) can be used to safely edit existing lists. The only caveat is that the list state is undefined while editing is in progress. Thus, it is advisable to prevent mail delivery by setting the ``sticky'' bit on the user's home directory while editing lists. ezmlm-make(1) will create the file DDIIRR//ccoonnffiigg. This files saves all the flags that were set at the last execution of ezmlm-make, as well as all the switch and command line arguments. When editing a list, only ``DIR'' and the non-default letter switches need to be specified. Other command line arguments and the ``digit switch'' arguments are read from DDIIRR//ccoonnffiigg. To remove a digit switch, simply use it with two single quotes as the argument. You can also easily determine how a list was set up by looking at DDIIRR//ccoonnffiigg. _N_o_t_e_: DDIIRR//tteexxtt// files will be created but not overwritten when using the ``-e'' or ``-+'' edit switches. This is to preserve manual customizations. To overwrite these and reset the files to the content specified by eezzmmllmmrrcc, use ``-ee'' or ``-++''. _N_o_t_e_: As of ezmlm-idx-0.40 the ezmlm-make(1) ``-c'' and ``-C file'' switches are sticky when using ``-+'' or ``-++'', so you do not need to specify them. This feature is disabled if ezmlm-make(1) is run as root. 44..4422.. WWhhaatt nnaammeess ccaann II uussee ffoorr mmyy lliissttss?? Rather than restrict you to a single E-mail address (user@host), qmail in the default setup gives you control over an infinite number of addresses user-*@host. Of course, you (normally) have no way of controlling elsewhere@host since that could lead to overlap between users' ``e-mail address space''. As a consequence, all you mailing lists have to be named user-xx@host where ``user'' is your user name and ``xx'' is anything. You cannot create e.g. mylist@host, only user- mylist@host. To create the list user-list@host do: % ezmlm-make ~/list ~/.qmail-list user-list host Notice that ``user'' is nnoott part of the ..qqmmaaiill file name. There are two way to create lists with names not starting with your user name: First, qmail can be set up so that you control a virtual domain (see below). Second, the system administrator can set up lists with arbitrary names within the ~~aalliiaass// directory. 44..4433.. LLiissttss iinn vviirrttuuaall ddoommaaiinnss If you use qmail>=1.02 and ezmlm-idx>=0.32, lists under virtual domains work just like other lists and require no adjustments. You can choose any local name for the list and the ezmlm-make(1) argument ``local'' is that name; ``host'' is the name of the virtual domain. 44..4444.. HHooww ddoo II mmaakkee ccuussttoommiizzaattiioonn ssiimmppllee ffoorr mmee//mmyy uusseerrss?? All non-default switches, ezmlm-issubn(1) setups, etc, can be made standard for new lists by customizing the ezmlm-make(1) configuration file named ``eezzmmllmmrrcc''. A default eezzmmllmmrrcc((55)) is installed in the ezmlm binary directory. If installed, a system-wide customized ezmlmrc file in //eettcc//eezzmmllmmrrcc (or symlinked from there) overrides this. Installing a ~~//..eezzmmllmmrrcc file in a user ddoottddiirr and using the ezmlm- make(1) ``-c'' switch allows further per user customization (see ``Customizing ezmlm-make operation''). 55.. eezzmmllmm ssuuppppoorrtt ffoorr SSQQLL ddaattaabbaasseess.. 55..11.. WWhhyy uussee aann SSQQLL ddaattaabbaassee wwiitthh eezzmmllmm?? The main advantages are that you are using an address database system that can easily be accessed from any number of other programs via ODBC, perl, java, PHP, ... You can easily hook up ezmlm with your customer database, etc. ezmlm programs compiled with SQL support (and when available also those compiled with support for other SQL servers) are entirely backwards compatible. You can mix SQL dbs with normal ezmlm dbs, and convert lists between them. 55..22.. WWhhyy nnoott ttoo uussee aann SSQQLL ddaattaabbaassee wwiitthh eezzmmllmm.. The main disadvantages of the SQL version are that you need to be familiar with the SQL server, the binaries are quite a bit larger, and you are trusting your addresses to a large database program, rather than a small and easily audited set of ezmlm programs. Also, the SQL server becomes a single point of failure. Ezmlm with SQL support continues to rely on qmail stability. If connection fails, ezmlm aborts with a temporary error causing redelivery at a later time point. 55..33.. TTaabblleess uusseedd ffoorr ((MMyy))SSQQLL ssuuppppoorrtt.. The basic philosophy is that the database can be on any host (if you use SENDER restrictions, connectivity to the main host is more important than to the sublists), and you choose the database and ``table root'' names. The default database is ``ezmlm'' and the default table root is ``list''. Each list has a separate table root. Any number of lists can share a database. The main list address table is named with the table root only, others have that name with various suffixes. In the following ``list'' is used as the table root. 55..33..11.. AAddddrreessss ttaabblleess.. lliisstt List subscriber addresses. lliisstt__ddiiggeesstt Digest list subscriber addresses. lliisstt__aallllooww List subscriber alias addresses. Used only if SENDER restrictions are used for the list. This is configured in the default SQL list setup, but a local (ezmlm-style non-SQL) database could also be used. lliisstt__ddeennyy List deny addresses. This table is created, but the default configuration, if it uses the ``deny'' addresses at all, will do so with a local database. lliisstt__mmoodd Moderator addresses. Created for completeness, but not used in the default configuration. If moderators are used, the addresses are stored in a local database. 55..33..22.. SSuubbssccrriibbeerr lloogg ttaabblleess.. For each of the above tables, there is a ``*_slog'' table that contains one row per transaction against the corresponding address table. The entries contain a time stamp, the subscription address; a direction indicator (``-'' for removals, ``+'' for additions); a type indicator (blank for ezmlm-manage, ``m'' for ``manual'', ``p'' for ``probe, i.e. bounce handling; and the subscriber ``From:'' line contents (only additions and only when made by ezmlm-manage or by ``ezmlm-sub(1) -n''). 55..33..33.. MMeessssaaggee llooggggiinngg ttaabblleess.. For both the list and the digest list, there are a pair of tables that log messages: lliisstt__ccooookkiiee The main list stores the message number and a pseudo-random cookie in this table when it processes the message. The cookie is derived from the secret DDIIRR//kkeeyy, the message sender and the message number. Thus, it is non-repeating and virtually impossible to guess beforehand. Sublists will check that the cookie sent with the message is the same as the one received with the message. The digest list is created similarly, except that it is ezmlm- get(1) that originates the message and creates the cookie. This is done in ``list_digest_cookie''. lliisstt__mmlloogg Both the main list and the sublists make entries in this table. Each entry consists of a time stamp, a message number, a list number, and a code. The code is 0 for message arrival, 1 for ``finished processing'', 2 for ``receipt received'' and -1 for bounce. The lists will refuse to process messages that do not have the correct cookie, or if the message already has an entry with a code of greater than 0. To inject a message at the sublist, an attacker would have to inject a message with the correct code before the list has processed the ``real'' message, or subvert the SQL server. In practice, this is very hard to do, unless the attacker has broken security at the database server or a sublist. This authentication mechanism is intended to make it safe to sublist moderated lists. It also blocks any message duplication between main list and sublist from being propagated to the subscribers. The codes 2 for ``receipt received'' and -1 for bounce are entered by ezmlm-receipt(1) at the main list. This program is configured instead of ezmlm-return(1) if the main list was set up with ``ezmlm-make -w6''. ezmlm-receipt(1) checks the cookie of messages addresses to mainlocal-return-receipt@mainhost and if correct enters the ``receipt received'' code. This address is normally in the subscriber database with a hash of 98, so that each list sends a message to the address _a_f_t_e_r all subscriber addresses. Bounces of sublist messages should not lead to removal of the sublist from the database. ezmlm-receipt(1) will instead log the bounce to the ``list_mlog'' table. It will also store up to 50 bounces in the bounce directory. This helps error detection and diagnosis. After the first 50 bounces, no more bounces are stored, until you manually remove the old ones. This is to prevent filling up your hard disk in case a configuration error causes a deluge of bounces. The digest list is treated in the same manner. Here, the tables is ``list_digest_mlog'' and the feedback address is mainlocal- digest-return-receipt@mainhost. 55..44.. HHooww ttoo sseett uupp aa ssiimmppllee lliisstt wwiitthh SSQQLL ssuuppppoorrtt.. To use SQL database support, you have to compile the programs with SQL support. Currently, only MySQL support is available. See IINNSSTTAALLLL..iiddxx in the package on how to do this. The programs with SQL support will work exactly like the normal programs for standard lists. However, if the file ssqqll exists in the basedir, it turns on the SQL mode and it is expected to contain SQL server connect info in the format ``host:port:user:password:database:table'' Here, ``Host'' is the SQL database server host, ``port'' can be left blank to use the default port, ``user'' and ``password'' are connec- tion credentials for a user you need to define and grant access to the database. ``Table'' is the name of the address table (``list'' in the examples above and ``list_digest'' for the corresponding digest list). For list clusters, ``:sublist'' is suffixed to this info and it is the name/address of the sublist. For each address database, you also need to create the address table as well as the ``*_slog'' subscription log table. In addition, you should create a ``*_cookie'' and ``*_mlog'' table for message logging. This is all it takes to start using an SQL database. 55..44..11.. HHeellppeerr pprrooggrraammss ffoorr SSQQLL--eennaabblleedd lliissttss.. Two programs are supplied in the distribution to make it easier to create the database user and tables. Also, ezmlm-make(1) has support for setting up SQL-enabled lists. CCrreeaattiinngg tthhee ttaabblleess ezmlm-mktab(1) will create the necessary tables: % ezmlm-mktab -d table Pipe this into the SQL client with the appropriate administrator credentials needed to create tables (see MySQL documentation, e.g. ). For most lists, the only addresses that are stored in the SQL database are the subscribers of list and digest, and the ``allow'' aliases. It is NOT normally advisable to store moderator addresses there, since they are needed only at the main list and secrecy is more important. ``Deny'' addresses are few and again only needed at the main list. ``Allow'' are put in the SQL database when using the default ezmlmrc file only to make all relevant addresses manipulatable via the SQL server. The other tables are created, in case they are wanted (the cost for having them as empty table is zero). The basedir/sql file is the decision point. If it exists, an SQL table is used; if not a local ezmlm db is used. CCrreeaattiinngg aa uusseerr eennttrryy Create a user that has full access to the database from the list host. How to do this depends on the RDBMS. CCrreeaattiinngg tthhee lliisstt ezmlm-make(1) supports SQL-enabled lists with the ``-6'' switch: % ezmlm-make other_switches -6 'host:port:user:pw:db:table' \ dir dot local host Will create an SQL-enabled list that uses the SQL server for the main list subscribers, digest list subscribers (if configured) and ``allow'' poster alias addresses (if configured). 55..55.. MMaannuuaallllyy mmaanniippuullaattiinngg tthhee ssuubbssccrriibbeerrss ooff aa SSQQLL--eennaabblleedd lliisstt.. ezmlm-sub(1), ezmlm-unsub(1), and ezmlm-list(1) work as you would expect also with a SQL-enabled list. ezmlm-list(1) may be minimally slower (depending on network speed) if the SQL server is not local. ezmlm-sub(1) and ezmlm-unsub(1) will be faster, but this is noticeable only with very large subscriber lists and addition/removal of large numbers of addresses (more than several thousands). 55..66.. CCoonnvveerrttiinngg ttoo aanndd ffrroomm aanndd SSQQLL ddaattaabbaassee.. Just like other programs, ezmlm-list(1), ezmlm-sub(1), and ezmlm- unsub(1) will work with normal address databases in the absence of DDIIRR//ssqqll. However, they also have a ``-M'' switch to force this behavior even in the presence of DDIIRR//ssqqll. This is used to convert an address database from the standard type to the SQL type: % ezmlm-list -M dir | xargs ezmlm-sub dir or from the SQL version to the standard type: % ezmlm-list dir | xargs ezmlm-sub -M dir To synchronize the two, remove one and then update it with ezmlm- sub(1) from the other. Alternatively, sort the ezmlm-list(1) output for both, use diff and sed/awk to get separate files of the differ- ences, and use ezmlm-sub(1) and ezmlm-unsub(1) to apply the differ- ences to the appropriate database. This type of conversion can serve as a convenient means to convert a list from one type to another, to back up databases, and to move subscriber addresses from a standard list to a SQL table for other purposes, or from a SQL database to a standard mailing list (you may need to use addresses from a SQL table, without wanting your lists to be dependent on an SQL server for day to day operation). _N_o_t_e_: This inter-conversion requires the DDIIRR//ssqqll file. If you do not run the list against an SQL server, you need to disable deliveries before you temporarily create this file. Otherwise, the list will run against the SQL database during the time DDIIRR//ssqqll exists. 55..77.. OOppttiimmiizziinngg MMyySSQQLL ffoorr eezzmmllmm.. 55..77..11.. AAddddrreessss SSEELLEECCTTss,, aaddddiittiioonnss,, rreemmoovvaallss.. ezmlm-idx-0.40 simplifies the SQL support and queries over ezmlm- idx-0.32 at the cost of dropping distributed sublist support. We have figured out a simpler way to support the latter, which hopefully will be incorporated into ezmlm in the future (written under contract). With the simplification, the queries are very straight forward, and tuning is indicated only under extreme circumstances (very many very large and busy lists or constant addition/removal of many addresses). 55..88.. MMaaiinntteennaannccee ooff tthhee MMyySSQQLL ddaattaabbaassee.. Weekly to monthly error checks on MySQL tables is recommended. Best is to use: # isamchk -s -O readbuffer=2M */*.ISM Other options allow automatic correction of errors, but are dangerous if tables are accessed while isamchk is running. Other isamchk options allow recovery of space after frequent insert/delete of addresses (can also be done with ``OPTIMIZE TABLE''), key optimization, etc. See the MySQL documentation ( ) for more info. 66.. PPoossssiibbllee eerrrroorr ccoonnddiittiioonnss iinn eezzmmllmm lliissttss.. 66..11.. WWhhaatt ddoo II ddoo iiff eezzmmllmm ddooeessnn''tt wwoorrkk?? Try to determine where the problem occurs and how to reproduce it: +o Do messages to ezmlm return an error message to the sender or not? +o What is/are the error message(s)? +o What does ezmlm log into the mail log? +o Are you using a setup with virtual domains, and qmail<1.02 or ezmlm-idx<0.31? If so, have you adjusted DDIIRR//iinnllooccaall (see ``Adapting ezmlm-make for virtual domains'')? +o Are posts sent out to the subscribers? +o Are there subscribers? % ezmlm-list DIR +o Are there moderators? % ezmlm-list moddir where ``moddir'' is the contents of DDIIRR//rreemmoottee (for remote admin lists), of DDIIRR//mmooddssuubb (for subscription moderated lists) or DDIIRR//mmoodd-- ppoosstt (for message moderation), if and only if the contents start with a forward slash. The default in all cases is DDIIRR//mmoodd//. If both DDIIRR//mmooddssuubb and DDIIRR//rreemmoottee contain directory names, the one in DDIIRR//mmoodd-- ssuubb is used for both subscription moderation and remote admin. +o Are the ownerships of all files correct, i.e. read/writable for the owner? % chown -R user DIR For lists under alias: % chown -R alias DIR If you use custom moderator databases, those directories and all their contents must also be readable for the user under which the list oper- ates (i.e. the user qmail changes to during the delivery). +o Read the qmail log and capture relevant parts. +o Did you customize the package at all? If so, try the default settings which are known to work. +o Did you customize eezzmmllmmrrcc((55))? Try to use the default copy (skip the -c switch). +o Did your customization of ..eezzmmllmmrrcc fail to have an effect? Remember to use the -c switch. The ..eezzmmllmmrrcc file used is the one in ``dotdir'', i.e. the directory where the ..qqmmaaiill files go, usually, but NOT necessarily, the one in your home directory. +o Make sure you followed the instructions in man pages and other documentati