[Bug 34927] New: Adding DMARC compatibility to mailing lists
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=34927 Bug ID: 34927 Summary: Adding DMARC compatibility to mailing lists Change sponsored?: --- Product: Koha Version: unspecified Hardware: All OS: All Status: NEW Severity: enhancement Priority: P5 - low Component: Websites, Mailing Lists, etc Assignee: wizzyrea@gmail.com Reporter: td-koha-bugs@agogme.com QA Contact: testopia@bugs.koha-community.org The issue of DMARC support for mailing lists is under consideration again after languishing for a while for time limitation as a recurring topic in Koha IRC development meetings now with Jitsi. There is an interest in knowing the availability and willingness of people who administer the mailing lists to support DMARC compatibility when sending messages from the mailing list to subscribers. Email is still not designed for significant authentication which leads some mailing list subscribers with email addresses on DMARC compliant systems to not receive email from Koha lists. The issue has been raised on the Koha mailing list and a vote has been pending about implementing DMARC for sending messages on the mailing lists. See https://lists.katipo.co.nz/pipermail/koha/2023-February/059036.html . The issue is about how mailing list messages are sent with the mailing list address authenticated and avoid having the mailing list server impersonate the author of the mailing list email message. No one is suggesting blocking mailing list subscribers who do not use DMARC which could be a substantial minority and is unrelated to the problem of mailing list impersonation of the message author. Mailing list impersonation of the message author is what is what mailing lists generally do by default when sending messages and is the problem for DMARC. The easiest way to obtain DMARC compatibility for Mailman 2 or 3 without the risk of breaking the message body for email clients is munging the from header, https://wiki.list.org/DEV/DMARC . MIME wrapping may be more elegant but seems only suitable for a company mailing list which users only read with a company supplied email client. In checking for actual implementations in the world, I did not find anyone reporting the choice of MIME wrapping. Having the message appear as only an attachment with an empty message body in an email client would not be good. Some major webmail provider might have avoided the problem but then break it for everyone for months without a care because it is too small a use case for their attention testing in addition to the issue that the webmail client is merely a gateway for many people. Using forum software as an alternative has been disfavoured by a few people commenting on the mailing list. If people would want forum software, Hyperkitty is part of Mailman 3 and better suited to mailing list compatibility than Discourse or other forum software. See https://lists.katipo.co.nz/pipermail/koha/2023-April/059289.html . -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=34927 Jonathan Druart <jonathan.druart+koha@gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |jonathan.druart+koha@gmail. | |com --- Comment #1 from Jonathan Druart <jonathan.druart+koha@gmail.com> --- Looks like I have access to the administration page of the koha-devel ML I see the following option: """ Replace the From: header address with the list's posting address to mitigate issues stemming from the original From: domain's DMARC or similar policies. (Details for from_is_list) * No Munge From Wrap Message """ Do you want me to change it? -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=34927 --- Comment #2 from Thomas Dukleth <td-koha-bugs@agogme.com> --- "Munge From" would be the recommended option for DMARC compatibility, however, you should announce your intention on the list to make the change as there will be difference from what people have come to expect in managing messages where messages will in future have the from header correspond to the mailing list email address and not the message author's email address. The DNS record for lists.koha-community.org seems to be set up correctly for DMARC. Please avoid the Wrap Message option. While MIME wrapping may be the more elegant solution, it can break some email clients message parsing with the message body parsed as an attachment to a message with an empty body. The number of email clients is relatively large. For people using popular proprietary webmail systems, if they are not broken today they may be tomorrow and fixing the issue for an extreme minority mailing list configuration may be perpetually at the bottom of the queue for fixing. I did not find anyone reporting using the MIME wrapping option presumably for the obvious reason of not wanting to break message parsing for some users' email clients. -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=34927 --- Comment #3 from Jonathan Druart <jonathan.druart+koha@gmail.com> --- I don't have any intentions but I can change the option if you want and if you think we should. Just let me know when you are ready. -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=34927 --- Comment #4 from Thomas Dukleth <td-koha-bugs@agogme.com> --- While there seems to be a consensus for adding DMARC support to all the mailing lists, the development meetings request has been to contact people and see whether someone with access was prepared to do the necessary work to alter the configuration and may be trivial. Explicit blessing for koha-devel and other lists should wait upon the next development meeting where people can be prepared to test the result. We should probably wait until after the next meeting. It may be better to try first with a Biblibre hosted mailing list, such as koha-devel, if it is easier to have attention to revert if a problem is identified and fewer people would be affected adversely. Mailman 2 Configuration. When DMARC is activated we should set both of the following variables. dmarc_is_list set to "Munge From". dmarc_moderation_action set to "Munge From". Preexisting or default values for other DMARC related variables should be fine. Koha General Mailing List. The only particular complaint about lack of DMARC support was for the general Koha mailing list where someone has a work institution which has a strict DMARC policy. The Katipo Mailman 2.1.18 installation maybe somewhat older than the Biblibre Mailman 2.1.26 installation and could use an upgrade but does not seem to have a critical problem for Mailman 2 DMARC support. We could find minor bug 1536816 for some messages in 2.1.18 which was fixed in 2.1.21rc1 is that non-ASCII names pulled from the registered user list when absent in the originating From header. See https://bugs.launchpad.net/mailman/+bug/1536816 . The DNS configuration for lists.katipo.co.nz needs the following line: _dmarc.lists.katipo.co.nz. IN TXT "v=DMARC1; p=none" ... if using BIND as DNS server or the equivalent where the policy is "p=none" matching the Biblibre DNS configuration and allows messages to the mailing list from non-DMARC compliant systems. -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=34927 --- Comment #5 from Thomas Dukleth <td-koha-bugs@agogme.com> --- Mailman bug 1334450 is another bug which we might find with Mailman 2.1.18 installed at Katipo was fixed in 2.1.19rc1. If a message is held for moderation, dmarc_moderation_action of munge From header is not applied after approval and thus the approved message would not be sent as DMARC. See https://bugs.launchpad.net/mailman/+bug/1334450 . Assuming messages are seldom held for moderation, bug 1334450 is not a problem which should necessarily prevent turning on DMARC for those who need it and are otherwise missing access to the Koha general mailing list from a work environment with a strict DMARC policy. Thus, waiting on upgrade to a more recent version of Mailman at Katipo and/or a possible administrative change to the mailing list provide such time should not be necessary. -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=34927 --- Comment #6 from Jonathan Druart <jonathan.druart+koha@gmail.com> --- #info Joubu could change settings on the Bblibre hosted lists Update of the following option from "No" to "Munge From" for koha-devel """ Replace the From: header address with the list's posting address to mitigate issues stemming from the original From: domain's DMARC or similar policies. (Details for from_is_list) No Munge From Wrap Message """ -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=34927 --- Comment #7 from Jonathan Druart <jonathan.druart+koha@gmail.com> --- And from "Accept" to "Munge From": """ Action to take when anyone posts to the list from a domain with a DMARC Reject/Quarantine Policy. (Details for dmarc_moderation_action) Accept Munge From Wrap Message Reject Discard """ -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=34927 --- Comment #8 from Thomas Dukleth <td-koha-bugs@agogme.com> --- 1. DKIM Not Working Properly for the Mailing List. For DMARC which is dependent upon SPF and DKIM, the DKIM signature should be re-signed using the mailing list sender, such as koha-devel@lists.koha-community.org . DKIM signatures from the message author should not validate because we change the From header, moving the original From header to Reply-To , adding "[koha-devel]" to the subject header, and add a mailing list footer to the body. I wrote about the issue on the koha-devel mailing list, "[Koha-devel] DMARC testing for koha-devel" - https://lists.koha-community.org/pipermail/koha-devel/2023-November/048405.h... . It is currently evident as probably expected that the DKIM signatures in the message headers are not being re-signed for lists.koha-community.org . Messages retain the DKIM signature of the original author and would not validate properly even without DMARC changes because every message is altered slightly for the mailing list. Such a DKIM signature problem may have been more of an issue in past years where popular mail systems had adopted excessive strictness in rejecting messages. Mailman 3 has code which might address the issue better when setting remove_dkim_headers to yes but removing the header does not ensure that the SMTP server will supply an appropriate DKIM header for the mailing list. Re-signing the outgoing message can be fixed by forcing Mailman to send to a special port for which messages are designated as originating from the system and re-signed. The mailing lists are also not DKIM signing the monthly mailing list membership messages despite a DKIM signature reported for lists.koha-community.org . I have communicated with people at BibLibre in recent weeks about the configuration details for fixing DKIM signing and mailing list message re-signing. The most important details of how to fix such a system as I sent it to people at BibLibre follow below but I have not had confirmation confirmation that Postfix as the Mailman default and most likely choice is what they are actually using. I had sent other details about fixing DKIM signing more generally for lists.koha-community.org which is broken. Priority had been low but I have discovered announcements from Google and Yahoo about changes coming in February 2024 which raise priority. 2. Settings for Mailing List DKIM Re-Signing. Postfix can be forced to re-sign mailing list messages by having Mailman send them to an additional Postfix listening port listening only on localhost set in /etc/postfix/master.cf and /etc/mailman/mm_cfg.py. Using a special port for assuring that messages are originating from the system and marked as such for DKIM using " -o milter_macro_daemon_name=ORIGINATING". Getting DKIM to work with Mailman and Postfix - https://groups.google.com/g/mailing.postfix.users/c/6xtGQNnfEWA . 2.1. Postfix master.cf Settings for Mailing List DKIM Re-Signing. In /etc/postfix/master.cf # Default smtp listening on port 25 unchanged. smtp inet n - - - - smtpd # Additional smtp listening port for resubmitting from Mailman to allow DKIM # re-signing. Connection over localhost 127.0.0.1:26 inet n - - - - smtpd # Mailman 2.1.26 running for lists.koha-community.org may # have sufficient authentication support that disabling Mailman authentication # for Postfix port 26 on localhost is not necessary # Use mailman authentication only if it is already being used. # Mailman 3 has much better authentication support # Uncomment if Mailman authentication fails. # -o smtpd_tls_security_level=none # Uncomment if Mailman authentication fails. # -o smtpd_sasl_auth_enable=no # Uncomment if Mailman authentication fails. # -o smtpd_client_restrictions= # Uncomment if Mailman authentication fails. # -o smtpd_helo_restrictions= # Uncomment if Mailman authentication fails. # -o smtpd_sender_restrictions= -o smtpd_recipient_restrictions=permit_mynetworks,reject # Uncomment if Mailman authentication fails. # -o smtpd_data_restrictions= -o milter_macro_daemon_name=ORIGINATING -o syslog_name=postfix/p26 2.2. Mailman mm_cfg.py Settings for Mailing List DKIM Re-Signing. In /etc/mailman/mm_cfg.py HOST = 'localhost' SMTPPORT = 26 # Changed to 26 for resubmitting to Postfix. MTA = 'Postfix' # Probably Postfix which may be the default if not set. # Check the configuration for authentication options if any. 2.3. OpenDKIM opendkim.cfg Settings for Mailing List DKIM Re-Signing. In /etc/opendkim.cfg # Sets the expected designation from the MTA to force signing of messages. MTA ORIGINATING -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=34927 --- Comment #9 from Thomas Dukleth <td-koha-bugs@agogme.com> --- By February 2024, fixing DKIM re-signing may be needed for lists.koha-community.org and more for the Koha genral mailing list when Gmail and Yahoo mail [with AOL] may start blocking messages for not being DKIM re-signed when sent from the mailing lists. It is uncertain whether Gmail and Yahoo mail will continue to give a false pass for messages using the original author's DKIM signature which does not match the mailing list sending server, nor the From header with DMARC support enabled. Gmail, Yahoo mail, AOL mail, Microsoft [Exchange and various names] etc. have all rejected messages for bad DKIM signature from mailing lists during past periods of extra DKIM strictness. I have given precise technical details of changes which may be made to resolve the issue of OpenDKIM signing with people at BibLibre responsible for lists.koha.org . I have also communicated with Rachael Rachel Hamilton-Williams about adding DMARC support for the Koha general mailing list. Rachel informed me that she would like to hand over hosting of the Koha mailing list for more attentive hosting than her partner is able to provide. At the end of last week, a radio broadcast brought my attention to changes coming in February 2024 which affect everyone in some manner. For large volume senders there additional requirements, beyond those affecting everyone, which might affect people subscribing to the Koha general mailing list if the number of mailing list subscribers is enough and enough people route mail through the popular choice of Gmail even on mobile despite having some other apparent domain of some subscriber wherever that may be hosted initially. Details about more stringent SPF, DKIM, DMARC, ARC, and one-click unsubscribe link requirements are available from Google. Mailing lists may be able to substitute DMARC support for lack of ARC support when rewriting the From header and thus re-originating and not merely forwarding messages but adding ARC is best addressed second. "Email sender guidelines : Requirements for all senders" - https://support.google.com/mail/answer/81126#zippy=%2Crequirements-for-all-s... . Another part of the same document has the requirements which may affect the Koha general mailing list "Requirements for sending 5,000 or more messages per day" - https://support.google.com/mail/answer/81126#requirements-5k&zippy=%2Crequirements-for-sending-or-more-messages-per-day . The Yahoo guide which I found has fewer details and does not refer to the coming February 2024 policy change: "Sender Best Practices" - https://senders.yahooinc.com/best-practices/ . There is no shortage of secondary sources such as from the support provider Proofpoint, "Google and Yahoo Set a Short Timeline to Meet New DMARC Policy & Setup Requirements. Are You Ready?" - https://www.proofpoint.com/us/blog/email-and-cloud-threats/google-and-yahoo-... . The general Koha mailing list may also have enough subscribers for which the most stringent requirements will be set. Sidenote on ARC Support. ARC is intended for authenticating the email chain when forwarding messages which is the basic function of mailing lists. Adding DMARC support should make the issue of ARC support for acceptable authentication for mailing lists might be moot because the mailing list is more clearly shown as re-originating email and not merely forwarding. However, the announcements for February 2024 do not state that case with explicit clarity and Gmail adds ARC headers to all mail on their system and people at Google may presume that everyone else should to especially when messages may retain headers showing that the message has been forwarded over the mailing list despite having been re-originated from mailing list with DMARC authentication. While Mailman 3 has functionality for ARC support which was added essentially experimentally a few years ago, the proper place for ARC support is in the MTA not in the mailing list software. When using ARC via Mailman 3 the mail envelope is sealed before DKIM re-signing which is the wrong order and has caused ARC authentication failure. OpenARC, like OpenDKIM, functions in the MTA for Postfix or Sendmail, https://github.com/trusteddomainproject/OpenARC . Mailing lists at https://openarc.org/ . OpenARC is not as fully developed as OpenDKIM and support for some nice things such as multiple sending domains on the system seems to have been abandoned. OpenARC has better support for BSD Unix and Red Hat than Debian based systems but is not as well developed, and although not robustly maintained for Debian based systems, there are openarc packages based on the OpenARC development branch for Debian 9 to 11, https://download.opensuse.org/repositories/home%3A/andreasschulze/ . [Mailman 2 which we are using for the mailing lists does not go past Debian 10 for lack of Python 2. Upgrading to Mailman 3 is non-trivial because of configuration changes, etc. and should not be the most immediate priority.] There is a very brief blog post about using the Andreas Schulze Debian packages, "OpenARC with Postfix on Debian 10 (buster)" / Matthieu - https://weber.fi.eu.org/blog/Informatique/openarc_with_postfix_on_debian_10.... . You can also build your own packages from source as I have. [In current testing of my source build, Postfix has a socket permissions error for OpenARC which may be from a mistake I had made with umask settings long ago on the system which runs my mailserver.] I would be pleased to help when I am available. My question about what MTA is being used with Mailman for lists.koha-community.org and a recommended fix for various configuration files are in my original message quoted further below. -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=34927 --- Comment #10 from Thomas Dukleth <td-koha-bugs@agogme.com> --- (In reply to Thomas Dukleth from comment #9)
for lists.koha.org .
Should have been lists.koha-community.org . -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=34927 --- Comment #11 from Thomas Dukleth <td-koha-bugs@agogme.com> ---
Sidenote on ARC Support.
If Gmail, etc. would still require ARC support for mailing lists despite mailing list use of DMARC, I have verified that OpenARC works in my own build from the development branch at least sufficiently well to satisfy Gmail currently. There are no designated release branches and people build from the OpenARC development branch especially given that the last commit to the development branch was three years ago and the development branch has some important bug fixes. [...]
current testing of my source build, Postfix has a socket permissions error for OpenARC
OpenARC lacks a umask configuration setting, which exists for OpenDKIM, even if other means should be available for controlling OpenDKIM socket permissions. Setting a chmod command to run following the start of the OpenARC daemon service is a simple workaround to add group write permission to the OpenARC socket fixing the Postfix permission problem for the OpenARC socket. On some other system other than my own mailserver, which I used for testing, the socket permission problem may not have appeared. I noticed that someone had filed an OpenARC bug requesting a umask configuration setting like the feature in OpenDKIM and with unimplemented supporting code present in the development branch. There are other helpful OpenDKIM features which are missing from OpenARC or only partly implemented with much less development work for OpenARC than OpenDKIM. Workarounds for the incomplete feature of supporting multiple domains or subdomains for OpenARC matching OpenDKIM with OpenARC using KeyTable and SigningTable with OpenARC do not seem as simple as adding a script for the startup service. Containerisation or additional VPS servers may be possible workarounds for multiple domains or subdomains but both workarounds may need additional IP addresses with good email reputation which are not readily available from most hosting providers. -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=34927 --- Comment #12 from Thomas Dukleth <td-koha-bugs@agogme.com> --- We should change the other BibLibre managed mailing lists to support DMARC by changing the following two configurations to match the changes for Koha-Devel before February 2024. An additional DNS change for Katipo is noted further below. In the Mailman web administration interface, change the following two settings. In the general list personality section change: from_is_list set to "Munge From" In the subscription rules section change: dmarc_moderation_action set to "Munge From". Alternatively, in etc/mailman/mm_cfg.py the same changes are: # 1 -> Rewrite the From: header of posts replacing the posters address with # that of the list. DEFAULT_FROM_IS_LIST = 1 # 1 = Munge From DEFAULT_DMARC_MODERATION_ACTION = 1 After the change, Mailman needs a restart which might be automatic when making the change via the web interface. Simon Blake or whoever has access and appropriate knowledge for Katipo also needs to add a line to the DNS for lists.katipo.co.nz with the leading underscore as formatted in BIND or in some web interface if one is used for managing DNS for Katipo. _dmarc.lists.katipo.co.nz. IN TXT "v=DMARC1; p=none" While we are working on the potentially large task of setting up, configuring, and testing Discourse we should minimise people missing mail in their spam box on Gmail etc. with changes to Gmail, etc. mail delivery policy coming in February 2024. Even without migration of archives or list members the task for Discourse is not trivial, while DMARC compatibility for the mailing lists sufficient to pass Gmail in my recent testing is trivial with just two or three settings. -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=34927 --- Comment #13 from Jonathan Druart <jonathan.druart+koha@gmail.com> --- (In reply to Thomas Dukleth from comment #12)
We should change the other BibLibre managed mailing lists to support DMARC by changing the following two configurations to match the changes for Koha-Devel before February 2024. An additional DNS change for Katipo is noted further below.
In the Mailman web administration interface, change the following two settings.
In the general list personality section change:
from_is_list set to "Munge From"
In the subscription rules section change:
dmarc_moderation_action set to "Munge From".
Done for: infos koha-i18n-german koha-patches koha-pt koha-sysop koha-translate koha-win32 (lol) koha-zebra mailman I do not have admin access to the other lists. -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=34927 --- Comment #14 from Katrin Fischer <katrin.fischer@bsz-bw.de> --- I see koha-de is missing in the list. Jonathan, could you maybe get in touch with Biblibre about the missing lists? I don't have admin access and I am not aware of anyone else who has. -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=34927 --- Comment #15 from Jonathan Druart <jonathan.druart+koha@gmail.com> --- Now done also on: koha-announce koha-br koha-bugs koha-commits koha-cvs koha-de Still missing koha-es koha-gr koha-i18n-ar koha-oz koha-security -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=34927 --- Comment #16 from Jonathan Druart <jonathan.druart+koha@gmail.com> --- (In reply to Jonathan Druart from comment #15)
Still missing koha-es koha-gr koha-i18n-ar koha-oz koha-security
Done now. -- You are receiving this mail because: You are watching all bug changes.
participants (1)
-
bugzilla-daemon＠bugs.koha-community.org