[Bug 30013] New: Limit/throttle SMTP connections
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=30013 Bug ID: 30013 Summary: Limit/throttle SMTP connections Change sponsored?: --- Product: Koha Version: master Hardware: All OS: All Status: NEW Severity: enhancement Priority: P5 - low Component: Notices Assignee: koha-bugs@lists.koha-community.org Reporter: magnus@libriotech.no QA Contact: testopia@bugs.koha-community.org We have a setup where Koha is sending email through the SMTP server of the library's mother institution. We are seeing a problem with sending lots of emails in one go, where the first emails are sent ok, but then the rest fail with an error like this: unable to establish SMTP connection to (mail.x.y) port 25 It looks like this is due to some limit on the server, making it unresponsive if there are too many connections from the same source in a short time. I don't know a lot about SMTP but I can imagine a couple of solutions: - Introduce a configurable "sleep" period between each call to the external SMTP server (or batches of a configurable number, with a configurable sleep between each batch) - Looking at the response from the server and if "unable to establish SMTP connection" is received, wait for a configurable amount of time before trying to resend the failed message and the rest of the queue -- You are receiving this mail because: You are watching all bug changes. You are the assignee for the bug.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=30013 --- Comment #1 from Magnus Enger <magnus@libriotech.no> --- Is this the sort of functionality that was taken care of when we used postfix or similar on the server, and that we loose when moving to using the SMTP functionality in Koha? -- You are receiving this mail because: You are the assignee for the bug. You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=30013 --- Comment #2 from Magnus Enger <magnus@libriotech.no> ---
From RFC5321, page 66:
The sender MUST delay retrying a particular destination after one attempt has failed. In general, the retry interval SHOULD be at least 30 minutes; -- You are receiving this mail because: You are the assignee for the bug. You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=30013 Magnus Enger <magnus@libriotech.no> changed: What |Removed |Added ---------------------------------------------------------------------------- Summary|Limit/throttle SMTP |Limit/throttle/retry SMTP |connections |connections -- You are receiving this mail because: You are watching all bug changes. You are the assignee for the bug.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=30013 David Cook <dcook@prosentient.com.au> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |dcook@prosentient.com.au -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=30013 Kyle M Hall <kyle@bywatersolutions.com> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |kyle@bywatersolutions.com --- Comment #3 from Kyle M Hall <kyle@bywatersolutions.com> --- How do we know what the error is? Koha::Email uses Email::Stuffer as a base with uses Email::Sender::Simple which throws an Email::Sender::Failure exception. If we know what the code for that is, we add a new 'retry' status, and a 'retry_after' timestamp such that process_message_queue.pl will only attempt to send retries after that timestamp. -- You are receiving this mail because: You are watching all bug changes. You are the assignee for the bug.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=30013 Liz Rea <wizzyrea@gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |wizzyrea@gmail.com --- Comment #4 from Liz Rea <wizzyrea@gmail.com> --- This also happens when using google as a provider of smtp. -- You are receiving this mail because: You are the assignee for the bug. You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=30013 Magnus Enger <magnus@libriotech.no> changed: What |Removed |Added ---------------------------------------------------------------------------- See Also| |https://bugs.koha-community | |.org/bugzilla3/show_bug.cgi | |?id=33360 --- Comment #5 from Magnus Enger <magnus@libriotech.no> --- Not sure if this will be solved by 33360. -- You are receiving this mail because: You are watching all bug changes. You are the assignee for the bug.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=30013 Marcel de Rooy <m.de.rooy@rijksmuseum.nl> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |m.de.rooy@rijksmuseum.nl --- Comment #6 from Marcel de Rooy <m.de.rooy@rijksmuseum.nl> --- (In reply to Liz Rea from comment #4)
This also happens when using google as a provider of smtp.
Yes, I wrote 33360 with mail deferrals from google in mind. -- You are receiving this mail because: You are the assignee for the bug. You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=30013 --- Comment #7 from Marcel de Rooy <m.de.rooy@rijksmuseum.nl> --- (In reply to Magnus Enger from comment #5)
Not sure if this will be solved by 33360.
Yes. But the current -limit is already doing that partially. The domain limits will be additional. -- You are receiving this mail because: You are watching all bug changes. You are the assignee for the bug.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=30013 --- Comment #8 from Marcel de Rooy <m.de.rooy@rijksmuseum.nl> --- (In reply to Kyle M Hall from comment #3)
How do we know what the error is? Koha::Email uses Email::Stuffer as a base with uses Email::Sender::Simple which throws an Email::Sender::Failure exception. If we know what the code for that is, we add a new 'retry' status, and a 'retry_after' timestamp such that process_message_queue.pl will only attempt to send retries after that timestamp.
In many cases Koha will just send to port 25. If the local mta receives it, Koha marks it as sent. But the mail is not yet outside. E.g. My local mta is now trying to send it to my mail container. Which also receives it and the problem comes up there. Bug 33360 tries to throttle a bit smarter than the current limit does by looking at recipient domains too. -- You are receiving this mail because: You are the assignee for the bug. You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=30013 Marcel de Rooy <m.de.rooy@rijksmuseum.nl> changed: What |Removed |Added ---------------------------------------------------------------------------- Resolution|--- |DUPLICATE Status|NEW |RESOLVED --- Comment #9 from Marcel de Rooy <m.de.rooy@rijksmuseum.nl> --- *** This bug has been marked as a duplicate of bug 33360 *** -- You are receiving this mail because: You are watching all bug changes. You are the assignee for the bug.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=30013 Liz Rea <wizzyrea@gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Resolution|DUPLICATE |--- Status|RESOLVED |REOPENED -- You are receiving this mail because: You are watching all bug changes. You are the assignee for the bug.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=30013 --- Comment #10 from Liz Rea <wizzyrea@gmail.com> --- I don't believe 33360 fixes this issue - this is about using google as a smtp server and the fact that our message queue opens a connection per message to the smtp server. Near as I can tell 33360 has to do with the recipients and sender limits of the emails, not the behaviour of the smtp server. Google wants each connection to send several mails, right now we only send one message per connection. Google does not publish the limits, but does say that smtp server connection limits are some number per rolling 24 hour period. They prefer the connection be kept open rather than making many individual connections. -- You are receiving this mail because: You are watching all bug changes. You are the assignee for the bug.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=30013 --- Comment #11 from Kyle M Hall <kyle@bywatersolutions.com> --- I think we need to implement https://metacpan.org/pod/Email::Sender::Transport::SMTP::Persistent and keep the instances of it in memory during each call to SendQueuedMessages -- You are receiving this mail because: You are the assignee for the bug. You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=30013 Kyle M Hall <kyle@bywatersolutions.com> changed: What |Removed |Added ---------------------------------------------------------------------------- See Also| |https://bugs.koha-community | |.org/bugzilla3/show_bug.cgi | |?id=33964 -- You are receiving this mail because: You are watching all bug changes. You are the assignee for the bug.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=30013 --- Comment #12 from David Cook <dcook@prosentient.com.au> --- (In reply to Kyle M Hall from comment #11)
I think we need to implement https://metacpan.org/pod/Email::Sender::Transport::SMTP::Persistent and keep the instances of it in memory during each call to SendQueuedMessages
+1 Since different libraries can have different SMTP servers, I reckon SendQueuedMessages should have an empty hashref that gets passed through to _send_message_by_email. (Looks like _send_message_by_email is only used in SendQueuedMessages so changes to the arguments should be easy.) Then the $stmp_server can be cached in that hashref for the life of SendQueuedMessages. -- You are receiving this mail because: You are the assignee for the bug. You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=30013 --- Comment #13 from David Cook <dcook@prosentient.com.au> --- (In reply to David Cook from comment #12)
(In reply to Kyle M Hall from comment #11)
I think we need to implement https://metacpan.org/pod/Email::Sender::Transport::SMTP::Persistent and keep the instances of it in memory during each call to SendQueuedMessages
+1
Since different libraries can have different SMTP servers, I reckon SendQueuedMessages should have an empty hashref that gets passed through to _send_message_by_email. (Looks like _send_message_by_email is only used in SendQueuedMessages so changes to the arguments should be easy.)
Then the $stmp_server can be cached in that hashref for the life of SendQueuedMessages.
And that's exactly what you've already done. Just saw the other report heh. -- You are receiving this mail because: You are the assignee for the bug. You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=30013 Marcel de Rooy <m.de.rooy@rijksmuseum.nl> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|REOPENED |RESOLVED Resolution|--- |DUPLICATE --- Comment #14 from Marcel de Rooy <m.de.rooy@rijksmuseum.nl> --- So we can close this one and look at 33964 instead.. *** This bug has been marked as a duplicate of bug 33964 *** -- You are receiving this mail because: You are the assignee for the bug. You are watching all bug changes.
participants (1)
-
bugzilla-daemon＠bugs.koha-community.org