[Bug 36977] New: Google does not read sitemaps with the name sitemapNNNN.xml
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=36977 Bug ID: 36977 Summary: Google does not read sitemaps with the name sitemapNNNN.xml Change sponsored?: --- Product: Koha Version: unspecified Hardware: All OS: All Status: NEW Severity: enhancement Priority: P5 - low Component: Command-line Utilities Assignee: januszop@gmail.com Reporter: januszop@gmail.com QA Contact: testopia@bugs.koha-community.org CC: michael.skarupianski@gmail.com, pedro.amorim@ptfs-europe.com, robin@catalyst.net.nz We have experienced problems with Google indexing: it seems (controlled in Google search console) that Google does not "like" sitemaps with a name like sitemapNNNN.xml (does not read them). Changing the name to sitemap_NNNN.xml miraculously resolves the issue: individual pieces of sitemap are read as they are declared in sitemapindex.xml. -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=36977 Janusz Kaczmarek <januszop@gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|NEW |Needs Signoff -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=36977 --- Comment #1 from Janusz Kaczmarek <januszop@gmail.com> --- Created attachment 167230 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=167230&action=edit Bug 36977: Google does not read sitemaps with the name sitemapNNNN.xml We have experienced problems with Google indexing: it seems (controlled in Google search console) that Google does not "like" sitemaps with a name like sitemapNNNN.xml (does not read them). Changing the name to sitemap_NNNN.xml miraculously resolves the issue: individual pieces of sitemap are read as they are declared in sitemapindex.xml. Test plan: ========== 1. Have your site configured to work with Google search console (cf. https://support.google.com/webmasters/answer/9008080). 2. Generate sitemap with: misc/cronjobs/sitemap.pl --dir /var/lib/koha/<instance>/sitemap (create the directory if necessary) 3. Check with you browser that the sitemap is generated: http://<OPAC-url>/sitemapindex.xml Check that the individual pieces are readeable: http://<OPAC-url>/sitemap0001.xml 4. Go to the Google seach console > Sitemaps > Add a new sitemap Enter sitemapindex.xml 5. Most probably Google won't read your sitemap chunks with the warning: "Couldn't fetch". 6. Apply the patch. Repeat p. 2, 3, 4, with the difference of the chunk name: http://<OPAC-url>/sitemap_0001.xml 7. You should see Google reading your entire sitemap with the info: "Success". -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=36977 David Nind <david@davidnind.com> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |david@davidnind.com --- Comment #2 from David Nind <david@davidnind.com> --- I'm not able to test this, and it's good that the change worked for you! However, I don't see any reason why the file name change affects whether Google tools can fetch the sitemap files (everything else being correct in the files): 1. Sitemap protocol information (https://www.sitemaps.org/protocol.html): the example on the page (you need to scroll down), doesn't have an underscore. 2. Google information (https://developers.google.com/search/docs/crawling-indexing/sitemaps/large-s...): Google's own information doesn't have an underscore. A quick search shows there can often be multiple reasons why Google tools can't fetch the site map files, including, but not limited to: - something in the site robots.txt file - validation issues (https://www.sitemaps.org/protocol.html#validating) It would be good to hear from others about whether they have issues (or not) with Google tools accessing their sitemap files. -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=36977 --- Comment #3 from Janusz Kaczmarek <januszop@gmail.com> --- David, thanks for commenting. This is the theory we all know. But the reality seems not to conform to the theory. Just try to provide to Google a sitemap for a catalogue with 200+ K records and see if it will read it. And then try with the patch. So you will be able to valuate this. We discussed this issue during recent Hackfest with a colleague from KIT -- we had both, independently, the same experience and we both found the same work around. Hence the patch. -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=36977 --- Comment #4 from Michael Skarupianski <michael.skarupianski@gmail.com> --- We at KIT (catalogue with over 1 million records) have independently noticed exactly the same behaviour from Google. Google did not want to index (or even download) our submitted sitemaps, only when we changed the file name (while the content was still the same) the sitemaps were indexed. We can therefore confirm both the problem and the patch. We also tried to contact Google but did not get any answer. -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=36977 --- Comment #5 from Janusz Kaczmarek <januszop@gmail.com> --- (In reply to Michael Skarupianski from comment #4)
We at KIT (catalogue with over 1 million records) have independently noticed exactly the same behaviour from Google. Google did not want to index (or even download) our submitted sitemaps, only when we changed the file name (while the content was still the same) the sitemaps were indexed. We can therefore confirm both the problem and the patch. We also tried to contact Google but did not get any answer.
Michael, thank you for your comment. So, could you--or somebody from KIT--sign off the patch, please? -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=36977 --- Comment #6 from Michael Skarupianski <michael.skarupianski@gmail.com> --- Created attachment 167345 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=167345&action=edit QA Screenshot from the Google Search console (both files are identical) -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=36977 Michael Skarupianski <michael.skarupianski@gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Needs Signoff |Signed Off -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=36977 --- Comment #7 from David Nind <david@davidnind.com> --- (In reply to Janusz Kaczmarek from comment #3)
David, thanks for commenting. This is the theory we all know. But the reality seems not to conform to the theory. Just try to provide to Google a sitemap for a catalogue with 200+ K records and see if it will read it. And then try with the patch. So you will be able to valuate this.
We discussed this issue during recent Hackfest with a colleague from KIT -- we had both, independently, the same experience and we both found the same work around. Hence the patch.
Thanks Janusz. It's good to have the confirmation from others that it is an issue (despite the theory!), and that renaming the site map files fixes the issue. David -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=36977 Michael Skarupianski <michael.skarupianski@gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #167230|0 |1 is obsolete| | --- Comment #8 from Michael Skarupianski <michael.skarupianski@gmail.com> --- Created attachment 167431 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=167431&action=edit Bug 36977: Google does not read sitemaps with the name sitemapNNNN.xml We have experienced problems with Google indexing: it seems (controlled in Google search console) that Google does not "like" sitemaps with a name like sitemapNNNN.xml (does not read them). Changing the name to sitemap_NNNN.xml miraculously resolves the issue: individual pieces of sitemap are read as they are declared in sitemapindex.xml. Test plan: ========== 1. Have your site configured to work with Google search console (cf. https://support.google.com/webmasters/answer/9008080). 2. Generate sitemap with: misc/cronjobs/sitemap.pl --dir /var/lib/koha/<instance>/sitemap (create the directory if necessary) 3. Check with you browser that the sitemap is generated: http://<OPAC-url>/sitemapindex.xml Check that the individual pieces are readeable: http://<OPAC-url>/sitemap0001.xml 4. Go to the Google seach console > Sitemaps > Add a new sitemap Enter sitemapindex.xml 5. Most probably Google won't read your sitemap chunks with the warning: "Couldn't fetch". 6. Apply the patch. Repeat p. 2, 3, 4, with the difference of the chunk name: http://<OPAC-url>/sitemap_0001.xml 7. You should see Google reading your entire sitemap with the info: "Success". Signed-off-by: Michael Skarupianski <michael.skarupianski@gmail.com> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=36977 Raphael Straub <raphael.straub@kit.edu> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |raphael.straub@kit.edu -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=36977 David Nind <david@davidnind.com> changed: What |Removed |Added ---------------------------------------------------------------------------- See Also| |https://bugs.koha-community | |.org/bugzilla3/show_bug.cgi | |?id=37352 -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=36977 Michał <schodkowy.omegi-0r@icloud.com> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |schodkowy.omegi-0r@icloud.c | |om -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=36977 Michał <schodkowy.omegi-0r@icloud.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #167431|0 |1 is obsolete| | --- Comment #9 from Michał <schodkowy.omegi-0r@icloud.com> --- Created attachment 169815 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=169815&action=edit Bug 36977: Google does not read sitemaps with the name sitemapNNNN.xml We have experienced problems with Google indexing: it seems (controlled in Google search console) that Google does not "like" sitemaps with a name like sitemapNNNN.xml (does not read them). Changing the name to sitemap_NNNN.xml miraculously resolves the issue: individual pieces of sitemap are read as they are declared in sitemapindex.xml. Test plan: ========== 1. Have your site configured to work with Google search console (cf. https://support.google.com/webmasters/answer/9008080). 2. Generate sitemap with: misc/cronjobs/sitemap.pl --dir /var/lib/koha/<instance>/sitemap (create the directory if necessary) 3. Check with you browser that the sitemap is generated: http://<OPAC-url>/sitemapindex.xml Check that the individual pieces are readeable: http://<OPAC-url>/sitemap0001.xml 4. Go to the Google seach console > Sitemaps > Add a new sitemap Enter sitemapindex.xml 5. Most probably Google won't read your sitemap chunks with the warning: "Couldn't fetch". 6. Apply the patch. Repeat p. 2, 3, 4, with the difference of the chunk name: http://<OPAC-url>/sitemap_0001.xml 7. You should see Google reading your entire sitemap with the info: "Success". Signed-off-by: Michael Skarupianski <michael.skarupianski@gmail.com> Signed-off-by: Michał Kula <148193449+mkibp@users.noreply.github.com> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=36977 --- Comment #10 from Michał <schodkowy.omegi-0r@icloud.com> --- I confirm, upon testing, that the patch indeed fixes the aforementioned issue, which we had also experienced. After applying it, we see that Google Search Console no longer shows 0 pages indexed in the sitemaps section, and now it shows up `/sitemapindex.xml => /sitemap_0001.xml` whereas previously it'd only show `/sitemapindex.xml`. -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=36977 Tomás Cohen Arazi (tcohen) <tomascohen@gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Signed Off |Passed QA Patch complexity|--- |Trivial patch -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=36977 Tomás Cohen Arazi (tcohen) <tomascohen@gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #169815|0 |1 is obsolete| | --- Comment #11 from Tomás Cohen Arazi (tcohen) <tomascohen@gmail.com> --- Created attachment 173612 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=173612&action=edit Bug 36977: Google does not read sitemaps with the name sitemapNNNN.xml We have experienced problems with Google indexing: it seems (controlled in Google search console) that Google does not "like" sitemaps with a name like sitemapNNNN.xml (does not read them). Changing the name to sitemap_NNNN.xml miraculously resolves the issue: individual pieces of sitemap are read as they are declared in sitemapindex.xml. Test plan: ========== 1. Have your site configured to work with Google search console (cf. https://support.google.com/webmasters/answer/9008080). 2. Generate sitemap with: misc/cronjobs/sitemap.pl --dir /var/lib/koha/<instance>/sitemap (create the directory if necessary) 3. Check with you browser that the sitemap is generated: http://<OPAC-url>/sitemapindex.xml Check that the individual pieces are readeable: http://<OPAC-url>/sitemap0001.xml 4. Go to the Google seach console > Sitemaps > Add a new sitemap Enter sitemapindex.xml 5. Most probably Google won't read your sitemap chunks with the warning: "Couldn't fetch". 6. Apply the patch. Repeat p. 2, 3, 4, with the difference of the chunk name: http://<OPAC-url>/sitemap_0001.xml 7. You should see Google reading your entire sitemap with the info: "Success". Signed-off-by: Michael Skarupianski <michael.skarupianski@gmail.com> Signed-off-by: Michał Kula <148193449+mkibp@users.noreply.github.com> Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=36977 --- Comment #12 from Tomás Cohen Arazi (tcohen) <tomascohen@gmail.com> --- Created attachment 173613 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=173613&action=edit Bug 36977: (QA follow-up) Adapt tests Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=36977 Tomás Cohen Arazi (tcohen) <tomascohen@gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |tomascohen@gmail.com QA Contact|testopia@bugs.koha-communit |tomascohen@gmail.com |y.org | --- Comment #13 from Tomás Cohen Arazi (tcohen) <tomascohen@gmail.com> --- I cannot confirm this solves an issue or not. But the change is harmless on its own. QA scripts are happy. I adapted the tests to this new filename schema. Note: always check for existing tests on features. This time your change had broken the sitemapper tests. And if there weren't tests, we would require you to add them to avoid regressions. -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=36977 Katrin Fischer <katrin.fischer@bsz-bw.de> changed: What |Removed |Added ---------------------------------------------------------------------------- Version(s)| |24.11.00 released in| | Status|Passed QA |Pushed to main -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=36977 --- Comment #14 from Katrin Fischer <katrin.fischer@bsz-bw.de> --- Pushed for 24.11! Well done everyone, thank you! -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=36977 Martin Renvoize (ashimema) <martin.renvoize@ptfs-europe.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Keywords| |release-notes-needed -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=36977 David Nind <david@davidnind.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Keywords|release-notes-needed | Text to go in the| |This changes the file name release notes| |for sitemap files[1] | |generated by | |misc/cronjobs/sitemap.pl | |from sitemapNNNN.xml to | |sitemap_NNNN.xml (it adds | |an underscore). For | |whatever reason, Google | |only seems to fetch sitemap | |files with an underscore, | |despite this not being a | |requirement in the sitemap | |protocol. This then | |resulted in pages not being | |crawled (reducing their | |discoverability). | | | |[1] Site | |map files are used by | |search engines to identify | |pages on websites that are | |available for crawling. See | |https://sitemaps.org/ for | |more information. -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=36977 Lucas Gass (lukeg) <lucas@bywatersolutions.com> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |lucas@bywatersolutions.com Status|Pushed to main |Needs documenting --- Comment #15 from Lucas Gass (lukeg) <lucas@bywatersolutions.com> --- Enhancement, no 24.05.x backport. -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=36977 Michał <schodkowy.omegi-0r@icloud.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Severity|enhancement |minor --- Comment #16 from Michał <schodkowy.omegi-0r@icloud.com> --- Actually I would say the "enhancement" category was wrong, I'm changing it to "minor". It is definitely a bug fix, since it actively prevented Google from reading the sitemap at all... might be worth backporting still -- You are receiving this mail because: You are watching all bug changes.
participants (1)
-
bugzilla-daemon＠bugs.koha-community.org