[Bug 33317] New: Add system preference to set meta robots for the OPAC
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=33317 Bug ID: 33317 Summary: Add system preference to set meta robots for the OPAC Change sponsored?: --- Product: Koha Version: master Hardware: All OS: All Status: NEW Severity: enhancement Priority: P5 - low Component: OPAC Assignee: oleonard@myacpl.org Reporter: hammat.wele@inlibro.com QA Contact: testopia@bugs.koha-community.org Websites must have a robots meta tag to improve search engine crawling, so we could tells search engines to not index OPAC pages for example. It needs to be added to koha-tmpl/opac-tmpl/bootstrap/en/includes/doc-head-close.inc The content should also be available as a system preference. -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=33317 Hammat wele <hammat.wele@inlibro.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Assignee|oleonard@myacpl.org |hammat.wele@inlibro.com -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=33317 Hammat wele <hammat.wele@inlibro.com> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |caroline.cyr-la-rose@inlibr | |o.com, | |hammat.wele@inlibro.com, | |philippe.blouin@inlibro.com -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=33317 --- Comment #1 from Hammat wele <hammat.wele@inlibro.com> --- Created attachment 148575 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=148575&action=edit Bug 33317: Add system preference to set meta robots for the OPAC Websites must have a robots meta tag to improve search engine crawling, so we could tell search engines to not index OPAC pages for example. To test: 1) Look at opac-main source and see that meta name=robots is missing. 2) Apply patch. 3) Go to system preferences > OPAC and add some directives to OpacMetaRobots (ex: noindex,nofollow). 4) Look at opac-main source and confirm that meta name=robots now has content equal to the text set in the OpacMetaRobots system preference. 5) This should Prevent search engines from indexing Opac pages (if OpacMetaRobots is set to noindex) -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=33317 Hammat wele <hammat.wele@inlibro.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|NEW |Needs Signoff -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=33317 David Cook <dcook@prosentient.com.au> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |dcook@prosentient.com.au -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=33317 ByWater Sandboxes <bws.sandboxes@gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #148575|0 |1 is obsolete| | --- Comment #2 from ByWater Sandboxes <bws.sandboxes@gmail.com> --- Created attachment 148593 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=148593&action=edit Bug 33317: Add system preference to set meta robots for the OPAC Websites must have a robots meta tag to improve search engine crawling, so we could tell search engines to not index OPAC pages for example. To test: 1) Look at opac-main source and see that meta name=robots is missing. 2) Apply patch. 3) Go to system preferences > OPAC and add some directives to OpacMetaRobots (ex: noindex,nofollow). 4) Look at opac-main source and confirm that meta name=robots now has content equal to the text set in the OpacMetaRobots system preference. 5) This should Prevent search engines from indexing Opac pages (if OpacMetaRobots is set to noindex) Signed-off-by: Sally <sally.healey@cheshiresharedservices.gov.uk> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=33317 Sally <sally.healey@cheshiresharedservices.gov.uk> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |sally.healey@cheshireshared | |services.gov.uk Status|Needs Signoff |Signed Off --- Comment #3 from Sally <sally.healey@cheshiresharedservices.gov.uk> --- I think this is 99% of the way there and I'm not sure if I'm being picky with this, so I've signed off so QA can take a look. Adding a value works perfectly. Changing a value works perfectly. But if you've added a value and you later decide to reset to the original defaults, you can't do so by deleting the values and saving the empty input field. To replicate: - Enter a value into OpacMetaRobots - Save - Open the OPAC and view the source; note the value matches - Change the value in OpacMetaRobots - Save - Refresh the OPAC and view the source; note the value matches - Remove the value from OpacMetaRobots so the input field is empty - Refresh the OPAC and view the source; this time, note that the last saved value still remains Given that 'index' is the default value if no robot meta tag is listed, maybe just a note to say 'to reset to default values, use: index' would work? -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=33317 --- Comment #4 from Sally <sally.healey@cheshiresharedservices.gov.uk> --- (In reply to Sally from comment #3)
I think this is 99% of the way there and I'm not sure if I'm being picky with this, so I've signed off so QA can take a look.
Adding a value works perfectly. Changing a value works perfectly.
But if you've added a value and you later decide to reset to the original defaults, you can't do so by deleting the values and saving the empty input field.
To replicate:
- Enter a value into OpacMetaRobots - Save - Open the OPAC and view the source; note the value matches - Change the value in OpacMetaRobots - Save - Refresh the OPAC and view the source; note the value matches - Remove the value from OpacMetaRobots so the input field is empty - Save - Refresh the OPAC and view the source; this time, note that the last saved value still remains
Given that 'index' is the default value if no robot meta tag is listed, maybe just a note to say 'to reset to default values, use: index' would work?
This error didn't make sense to me, so I wondered if I was having sandbox woes again. I repeated the tests on a different sandbox and it all worked perfectly. So please ignore the above comment - it's working as described. Sorry for the noise. -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=33317 Lucas Gass <lucas@bywatersolutions.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Signed Off |Failed QA CC| |lucas@bywatersolutions.com --- Comment #5 from Lucas Gass <lucas@bywatersolutions.com> --- Hammat, This is missing a DB update. Have a look at Bug 30403 for an example. -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=33317 Hammat wele <hammat.wele@inlibro.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #148593|0 |1 is obsolete| | --- Comment #6 from Hammat wele <hammat.wele@inlibro.com> --- Created attachment 150420 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=150420&action=edit Bug 33317: Add system preference to set meta robots for the OPAC Websites must have a robots meta tag to improve search engine crawling, so we could tell search engines to not index OPAC pages for example. To test: 1) Look at opac-main source and see that meta name=robots is missing. 2) Apply patch. 3) Run ./installer/data/mysql/updatedatabase.pl 4) Go to system preferences > OPAC and add some directives to OpacMetaRobots (ex: noindex,nofollow). 5) Look at opac-main source and confirm that meta name=robots now has content equal to the text set in the OpacMetaRobots system preference. 6) This should Prevent search engines from indexing Opac pages (if OpacMetaRobots is set to noindex) -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=33317 Hammat wele <hammat.wele@inlibro.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Failed QA |Needs Signoff -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=33317 --- Comment #7 from Lucas Gass <lucas@bywatersolutions.com> --- I have a few follow-up questions: 1. Should a default value of noindex, nofollow be used here? 2. Is freetext the right thing to use here? Using conflicting or inconsistent values can confuse search engine crawlers and cause them to make incorrect assumptions about how to treat your pages in their index. -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=33317 Lucas Gass <lucas@bywatersolutions.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Needs Signoff |Failed QA --- Comment #8 from Lucas Gass <lucas@bywatersolutions.com> --- Failing QA here until my questions are addressed. -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=33317 --- Comment #9 from David Cook <dcook@prosentient.com.au> --- (In reply to Lucas Gass from comment #7)
I have a few follow-up questions:
1. Should a default value of noindex, nofollow be used here?
2. Is freetext the right thing to use here? Using conflicting or inconsistent values can confuse search engine crawlers and cause them to make incorrect assumptions about how to treat your pages in their index.
Good questions... With Google is looks like there are quite a few valid rules: https://developers.google.com/search/docs/crawling-indexing/robots-meta-tag#... But then free-text is error prone as Lucas says... -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=33317 --- Comment #10 from Hammat wele <hammat.wele@inlibro.com> --- Created attachment 158820 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=158820&action=edit Bug 33317: Using checkoxes in OpacMetaRobots preference system to prevent incorrect data entry -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=33317 Hammat wele <hammat.wele@inlibro.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Failed QA |Needs Signoff -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=33317 Lucas Gass <lucas@bywatersolutions.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Needs Signoff |Failed QA --- Comment #11 from Lucas Gass <lucas@bywatersolutions.com> --- Looks good Hammat. Only issue now is to perltidy 'installer/data/mysql/atomicupdate/bug_33317.pl'. -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=33317 --- Comment #12 from Hammat wele <hammat.wele@inlibro.com> --- Created attachment 159653 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=159653&action=edit Bug 33317: (follow-up) Perltidy -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=33317 Hammat wele <hammat.wele@inlibro.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Failed QA |Needs Signoff -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=33317 Owen Leonard <oleonard@myacpl.org> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Needs Signoff |Failed QA --- Comment #13 from Owen Leonard <oleonard@myacpl.org> --- The qa tool still has complaints about the atomicupdate. Please take another look. -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=33317 Liz Rea <wizzyrea@gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |wizzyrea@gmail.com --- Comment #14 from Liz Rea <wizzyrea@gmail.com> --- This is a very good feature - put the power in the user's hands instead of the sysadmin. Strong approval. Agree with Lucas that freetext is probably better avoided, since it can be error prone. Liz -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=33317 Hammat wele <hammat.wele@inlibro.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #159653|0 |1 is obsolete| | -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=33317 --- Comment #15 from Hammat wele <hammat.wele@inlibro.com> --- Created attachment 160125 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=160125&action=edit Bug 33317: (follow-up) Perltidy -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=33317 Hammat wele <hammat.wele@inlibro.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #160125|0 |1 is obsolete| | -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=33317 --- Comment #16 from Hammat wele <hammat.wele@inlibro.com> --- Created attachment 160126 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=160126&action=edit Bug 33317: (follow-up) Perltidy -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=33317 Hammat wele <hammat.wele@inlibro.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Failed QA |Needs Signoff -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=33317 Lucas Gass <lucas@bywatersolutions.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Needs Signoff |Failed QA --- Comment #17 from Lucas Gass <lucas@bywatersolutions.com> --- Hammat, The qa tool still complains. Please fix: FAIL installer/data/mysql/atomicupdate/bug_33317.pl FAIL file permissions File must have the exec flag WARN tidiness The file is less tidy than before (bad/messy lines before: 0, now: 1) -perltidy all of bug_33317.pl -Also make sure bug_33317.pl has the proper permissions set ( File must have the exec flag ) sudo chmod 755 bug_33317.pl should do it. -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=33317 Hammat wele <hammat.wele@inlibro.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #160126|0 |1 is obsolete| | -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=33317 --- Comment #18 from Hammat wele <hammat.wele@inlibro.com> --- Created attachment 160534 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=160534&action=edit Bug 33317: (follow-up) Perltidy -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=33317 Hammat wele <hammat.wele@inlibro.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Failed QA |Needs Signoff -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=33317 Owen Leonard <oleonard@myacpl.org> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Needs Signoff |Signed Off Patch complexity|--- |Small patch -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=33317 Owen Leonard <oleonard@myacpl.org> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #150420|0 |1 is obsolete| | Attachment #158820|0 |1 is obsolete| | Attachment #160534|0 |1 is obsolete| | --- Comment #19 from Owen Leonard <oleonard@myacpl.org> --- Created attachment 160539 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=160539&action=edit Bug 33317: Add system preference to set meta robots for the OPAC Websites must have a robots meta tag to improve search engine crawling, so we could tell search engines to not index OPAC pages for example. To test: 1) Look at opac-main source and see that meta name=robots is missing. 2) Apply patch. 3) Run ./installer/data/mysql/updatedatabase.pl 4) Go to system preferences > OPAC and add some directives to OpacMetaRobots (ex: noindex,nofollow). 5) Look at opac-main source and confirm that meta name=robots now has content equal to the text set in the OpacMetaRobots system preference. 6) This should Prevent search engines from indexing Opac pages (if OpacMetaRobots is set to noindex) Signed-off-by: Owen Leonard <oleonard@myacpl.org> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=33317 --- Comment #20 from Owen Leonard <oleonard@myacpl.org> --- Created attachment 160540 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=160540&action=edit Bug 33317: Using checkoxes in OpacMetaRobots preference system to prevent incorrect data entry Signed-off-by: Owen Leonard <oleonard@myacpl.org> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=33317 --- Comment #21 from Owen Leonard <oleonard@myacpl.org> --- Created attachment 160541 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=160541&action=edit Bug 33317: (follow-up) Perltidy Signed-off-by: Owen Leonard <oleonard@myacpl.org> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=33317 Lucas Gass <lucas@bywatersolutions.com> changed: What |Removed |Added ---------------------------------------------------------------------------- QA Contact|testopia@bugs.koha-communit |lucas@bywatersolutions.com |y.org | -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=33317 --- Comment #22 from Lucas Gass <lucas@bywatersolutions.com> --- Looks good now, passing QA. -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=33317 Lucas Gass <lucas@bywatersolutions.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Signed Off |Passed QA -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=33317 Lucas Gass <lucas@bywatersolutions.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #160539|0 |1 is obsolete| | --- Comment #23 from Lucas Gass <lucas@bywatersolutions.com> --- Created attachment 160544 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=160544&action=edit Bug 33317: Add system preference to set meta robots for the OPAC Websites must have a robots meta tag to improve search engine crawling, so we could tell search engines to not index OPAC pages for example. To test: 1) Look at opac-main source and see that meta name=robots is missing. 2) Apply patch. 3) Run ./installer/data/mysql/updatedatabase.pl 4) Go to system preferences > OPAC and add some directives to OpacMetaRobots (ex: noindex,nofollow). 5) Look at opac-main source and confirm that meta name=robots now has content equal to the text set in the OpacMetaRobots system preference. 6) This should Prevent search engines from indexing Opac pages (if OpacMetaRobots is set to noindex) Signed-off-by: Owen Leonard <oleonard@myacpl.org> Signed-off-by: Lucas Gass <lucas@bywatersolutions.com> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=33317 Lucas Gass <lucas@bywatersolutions.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #160540|0 |1 is obsolete| | --- Comment #24 from Lucas Gass <lucas@bywatersolutions.com> --- Created attachment 160545 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=160545&action=edit Bug 33317: Using checkoxes in OpacMetaRobots preference system to prevent incorrect data entry Signed-off-by: Owen Leonard <oleonard@myacpl.org> Signed-off-by: Lucas Gass <lucas@bywatersolutions.com> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=33317 Lucas Gass <lucas@bywatersolutions.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #160541|0 |1 is obsolete| | --- Comment #25 from Lucas Gass <lucas@bywatersolutions.com> --- Created attachment 160546 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=160546&action=edit Bug 33317: (follow-up) Perltidy Signed-off-by: Owen Leonard <oleonard@myacpl.org> Signed-off-by: Lucas Gass <lucas@bywatersolutions.com> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=33317 Lucas Gass <lucas@bywatersolutions.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Text to go in the| |This enhancement adds a new release notes| |system preference, | |OpacMetaRobots which allows | |libraries to tell search | |engine robots how to crawl | |and index OPAC pages. -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=33317 Katrin Fischer <katrin.fischer@bsz-bw.de> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Passed QA |Failed QA --- Comment #26 from Katrin Fischer <katrin.fischer@bsz-bw.de> --- I feel like the system preference description is a bit sparse. Could we add an example or a link to documentation to make it clearer what you are supposed to add? + - + - "Improve search engine crawling." + - pref: OpacMetaRobots + type: textarea Also, the pref description shoudl be formed as a sentencee and connect with the input field. -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=33317 M <schodkowy.omegi-0r@icloud.com> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |schodkowy.omegi-0r@icloud.c | |om --- Comment #27 from M <schodkowy.omegi-0r@icloud.com> --- Wait, so this simply adds the same robots tag for the entirety of OPAC? I've been redirected here from Bug 35812 due to conflicts. And I think this bug right here is questionable and I'm not sure if it should be merged, at least as-is.
Websites must have a robots meta tag
That is not true, this tag is very much optional and meant for granular page-level steering of crawling bots. The way this preference is implemented, the rules are going to apply to ALL opac pages, which I'm not sure if there's any reasonable use-case. The author mentions example "noindex,nofollow" to prevent ALL opac pages from being indexed. I think that if some library wants that, they'd be better off using the more widely used and known robots.txt file, which is more likely to be supported by various crawlers, and will prevent them from downloading the pages in the first place (instead of downloading the pages it wanted to and then discarding them upon discovering the meta tag for particular page). I think the better direction is to diversify manually which pages should be crawlable by default and which shouldn't like in Bug 35812, ie. search results and so on (dynamic pages) shouldn't be indexed to decreate the amount of junk (but they should be crawled to extract links from it), while main page/info subpages/user-created lists/biblio records should probably be indexed by default and it's probably what most libraries would want by default. With that said, there currently there's no "obvious"/"easy" way of specifying custom robots.txt file, apart from doing something like `Alias /robots.txt /var/www/html/robots.txt` in Apache config for opac (it works well enough btw). So, in the spirit of what OP originally wanted, I believe it could be better to consider instead adding a user preference for textarea of robots.txt file contents, in place of site-wide robots meta tag contents. This would allow libraries to set more granular rules, but someone who wants to block everything could still just do: User-agent: * Disallow: / Btw this is already documented in README.robots file in Koha's main git directory (last edited 13 years ago, the last paragraph there is probably outdated). So I believe my idea above could solve the conflict between our two patches, robots.txt usage is more widely documented on the Internet I believe, and that'd override any rules that Koha devs could specify on per-template basis manually in robots tag on pages like I did in Bug 35812... -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=33317 --- Comment #28 from M <schodkowy.omegi-0r@icloud.com> --- By the way, having programmatic robots.txt generation could allow having sitemaps linked there with "Sitemap:" directive, provided the generated files are there, and that would remove one obstacle from having the sitemap generation enabled by default in the Debian installation. -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=33317 Blou <philippe.blouin@inlibro.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Failed QA |Signed Off --- Comment #29 from Blou <philippe.blouin@inlibro.com> --- (In reply to Michał from comment #27)
which I'm not sure if there's any reasonable use-case. The author mentions example "noindex,nofollow" to prevent ALL opac pages from being indexed. I think that if some library wants that, they'd be better off using the more widely used and known robots.txt file
Your solution doesn't work. Robots.txt doesn't prevent sites from being indexed, Google itself instructs https://developers.google.com/search/docs/crawling-indexing/block-indexing Our use case is very valid and a basic option in Wordpress: websites wants ways to not be found (while developping). Here we have test and production systems, and of course our customers do not want their users to find the test system. Two koha, similar data, user search google for their library and click on the test link. -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=33317 --- Comment #30 from Michał <schodkowy.omegi-0r@icloud.com> --- Hmm, I think for your use-case it might be the best to set the robots configuration as a header, by adding something like this to Apache config:
Header set X-Robots-Tag "noindex, nofollow"
This way there's no need to modify all Koha templates, and I think it's better off there, as custom ways of hiding a test system, I think, would technically belong to a web server configuration on a layer above Koha, independent of its specific configuration (for example if it's a test system, someone might re-setup Koha and forget to set that setting up again etc. too). Alternatively, `OPACPublic` preference might be an alternative, although it won't really *fully* hide it, it would do the job for the most part at least too. -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=33317 --- Comment #31 from Lucas Gass <lucas@bywatersolutions.com> --- Created attachment 168071 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=168071&action=edit Bug 33317: (follow-up) Improve system preference description -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=33317 --- Comment #32 from Lucas Gass <lucas@bywatersolutions.com> --- (In reply to Katrin Fischer from comment #26)
I feel like the system preference description is a bit sparse. Could we add an example or a link to documentation to make it clearer what you are supposed to add?
+ - + - "Improve search engine crawling." + - pref: OpacMetaRobots + type: textarea
Also, the pref description shoudl be formed as a sentencee and connect with the input field.
I added a follow-up to improve the sys pref description. Let me know if you agree with it. -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=33317 Lucas Gass <lucas@bywatersolutions.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Signed Off |Passed QA -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=33317 Lucas Gass <lucas@bywatersolutions.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #160544|0 |1 is obsolete| | --- Comment #33 from Lucas Gass <lucas@bywatersolutions.com> --- Created attachment 168072 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=168072&action=edit Bug 33317: Add system preference to set meta robots for the OPAC Websites must have a robots meta tag to improve search engine crawling, so we could tell search engines to not index OPAC pages for example. To test: 1) Look at opac-main source and see that meta name=robots is missing. 2) Apply patch. 3) Run ./installer/data/mysql/updatedatabase.pl 4) Go to system preferences > OPAC and add some directives to OpacMetaRobots (ex: noindex,nofollow). 5) Look at opac-main source and confirm that meta name=robots now has content equal to the text set in the OpacMetaRobots system preference. 6) This should Prevent search engines from indexing Opac pages (if OpacMetaRobots is set to noindex) Signed-off-by: Owen Leonard <oleonard@myacpl.org> Signed-off-by: Lucas Gass <lucas@bywatersolutions.com> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=33317 Lucas Gass <lucas@bywatersolutions.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #160545|0 |1 is obsolete| | --- Comment #34 from Lucas Gass <lucas@bywatersolutions.com> --- Created attachment 168073 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=168073&action=edit Bug 33317: Using checkoxes in OpacMetaRobots preference system to prevent incorrect data entry Signed-off-by: Owen Leonard <oleonard@myacpl.org> Signed-off-by: Lucas Gass <lucas@bywatersolutions.com> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=33317 Lucas Gass <lucas@bywatersolutions.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #160546|0 |1 is obsolete| | --- Comment #35 from Lucas Gass <lucas@bywatersolutions.com> --- Created attachment 168074 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=168074&action=edit Bug 33317: (follow-up) Perltidy Signed-off-by: Owen Leonard <oleonard@myacpl.org> Signed-off-by: Lucas Gass <lucas@bywatersolutions.com> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=33317 Lucas Gass <lucas@bywatersolutions.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #168071|0 |1 is obsolete| | --- Comment #36 from Lucas Gass <lucas@bywatersolutions.com> --- Created attachment 168075 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=168075&action=edit Bug 33317: (follow-up) Improve system preference description Signed-off-by: Lucas Gass <lucas@bywatersolutions.com> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=33317 --- Comment #37 from Lucas Gass <lucas@bywatersolutions.com> --- (In reply to Michał from comment #30)
Hmm, I think for your use-case it might be the best to set the robots configuration as a header, by adding something like this to Apache config:
Header set X-Robots-Tag "noindex, nofollow"
The idea is to put more power in the hands of librarians and less burden on system admins. -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=33317 Katrin Fischer <katrin.fischer@bsz-bw.de> changed: What |Removed |Added ---------------------------------------------------------------------------- Version(s)| |24.11.00 released in| | Status|Passed QA |Pushed to main -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=33317 --- Comment #38 from Katrin Fischer <katrin.fischer@bsz-bw.de> --- Pushed for 24.11! Well done everyone, thank you! -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=33317 Lucas Gass <lucas@bywatersolutions.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Pushed to main |Needs documenting --- Comment #39 from Lucas Gass <lucas@bywatersolutions.com> --- Enhancement will not be included in the 24.05.x series -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=33317 Caroline Cyr La Rose <caroline.cyr-la-rose@inlibro.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Documentation| |Caroline Cyr La Rose contact| | Status|Needs documenting |RESOLVED Documentation| |https://gitlab.com/koha-com submission| |munity/koha-manual/-/merge_ | |requests/1034 Resolution|--- |FIXED -- You are receiving this mail because: You are watching all bug changes.
participants (1)
-
bugzilla-daemon＠bugs.koha-community.org