[Bug 27667] New: Display the number of non-indexed records
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27667 Bug ID: 27667 Summary: Display the number of non-indexed records Change sponsored?: --- Product: Koha Version: master Hardware: All OS: All Status: NEW Severity: normal Priority: P5 - low Component: Searching - Elasticsearch Assignee: koha-bugs@lists.koha-community.org Reporter: severine.queune@bulac.fr On "About Koha" page are displayed the numbers of indexed biblios and authorities with ES. It could be great if we could have comparison with the number of records in the database to display the number of non-indexed datas. We could have on the same page something like : Indices: koha_robin_biblios (count: 834999 on 1056480 = 221481 missing) Or on the "System information" tab with the other "Data problems". -- You are receiving this mail because: You are watching all bug changes. You are the assignee for the bug.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27667 Jonathan Druart <jonathan.druart@bugs.koha-community.org> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |jonathan.druart@bugs.koha-c | |ommunity.org Severity|normal |enhancement -- You are receiving this mail because: You are watching all bug changes. You are the assignee for the bug.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27667 Fridolin Somers <fridolin.somers@biblibre.com> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |fridolin.somers@biblibre.co | |m Assignee|koha-bugs@lists.koha-commun |fridolin.somers@biblibre.co |ity.org |m -- You are receiving this mail because: You are watching all bug changes. You are the assignee for the bug.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27667 Fridolin Somers <fridolin.somers@biblibre.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|NEW |ASSIGNED -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27667 Fridolin Somers <fridolin.somers@biblibre.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|ASSIGNED |Needs Signoff -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27667 --- Comment #1 from Fridolin Somers <fridolin.somers@biblibre.com> --- Created attachment 134904 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=134904&action=edit Bug 27667: Display a warning for records missing in Elasticsearch On "About Koha" page are displayed the numbers of indexed biblios and authorities with ES. It could be great if we could have comparison with the number of records in the database to display the number of missing records. In the "System information" tab with the other "Data problems". If Searchengine has any issue (index count will be -1), we do not display missing records. Test plan : 1) Use Elasticsearch searchengine 2) Rebuild all records : koha-elasticsearch --rebuild -d kohadev 3) Go to "About Koha" page and look at System information tab => you see no warning 4) On a biblio record delete the MARCXML in biblio_metadata.metadata 5) Rebuild all records : koha-elasticsearch --rebuild -d kohadev 6) Go to "About Koha" page and look at System information tab => you see warning : 1 record(s) missing on a total of 435 in indice koha_kohadev_biblios. 7) On a authority record delete the MARCXML in auth_header.marcxml 8) Rebuild all records : koha-elasticsearch --rebuild -d kohadev 9) Go to "About Koha" page and look at System information tab => you see warning : 1 record(s) missing on a total of 1705 in indice koha_kohadev_authorities. -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27667 --- Comment #2 from Fridolin Somers <fridolin.somers@biblibre.com> --- I was digging into ES bugs/enh and found this easy one. -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27667 Fridolin Somers <fridolin.somers@biblibre.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Patch complexity|--- |Small patch -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27667 Fridolin Somers <fridolin.somers@biblibre.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Patch complexity|Small patch |Trivial patch -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27667 David Nind <david@davidnind.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Needs Signoff |Signed Off -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27667 David Nind <david@davidnind.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #134904|0 |1 is obsolete| | --- Comment #3 from David Nind <david@davidnind.com> --- Created attachment 136565 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=136565&action=edit Bug 27667: Display a warning for records missing in Elasticsearch On "About Koha" page are displayed the numbers of indexed biblios and authorities with ES. It could be great if we could have comparison with the number of records in the database to display the number of missing records. In the "System information" tab with the other "Data problems". If Searchengine has any issue (index count will be -1), we do not display missing records. Test plan : 1) Use Elasticsearch searchengine 2) Rebuild all records : koha-elasticsearch --rebuild -d kohadev 3) Go to "About Koha" page and look at System information tab => you see no warning 4) On a biblio record delete the MARCXML in biblio_metadata.metadata 5) Rebuild all records : koha-elasticsearch --rebuild -d kohadev 6) Go to "About Koha" page and look at System information tab => you see warning : 1 record(s) missing on a total of 435 in indice koha_kohadev_biblios. 7) On a authority record delete the MARCXML in auth_header.marcxml 8) Rebuild all records : koha-elasticsearch --rebuild -d kohadev 9) Go to "About Koha" page and look at System information tab => you see warning : 1 record(s) missing on a total of 1705 in indice koha_kohadev_authorities. Signed-off-by: David Nind <david@davidnind.com> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27667 David Nind <david@davidnind.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Text to go in the| |This enhancement adds release notes| |information about | |non-indexed records when | |using the Elasticsearch | |search engine on the About | |Koha > System information | |page. | | | |For example: | | | |Records are missing in | |Elasticsearch search engine | |- Warning 1 record(s) | |missing on a total of 435 | |in indice | |koha_kohadev_biblios. | |- | |Warning 1 record(s) | |missing on a total of 1705 | |in indice | |koha_kohadev_authorities. CC| |david@davidnind.com --- Comment #4 from David Nind <david@davidnind.com> --- Testing notes (using koha-testing-docker with sample data): - For step 4, SQL query is: update biblio_metadata set metadata="" where id=436; - For step 7, SQL query is: update auth_header set marcxml="" where authid=1707; -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27667 Victor Grousset/tuxayo <victor@tuxayo.net> changed: What |Removed |Added ---------------------------------------------------------------------------- QA Contact| |victor@tuxayo.net CC| |victor@tuxayo.net --- Comment #5 from Victor Grousset/tuxayo <victor@tuxayo.net> ---
[% index.missing_count | html %] record(s) missing on a total of [% index.index_count | html %] in indice [% index.index_name | html %].
Any reason to use indice instead of index? IIUC, only the plural form is used, for the singular, it's index. Otherwise, all looks good. -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27667 Victor Grousset/tuxayo <victor@tuxayo.net> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #136565|0 |1 is obsolete| | --- Comment #6 from Victor Grousset/tuxayo <victor@tuxayo.net> --- Created attachment 137458 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=137458&action=edit Bug 27667: Display a warning for records missing in Elasticsearch On "About Koha" page are displayed the numbers of indexed biblios and authorities with ES. It could be great if we could have comparison with the number of records in the database to display the number of missing records. In the "System information" tab with the other "Data problems". If Searchengine has any issue (index count will be -1), we do not display missing records. Test plan : 1) Use Elasticsearch searchengine 2) Rebuild all records : koha-elasticsearch --rebuild -d kohadev 3) Go to "About Koha" page and look at System information tab => you see no warning 4) On a biblio record delete the MARCXML in biblio_metadata.metadata 5) Rebuild all records : koha-elasticsearch --rebuild -d kohadev 6) Go to "About Koha" page and look at System information tab => you see warning : 1 record(s) missing on a total of 435 in indice koha_kohadev_biblios. 7) On a authority record delete the MARCXML in auth_header.marcxml 8) Rebuild all records : koha-elasticsearch --rebuild -d kohadev 9) Go to "About Koha" page and look at System information tab => you see warning : 1 record(s) missing on a total of 1705 in indice koha_kohadev_authorities. Signed-off-by: David Nind <david@davidnind.com> Signed-off-by: Victor Grousset/tuxayo <victor@tuxayo.net> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27667 Katrin Fischer <katrin.fischer@bsz-bw.de> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Signed Off |Failed QA --- Comment #7 from Katrin Fischer <katrin.fischer@bsz-bw.de> --- I followed the test plan, but the system information note doesn't show up for me. I deleted the MARCXML as follows: update biblio_metadata set metadata = "" where biblionumber = 2; Reindexing then gives an error: No record passed at /kohadevbox/koha/Koha/MetadataIterator.pm line 71. Something went wrong reading record for biblio Koha::Schema::Result::Biblio=HASH(0x557fea5e8d78)->biblionumber: Can't bless non-reference value at /kohadevbox/koha/Koha/BiblioUtils.pm line 69. Cannot determine authority type for record: 1 at /kohadevbox/koha/Koha/SearchEngine/Elasticsearch.pm line 564. BUT: The system information tab remains empty. -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27667 Katrin Fischer <katrin.fischer@bsz-bw.de> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Failed QA |Signed Off --- Comment #8 from Katrin Fischer <katrin.fischer@bsz-bw.de> --- Ok, good test: you also need to set the SearchEngine to Elasticsearch :) -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27667 Katrin Fischer <katrin.fischer@bsz-bw.de> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Signed Off |Passed QA -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27667 Katrin Fischer <katrin.fischer@bsz-bw.de> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #137458|0 |1 is obsolete| | --- Comment #9 from Katrin Fischer <katrin.fischer@bsz-bw.de> --- Created attachment 137755 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=137755&action=edit Bug 27667: Display a warning for records missing in Elasticsearch On "About Koha" page are displayed the numbers of indexed biblios and authorities with ES. It could be great if we could have comparison with the number of records in the database to display the number of missing records. In the "System information" tab with the other "Data problems". If Searchengine has any issue (index count will be -1), we do not display missing records. Test plan : 1) Use Elasticsearch searchengine 2) Rebuild all records : koha-elasticsearch --rebuild -d kohadev 3) Go to "About Koha" page and look at System information tab => you see no warning 4) On a biblio record delete the MARCXML in biblio_metadata.metadata 5) Rebuild all records : koha-elasticsearch --rebuild -d kohadev 6) Go to "About Koha" page and look at System information tab => you see warning : 1 record(s) missing on a total of 435 in indice koha_kohadev_biblios. 7) On a authority record delete the MARCXML in auth_header.marcxml 8) Rebuild all records : koha-elasticsearch --rebuild -d kohadev 9) Go to "About Koha" page and look at System information tab => you see warning : 1 record(s) missing on a total of 1705 in indice koha_kohadev_authorities. Signed-off-by: David Nind <david@davidnind.com> Signed-off-by: Victor Grousset/tuxayo <victor@tuxayo.net> Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27667 --- Comment #10 from Katrin Fischer <katrin.fischer@bsz-bw.de> --- Created attachment 137756 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=137756&action=edit Bug 27667: (QA follow-up) Fix singular index and rephrase heading -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27667 Tomás Cohen Arazi <tomascohen@gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Passed QA |Pushed to master Version(s)| |22.11.00 released in| | -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27667 --- Comment #11 from Tomás Cohen Arazi <tomascohen@gmail.com> --- Pushed to master for 22.11. Nice work everyone, thanks! -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27667 David Nind <david@davidnind.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Text to go in the|This enhancement adds |This enhancement adds release notes|information about |information about |non-indexed records when |non-indexed records when |using the Elasticsearch |using the Elasticsearch |search engine on the About |search engine on the About |Koha > System information |Koha > System information |page. |page. | | |For example: |For example: | | |Records are missing in |Records are not indexed in |Elasticsearch search engine |Elasticsearch |- Warning 1 record(s) |- Warning 1 |missing on a total of 435 |record(s) missing on a |in indice |total of 435 in index |koha_kohadev_biblios. |koha_kohadev_biblios.record |- |(s). |Warning 1 record(s) |- Warning 1 record(s) |missing on a total of 1705 |missing on a total of 1705 |in indice |in index |koha_kohadev_authorities. |koha_kohadev_authorities. -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27667 David Cook <dcook@prosentient.com.au> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |dcook@prosentient.com.au --- Comment #12 from David Cook <dcook@prosentient.com.au> --- Some day, it would be nice to make about.pl use more asynchronous calls. I often look at about.pl when I want to double-check a Koha instance's version and it takes so long to load these days... -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27667 --- Comment #13 from Fridolin Somers <fridolin.somers@biblibre.com> --- That is why I proposed Bug 27497 :) We at Biblibre have patch to add a toolip on Koha logo showing version, marcflavour and searchengine. Those info are super useful when managing hundreds of sites. One can have them even without login into staff interface, via logo in auth page. There is also this plugin, we use it for demo and dev envs : https://github.com/biblibre/koha-plugin-intranet-version-in-header -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27667 Lucas Gass <lucas@bywatersolutions.com> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |lucas@bywatersolutions.com Version(s)|22.11.00 |22.11.00, 22.05.05 released in| | Status|Pushed to master |Pushed to stable --- Comment #14 from Lucas Gass <lucas@bywatersolutions.com> --- Backported to 22.05.x for 22.05.05 -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27667 Arthur Suzuki <arthur.suzuki@biblibre.com> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |arthur.suzuki@biblibre.com Version(s)|22.11.00, 22.05.05 |22.11.00, 22.05.05, released in| |21.11.12 Status|Pushed to stable |Pushed to oldstable --- Comment #15 from Arthur Suzuki <arthur.suzuki@biblibre.com> --- thx pushed to 21.11.x for 21.11.12 -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27667 Victor Grousset/tuxayo <victor@tuxayo.net> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Pushed to oldstable |Needs documenting --- Comment #16 from Victor Grousset/tuxayo <victor@tuxayo.net> --- Not backported to oldoldstable (21.05.x). Feel free to ask if it's needed. Maybe worth documenting? -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27667 Philip Orr <philip.orr@lmscloud.de> changed: What |Removed |Added ---------------------------------------------------------------------------- Documentation| |https://gitlab.com/koha-com submission| |munity/koha-manual/-/merge_ | |requests/760 Documentation| |Philip Orr contact| | Status|Needs documenting |RESOLVED Resolution|--- |FIXED CC| |philip.orr@lmscloud.de --- Comment #17 from Philip Orr <philip.orr@lmscloud.de> --- Added to the manual. -- You are receiving this mail because: You are watching all bug changes.
participants (1)
-
bugzilla-daemon＠bugs.koha-community.org