https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=32707 Janusz Kaczmarek <januszop@gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Assignee|koha-bugs@lists.koha-commun |januszop@gmail.com |ity.org | Status|NEW |Needs Signoff --- Comment #1 from Janusz Kaczmarek <januszop@gmail.com> --- Created attachment 145606 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=145606&action=edit [PATCH] Bug 32707: ElasticSearch should not auto truncate (even if QueryAutoTruncate = 1) for identifiers (and some other fields) Koha with Zebra prevented auto truncation in some circumstances -- see the first return for ccl inside C4::Search:: buildQuery, before applying auto truncation, and setting $auto_truncation = 0 for some search fields. Koha with ElasticSearch applies auto truncation for all search fields, not paying attention to these special cases when it should not be done. This leads to various problems as described in bug 26508, 26608, etc. The solution would be to prevent auto truncation for certain search fields, above all – the identifiers. In addition, under no circumstances should a search field other than of text type be truncated (an attempt to truncate would generate an exception from ElasticSearch, e.g. number_format_exception for integer). Test plan ========= Scenario A (authority) ---------------------- 1. Enable Elasticsearch engine. 2. Have like 100+ bibliographic records and properly linked authority records. 3. Reindex ElasticSearch if needed. 4. Enable QueryAutoTruncate systempreference. 5. Find the authority record #1 and note the number of linked biblio records. 6. Show the detail of authority #1 and compare the number of linked biblio records. 7. If in the database there are authority records with ids =~ /^1/ (i.e. 10, 11, 12, ..., 100, 101, ...) linked to the biblio records you get two different numbers of linked records. 8. Also, as lists of linked biblio records (via link: Used in N record(s) from results view and detail view) you will get two different sets of biblio records. In particular, on the list generated from detail view (authorities/detail.pl?authid=1) you will get biblio records that are in fact not linked to the auth #1 (the list is generated with &q=an:1). 9. Apply this patch. 10. Counts and list of linked biblios should be ok. Scenario B (analytics) ---------------------- 1. Enable Elasticsearch engine. 2. Have three monographic bibliographic records with 001 = 1, 10, 100 (i.e. =~ /^1/). 3. Have an analytical record with 773 $w = 1. 4. Reindex ElasticSearch if needed. 5. Enable QueryAutoTruncate and UseControlNumber systempreference. 6. Find the analytical record and click on the link generated from 773, i.e. (In: ... --> catalogue/search.pl?q=Control-number:1). 7. You should see a list of three records (001 = 1, 10, 100) instead of one (001 = 1). 8. Apply this patch. 9. Repeat step 6. You should arrive at the right host record. -- You are receiving this mail because: You are watching all bug changes. You are the assignee for the bug.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=32707 Amaury GAU <amaury.gau@bulac.fr> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |amaury.gau@bulac.fr --- Comment #2 from Amaury GAU <amaury.gau@bulac.fr> --- Hi, For now, I tried Scenario A : 1. Checked linked authorities with Zebra : OK 2. Change to ES : OK 3. Reindex Es on the sandbox : OK 4. Expected authorities to be properly linked, but they weren't, even though the reindexation worked : " [11721] Checking state of biblios index [11721] Dropping and recreating biblios index [11721] Checking state of authorities index [11721] Dropping and recreating authorities index [11721] Indexing biblios [...] [11721] Indexing authorities [...] [11721] Commit complete [...] [11721] Total 6635 records indexed " Did I miss something on the way here ? -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=32707 Fridolin Somers <fridolin.somers@biblibre.com> changed: What |Removed |Added ---------------------------------------------------------------------------- See Also| |https://bugs.koha-community | |.org/bugzilla3/show_bug.cgi | |?id=24143 CC| |fridolin.somers@biblibre.co | |m --- Comment #4 from Fridolin Somers <fridolin.somers@biblibre.com> --- Looks like this is related to Bug 24143 Please have a look. Don't you think we should avoid all query changes (truncation, ...) for search in identifiers ? -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=32707 David Cook <dcook@prosentient.com.au> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |dcook@prosentient.com.au -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=32707 Katrin Fischer <katrin.fischer@bsz-bw.de> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |andrewfh@dubcolib.org --- Comment #6 from Katrin Fischer <katrin.fischer@bsz-bw.de> --- *** Bug 24129 has been marked as a duplicate of this bug. *** -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=32707 Janusz Kaczmarek <januszop@gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|In Discussion |Needs Signoff Patch complexity|--- |Small patch -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=32707 --- Comment #9 from Janusz Kaczmarek <januszop@gmail.com> --- Created attachment 158008 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=158008&action=edit Bug 32707: ElasticSearch should not auto truncate (even if QueryAutoTruncate = 1) for identifiers (and some other fields) Koha with Zebra prevented auto truncation in some circumstances -- see the first return for ccl inside C4::Search:: buildQuery, before applying auto truncation, and setting $auto_truncation = 0 for some search fields. Koha with ElasticSearch applies auto truncation for all search fields, not paying attention to these special cases when it should not be done. This leads to various problems as described in bug 26508, 26608, etc. The solution would be to prevent auto truncation for certain search fields, above all – the identifiers. In addition, under no circumstances should a search field other than of text type be truncated (an attempt to truncate would generate an exception from ElasticSearch, e.g. number_format_exception for integer). Test plan ========= Scenario A (authority) ---------------------- 1. Enable Elasticsearch engine. 2. Have like 100+ bibliographic records and properly linked authority records. 3. Reindex ElasticSearch if needed. 4. Enable QueryAutoTruncate systempreference. 5. Find the authority record #1 and note the number of linked biblio records. 6. Show the detail of authority #1 and compare the number of linked biblio records. 7. If in the database there are authority records with ids =~ /^1/ (i.e. 10, 11, 12, ..., 100, 101, ...) linked to the biblio records you get two different numbers of linked records. 8. Also, as lists of linked biblio records (via link: Used in N record(s) from results view and detail view) you will get two different sets of biblio records. In particular, on the list generated from detail view (authorities/detail.pl?authid=1) you will get biblio records that are in fact not linked to the auth #1 (the list is generated with &q=an:1). 9. Apply this patch. 10. Counts and list of linked biblios should be ok. Scenario B (analytics) ---------------------- 1. Enable Elasticsearch engine. 2. Have three monographic bibliographic records with 001 = 1, 10, 100 (i.e. =~ /^1/). 3. Have an analytical record with 773 $w = 1. 4. Reindex ElasticSearch if needed. 5. Enable QueryAutoTruncate and UseControlNumber systempreference. 6. Find the analytical record and click on the link generated from 773, i.e. (In: ... --> catalogue/search.pl?q=Control-number:1). 7. You should see a list of three records (001 = 1, 10, 100) instead of one (001 = 1). 8. Apply this patch. Make sure that record-control-number is included in ESPreventAutoTruncate syspref. 9. Repeat step 6. You should arrive at the right host record. -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=32707 --- Comment #11 from Janusz Kaczmarek <januszop@gmail.com> --- Koha with Zebra prevented auto truncation in some circumstances -- see the first return for ccl inside C4::Search:: buildQuery, before applying auto truncation, and setting $auto_truncation = 0 for some search fields. Koha with ElasticSearch applies auto truncation for all search fields, not paying attention to these special cases when it should not be done. This leads to various problems as described in bug 26508, 26608, etc. The solution would be to prevent auto truncation for certain search fields, above all – the identifiers. In addition, under no circumstances should a search field other than of text type be truncated (an attempt to truncate would generate an exception from ElasticSearch, e.g. number_format_exception for integer). Test plan ========= Scenario A (authority) ---------------------- 1. Enable Elasticsearch engine. 2. Have like 100+ bibliographic records and properly linked authority records. 3. Reindex ElasticSearch if needed. 4. Enable QueryAutoTruncate systempreference. 5. Find the authority record #1 and note the number of linked biblio records. 6. Show the detail of authority #1 and compare the number of linked biblio records. 7. If in the database there are authority records with ids =~ /^1/ (i.e. 10, 11, 12, ..., 100, 101, ...) linked to the biblio records you get two different numbers of linked records. 8. Also, as lists of linked biblio records (via link: Used in N record(s) from results view and detail view) you will get two different sets of biblio records. In particular, on the list generated from detail view (authorities/detail.pl?authid=1) you will get biblio records that are in fact not linked to the auth #1 (the list is generated with &q=an:1). 9. Apply this patch. 10. Counts and list of linked biblios should be ok. Scenario B (analytics) ---------------------- 1. Enable Elasticsearch engine. 2. Have three monographic bibliographic records with 001 = 1, 10, 100 (i.e. =~ /^1/). 3. Have an analytical record with 773 $w = 1. 4. Reindex ElasticSearch if needed. 5. Enable QueryAutoTruncate and UseControlNumber systempreference. 6. Find the analytical record and click on the link generated from 773, i.e. (In: ... --> catalogue/search.pl?q=Control-number:1). 7. You should see a list of three records (001 = 1, 10, 100) instead of one (001 = 1). 8. Apply this patch. Make sure that record-control-number is included in ESPreventAutoTruncate syspref. 9. Repeat step 6. You should arrive at the right host record. -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=32707 Katrin Fischer <katrin.fischer@bsz-bw.de> changed: What |Removed |Added ---------------------------------------------------------------------------- See Also| |https://bugs.koha-community | |.org/bugzilla3/show_bug.cgi | |?id=33842 -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=32707 --- Comment #13 from Janusz Kaczmarek <januszop@gmail.com> --- Created attachment 161035 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=161035&action=edit biblio and auth data sample to test the patch Since the biblio and auth data in ktd are not linked (and even not related), this can be a difficulty in testing the patch. I am attaching 500 biblio records with almost 2,000 auth records already linked by subfield $9 in biblio heading fields. This data should allow you to test the patch according to the proposed scenarios. Data import (in ktd): ./misc/migration_tools/bulkmarcimport.pl -a -d -file bug_32707_auths_sample.mrc ./misc/migration_tools/bulkmarcimport.pl -b -d -file bug_32707_bibs_sample.mrc sudo koha-elasticsearch --rebuild -r -d -a -b kohadev -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=32707 Victor Grousset/tuxayo <victor@tuxayo.net> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |victor@tuxayo.net --- Comment #14 from Victor Grousset/tuxayo <victor@tuxayo.net> --- Hi :)

Scenario A (authority) 5. Find the authority record #1 and note the number of linked biblio records. 6. Show the detail of authority #1 and compare the number of linked biblio records. 7. If in the database there are authority records with ids =~ /^1/ (i.e. 10, 11, 12, ..., 100, 101, ...) linked to the biblio records you get two different numbers of linked records.

In sample data an:1 isn't linked to anything and doesn't have a name. I completed it with a name. Then linked it to a 100$a of a record, replacing the current author. When searching the auth name I get it, it tells me 1 linked record. Auth details page tells me 1 linked record also. Both are the same, so it fails to reproduce the issue :( Auths 10 to 19 have linked records so it should cause the bug, right? SearchEngine is indeed set to ES. I've started KTD with it and catalogue search is working. Anything else to check to be able to reproduce? -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=32707 --- Comment #15 from Janusz Kaczmarek <januszop@gmail.com> --- (In reply to Victor Grousset/tuxayo from comment #14)

In sample data an:1 isn't linked to anything and doesn't have a name. I completed it with a name. Then linked it to a 100$a of a record, replacing the current author.

When searching the auth name I get it, it tells me 1 linked record. Auth details page tells me 1 linked record also. Both are the same, so it fails to reproduce the issue :( Auths 10 to 19 have linked records so it should cause the bug, right? SearchEngine is indeed set to ES. I've started KTD with it and catalogue search is working.

Anything else to check to be able to reproduce?

Hi! Sorry I've missed the message. That's somehow strange. I just repeated the test scenario and it seems OK with the data provided. The authid #1 should be 110 $a Institute of Jesuit Sources (Boston). -- linked to 453 biblio records. While importing, did you delete the records provided with ktd (bulkmarcimport.pl with -d -fk switches)? Anyway, I'm going to prepare a refreshed version of sample data, since the problem of linked analytical records is not well illustrated with the current set. -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=32707 Janusz Kaczmarek <januszop@gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #158007|0 |1 is obsolete| | Attachment #158008|0 |1 is obsolete| | Attachment #158009|0 |1 is obsolete| | --- Comment #17 from Janusz Kaczmarek <januszop@gmail.com> --- Created attachment 163749 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=163749&action=edit Bug 32707: Add 'ESPreventAutoTruncate' preference This patch adds a new 'ESPreventAutoTruncate' preference allowing to define Elasticsearch search fields that should not be autotruncated when 'QueryAutoTruncate' is active (e.g. barcode). -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=32707 --- Comment #19 from Janusz Kaczmarek <januszop@gmail.com> --- Created attachment 163751 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=163751&action=edit Bug 32707: Unit tests NB in t/db_dependent/Koha/SearchEngine/Elasticsearch/QueryBuilder.t, I had to modify four existing test by changing Local-number:123456 to Personal-name:donald -- since Local-number should never be auto truncated according to the submitted patch. -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=32707 --- Comment #21 from Victor Grousset/tuxayo <victor@tuxayo.net> --- For test plan B, what is a clean way to have an analytical record with 773 $w = 1 ? I after learning about analytics records, I tried to fill the gaps but in the end I still get 111 results with that link: http://localhost:8081/cgi-bin/koha/catalogue/search.pl?q=Control-number:1 Which for step B 7. doesn't make sense. --- Scenario A worked as expected :) --- testing notes: after test data import + reindex, restart services to get the right counts of linked records. -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=32707 --- Comment #23 from Janusz Kaczmarek <januszop@gmail.com> --- (In reply to Janusz Kaczmarek from comment #22)

(In reply to Victor Grousset/tuxayo from comment #21)

...

For test plan B, what is a clean way to have an analytical record with 773 $w = 1 ? I after learning about analytics records, I tried to fill the gaps but in the end I still get 111 results with that link: http://localhost:8081/cgi-bin/koha/catalogue/search.pl?q=Control-number:1 Which for step B 7. doesn't make sense.

Did you put "control-number" in the new ESPreventAutoTruncate preference?

I'm sorry--for analytical records to work correctly, you should put both: control-number and record-control-number in the ESPreventAutoTruncate preference. Explanation: with searchfield control-number you go "up", i.e. from 773 $w to the host record; with record-control-number you go "down", i.e. from the host record the linked analytical records. A patch with corrected test plan follows. -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=32707 --- Comment #25 from Victor Grousset/tuxayo <victor@tuxayo.net> ---

Did you put "control-number" in the new ESPreventAutoTruncate preference?

Oh no I forgot to updatedatabase >_< Still, without patch, I should get 3 results, not 111. That's why my improvisation in making the analytical records might be wrong. Ok you fixed the test plan on that, also, thanks :D Let's retry :) -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=32707 --- Comment #27 from Victor Grousset/tuxayo <victor@tuxayo.net> ---

3. Have an analytical record with 773 $w = 1 (in the test data set - biblio 896).

Oh nice! I don't even have to mess around unreliably trying to create analytical records :) -- Scenario B works :) I'll retest both after the tests are fixed in case it requires changes in the implementation. -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=32707 Janusz Kaczmarek <januszop@gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Failed QA |Needs Signoff -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=32707 --- Comment #30 from Janusz Kaczmarek <januszop@gmail.com> --- Created attachment 163823 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=163823&action=edit Bug 32707: ElasticSearch should not auto truncate (even if QueryAutoTruncate = 1) for identifiers (and some other fields) Koha with Zebra prevented auto truncation in some circumstances -- see the first return for ccl inside C4::Search:: buildQuery, before applying auto truncation, and setting $auto_truncation = 0 for some search fields. Koha with ElasticSearch applies auto truncation for all search fields, not paying attention to these special cases when it should not be done. This leads to various problems as described in bug 26508, 26608, etc. The solution would be to prevent auto truncation for certain search fields, above all – the identifiers. In addition, under no circumstances should a search field other than of text type be truncated (an attempt to truncate would generate an exception from ElasticSearch, e.g. number_format_exception for integer). Test plan ========= 0. Use the test data sample provided in the bug report. Scenario A (authority) ---------------------- 1. Enable Elasticsearch engine. 2. Have like 100+ bibliographic records and properly linked authority records. 3. Reindex ElasticSearch if needed. 4. Enable QueryAutoTruncate systempreference. 5. Find the authority record #1 and note the number of linked biblio records. 6. Show the detail of authority #1 and compare the number of linked biblio records. 7. If in the database there are authority records with ids =~ /^1/ (i.e. 10, 11, 12, ..., 100, 101, ...) linked to the biblio records you get two different numbers of linked records. 8. Also, as lists of linked biblio records (via link: Used in N record(s) from results view and detail view) you will get two different sets of biblio records. In particular, on the list generated from detail view (authorities/detail.pl?authid=1) you will get biblio records that are in fact not linked to the auth #1 (the list is generated with &q=an:1). 9. Apply this patch. 10. Counts and list of linked biblios should be ok. Scenario B (analytics) ---------------------- 1. Enable Elasticsearch engine. 2. Have three monographic bibliographic records with 001 = 1, 10, 100 (i.e. =~ /^1/). 3. Have an analytical record with 773 $w = 1 (in the test data set - biblio 896). 4. Enable QueryAutoTruncate and UseControlNumber systempreference. 6. Find the analytical record and click on the link generated from 773, i.e. (In: ... --> catalogue/search.pl?q=Control-number:1). You should see a list of 100+ records (001 = 1, 10, 100) instead of one (001 = 1). 7. From the biblio # 1 try to go to the analytice records (with Show analytics link) - you should get 60+ records with from different host records (773 $t) -- instead of one. 8. Apply this patch. Make sure that control-number and record-control-number are included in ESPreventAutoTruncate syspref. 9. Repeat steps 6 and 7. You should arrive at the right host record in p. 6 and at one analytical records when looking for analytical records in p. 7 (or more than one, but right, if you modified the test data set). -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=32707 --- Comment #32 from Janusz Kaczmarek <januszop@gmail.com> --- (In reply to Victor Grousset/tuxayo from comment #26)

Regardless of me retesting, there are other issues:

Tests don't pass:

t/Koha/SearchEngine/Elasticsearch/QueryBuilder.t ............... 1/7 # Failed test 'no truncation for non non test fields (date) - OK' # at t/Koha/SearchEngine/Elasticsearch/QueryBuilder.t line 171. # got: '1' # expected: '0'

# Failed test 'no truncation for non non test fields (date) - OK' # at t/Koha/SearchEngine/Elasticsearch/QueryBuilder.t line 183. # got: '1' # expected: '0' # Looks like you failed 2 tests of 9. t/Koha/SearchEngine/Elasticsearch/QueryBuilder.t ............... 4/7

Updated version provided. Unfortunately, after modifying the patch in October, I didn't update the tests. I'm sorry for this. -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=32707 Victor Grousset/tuxayo <victor@tuxayo.net> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #163822|0 |1 is obsolete| | --- Comment #34 from Victor Grousset/tuxayo <victor@tuxayo.net> --- Created attachment 163881 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=163881&action=edit Bug 32707: Add 'ESPreventAutoTruncate' preference This patch adds a new 'ESPreventAutoTruncate' preference allowing to define Elasticsearch search fields that should not be autotruncated when 'QueryAutoTruncate' is active (e.g. barcode). Signed-off-by: Victor Grousset/tuxayo <victor@tuxayo.net> -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=32707 Victor Grousset/tuxayo <victor@tuxayo.net> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Needs Signoff |Signed Off --- Comment #36 from Victor Grousset/tuxayo <victor@tuxayo.net> --- It works :) --- Thanks for the other fixes. To spare a QA roundtrip, you can fix the remaining issues: WARN Koha/SearchEngine/Elasticsearch/QueryBuilder.pm WARN tidiness The file is less tidy than before (bad/messy lines before: 318, now: 333) WARN installer/data/mysql/atomicupdate/bug_32707.pl WARN tidiness The file is less tidy than before (bad/messy lines before: 0, now: 5) WARN t/Koha/SearchEngine/Elasticsearch/QueryBuilder.t WARN tidiness The file is less tidy than before (bad/messy lines before: 72, now: 85) It's a WARN because it's prone to false positives, like at least for autogenerated code or when moving code. But in this case it seems valid. -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=32707 --- Comment #38 from Janusz Kaczmarek <januszop@gmail.com> --- Created attachment 163893 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=163893&action=edit Bug 32707: ElasticSearch should not auto truncate (even if QueryAutoTruncate = 1) for identifiers (and some other fields) Koha with Zebra prevented auto truncation in some circumstances -- see the first return for ccl inside C4::Search:: buildQuery, before applying auto truncation, and setting $auto_truncation = 0 for some search fields. Koha with ElasticSearch applies auto truncation for all search fields, not paying attention to these special cases when it should not be done. This leads to various problems as described in bug 26508, 26608, etc. The solution would be to prevent auto truncation for certain search fields, above all – the identifiers. In addition, under no circumstances should a search field other than of text type be truncated (an attempt to truncate would generate an exception from ElasticSearch, e.g. number_format_exception for integer). Test plan ========= 0. Use the test data sample provided in the bug report. Comment 16 in the ticket. Scenario A (authority) ---------------------- 1. Enable Elasticsearch engine. 2. Have like 100+ bibliographic records and properly linked authority records. 3. Reindex ElasticSearch if needed. 4. Enable QueryAutoTruncate systempreference. 5. Find the authority record #1 and note the number of linked biblio records. 6. Show the detail of authority #1 and compare the number of linked biblio records. 7. If in the database there are authority records with ids =~ /^1/ (i.e. 10, 11, 12, ..., 100, 101, ...) linked to the biblio records you get two different numbers of linked records. 8. Also, as lists of linked biblio records (via link: Used in N record(s) from results view and detail view) you will get two different sets of biblio records. In particular, on the list generated from detail view (authorities/detail.pl?authid=1) you will get biblio records that are in fact not linked to the auth #1 (the list is generated with &q=an:1). 8.99. Skip to scenario B and come back here after finishing to not have to unapply the patches and restart services for nothing. 9. Apply this patch. 10. Counts and list of linked biblios should be ok. Scenario B (analytics) ---------------------- 1. Enable Elasticsearch engine. 2. Have three monographic bibliographic records with 001 = 1, 10, 100 (i.e. =~ /^1/). 3. Have an analytical record with 773 $w = 1 (in the test data set - biblio 896). 4. Enable QueryAutoTruncate and UseControlNumber systempreference. 6. Find the analytical record and click on the link generated from 773, i.e. (In: ... --> catalogue/search.pl?q=Control-number:1). You should see a list of 100+ records (001 = 1, 10, 100) instead of one (001 = 1). 7. From the biblio # 1 try to go to the analytice records (with Show analytics link) - you should get 60+ records with from different host records (773 $t) -- instead of one. 8. Apply this patch. Make sure that control-number and record-control-number are included in ESPreventAutoTruncate syspref. 9. Repeat steps 6 and 7. You should arrive at the right host record in p. 6 and at one analytical records when looking for analytical records in p. 7 (or more than one, but right, if you modified the test data set). Signed-off-by: Victor Grousset/tuxayo <victor@tuxayo.net> -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=32707 --- Comment #40 from Janusz Kaczmarek <januszop@gmail.com> --- (In reply to Victor Grousset/tuxayo from comment #36)

It's a WARN because it's prone to false positives, like at least for autogenerated code or when moving code. But in this case it seems valid.

Thank you. Tidiness corrected. -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=32707 Nick Clemens <nick@bywatersolutions.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Signed Off |Passed QA -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=32707 --- Comment #42 from Nick Clemens <nick@bywatersolutions.com> --- Created attachment 164141 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=164141&action=edit Bug 32707: ElasticSearch should not auto truncate (even if QueryAutoTruncate = 1) for identifiers (and some other fields) Koha with Zebra prevented auto truncation in some circumstances -- see the first return for ccl inside C4::Search:: buildQuery, before applying auto truncation, and setting $auto_truncation = 0 for some search fields. Koha with ElasticSearch applies auto truncation for all search fields, not paying attention to these special cases when it should not be done. This leads to various problems as described in bug 26508, 26608, etc. The solution would be to prevent auto truncation for certain search fields, above all – the identifiers. In addition, under no circumstances should a search field other than of text type be truncated (an attempt to truncate would generate an exception from ElasticSearch, e.g. number_format_exception for integer). Test plan ========= 0. Use the test data sample provided in the bug report. Comment 16 in the ticket. Scenario A (authority) ---------------------- 1. Enable Elasticsearch engine. 2. Have like 100+ bibliographic records and properly linked authority records. 3. Reindex ElasticSearch if needed. 4. Enable QueryAutoTruncate systempreference. 5. Find the authority record #1 and note the number of linked biblio records. 6. Show the detail of authority #1 and compare the number of linked biblio records. 7. If in the database there are authority records with ids =~ /^1/ (i.e. 10, 11, 12, ..., 100, 101, ...) linked to the biblio records you get two different numbers of linked records. 8. Also, as lists of linked biblio records (via link: Used in N record(s) from results view and detail view) you will get two different sets of biblio records. In particular, on the list generated from detail view (authorities/detail.pl?authid=1) you will get biblio records that are in fact not linked to the auth #1 (the list is generated with &q=an:1). 8.99. Skip to scenario B and come back here after finishing to not have to unapply the patches and restart services for nothing. 9. Apply this patch. 10. Counts and list of linked biblios should be ok. Scenario B (analytics) ---------------------- 1. Enable Elasticsearch engine. 2. Have three monographic bibliographic records with 001 = 1, 10, 100 (i.e. =~ /^1/). 3. Have an analytical record with 773 $w = 1 (in the test data set - biblio 896). 4. Enable QueryAutoTruncate and UseControlNumber systempreference. 6. Find the analytical record and click on the link generated from 773, i.e. (In: ... --> catalogue/search.pl?q=Control-number:1). You should see a list of 100+ records (001 = 1, 10, 100) instead of one (001 = 1). 7. From the biblio # 1 try to go to the analytice records (with Show analytics link) - you should get 60+ records with from different host records (773 $t) -- instead of one. 8. Apply this patch. Make sure that control-number and record-control-number are included in ESPreventAutoTruncate syspref. 9. Repeat steps 6 and 7. You should arrive at the right host record in p. 6 and at one analytical records when looking for analytical records in p. 7 (or more than one, but right, if you modified the test data set). Signed-off-by: Victor Grousset/tuxayo <victor@tuxayo.net> Signed-off-by: Nick Clemens <nick@bywatersolutions.com> -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=32707 Katrin Fischer <katrin.fischer@bsz-bw.de> changed: What |Removed |Added ---------------------------------------------------------------------------- Summary|ElasticSearch should not |Elasticsearch should not |auto truncate (even if |auto truncate (even if |QueryAutoTruncate = 1) for |QueryAutoTruncate = 1) for |identifiers (and some other |identifiers (and some other |fields) |fields) Keywords| |release-notes-needed -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=32707 Katrin Fischer <katrin.fischer@bsz-bw.de> changed: What |Removed |Added ---------------------------------------------------------------------------- Version(s)| |24.05.00 released in| | Status|Passed QA |Pushed to master -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=32707 Fridolin Somers <fridolin.somers@biblibre.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Pushed to main |Pushed to stable Version(s)|24.05.00 |24.05.00,23.11.06 released in| | --- Comment #46 from Fridolin Somers <fridolin.somers@biblibre.com> --- Pushed to 23.11.x for 23.11.06 -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=32707 Caroline Cyr La Rose <caroline.cyr-la-rose@inlibro.com> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |caroline.cyr-la-rose@inlibr | |o.com Documentation| |Caroline Cyr La Rose contact| | Documentation| |https://gitlab.com/koha-com submission| |munity/koha-manual/-/merge_ | |requests/933 Keywords|no-sandbox, | |release-notes-needed | --- Comment #48 from Caroline Cyr La Rose <caroline.cyr-la-rose@inlibro.com> --- I tried to document this new system preference to the best of my ability. I thought it would be useful to list the Zebra indexes that are not affected by QueryAutoTruncate and tried to find them in C4/Search.pm as mentioned in the description. This is what I found, I hope it is correct. - yr (date of publication) - acqdate (date of acquisition) - nb (ISBN) - ns (ISSN) - hi (host itemnumber) Let me know if there is anything wrong or anything else I should add. https://gitlab.com/koha-community/koha-manual/-/merge_requests/933/diffs -- You are receiving this mail because: You are watching all bug changes.