http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=9579 Katrin Fischer <katrin.fischer@bsz-bw.de> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|NEW |RESOLVED Resolution|--- |WORKSFORME --- Comment #2 from Katrin Fischer <katrin.fischer@bsz-bw.de> --- Please reopen if that does not solve your problem. -- You are receiving this mail because: You are watching all bug changes.

http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=9579 --- Comment #4 from Tigran <tigran@flib.sci.am> --- (In reply to comment #2)

Please reopen if that does not solve your problem.

(In reply to comment #0)

Created attachment 15202 [details] 3 iamges with incorrect UNICODE

Attaching 3 images - 2 for Cyrillic records and one or Armenian. You will notice that instead of the letter some other code is displayed and when pressing on the names from left pan (Refine your search), the search is returning null results (see arm1.png). Another interesting detail. When pressing om left pan name, the display on search window of Armenian name is displayed correctly (see arm1.png) but on the top (URL window) some of letters are displayed incorrectly. When we are just typing Armenian or Cyrillic scripts in the search window everything is working fine. So the problem is how is encoding/indexing going with the 'Refine your search' pan. Tigran

(In reply to comment #1)

Hi Tigran,

try increasing the value in the system preference FacetLabelTruncationLength. Your broken characters always show up where the system cuts of the string - so this could fix it.

Hi Katrin After changes in system preference the problem is not solved (we have done reindexing). Attaching images wirh explanations. Letters are displayed in not correct way in the URL, although in the left pan the display is normal. Tgran -- You are receiving this mail because: You are watching all bug changes.

http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=9579 --- Comment #6 from Tigran <tigran@flib.sci.am> --- Noticed that for Cyrillic 2 capital letters are displayed incorrectly (the letters with the codes 041F and 0421). and for Armenian 2 small letters are displayed incorrectly (the letters with the cods 0561 and 0576) Tigran -- You are receiving this mail because: You are watching all bug changes.

http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=9579 Jared Camins-Esakov <jcamins@cpbibliography.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Severity|enhancement |major -- You are receiving this mail because: You are watching all bug changes.

http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=9579 --- Comment #8 from Karam Qubsi <karamqubsi@gmail.com> --- Hi Fridolyn, Thanks a lot for your efforts ^_^ I just have a basic information about git so I've tried to apply the changes manually and installed this from cpan TruncateByWord, see http://search.cpan.org/~kawabata/Template-Plugin-TruncateByWord-0.11/ but the results is still not good maybe I miss something :( can you help me to test this with you ? some links to read about applying a patch with git . and I wish you test it with the attached Arabic records or any one if want to test .file name : arabicrecords.mrc Many thanks for you Karam. -- You are receiving this mail because: You are watching all bug changes.

http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=9579 mathieu saby <mathieu.saby@univ-rennes2.fr> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |mathieu.saby@univ-rennes2.f | |r -- You are receiving this mail because: You are watching all bug changes.

http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=9579 --- Comment #11 from mathieu saby <mathieu.saby@univ-rennes2.fr> --- Fridolyn, your patch does not fix the problem. Mathieu -- You are receiving this mail because: You are watching all bug changes.

http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=9579 Bernardo Gonzalez Kriegel <bgkriegel@gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |bgkriegel@gmail.com --- Comment #13 from Bernardo Gonzalez Kriegel <bgkriegel@gmail.com> --- Hi, could you please test Bug 6554. I found that that bug solves this problem. -- You are receiving this mail because: You are watching all bug changes.

http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=9579 --- Comment #14 from Karam Qubsi <karamqubsi@gmail.com> --- The patch in http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=6554 was reverted and this problem is come back again to 3.12 . -- You are receiving this mail because: You are watching all bug changes.

http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=9579 --- Comment #15 from Bernardo Gonzalez Kriegel <bgkriegel@gmail.com> --- Hi, this is no more in current master, fixed by Bug 10176. I did test using Karam records, for me facets are working. Could someone else test? -- You are receiving this mail because: You are watching all bug changes.

http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=9579 --- Comment #17 from Bernardo Gonzalez Kriegel <bgkriegel@gmail.com> --- Hi Tigran, you can have two problems: wrong display and wrong links, 10176 fixes the link, not the display :-) To fix the display we need to rewrite how facets are trimmed, at this moment it use preference FacetLabelTruncationLength, that truncates facet text at that lenght, which is alright but only for scripts that use latin or not ligate chars. It needs to be rewritten so it cut facet text on " " (space) position <= FacetLabelTruncationLength. A workaround is set FacetLabelTruncationLength to a bigger value, 50 for example, then facets will not show those <?>. The test I was talking about involves the links :-) -- You are receiving this mail because: You are watching all bug changes.

http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=9579 Tomás Cohen Arazi <tomascohen@gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |tomascohen@gmail.com Version|3.12 |master -- You are receiving this mail because: You are watching all bug changes.

http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=9579 Tomás Cohen Arazi <tomascohen@gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Failed QA |Needs Signoff Assignee|fridolyn.somers@biblibre.co |tomascohen@gmail.com |m | --- Comment #19 from Tomás Cohen Arazi <tomascohen@gmail.com> --- Created attachment 24426 --> http://bugs.koha-community.org/bugzilla3/attachment.cgi?id=24426&action=edit Bug 9579: (followup) add dependency for String::Multibyte Bug 9579: (followup) add dependency for String::Multibyte -- You are receiving this mail because: You are watching all bug changes.

http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=9579 Tomás Cohen Arazi <tomascohen@gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #24425|0 |1 is obsolete| | Attachment #24426|0 |1 is obsolete| | --- Comment #20 from Tomás Cohen Arazi <tomascohen@gmail.com> --- Created attachment 24430 --> http://bugs.koha-community.org/bugzilla3/attachment.cgi?id=24430&action=edit Bug 9579: Facets truncation broken for multi-byte characters We seem to be relying on whatever Zoom::Results->render return, and Perl doesn't explicitly consider it UNICODE data. That's why CORE::substr (and probably CORE::length too) cut the bytes wrong. This patch just decodes the UTF-8 data that render() returns and then Perl behaves, heh. It uses Encode::decode_utf8 which is already a dependency for the current stable Koha releases. To test: - Set FacetLabelTruncationLength = 4 to make sure it will try to truncate the facet label. - Add a record with multiple multibyte characters in the author name (copy and paste a cyrillic character several times so it is looong). Note: make sure there are at least two records so you're presented the search results. - Reindex zebra to make it possible to search the record. - Search for the record. - Notice the facet with cyrillic characters is broken in the place it gets truncated. - Apply the patch and repeat the search. - Notice the string is not broken. - Sign off. Regards To+ Sponsored-by: Universidad Nacional de Cordoba -- You are receiving this mail because: You are watching all bug changes.

http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=9579 M. Tompsett <mtompset@hotmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #24430|0 |1 is obsolete| | CC| |mtompset@hotmail.com --- Comment #22 from M. Tompsett <mtompset@hotmail.com> --- Created attachment 24684 --> http://bugs.koha-community.org/bugzilla3/attachment.cgi?id=24684&action=edit Facets truncation broken for multi-byte characters You may have issues trying to git bz this. I had to upload this old-school, because I have يكيمكتبات : قبسي ، كرم in the commit message. -- You are receiving this mail because: You are watching all bug changes.

http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=9579 --- Comment #24 from Katrin Fischer <katrin.fischer@bsz-bw.de> --- Created attachment 24725 --> http://bugs.koha-community.org/bugzilla3/attachment.cgi?id=24725&action=edit [PASSED QA] Bug 9579: Facets truncation broken for multi-byte characters We seem to be relying on whatever Zoom::Results->render return, and Perl doesn't explicitly consider it UNICODE data. That's why CORE::substr (and probably CORE::length too) cut the bytes wrong. This patch just decodes the UTF-8 data that render() returns and then Perl behaves, heh. It uses Encode::decode_utf8 which is already a dependency for the current stable Koha releases. REVISED TEST PLAN ----------------- 1) Import the attached sample records. 2) Rebuild your indexes 3) In OPAC search for يكيمكتبات : قبسي ، كرم -- There will be ugly diamonds with question marks in the facets 4) apply the patch 5) Search again. -- The names will be properly truncated. NOTE: This test assumes FacetLabelTruncationLength = 20. Regards To+ Sponsored-by: Universidad Nacional de Cordoba Signed-off-by: Mark Tompsett <mtompset@hotmail.com> Signed-off-by: Katrin Fischer <Katrin.Fischer.83@web.de> Passes all tests and QA script. Works as described, tested with several German, English and the Arabic test record. Arabic strings now display correctly and no regression was found. -- You are receiving this mail because: You are watching all bug changes.

http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=9579 Katrin Fischer <katrin.fischer@bsz-bw.de> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #24684|0 |1 is obsolete| | -- You are receiving this mail because: You are watching all bug changes.

http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=9579 --- Comment #25 from Galen Charlton <gmcharlt@gmail.com> --- Created attachment 24959 --> http://bugs.koha-community.org/bugzilla3/attachment.cgi?id=24959&action=edit Bug 9579: fix truncation of facets containing multi-byte characters We seem to be relying on whatever Zoom::Results->render return, and Perl doesn't explicitly consider it UNICODE data. That's why CORE::substr (and probably CORE::length too) cut the bytes wrong. This patch just decodes the UTF-8 data that render() returns and then Perl behaves, heh. It uses Encode::decode_utf8 which is already a dependency for the current stable Koha releases. REVISED TEST PLAN ----------------- 1) Import the attached sample records. 2) Rebuild your indexes 3) In OPAC search for يكيمكتبات : قبسي ، كرم -- There will be ugly diamonds with question marks in the facets 4) apply the patch 5) Search again. -- The names will be properly truncated. NOTE: This test assumes FacetLabelTruncationLength = 20. Sponsored-by: Universidad Nacional de Cordoba Signed-off-by: Mark Tompsett <mtompset@hotmail.com> Signed-off-by: Katrin Fischer <Katrin.Fischer.83@web.de> Passes all tests and QA script. Works as described, tested with several German, English and the Arabic test record. Arabic strings now display correctly and no regression was found. Signed-off-by: Galen Charlton <gmc@esilibrary.com> I've reviewed it and approve its inclusion in 3.14.x and earlier. I will use the patches for bug 11096, once they pass QA, for the master branch. Signed-off-by: Galen Charlton <gmc@esilibrary.com> -- You are receiving this mail because: You are watching all bug changes.

http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=9579 Galen Charlton <gmcharlt@gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #24725|0 |1 is obsolete| | -- You are receiving this mail because: You are watching all bug changes.

http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=9579 Fridolin SOMERS <fridolyn.somers@biblibre.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Version|master |3.14 -- You are receiving this mail because: You are watching all bug changes.