Re: [Koha-devel] koha-elasticsearch verbosity issue
Can you provide some examples of the log messages? That might help identify where they are coming from. You could also raise the commit number to generate fewer requests/responses? On Fri, Sep 8, 2023 at 9:13 AM Nick Clemens <nick@bywatersolutions.com> wrote:
Hi Udo,
What versions of ES are you using? We currently face an issue of deprecation warnings as we have not resolved use of 'types' in ES
Our systems found this was causing a large log on each reindex and piped the logs through sed to remove the warnings
-Nick
On Thu, Sep 7, 2023 at 8:04 AM Udo Willke <udo.willke@kit.edu> wrote:
Hi koha developers,
Quick introduction: As a system admin, I am part of the team that rolls out the koha software at KIT Karlsruhe.
Currently I have an issue with the koha-elasticsearch script that is used to index the biblio and authorities data in elasticsearch. We have different test and development servers and on some of the severs I get extensive diagnostic output when running the command while on others I only get the errors and warnings.
In every run /usr/sbin/koha-elasticsearch (which subsequently calls /usr/share/koha/bin/search_tools/rebuild_elasticsearch.pl) is definitely invoked without any "verbose" flag
Unfortunately there is no output on stdout, but everything gets written to stderr. As we have 1.3M biblios and 600k authorities I would get tens of gigabytes of log file, which I would have to check.
My question: is there somewhere a setting in koha that globally controls the debug level? Or how can the different behaviours be explained?
All installations run koha Release 23.05.X on Ubuntu 22.04.3 LTS
Any thoughts are welcome.
Many thanks in advance
Udo
_______________________________________________ Koha-devel mailing list Koha-devel@lists.koha-community.org https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-devel website : https://www.koha-community.org/ git : https://git.koha-community.org/ bugs : https://bugs.koha-community.org/
-- Nick Clemens ByWater Solutions bywatersolutions.com Phone: (888) 900-8944 Pronouns: (he/him/his) Timezone: Eastern Follow us: <https://www.facebook.com/ByWaterSolutions/> <https://www.instagram.com/bywatersolutions/> <https://www.youtube.com/user/bywatersolutions> <https://twitter.com/ByWaterSolution>
-- Nick Clemens ByWater Solutions bywatersolutions.com Phone: (888) 900-8944 Pronouns: (he/him/his) Timezone: Eastern Follow us: <https://www.facebook.com/ByWaterSolutions/> <https://www.instagram.com/bywatersolutions/> <https://www.youtube.com/user/bywatersolutions> <https://twitter.com/ByWaterSolution>
Hi Nick, the start of the log data looks like this:
[DEPRECATION] [types removal] Using include_type_name in put mapping requests is deprecated. The parameter will be removed in the next major version. - In request: {body => {data => {properties => {abl => {analyzer => "analyzer_standard",fields => {ci_raw => {normalizer => "icu_folding_normalizer",type => "keyword"},phrase => {analyzer => "analyzer_phrase",search_analyzer => "analyzer_phrase",type => "text"},raw => {normalizer => "nfkc_cf_normalizer",type => "keyword"}},search_analyzer => "analyzer_standard",type => "text"},abl__sort => {index => bless( do{\(my $o = 0)}, 'JSON::PP::Boolean' ),numeric => bless( do{\(my $o = 1)}, 'JSON::PP::Boolean' ),type => "icu_collation_keyword"},abstract => $VAR1->{body}{data}{properties}{abl},abstract__sort => $VAR1->{body}{data}{properties}{abl__sort},acqsource => $VAR1->{body}{data}{properties}{abl},acqsource__sort => $VAR1->{body}{data}{properties}{abl__sort},arl => $VAR1->{body}{data}{properties}{abl},arl__sort => $VAR1->{body}{data}{properties}{abl__sort},arp => $VAR1->{body}{data}{properties}{abl},arp__sort => $VAR1->{body}{data}{properties}{abl__sort},author => $VAR1->{body}{data}{properties}{abl},"author-in-order" => $VAR1->{body}{data}{properties}{abl},"author-in-order__sort" => $VAR1->{body}{data}{properties}{abl__sort},"author-name-corporate" => $VAR1->{body}{data}{properties}{abl},"author-name-corporate__sort" => $VAR1->{body}{data}{properties}{abl__sort},"author-name-personal" => $VAR1->{body}{data}{properties}{abl},"author-name-personal__sort" => $VAR1->{body}{data}{properties}{abl__sort},"author-personal-bibliography" => $VAR1->{body}{data}{properties}{abl},"author-personal-bibliography__sort" => $VAR1->{body}{data}{properties}{abl__sort},"author-title" => $VAR1->{body}{data}{properties}{abl},"author-title__sort" => $VAR1->{body}{data}{properties}{abl__sort},author__facet => {normalizer => "facet_normalizer",type => "keyword"},author__sort => $VAR1->{body}{data}{properties}{abl__sort},author__suggestion => {analyzer => "simple",max_input_length => 100,search_analyzer => "simple",type => "completion"},barcode => $VAR1->{body}{data}{properties}{abl},barcode__sort => $VAR1->{body}{data}{properties}{abl__sort},"bgf-number" => $VAR1->{body}{data}{properties}{abl},"bgf-number__sort" => $VAR1->{body}{data}{properties}{abl__sort},"bib-level" => $VAR1->{body}{data}{properties}{abl},"bib-level__sort" => $VAR1->{body}{data}{properties}{abl__sort},biblioitemnumber => $VAR1->{body}{data}{properties}{abl},biblioitemnumber__sort => $VAR1->{body}{data}{properties}{abl__sort},bio => $VAR1->{body}{data}{properties}{abl},bio__sort => $VAR1->{body}{data}{properties}{abl__sort},"bnb-card-number" => $VAR1->{body}{data}{properties}{abl},"bnb-card-number__sort" => $VAR1->{body}{data}{properties}{abl__sort},ccode => $VAR1->{body}{data}{properties}{abl},ccode__facet => $VAR1->{body}{data}{properties}{author__facet},ccode__sort => $VAR1->{body}{data}{properties}{abl__sort},"classification-source" => $VAR1->{body}{data}{properties}{abl},"classification-source__sort" => $VAR1->{body}{data}{properties}{abl__sort},"cn-bib-sort" => $VAR1->{body}{data}{properties}{abl},"cn-bib-sort__sort" => $VAR1->{body}{data}{properties}{abl__sort},"cn-bib-source" => $VAR1->{body}{data}{properties}{abl},"cn-bib-source__sort" => $VAR1->{body}{data}{properties}{abl__sort},"cn-class" => $VAR1->{body}{data}{properties}{abl},"cn-class__sort" => $VAR1->{body}{data}{properties}{abl__sort},"cn-item" => $VAR1->{body}{data}{properties}{abl},"cn-item__sort" => $VAR1->{body}{data}{properties}{abl__sort},"cn-prefix" => $VAR1->{body}{data}{properties}{abl},"cn-prefix__sort" => $VAR1->{body}{data}{properties}{abl__sort},"cn-sort" => $VAR1->{body}{data}{properties}{abl},"cn-sort__sort" => {index => bless( do{\(my $o = 0)}, 'JSON::PP::Boolean' ),numeric => bless( do{\(my $o = 0)}, 'JSON::PP::Boolean' ),type => "icu_collation_keyword"},"cn-suffix" => $VAR1->{body}{data}{properties}{abl},"cn-suffix__sort" => $VAR1->{body}{data}{properties}{abl__sort},"code-geographic" => $VAR1->{body}{data}{properties}{abl},"code-geographic__sort" => $VAR1->{body}{data}{properties}{abl__sort},"code-institution" => $VAR1->{body}{data}{properties}{abl},"code-institution__sort" => $VAR1->{body}{data}{properties}{abl__sort},"coded-location-qualifier" => $VAR1->{body}{data}{properties}{abl},"coded-location-qualifier__sort" => $VAR1->{body}{data}{properties}{abl__sort},coden => $VAR1->{body}{data}{properties}{abl},coden__sort => $VAR1->{body}{data}{properties}{abl__sort},"conference-name" => $VAR1->{body}{data}{properties}{abl},"conference-name__sort" => $VAR1->{body}{data}{properties}{abl__sort},"control-number" => $VAR1->{body}{data}{properties}{abl},"control-number-identifier" => $VAR1->{body}{data}{properties}{abl},"control-number-identifier__sort" => $VAR1->{body}{data}{properties}{abl__sort},"control-number__sort" => $VAR1->{body}{data}{properties}{abl__sort},copydate => $VAR1->{body}{data}{properties}{abl},copydate__facet => $VAR1->{body}{data}{properties}{author__facet},copydate__sort => $VAR1->{body}{data}{properties}{abl__sort},copynumber => $VAR1->{body}{data}{properties}{abl},copynumber__sort => $VAR1->{body}{data}{properties}{abl__sort},"corporate-name" => $VAR1->{body}{data}{properties}{abl},"corporate-name__sort" => $VAR1->{body}{data}{properties}{abl__sort},"cross-reference" => $VAR1->{body}{data}{properties}{abl},"cross-reference__sort" => $VAR1->{body}{data}{properties}{abl__sort},ctype => $VAR1->
I think this is what you were pointing out earlier. Maybe I don't need to send a bigger chunk of the output. You are right: piping the log through sed would be a workaround. As I am away next week my colleagues in CC: of this mail will take over. Best Regards Udo Am 08.09.2023 um 15:58 schrieb Nick Clemens:
Can you provide some examples of the log messages? That might help identify where they are coming from.
You could also raise the commit number to generate fewer requests/responses?
On Fri, Sep 8, 2023 at 9:13 AM Nick Clemens <nick@bywatersolutions.com> wrote:
Hi Udo,
What versions of ES are you using? We currently face an issue of deprecation warnings as we have not resolved use of 'types' in ES
Our systems found this was causing a large log on each reindex and piped the logs through sed to remove the warnings
-Nick
On Thu, Sep 7, 2023 at 8:04 AM Udo Willke <udo.willke@kit.edu> wrote:
Hi koha developers,
Quick introduction: As a system admin, I am part of the team that rolls out the koha software at KIT Karlsruhe.
Currently I have an issue with the koha-elasticsearch script that is used to index the biblio and authorities data in elasticsearch. We have different test and development servers and on some of the severs I get extensive diagnostic output when running the command while on others I only get the errors and warnings.
In every run /usr/sbin/koha-elasticsearch (which subsequently calls /usr/share/koha/bin/search_tools/rebuild_elasticsearch.pl <http://rebuild_elasticsearch.pl>) is definitely invoked without any "verbose" flag
Unfortunately there is no output on stdout, but everything gets written to stderr. As we have 1.3M biblios and 600k authorities I would get tens of gigabytes of log file, which I would have to check.
My question: is there somewhere a setting in koha that globally controls the debug level? Or how can the different behaviours be explained?
All installations run koha Release 23.05.X on Ubuntu 22.04.3 LTS
Any thoughts are welcome.
Many thanks in advance
Udo
_______________________________________________ Koha-devel mailing list Koha-devel@lists.koha-community.org https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-devel website : https://www.koha-community.org/ git : https://git.koha-community.org/ bugs : https://bugs.koha-community.org/
-- Nick Clemens ByWater Solutions bywatersolutions.com <http://bywatersolutions.com> Phone: (888) 900-8944 Pronouns: (he/him/his) Timezone: Eastern Follow us: <https://www.facebook.com/ByWaterSolutions/> <https://www.instagram.com/bywatersolutions/> <https://www.youtube.com/user/bywatersolutions> <https://twitter.com/ByWaterSolution>
-- Nick Clemens ByWater Solutions bywatersolutions.com <http://bywatersolutions.com> Phone: (888) 900-8944 Pronouns: (he/him/his) Timezone: Eastern Follow us: <https://www.facebook.com/ByWaterSolutions/> <https://www.instagram.com/bywatersolutions/> <https://www.youtube.com/user/bywatersolutions> <https://twitter.com/ByWaterSolution>
-- +---------------------------------------------------------------+ | Karlsruher Institut für Technologie (KIT) | | KIT-Bibliothek | | Dr. Udo Willke | | Straße am Forum 2, Geb. 30.50/30.51 | | 76131 Karlsruhe | | Tel.: 0721/608-47939, Fax: 0721/608-44886,udo.willke@kit.edu | |http://www.bibliothek.kit.edu/ | | KIT - Universität des Landes Baden-Württemberg und nationales | | Forschungszentrum in der Helmholtz-Gemeinschaft | +---------------------------------------------------------------+
participants (2)
-
Nick Clemens -
Udo Willke