[Koha-devel] koha-elasticsearch verbosity issue
Udo Willke
udo.willke at kit.edu
Fri Sep 8 16:39:46 CEST 2023
Hi Nick,
the start of the log data looks like this:
> [DEPRECATION] [types removal] Using include_type_name in put mapping
> requests is deprecated. The parameter will be removed in the next
> major version. - In request: {body => {data => {properties => {abl =>
> {analyzer => "analyzer_standard",fields => {ci_raw => {normalizer =>
> "icu_folding_normalizer",type => "keyword"},phrase => {analyzer =>
> "analyzer_phrase",search_analyzer => "analyzer_phrase",type =>
> "text"},raw => {normalizer => "nfkc_cf_normalizer",type =>
> "keyword"}},search_analyzer => "analyzer_standard",type =>
> "text"},abl__sort => {index => bless( do{\(my $o = 0)},
> 'JSON::PP::Boolean' ),numeric => bless( do{\(my $o = 1)},
> 'JSON::PP::Boolean' ),type => "icu_collation_keyword"},abstract =>
> $VAR1->{body}{data}{properties}{abl},abstract__sort =>
> $VAR1->{body}{data}{properties}{abl__sort},acqsource =>
> $VAR1->{body}{data}{properties}{abl},acqsource__sort =>
> $VAR1->{body}{data}{properties}{abl__sort},arl =>
> $VAR1->{body}{data}{properties}{abl},arl__sort =>
> $VAR1->{body}{data}{properties}{abl__sort},arp =>
> $VAR1->{body}{data}{properties}{abl},arp__sort =>
> $VAR1->{body}{data}{properties}{abl__sort},author =>
> $VAR1->{body}{data}{properties}{abl},"author-in-order" =>
> $VAR1->{body}{data}{properties}{abl},"author-in-order__sort" =>
> $VAR1->{body}{data}{properties}{abl__sort},"author-name-corporate" =>
> $VAR1->{body}{data}{properties}{abl},"author-name-corporate__sort" =>
> $VAR1->{body}{data}{properties}{abl__sort},"author-name-personal" =>
> $VAR1->{body}{data}{properties}{abl},"author-name-personal__sort" =>
> $VAR1->{body}{data}{properties}{abl__sort},"author-personal-bibliography"
> =>
> $VAR1->{body}{data}{properties}{abl},"author-personal-bibliography__sort"
> => $VAR1->{body}{data}{properties}{abl__sort},"author-title" =>
> $VAR1->{body}{data}{properties}{abl},"author-title__sort" =>
> $VAR1->{body}{data}{properties}{abl__sort},author__facet =>
> {normalizer => "facet_normalizer",type => "keyword"},author__sort =>
> $VAR1->{body}{data}{properties}{abl__sort},author__suggestion =>
> {analyzer => "simple",max_input_length => 100,search_analyzer =>
> "simple",type => "completion"},barcode =>
> $VAR1->{body}{data}{properties}{abl},barcode__sort =>
> $VAR1->{body}{data}{properties}{abl__sort},"bgf-number" =>
> $VAR1->{body}{data}{properties}{abl},"bgf-number__sort" =>
> $VAR1->{body}{data}{properties}{abl__sort},"bib-level" =>
> $VAR1->{body}{data}{properties}{abl},"bib-level__sort" =>
> $VAR1->{body}{data}{properties}{abl__sort},biblioitemnumber =>
> $VAR1->{body}{data}{properties}{abl},biblioitemnumber__sort =>
> $VAR1->{body}{data}{properties}{abl__sort},bio =>
> $VAR1->{body}{data}{properties}{abl},bio__sort =>
> $VAR1->{body}{data}{properties}{abl__sort},"bnb-card-number" =>
> $VAR1->{body}{data}{properties}{abl},"bnb-card-number__sort" =>
> $VAR1->{body}{data}{properties}{abl__sort},ccode =>
> $VAR1->{body}{data}{properties}{abl},ccode__facet =>
> $VAR1->{body}{data}{properties}{author__facet},ccode__sort =>
> $VAR1->{body}{data}{properties}{abl__sort},"classification-source" =>
> $VAR1->{body}{data}{properties}{abl},"classification-source__sort" =>
> $VAR1->{body}{data}{properties}{abl__sort},"cn-bib-sort" =>
> $VAR1->{body}{data}{properties}{abl},"cn-bib-sort__sort" =>
> $VAR1->{body}{data}{properties}{abl__sort},"cn-bib-source" =>
> $VAR1->{body}{data}{properties}{abl},"cn-bib-source__sort" =>
> $VAR1->{body}{data}{properties}{abl__sort},"cn-class" =>
> $VAR1->{body}{data}{properties}{abl},"cn-class__sort" =>
> $VAR1->{body}{data}{properties}{abl__sort},"cn-item" =>
> $VAR1->{body}{data}{properties}{abl},"cn-item__sort" =>
> $VAR1->{body}{data}{properties}{abl__sort},"cn-prefix" =>
> $VAR1->{body}{data}{properties}{abl},"cn-prefix__sort" =>
> $VAR1->{body}{data}{properties}{abl__sort},"cn-sort" =>
> $VAR1->{body}{data}{properties}{abl},"cn-sort__sort" => {index =>
> bless( do{\(my $o = 0)}, 'JSON::PP::Boolean' ),numeric => bless(
> do{\(my $o = 0)}, 'JSON::PP::Boolean' ),type =>
> "icu_collation_keyword"},"cn-suffix" =>
> $VAR1->{body}{data}{properties}{abl},"cn-suffix__sort" =>
> $VAR1->{body}{data}{properties}{abl__sort},"code-geographic" =>
> $VAR1->{body}{data}{properties}{abl},"code-geographic__sort" =>
> $VAR1->{body}{data}{properties}{abl__sort},"code-institution" =>
> $VAR1->{body}{data}{properties}{abl},"code-institution__sort" =>
> $VAR1->{body}{data}{properties}{abl__sort},"coded-location-qualifier"
> =>
> $VAR1->{body}{data}{properties}{abl},"coded-location-qualifier__sort"
> => $VAR1->{body}{data}{properties}{abl__sort},coden =>
> $VAR1->{body}{data}{properties}{abl},coden__sort =>
> $VAR1->{body}{data}{properties}{abl__sort},"conference-name" =>
> $VAR1->{body}{data}{properties}{abl},"conference-name__sort" =>
> $VAR1->{body}{data}{properties}{abl__sort},"control-number" =>
> $VAR1->{body}{data}{properties}{abl},"control-number-identifier" =>
> $VAR1->{body}{data}{properties}{abl},"control-number-identifier__sort"
> => $VAR1->{body}{data}{properties}{abl__sort},"control-number__sort"
> => $VAR1->{body}{data}{properties}{abl__sort},copydate =>
> $VAR1->{body}{data}{properties}{abl},copydate__facet =>
> $VAR1->{body}{data}{properties}{author__facet},copydate__sort =>
> $VAR1->{body}{data}{properties}{abl__sort},copynumber =>
> $VAR1->{body}{data}{properties}{abl},copynumber__sort =>
> $VAR1->{body}{data}{properties}{abl__sort},"corporate-name" =>
> $VAR1->{body}{data}{properties}{abl},"corporate-name__sort" =>
> $VAR1->{body}{data}{properties}{abl__sort},"cross-reference" =>
> $VAR1->{body}{data}{properties}{abl},"cross-reference__sort" =>
> $VAR1->{body}{data}{properties}{abl__sort},ctype => $VAR1->
I think this is what you were pointing out earlier. Maybe I don't need
to send a bigger chunk of the output.
You are right: piping the log through sed would be a workaround.
As I am away next week my colleagues in CC: of this mail will take over.
Best Regards
Udo
Am 08.09.2023 um 15:58 schrieb Nick Clemens:
> Can you provide some examples of the log messages? That might help
> identify where they are coming from.
>
> You could also raise the commit number to generate fewer
> requests/responses?
>
> On Fri, Sep 8, 2023 at 9:13 AM Nick Clemens
> <nick at bywatersolutions.com> wrote:
>
> Hi Udo,
>
> What versions of ES are you using? We currently face an issue of
> deprecation warnings as we have not resolved use of 'types' in ES
>
> Our systems found this was causing a large log on each reindex and
> piped the logs through sed to remove the warnings
>
> -Nick
>
> On Thu, Sep 7, 2023 at 8:04 AM Udo Willke <udo.willke at kit.edu> wrote:
>
> Hi koha developers,
>
> Quick introduction: As a system admin, I am part of the team
> that rolls
> out the koha software at KIT Karlsruhe.
>
> Currently I have an issue with the koha-elasticsearch script
> that is
> used to index the biblio and authorities data in
> elasticsearch. We have
> different test and development servers and on some of the
> severs I get
> extensive diagnostic output when running the command while on
> others I
> only get the errors and warnings.
>
> In every run /usr/sbin/koha-elasticsearch (which subsequently
> calls
> /usr/share/koha/bin/search_tools/rebuild_elasticsearch.pl
> <http://rebuild_elasticsearch.pl>) is definitely
> invoked without any "verbose" flag
>
> Unfortunately there is no output on stdout, but everything
> gets written
> to stderr. As we have 1.3M biblios and 600k authorities I
> would get tens
> of gigabytes of log file, which I would have to check.
>
> My question: is there somewhere a setting in koha that
> globally controls
> the debug level? Or how can the different behaviours be explained?
>
> All installations run koha Release 23.05.X on Ubuntu 22.04.3 LTS
>
> Any thoughts are welcome.
>
> Many thanks in advance
>
> Udo
>
>
>
>
>
>
>
>
>
> _______________________________________________
> Koha-devel mailing list
> Koha-devel at lists.koha-community.org
> https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-devel
> website : https://www.koha-community.org/
> git : https://git.koha-community.org/
> bugs : https://bugs.koha-community.org/
>
>
>
> --
> Nick Clemens
> ByWater Solutions
> bywatersolutions.com <http://bywatersolutions.com>
> Phone: (888) 900-8944
> Pronouns: (he/him/his)
> Timezone: Eastern
> Follow us:
> <https://www.facebook.com/ByWaterSolutions/>
> <https://www.instagram.com/bywatersolutions/>
> <https://www.youtube.com/user/bywatersolutions>
> <https://twitter.com/ByWaterSolution>
>
>
>
> --
> Nick Clemens
> ByWater Solutions
> bywatersolutions.com <http://bywatersolutions.com>
> Phone: (888) 900-8944
> Pronouns: (he/him/his)
> Timezone: Eastern
> Follow us:
> <https://www.facebook.com/ByWaterSolutions/>
> <https://www.instagram.com/bywatersolutions/>
> <https://www.youtube.com/user/bywatersolutions>
> <https://twitter.com/ByWaterSolution>
--
+---------------------------------------------------------------+
| Karlsruher Institut für Technologie (KIT) |
| KIT-Bibliothek |
| Dr. Udo Willke |
| Straße am Forum 2, Geb. 30.50/30.51 |
| 76131 Karlsruhe |
| Tel.: 0721/608-47939, Fax: 0721/608-44886,udo.willke at kit.edu |
|http://www.bibliothek.kit.edu/ |
| KIT - Universität des Landes Baden-Württemberg und nationales |
| Forschungszentrum in der Helmholtz-Gemeinschaft |
+---------------------------------------------------------------+
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.koha-community.org/pipermail/koha-devel/attachments/20230908/e2cb673a/attachment-0001.htm>
More information about the Koha-devel
mailing list