[Koha-bugs] [Bug 21872] Elasticsearch indexing faster by making it multi-threaded

[bugzilla-daemon at bugs.koha-community.org]

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=21872

David Cook <dcook at prosentient.com.au> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |dcook at prosentient.com.au

--- Comment #1 from David Cook <dcook at prosentient.com.au> ---
(In reply to Joonas Kylmälä from comment #0)
> The record prosessing in misc/search_tools/rebuild_elastic_search.pl happens
> currently single threadeadly, it could be made multithreaded to take full
> advantage of multicore systems.
> 
> <ere> says on IRC following about this: "[..] a simple way would be to add
> start offset and skip count to the indexing script so you could run multiple
> in parallel"
> 
> So in the line "while ( my $record = $next->() ) {" the next->() function
> gets called and that should be possible to multithread.

I'll just split hairs and mentioning that multithreading in Perl is not
recommended and never really done, but you could achieve the thing by forking
workers. 

In #10662, I use the following modules to perform rapid event-driven processing
of job queues:

https://metacpan.org/pod/POE::Component::JobQueue
https://metacpan.org/pod/POE::Wheel::Run

Another option would be to use a message queue and separate workers for doing
the indexing. 

Just a thought.

-- 
You are receiving this mail because:
You are watching all bug changes.
You are the assignee for the bug.