[For those not subscribed or giving attention to the Koha mailing list, I repeat the announcement here without in message cross-posting.] The Koha Wiki now running MediaWiki Canasta has been up for a few hours at the usual DNS subdomain https://wiki.test.koha-community.org . The wiki is now up to date with MediaWiki 1.35.07 long term stable using a MySQL database and ElasticSearch with many fine enhancements such as xVisualEditor, customised AdvancedSearch, and dynamic archiving of obsolete pages (which often still have useful information). Please see further below for details. Unfortunately, the mail system on the server for the wiki for resetting wiki login passwords, or creating new login users, etc. had previously become broken and was missed for fixing amidst all the work which people have been doing for releasing a new Koha version. Someone will send a message when the mail system is fixed. If you know your Koha wiki login username and password from previously, they will work. There are problably many problems with result set relvance for search queries within the wiki. We will fix them over time and relevance ranking should automatically improve with use, although, maintenance changes may often count as use eroding recently updated relevance. The wiki is now using ElasticSearch which is used by Wikipedia and has better extension support than database based indexing. See some details about the customised AdvancedSearch and the need for careful consideration in improving search query indexing further below. Sitemap creation, which assists Google and other web indexing systems in indexing the Koha wiki, may not be working correctly in the way in which we have configured the MediaWiki Canasta Docker container. Google and others can still index the content without the sitemap but the process functions better with a sitemap. The Canasta Docker container does some things differently than the way they would function in a standard environment such that less effort should be required for maintenance tasks but we need a little more time to examine how some things such as sitemap creation are intended to function in Canasta. We should always be able to use methods ordinarily used for sitemap creation in a standard environment if necessary. The Koha MediaWiki Canasta test instance should continue to be available for first testing significant changes and bug fixes, at https://wiki.test.koha-community.org . Please do not make wiki contributions that you want to save in the MediaWiki Canasta test instance as they will not be carried over to the production wiki. Continue to make lasting contributions to the production wiki at https://wiki.koha-community.org . Please read below for an understanding of what to expect before reporting issues about which we are already aware, such as the test database is not a current copy of the wiki and the mail system for resetting login passwords and creating new login users is not working. You may report bugs to the bug "wiki needs updating to a later version", https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=23073 . WIKI DATABASE MIGRATION, UPGRADE, AND CONTAINER MANAGEMENT. Migrating the Koha MediaWiki database from Postgres to MySQL and upgrading to MediaWiki 1.35.07, the current long term stable version used a repeatable process managed with a set of scripts which I developed in bash, Perl, and Python as appropriate for the task and previous code in the case of Python. Choosing Postgres as the database for a test instance for MediaWiki had left us with a mistake in database choice complicating compatibility and future upgrades when the MediaWiki test suddenly became the only Koha wiki running when the previous Koha wiki went down in the midst of a community schism with LibLime long ago. The database migration and upgrade process has been developed and progressively tested over the course over time from 2019 for ensuring that the database is migrated correctly, etc. I built the database migration process upon the originally incomplete and sometimes mistaken Python script of Philipp Spitzer which was a fantastic proven starting point without which the task may have been some degree too much. Mason James ran a web crawl and diff test to verify that the production and another test database migration and upgrade of the wiki had the same content except for evident changes where the production wiki had been updated with new content. The database was imported to MediaWiki Canasta which Tomás Cohen Arazi identified and customised to connect to the Koha Portainer Docker container management to provide MediaWiki in a Docker container with a large set of important extensions to help make managing the MediaWiki software easier. See https://github.com/CanastaWiki for more about MediaWiki Canasta. After a minimial final testing period with Canasta in Koha Portainer, I marked the old wiki instance which had been using Postgres readonly and proceded with the database migration and upgrade using an up to date copy of the database. Tomás made further modifications in Portainer and Chris Cormack redirected the DNS record for wiki.koha-community.org DNS to the current server for the wiki. Although you may never notice an interruption in service for the wiki. We may have to restart it to fix things which function a little differently and a little more complicated to fix for the MediaWiki Canasta Docker container than a standard operating system environment. Once fixed, maintenance of a Docker container should be easier than a standard environment. We may even move the server or some functions back to the server where the wiki had been hosted for years thanks to Galen Charleton and Equinox. See further below for a little about other modifications which I made to support dynamic archiving, etc. KNOWN ISSUES. The mail system on the server for the wiki for resetting wiki login passwords, or creating new logins, etc. had previously become broken and needs fixing as a matter of priority. The test instance at https://wiki.test.koha-community.org has copy of the database which will become outdated over time. In future, we may set up a process to update the database periodically. The purpose of the test instance is to support testing significant wiki changes or wiki bug fixes first without the hazard of harming the production wiki. Bug fixes need testing and can at least temporarily break the wiki. Significant changes may fail to work as expected and might not be easily undone particularly if the changes have been created by a script for mass editing. Having the latest wiki revisions is not usually needed for testing. There have been bugs specific to MediaWiki Canasta rearranging some standard files for the Docker container which have been addressed. However, there are at least some Canasta Docker specific bugs relating to the Docker container environment. Please report any instances of "Error creating thumbnail: Unable to save thumbnail to destination" which I found in the Koha History page https://wiki.koha-community.org/wiki/History . Instances of the bug can be fixed with command line shell access by removing the images/thumb/$buggy_image_name subdirectory for the image and making non-changing edit to the page which allows MediaWiki to recreate the images/thumb subdirectory without a problem and the bug goes away. We should probably remove all the subdirectories in the images/thumb directory proactively. Yet, why is there a special problem for the Docker container which does not exist in other test instances when not using a Docker container and the container environment is running as the root user as standard for Docker containers and the root user should have all the permission necessary to access or create a thumbnail directory? Changing the ownership of the images directory and subdirectories back and forth to test the effect temporarily broke a test instance of the wiki until the container was restarted. MAJOR ENHANCEMENTS FROM UPGRADING. The VisualEditor extension used by Wikipedia is a WYSIWIG and guided forms aid for visually editing the underlying wikitext for a page and using guided forms for adding some features to a page. Users can switch back and forth between source editing in all wikitext syntax and VisualEditor, however, it may be best to save the current edit before switching back and forth to avoid problems of imperfect correspondence between wikitext syntax and the VisualEditor model of wikitext. The AdvancedSearch extension used by Wikipedia is helpful for a user friendly interface to construct search queries and modify them by removing terms which appear in a bubble with an [x] to remove the term. AdvancedSearch depends on ElasticSearch which performs remarkably well in testing and allows the wiki to be reindexed in a couple of minutes if necessary. See further below for modifications to the AdvancedSearch extension. SemanticMediaWiki was reinstalled after copying the upgraded database. Modifying the AdvancedSearch extension in conjunction with special AdvancedSearch navigation links and custom queries using carefully managed standard wiki categories may be more helpful than SemanticMediaWiki. Furthermore, anyone experimenting with SemanticMediaWiki should be aware that verbose syntax is required to avoid breaking most wikis with SemanticMediaWiki after forthcoming MediaWiki updates in which a hook commonly relied upon for SemanticMediaWiki which has been deprecated will be removed. Wikipedia does not use SemanticMediaWiki and thus some MediaWiki developers may not have given sufficient consideration to managing the issue. The workaround may involve a potential performance deficit when using SemanticMediaWiki search queries. The MassEditRegex extension has power one might hope for in the name for using regular expressions to modify a list of pages. However, given its power it remains commented out in LocalSettings.php for the production system. Use is intended to be for some special group of users such as wiki administrors, however even they should be most strongly cautioned to first test their process on a test instance of the wiki. Furthermore, use should be with a bot subaccount set up by the user so that they may be identified as the work of a bot process and those mass changes may avoid adversely affecting page modification priorities in search result sets. The creation of user bot subaccounts should be documented. In testing, MassEditRegex works fantastically well for adding categories to the bottom of pages and templates to the top of pages which can be done without risk of an inadequately debugged regular expression breaking page content in the middle. MODIFIED FEATURES. I modified the following to support dynamic archiving in which obsolete content does not appear by default for search results unless the user goes directly to the advanced search page without following provided navigation links or changes the default VectorMod skin affecting the basic search box. ADVANCEDSEARCH EXTENSION WITH MODIFICATIONS. The AdvancedSearch extension has been modified to include two additional form elements: one for excluding particular categories and another for excluding particular templates. These additional elements appear in the user friendly AdvancedSearch term bubbles which can be individually removed from a query by clicking on the [x] for the particular bubble. Editing the non-English localisation files is still pending. For languages for which a non-English localisation file has not been edited, the custom fields for category and template exclusion display a description in English. DeepCategory searches for subcategories of a category is disabled because it requires a sparkle database and is only updated on a weekly basis for Wikipedia. Searching subcategories of a category should be less of an issue with faceted use of categories which we should be carefully moving towards. Excluding particular categories supports dynamic archiving by supporting search queries excluding obsolete pages with -incategory:"Obsolete", which is automatically invoked from the navigation link "Advanced Search current" or from simple search box when using the modified Vector skin, VectorMod. Obsolete pages are also noted with a prominent notice using the Obsolete template. Such pages should be updated if they can be, but are otherwise available to consult most importantly for valuable information they often contain which is not yet present in current pages. Archived obsolete pages can be found exclusively by following the navigation link "Advanced search obsolete archive" which includes incategory:"Obsolete" automatically. The result set for search queries with incategory:"Obsolete" can be used to identify the type of pages which should have the Obsolete category and Obsolete template but do not yet, such as installation information for some particular old Debian versions. Various combinations of including and excluding categories and templates can be easily used in the modified AdvancedSearch to find pages which only have one of either the Obsolete category or Obsolete template which should be used together or both removed if the page has been updated to be current. All wiki pages should have some category even if it may be [[Category:Empty]] for people uncertain of what may be appropriate in the moment. Pages missing categories may not be disappearing from query results by category when using ElasticSearch indexing as they had been when using database based search indexing. We can also query for pages missing categories using https://wiki.koha-community.org/w/index.php?title=Special:UncategorizedPages and correct the issue which has been neglected due to loss of time where migrating and upgrading the wiki has been the priority with much less time available otherwise especially since the pandemic. We should take some care when thinking about faceted category use as no wiki software uses fielded categories. Thus there may be no concise way to query for pages which address a topic in a general way or supplement other documentation on a topic containing a lone category such as [[Category:Circulation]], if we then have many other pages with [[Category:RFCs]] and [[Category:Circulation]] but no longer [[Category:Circulation RFCs]] as a possible change for faceting. In such an example, the search results of a query for incategory:"Circulation" might have a result set in which pages for RFCs relating to circulation issues containing both [[Category:RFCs]] and [[Category:Circulation]] might crowd out more generally helpful pages with [[Category:Circulation]] alone. The problem may indicate a need for a navigation link to exclude RFCs from a search query; designating old RFCs as obsolete; or both. Alternatively or additionally, we may be able to adjust the weighting of the ElasticSearch indexing options such that pages containing [[Category:RFCs]] have a lower weight and appear further down the result set or pages with a single category such as [[Category:Circulation]] alone or some particular additional categories such as [[Category:Documentation]] have higher weight and appear further up the result set. VECTORMOD SKIN. Users are free to choose their own preferred MediaWiki skin and we can add others. VectorMod is merely set as the default to help people avoid obsolete pages when submitting search queries from the simple search box which appears on every page. VectorMod is a custom version of the Vector skin which includes a modified version of Vector/includes/templates/SearchBox.mustache supporting dynamic archiving of obsolete content by excluding pages which have been designated obsolete by automatically adding -inCategory:"Obsolete" to basic search querries. The syntax incategory requires using ElasticSearch. Previously, I replaced the SearchBox.mustache file in the Vector skin directly, which certainly worked without the extra effort of creating a custom skin. Automatically inserting -inCategory:"Obsolete" in the basic search box is now somewhat elegant in conjunction with the modified AvancedSearch extension as it uses explanatory language labels with a bubble which has a removal [x] and allows autocompletion of query terms. Significant renaming of references to Vector as VectorMod and vector as vectormod has been scripted allows both Vector and VectorMod to be loaded and available to users. Thomas Dukleth Agogme 109 E 9th Street, 3D New York, NY 10003 USA http://www.agogme.com +1 212-674-3783

Thanks a lot! :) Looking forward to using the new features and doing some clean-up work! On 27.10.22 14:29, Thomas Dukleth wrote:

[For those not subscribed or giving attention to the Koha mailing list, I repeat the announcement here without in message cross-posting.]

The Koha Wiki now running MediaWiki Canasta has been up for a few hours at the usual DNS subdomain https://wiki.test.koha-community.org . The wiki is now up to date with MediaWiki 1.35.07 long term stable using a MySQL database and ElasticSearch with many fine enhancements such as xVisualEditor, customised AdvancedSearch, and dynamic archiving of obsolete pages (which often still have useful information). Please see further below for details.

Unfortunately, the mail system on the server for the wiki for resetting wiki login passwords, or creating new login users, etc. had previously become broken and was missed for fixing amidst all the work which people have been doing for releasing a new Koha version. Someone will send a message when the mail system is fixed. If you know your Koha wiki login username and password from previously, they will work.

There are problably many problems with result set relvance for search queries within the wiki. We will fix them over time and relevance ranking should automatically improve with use, although, maintenance changes may often count as use eroding recently updated relevance. The wiki is now using ElasticSearch which is used by Wikipedia and has better extension support than database based indexing. See some details about the customised AdvancedSearch and the need for careful consideration in improving search query indexing further below.

Sitemap creation, which assists Google and other web indexing systems in indexing the Koha wiki, may not be working correctly in the way in which we have configured the MediaWiki Canasta Docker container. Google and others can still index the content without the sitemap but the process functions better with a sitemap. The Canasta Docker container does some things differently than the way they would function in a standard environment such that less effort should be required for maintenance tasks but we need a little more time to examine how some things such as sitemap creation are intended to function in Canasta. We should always be able to use methods ordinarily used for sitemap creation in a standard environment if necessary.

The Koha MediaWiki Canasta test instance should continue to be available for first testing significant changes and bug fixes, at https://wiki.test.koha-community.org . Please do not make wiki contributions that you want to save in the MediaWiki Canasta test instance as they will not be carried over to the production wiki. Continue to make lasting contributions to the production wiki at https://wiki.koha-community.org .

Please read below for an understanding of what to expect before reporting issues about which we are already aware, such as the test database is not a current copy of the wiki and the mail system for resetting login passwords and creating new login users is not working.

You may report bugs to the bug "wiki needs updating to a later version", https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=23073 .

WIKI DATABASE MIGRATION, UPGRADE, AND CONTAINER MANAGEMENT.

Migrating the Koha MediaWiki database from Postgres to MySQL and upgrading to MediaWiki 1.35.07, the current long term stable version used a repeatable process managed with a set of scripts which I developed in bash, Perl, and Python as appropriate for the task and previous code in the case of Python. Choosing Postgres as the database for a test instance for MediaWiki had left us with a mistake in database choice complicating compatibility and future upgrades when the MediaWiki test suddenly became the only Koha wiki running when the previous Koha wiki went down in the midst of a community schism with LibLime long ago. The database migration and upgrade process has been developed and progressively tested over the course over time from 2019 for ensuring that the database is migrated correctly, etc. I built the database migration process upon the originally incomplete and sometimes mistaken Python script of Philipp Spitzer which was a fantastic proven starting point without which the task may have been some degree too much.

Mason James ran a web crawl and diff test to verify that the production and another test database migration and upgrade of the wiki had the same content except for evident changes where the production wiki had been updated with new content.

The database was imported to MediaWiki Canasta which Tomás Cohen Arazi identified and customised to connect to the Koha Portainer Docker container management to provide MediaWiki in a Docker container with a large set of important extensions to help make managing the MediaWiki software easier. See https://github.com/CanastaWiki for more about MediaWiki Canasta.

After a minimial final testing period with Canasta in Koha Portainer, I marked the old wiki instance which had been using Postgres readonly and proceded with the database migration and upgrade using an up to date copy of the database. Tomás made further modifications in Portainer and Chris Cormack redirected the DNS record for wiki.koha-community.org DNS to the current server for the wiki.

Although you may never notice an interruption in service for the wiki. We may have to restart it to fix things which function a little differently and a little more complicated to fix for the MediaWiki Canasta Docker container than a standard operating system environment. Once fixed, maintenance of a Docker container should be easier than a standard environment. We may even move the server or some functions back to the server where the wiki had been hosted for years thanks to Galen Charleton and Equinox.

See further below for a little about other modifications which I made to support dynamic archiving, etc.

KNOWN ISSUES.

The mail system on the server for the wiki for resetting wiki login passwords, or creating new logins, etc. had previously become broken and needs fixing as a matter of priority.

The test instance at https://wiki.test.koha-community.org has copy of the database which will become outdated over time. In future, we may set up a process to update the database periodically. The purpose of the test instance is to support testing significant wiki changes or wiki bug fixes first without the hazard of harming the production wiki. Bug fixes need testing and can at least temporarily break the wiki. Significant changes may fail to work as expected and might not be easily undone particularly if the changes have been created by a script for mass editing. Having the latest wiki revisions is not usually needed for testing.

There have been bugs specific to MediaWiki Canasta rearranging some standard files for the Docker container which have been addressed. However, there are at least some Canasta Docker specific bugs relating to the Docker container environment. Please report any instances of "Error creating thumbnail: Unable to save thumbnail to destination" which I found in the Koha History page https://wiki.koha-community.org/wiki/History . Instances of the bug can be fixed with command line shell access by removing the images/thumb/$buggy_image_name subdirectory for the image and making non-changing edit to the page which allows MediaWiki to recreate the images/thumb subdirectory without a problem and the bug goes away. We should probably remove all the subdirectories in the images/thumb directory proactively. Yet, why is there a special problem for the Docker container which does not exist in other test instances when not using a Docker container and the container environment is running as the root user as standard for Docker containers and the root user should have all the permission necessary to access or create a thumbnail directory? Changing the ownership of the images directory and subdirectories back and forth to test the effect temporarily broke a test instance of the wiki until the container was restarted.

MAJOR ENHANCEMENTS FROM UPGRADING.

The VisualEditor extension used by Wikipedia is a WYSIWIG and guided forms aid for visually editing the underlying wikitext for a page and using guided forms for adding some features to a page. Users can switch back and forth between source editing in all wikitext syntax and VisualEditor, however, it may be best to save the current edit before switching back and forth to avoid problems of imperfect correspondence between wikitext syntax and the VisualEditor model of wikitext.

The AdvancedSearch extension used by Wikipedia is helpful for a user friendly interface to construct search queries and modify them by removing terms which appear in a bubble with an [x] to remove the term. AdvancedSearch depends on ElasticSearch which performs remarkably well in testing and allows the wiki to be reindexed in a couple of minutes if necessary. See further below for modifications to the AdvancedSearch extension.

SemanticMediaWiki was reinstalled after copying the upgraded database. Modifying the AdvancedSearch extension in conjunction with special AdvancedSearch navigation links and custom queries using carefully managed standard wiki categories may be more helpful than SemanticMediaWiki. Furthermore, anyone experimenting with SemanticMediaWiki should be aware that verbose syntax is required to avoid breaking most wikis with SemanticMediaWiki after forthcoming MediaWiki updates in which a hook commonly relied upon for SemanticMediaWiki which has been deprecated will be removed. Wikipedia does not use SemanticMediaWiki and thus some MediaWiki developers may not have given sufficient consideration to managing the issue. The workaround may involve a potential performance deficit when using SemanticMediaWiki search queries.

The MassEditRegex extension has power one might hope for in the name for using regular expressions to modify a list of pages. However, given its power it remains commented out in LocalSettings.php for the production system. Use is intended to be for some special group of users such as wiki administrors, however even they should be most strongly cautioned to first test their process on a test instance of the wiki. Furthermore, use should be with a bot subaccount set up by the user so that they may be identified as the work of a bot process and those mass changes may avoid adversely affecting page modification priorities in search result sets. The creation of user bot subaccounts should be documented. In testing, MassEditRegex works fantastically well for adding categories to the bottom of pages and templates to the top of pages which can be done without risk of an inadequately debugged regular expression breaking page content in the middle.

MODIFIED FEATURES.

I modified the following to support dynamic archiving in which obsolete content does not appear by default for search results unless the user goes directly to the advanced search page without following provided navigation links or changes the default VectorMod skin affecting the basic search box.

ADVANCEDSEARCH EXTENSION WITH MODIFICATIONS.

The AdvancedSearch extension has been modified to include two additional form elements: one for excluding particular categories and another for excluding particular templates. These additional elements appear in the user friendly AdvancedSearch term bubbles which can be individually removed from a query by clicking on the [x] for the particular bubble.

Editing the non-English localisation files is still pending. For languages for which a non-English localisation file has not been edited, the custom fields for category and template exclusion display a description in English.

DeepCategory searches for subcategories of a category is disabled because it requires a sparkle database and is only updated on a weekly basis for Wikipedia. Searching subcategories of a category should be less of an issue with faceted use of categories which we should be carefully moving towards.

Excluding particular categories supports dynamic archiving by supporting search queries excluding obsolete pages with -incategory:"Obsolete", which is automatically invoked from the navigation link "Advanced Search current" or from simple search box when using the modified Vector skin, VectorMod. Obsolete pages are also noted with a prominent notice using the Obsolete template. Such pages should be updated if they can be, but are otherwise available to consult most importantly for valuable information they often contain which is not yet present in current pages. Archived obsolete pages can be found exclusively by following the navigation link "Advanced search obsolete archive" which includes incategory:"Obsolete" automatically.

The result set for search queries with incategory:"Obsolete" can be used to identify the type of pages which should have the Obsolete category and Obsolete template but do not yet, such as installation information for some particular old Debian versions. Various combinations of including and excluding categories and templates can be easily used in the modified AdvancedSearch to find pages which only have one of either the Obsolete category or Obsolete template which should be used together or both removed if the page has been updated to be current.

All wiki pages should have some category even if it may be [[Category:Empty]] for people uncertain of what may be appropriate in the moment. Pages missing categories may not be disappearing from query results by category when using ElasticSearch indexing as they had been when using database based search indexing. We can also query for pages missing categories using https://wiki.koha-community.org/w/index.php?title=Special:UncategorizedPages and correct the issue which has been neglected due to loss of time where migrating and upgrading the wiki has been the priority with much less time available otherwise especially since the pandemic.

We should take some care when thinking about faceted category use as no wiki software uses fielded categories. Thus there may be no concise way to query for pages which address a topic in a general way or supplement other documentation on a topic containing a lone category such as [[Category:Circulation]], if we then have many other pages with [[Category:RFCs]] and [[Category:Circulation]] but no longer [[Category:Circulation RFCs]] as a possible change for faceting. In such an example, the search results of a query for incategory:"Circulation" might have a result set in which pages for RFCs relating to circulation issues containing both [[Category:RFCs]] and [[Category:Circulation]] might crowd out more generally helpful pages with [[Category:Circulation]] alone. The problem may indicate a need for a navigation link to exclude RFCs from a search query; designating old RFCs as obsolete; or both. Alternatively or additionally, we may be able to adjust the weighting of the ElasticSearch indexing options such that pages containing [[Category:RFCs]] have a lower weight and appear further down the result set or pages with a single category such as [[Category:Circulation]] alone or some particular additional categories such as [[Category:Documentation]] have higher weight and appear further up the result set.

VECTORMOD SKIN.

Users are free to choose their own preferred MediaWiki skin and we can add others. VectorMod is merely set as the default to help people avoid obsolete pages when submitting search queries from the simple search box which appears on every page.

VectorMod is a custom version of the Vector skin which includes a modified version of Vector/includes/templates/SearchBox.mustache supporting dynamic archiving of obsolete content by excluding pages which have been designated obsolete by automatically adding -inCategory:"Obsolete" to basic search querries. The syntax incategory requires using ElasticSearch. Previously, I replaced the SearchBox.mustache file in the Vector skin directly, which certainly worked without the extra effort of creating a custom skin.

Automatically inserting -inCategory:"Obsolete" in the basic search box is now somewhat elegant in conjunction with the modified AvancedSearch extension as it uses explanatory language labels with a bubble which has a removal [x] and allows autocompletion of query terms.

Significant renaming of references to Vector as VectorMod and vector as vectormod has been scripted allows both Vector and VectorMod to be loaded and available to users.

Thomas Dukleth Agogme 109 E 9th Street, 3D New York, NY 10003 USA http://www.agogme.com +1 212-674-3783

_______________________________________________ Koha-devel mailing list Koha-devel@lists.koha-community.org https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-devel website : https://www.koha-community.org/ git : https://git.koha-community.org/ bugs : https://bugs.koha-community.org/