- Koha-devel - lists.koha-community.org

MySQL/MariaDB index prefix length
by David Cook 09 Dec '22

09 Dec '22

Hi all, We often create indexes with a prefix length of 191 because we use 4 byte encoded fields. 191 * 4 = 764, which comes in under the prefix index length threshold of 767. But that looks like a limit that was imposed due to the COMPACT row format. Most newer MySQL/MariaDB databases will be using the DYNAMIC row format, which can support an index key prefix limit of 3072 bytes (or 768 4-byte encoded characters). On Bugzilla, there have been times where I've speculated that maybe it's the sysadmin's job to update the database row format, but perhaps we should take a more proactive effort to move row format from COMPACT to DYNAMIC, so that we can more systematically take advantage of longer index prefixes. I often think about how to index URLs, since the maximum length of URLs are very large. Web browsers hover around the 2000 character mark while web servers can be much higher than that. Of course, even with 3072 bytes, that still wouldn't be enough for a full URL. But it would get a person a lot farther than 191 characters. David Cook Senior Software Engineer Prosentient Systems Suite 7.03 6a Glen St Milsons Point NSW 2061 Australia Office: 02 9212 0899 Online: 02 8005 0595

1 0

Error - browsing Patrons by Last Name
by Wainui Witika-Park 08 Dec '22

08 Dec '22

Hi everyone, I was wondering if anyone can help me please, I am getting an error when Browsing Patrons by Last Name. When clicking on some letters, the search gets stuck on "Processing". The letters that it seems to get stuck on are: A, H, N, T, V and W It also gives a pop up error when clicking on S that sometimes says "Something went wrong when loading the table. 500: Internal Server Error" and sometimes says "Something went wrong. The note has not been saved". The plack-error.log says: [2022-12-06 14:47:41.21998] [19070] [warn] OpenAPI >>> GET api/v1/patrons [{"message":"Expected string - got null.","path":"\/body\/4\/extended_attributes\/0\/value"}] Any help would very much appreciated :) Wainui

2 1

Good enough?
by Jonathan Druart 06 Dec '22

06 Dec '22

Hi devs, I was wondering... How good is your "good enough"? It's a question I often ask myself, when writing patches or QAing yours: is it good enough to be into Koha? It does not have to be perfect or it may never reach master, but it must meet certain standards. When I was RM I tried to put that limit quite high. Not necessarily by asking the author for improving the follow-up patches, but also by adding the missing bits myself. There are various types of "good enough", depending on what we look at: good enough to be reviewed, good enough to be tested, to be put in production, get feedback, try an idea, etc. Most of the time, what is driving the limit is answering the questions "Is it maintainable?" / "Is it future proof?". Making sure the code you are writing won't be broken inadvertently is very important and must be part of the reflection. Katrin asked the QA team members what were our plans for 23.05. In my opinion we should enforce this "be future proof" concept. Writing code is easy, especially in Koha (yes it is!). Writing maintainable and robust code, following our guidelines is a bit harder. That's why we have a QA process that tries to catch inconsistencies or edge cases you may have missed. But I think we can be even more rigorous, and aim for better standards. We can elevate our ambitions and do better. When we see the number of new enhancements we are releasing every 6 months, it shows well that writing code is definitely not a problem. However sometimes developers are tempted to abandon their work once they are pushed to master. Pushed does not mean "done". Providing bug fixes, following-up with patches when needed, fixing CI jenkins, etc. is part of job (/fun) As a Koha developer for a long time now, I know how frustrating it can be to be asked for follow-ups/rewrite/tests to have our patches stamped with the precious PQA mark. But from the other point of view (RM, RMaints, QA team), I also know it's very frustrating when you are in charge of the release and you do not get the appropriate follow-up work once it's pushed to master. There are some easy steps to write/review better patches. All have been discussed already several times, but that can be enforced even more: 1. Perltidy (!) This is really a very trivial step. Please perltidy your code. There are hundreds of commits that have been pushed in the last months that are not tidy (alignment, indentation, lines too long, etc.) This can easily be configured in your IDE! [1] 2. Provide clean code. As said it's not necessarily easy, but the QA team and RM are supposed to know if the code is clean regarding Koha guidelines. If the code is not clean, don't PQA, don't push. Either clean yourself, or ask the original author of the patch to do it (explaining to them how it can be improved ofc). 3. Squash! I have been away for a couple of months and had to read the git history to know what I missed. And it was really hard to follow what was going on. First of all, we are not consistent: the commit message must tell what the patch is doing, not what the bug was (if you are writing a bug fix). Then, there are way too many follow-ups: tidiness, indentation fix, typo, spelling, etc. All those tiny follow-ups could be squashed into the original patch. We don't need unnecessary tons of entries in our git log for that. For instance, I usually add a "JD Amended patch: perltidy" for instance when I tidy the original patch, to keep track of the modification. Squash can be done by the original author, the QAer, the RM. So yes, you are losing one commit in the stats but the git log is easy to read! We could have an "Amended-by" marker if we really want to add credit on the dashboard (and/or release notes). 4. Run tests. Don't wait for Jenkins to fail. This is valid for the author and QA. Anticipate the failures by running more tests. If you are modifying C4::Circulation, then run prove on t/db_dependent/Circulation*, not only Circulation.t. It will help you catch edge cases. When something is pushed, track down jenkins failures that could be caused by your patches. 6. Be strict if you are QAing. Each QA member has their own "good enough", and the RM as well (either relying on the QAer or providing a full review). But QA must fail if the code is old Koha style code, or not "good enough". 7. Provide support for failing tests, fix things you broke. The QA team will be more comfortable with your patches if you show them you are providing support for your stuff. It's not because it's pushed that you don't have any more efforts to make. Provide follow-up patches you promised, provide bug fixes, etc. We don't have a good way to keep track of such demands, which does not make tracking easier for devs, QA and RM. Any suggestions? 8. QA team MUST NEVER* pass QA a change that is not covered by tests, never. You should not provide change to modules without tests! * almost never... 9. Stick to existing patterns. We should not have different ways to do the same thing. We should not have different places where a code is doing the same thing. Ask for help or advice on the list or IRC before you start coding. We will be happy to guide you. Even if you are a regular Koha developer it's not always easy to be aware of the latest master changes. We will tell you what's the current good practice, or point you to examples you could reuse for what you want to implement. 10. CI should drive the pushes. No more push if CI is not green. The more we wait the harder it is to track down the origin of the problem. Last cycle some jobs have been red for months, and we released 22.11.00 with D10, D11, D12 marked unstable... What will I do next cycle? All of that, and more. I will track down jenkins failures and responsibilize developers telling them when they break tests (and won't fix them anymore as I have been doing for years). I will raise on the bug reports what could have been improved. Yes, read that I will be even more annoying (to put it politely) than before. I've noticed that the pre-commit git hook on the wiki has been broken for more than 3 years. And also caught some core developers that do not have it in place. I am relying on it to keep Vue files tidy so it's important to have it set up properly. I am planning to force its usage for ktd users [2]. Adding more checks to it will help us to catch inconsistencies from the beginning. To summarize, writing code is cheap, maintaining code is way more expensive! It is easier to get the attention of developers before the patches are pushed to master than after, so we could be more ambitious and ask more. For discussion :) Cheers, Jonathan [1] If you are using vim, open ~/vimrc, add vmap <F8> :!perltidy -q<CR> Reload vim, select code in visual mode [2] https://gitlab.com/koha-community/koha-misc4dev/-/issues/59

5 13

Maximum execution time for background jobs?
by David Cook 06 Dec '22

06 Dec '22

Hi all, I was closing old Bugzilla bugs when I bumped into "Bug 10460 - way to stop a run away report". It got me thinking about maximum execution time for background jobs. What do people think about background jobs workers killing their child processes if they exceed a maximum execution time? I don't have a particular use case for it at the moment, so not a priority by any means, but thoughts on the concept? David Cook Senior Software Engineer Prosentient Systems Suite 7.03 6a Glen St Milsons Point NSW 2061 Australia Office: 02 9212 0899 Online: 02 8005 0595

1 0

Fundamental flaw in Koha REST API
by David Cook 06 Dec '22

06 Dec '22

Hi all, Just wondering if my email titled “x-koha-query header doesn't support UTF-8 data” actually made it to the koha-devel list as I didn’t get any replies to what I perceive as a significant issue. Basically, the x-koha-query header can only support ISO 8859-1/Latin-1 encoded characters. If you try to send UTF-8 encoded data to the API via Javascript, it will silently fail. See Bug 32401 for a practical example of this. I encountered this issue when using Arabic, but it is also obvious when using Hebrew, Chinese, Devanagari, or any non-Latin based script. It is less obvious – but still a problem – for some languages like Dutch, Finnish, French, and German. For instance, a x-koha-query header containing the French word “cœur” will fail, because œ is not a ISO 8859-1/Latin-1 character. (See the Wikipedia entry for ISO 8859-1 for more examples of problems with languages mostly covered by Latin-1.) Koha devs need to discuss this and possibly remove “x-koha-query” from Koha. At the moment, it’s not widely used by Koha itself, so I don’t think it will be hard to remove from Koha, but any third-party integrations would need to refactor to use a different option. Anyway, it would be great if someone could acknowledge receipt of this email. I have figured out how to get DataTables to stop using x-koha-query, so I’m going to do that for the particular acquisitions problem mentioned in Bug 32401, but we do need to talk about this issue. David Cook Senior Software Engineer Prosentient Systems Suite 7.03 6a Glen St Milsons Point NSW 2061 Australia Office: 02 9212 0899 Online: 02 8005 0595

3 4

Setting up Koha Testing Docker
by Lamancusa, Emily 05 Dec '22

05 Dec '22

Hello all, I'm trying to set up a Koha Testing Docker on my local machine (running Pop!_OS 22.04). I haven't used Docker before, and I have a few questions about how to get it set up. 1. It looks like Linux users have the option to install the whole Docker Desktop VM, or install just Docker Engine, Docker CLI, and the Compose plugin. Do I need Docker Desktop to run KTD, or will the latter installation option work on its own? (Aside from not wanting to install a bunch of extra stuff that I don't need, I would also need to request that my library system purchase a license for Docker Desktop, so I want to check if that is actually necessary.) 2. Is the idea to do all of the code editing, git commands, etc, from inside the container shell? Or would all of that be done outside the container, and the container be used only to run code/tests? 3. From what I've read in the wiki/getting started resources, I need Docker, the KTD itself, git-bz, and the git hooks from https://wiki.koha-community.org/wiki/Tips_and_tricks#git_tips_and_tricks (other than the pre-commit hook that is currently broken?) And then there's a tutorial on creating and submitting patches here (https://gitlab.com/koha-community/koha-howto/blob/master/README.md) that I can run once I have the KTD working. Am I missing anything? Hopefully this will make more sense once I have everything installed and can actually start playing with it, but I'll admit I'm feeling a little overwhelmed right now. Thank you so much to anyone who can give this newbie a little extra guidance on how to get started! Emily Lamancusa (she/her) IT Specialist III Montgomery County Public Libraries 21 Maryland Ave, Suite 310 Rockville, MD 20850 (240) 777-0676 [https://www.montgomerycountymd.gov/mcg/Resources/Images/Cybersecurity-foote…] For more helpful Cybersecurity Resources, visit: https://www.montgomerycountymd.gov/cybersecurity

3 2

x-koha-query header doesn't support UTF-8 data
by David Cook 05 Dec '22

05 Dec '22

Hi all, It turns out the “x-koha-query” header can’t support any value that isn’t ISO-8859-1 encoded. That really limits the number of languages that can use that header, so I think it has to be removed from Koha. See https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=32401 for more details. I bumped into the problem via the DataTables in the “Pending Orders” table when I was trying to search through the pending orders using an Arabic string. It just silently failed, so I started digging. When I opened the F12 tools, clicked “Sources”, and clicked “Pause on caught exceptions”, and then ran my search… I got a nice “Paused in debugger” message along with the caught exception: Failed to execute 'setRequestHeader' on 'XMLHttpRequest': String contains non ISO-8859-1 code point. Feel free to try it yourself using Arabic (خمسة), Chinese (五), or whatever language you like that requires UTF-8. I find the kohaTable code in koha-tmpl/intranet-tmpl/prog/js/datatables.js to be unwieldy so hesitant to touch it myself. I think Agustin, Tomas, and Jonathan have the best understanding of it, so they might be best positioned to fix this one. Not sure if everything will need to be refactored to POSTs or if the queries can be put into the “q” parameter, but anyway… just raising awareness about it here for now. Curious to hear what people think. Cheers, David Cook Senior Software Engineer Prosentient Systems Suite 7.03 6a Glen St Milsons Point NSW 2061 Australia Office: 02 9212 0899 Online: 02 8005 0595

1 0

OS upgrade for my linode (running bugs, splinter, the dashboard etc)
by Chris Cormack 04 Dec '22

04 Dec '22

Hi All I am about to do any OS upgrade for the server that runs a bunch of our services. I figure now is the best time post the major release. It will be going from 18.04 to 20.04 (maybe to 22.04 later also) Chris

1 1

OAI set mappings
by David Cook 02 Dec '22

02 Dec '22

Hi all, Do many people use "OAI set mappings"? I was feeling a bit frustrated because there's no obvious way to group rules using the Boolean AND/OR rules. I wanted a rule like the following but there was no clear mechanism to add those groupings. (952$a eq BRANCH) && ( (952$c eq GEN) || (952$c eq AV) ) After some trial and error, I realized you have to do the following: (952$a eq BRANCH && 952$c eq GEN) || (952$a eq BRANCH && 952$c eq AV) -- The OR rule seems to just bind together single rules or rules joined together with AND. I mean it's workable but it was a bit confusing and I don't think the documentation makes that clear. What do you folk think? David Cook Senior Software Engineer Prosentient Systems Suite 7.03 6a Glen St Milsons Point NSW 2061 Australia Office: 02 9212 0899 Online: 02 8005 0595

1 0

Tomorrow (Friday 2): Patch testing session to help enhancements and bug fixes to be released
by Victor Grousset/tuxayo 01 Dec '22

01 Dec '22

Hi :) This is for the support providers that aren't used to testing patches and any provider that knows libraries where someone could be interested. To be forwarded to librarians, system librarians, educators, devs, sysadmins and other people that could be interested in contributing to Koha. Let's meet Friday the 2nd at any moment in the following timeframe: 13h-20h UTC&UK / 8h-15h US East / 5h-12h US West The place is https://visio.chapril.org/koha (the text chat can be opened with the 4th button at the bottom: 🗨️) Reminder of the modalities: The idea is to test patches to contribute to Koha. The biggest bottleneck for having more bugs fixes and enhancements is patch testing. Thus this call for getting together as Koha users. No need for particular skills, just basic Koha usage. You can come an leave at any moment, even for a short time it's possible to get simple stuff moving forward. We will use a VoIP call with chat tool to speak and eventually screenshare. It's possible to join without sound and only use chat. So if you are at the front desk and there are not much patrons you can still test patches ^^ So, whichever communication mode is the most convenient for you. No need to have a local Koha instance. Just with a web browser we can use the sandboxes [1] to have temporary Kohas to test most patches and do whatever we want without bad consequences. Here is a resource about how to test patches using sandboxes: https://wiki.koha-community.org/wiki/Sandboxes Cheers, [1] https://wiki.koha-community.org/wiki/Sandboxes -- Victor Grousset/tuxayo

1 0