From victor at tuxayo.net Thu Dec 1 06:51:33 2022 From: victor at tuxayo.net (Victor Grousset/tuxayo) Date: Thu, 1 Dec 2022 06:51:33 +0100 Subject: [Koha-devel] Tomorrow (Friday 2): Patch testing session to help enhancements and bug fixes to be released Message-ID: Hi :) This is for the support providers that aren't used to testing patches and any provider that knows libraries where someone could be interested. To be forwarded to librarians, system librarians, educators, devs, sysadmins and other people that could be interested in contributing to Koha. Let's meet Friday the 2nd at any moment in the following timeframe: 13h-20h UTC&UK / 8h-15h US East / 5h-12h US West The place is https://visio.chapril.org/koha (the text chat can be opened with the 4th button at the bottom: đŸ—šïž) Reminder of the modalities: The idea is to test patches to contribute to Koha. The biggest bottleneck for having more bugs fixes and enhancements is patch testing. Thus this call for getting together as Koha users. No need for particular skills, just basic Koha usage. You can come an leave at any moment, even for a short time it's possible to get simple stuff moving forward. We will use a VoIP call with chat tool to speak and eventually screenshare. It's possible to join without sound and only use chat. So if you are at the front desk and there are not much patrons you can still test patches ^^ So, whichever communication mode is the most convenient for you. No need to have a local Koha instance. Just with a web browser we can use the sandboxes [1] to have temporary Kohas to test most patches and do whatever we want without bad consequences. Here is a resource about how to test patches using sandboxes: https://wiki.koha-community.org/wiki/Sandboxes Cheers, [1] https://wiki.koha-community.org/wiki/Sandboxes -- Victor Grousset/tuxayo From dcook at prosentient.com.au Fri Dec 2 06:41:07 2022 From: dcook at prosentient.com.au (David Cook) Date: Fri, 2 Dec 2022 16:41:07 +1100 Subject: [Koha-devel] OAI set mappings Message-ID: <063201d90610$af312fe0$0d938fa0$@prosentient.com.au> Hi all, Do many people use "OAI set mappings"? I was feeling a bit frustrated because there's no obvious way to group rules using the Boolean AND/OR rules. I wanted a rule like the following but there was no clear mechanism to add those groupings. (952$a eq BRANCH) && ( (952$c eq GEN) || (952$c eq AV) ) After some trial and error, I realized you have to do the following: (952$a eq BRANCH && 952$c eq GEN) || (952$a eq BRANCH && 952$c eq AV) -- The OR rule seems to just bind together single rules or rules joined together with AND. I mean it's workable but it was a bit confusing and I don't think the documentation makes that clear. What do you folk think? David Cook Senior Software Engineer Prosentient Systems Suite 7.03 6a Glen St Milsons Point NSW 2061 Australia Office: 02 9212 0899 Online: 02 8005 0595 -------------- next part -------------- An HTML attachment was scrubbed... URL: From jonathan.druart at bugs.koha-community.org Fri Dec 2 15:42:39 2022 From: jonathan.druart at bugs.koha-community.org (Jonathan Druart) Date: Fri, 2 Dec 2022 15:42:39 +0100 Subject: [Koha-devel] Good enough? Message-ID: Hi devs, I was wondering... How good is your "good enough"? It's a question I often ask myself, when writing patches or QAing yours: is it good enough to be into Koha? It does not have to be perfect or it may never reach master, but it must meet certain standards. When I was RM I tried to put that limit quite high. Not necessarily by asking the author for improving the follow-up patches, but also by adding the missing bits myself. There are various types of "good enough", depending on what we look at: good enough to be reviewed, good enough to be tested, to be put in production, get feedback, try an idea, etc. Most of the time, what is driving the limit is answering the questions "Is it maintainable?" / "Is it future proof?". Making sure the code you are writing won't be broken inadvertently is very important and must be part of the reflection. Katrin asked the QA team members what were our plans for 23.05. In my opinion we should enforce this "be future proof" concept. Writing code is easy, especially in Koha (yes it is!). Writing maintainable and robust code, following our guidelines is a bit harder. That's why we have a QA process that tries to catch inconsistencies or edge cases you may have missed. But I think we can be even more rigorous, and aim for better standards. We can elevate our ambitions and do better. When we see the number of new enhancements we are releasing every 6 months, it shows well that writing code is definitely not a problem. However sometimes developers are tempted to abandon their work once they are pushed to master. Pushed does not mean "done". Providing bug fixes, following-up with patches when needed, fixing CI jenkins, etc. is part of job (/fun) As a Koha developer for a long time now, I know how frustrating it can be to be asked for follow-ups/rewrite/tests to have our patches stamped with the precious PQA mark. But from the other point of view (RM, RMaints, QA team), I also know it's very frustrating when you are in charge of the release and you do not get the appropriate follow-up work once it's pushed to master. There are some easy steps to write/review better patches. All have been discussed already several times, but that can be enforced even more: 1. Perltidy (!) This is really a very trivial step. Please perltidy your code. There are hundreds of commits that have been pushed in the last months that are not tidy (alignment, indentation, lines too long, etc.) This can easily be configured in your IDE! [1] 2. Provide clean code. As said it's not necessarily easy, but the QA team and RM are supposed to know if the code is clean regarding Koha guidelines. If the code is not clean, don't PQA, don't push. Either clean yourself, or ask the original author of the patch to do it (explaining to them how it can be improved ofc). 3. Squash! I have been away for a couple of months and had to read the git history to know what I missed. And it was really hard to follow what was going on. First of all, we are not consistent: the commit message must tell what the patch is doing, not what the bug was (if you are writing a bug fix). Then, there are way too many follow-ups: tidiness, indentation fix, typo, spelling, etc. All those tiny follow-ups could be squashed into the original patch. We don't need unnecessary tons of entries in our git log for that. For instance, I usually add a "JD Amended patch: perltidy" for instance when I tidy the original patch, to keep track of the modification. Squash can be done by the original author, the QAer, the RM. So yes, you are losing one commit in the stats but the git log is easy to read! We could have an "Amended-by" marker if we really want to add credit on the dashboard (and/or release notes). 4. Run tests. Don't wait for Jenkins to fail. This is valid for the author and QA. Anticipate the failures by running more tests. If you are modifying C4::Circulation, then run prove on t/db_dependent/Circulation*, not only Circulation.t. It will help you catch edge cases. When something is pushed, track down jenkins failures that could be caused by your patches. 6. Be strict if you are QAing. Each QA member has their own "good enough", and the RM as well (either relying on the QAer or providing a full review). But QA must fail if the code is old Koha style code, or not "good enough". 7. Provide support for failing tests, fix things you broke. The QA team will be more comfortable with your patches if you show them you are providing support for your stuff. It's not because it's pushed that you don't have any more efforts to make. Provide follow-up patches you promised, provide bug fixes, etc. We don't have a good way to keep track of such demands, which does not make tracking easier for devs, QA and RM. Any suggestions? 8. QA team MUST NEVER* pass QA a change that is not covered by tests, never. You should not provide change to modules without tests! * almost never... 9. Stick to existing patterns. We should not have different ways to do the same thing. We should not have different places where a code is doing the same thing. Ask for help or advice on the list or IRC before you start coding. We will be happy to guide you. Even if you are a regular Koha developer it's not always easy to be aware of the latest master changes. We will tell you what's the current good practice, or point you to examples you could reuse for what you want to implement. 10. CI should drive the pushes. No more push if CI is not green. The more we wait the harder it is to track down the origin of the problem. Last cycle some jobs have been red for months, and we released 22.11.00 with D10, D11, D12 marked unstable... What will I do next cycle? All of that, and more. I will track down jenkins failures and responsibilize developers telling them when they break tests (and won't fix them anymore as I have been doing for years). I will raise on the bug reports what could have been improved. Yes, read that I will be even more annoying (to put it politely) than before. I've noticed that the pre-commit git hook on the wiki has been broken for more than 3 years. And also caught some core developers that do not have it in place. I am relying on it to keep Vue files tidy so it's important to have it set up properly. I am planning to force its usage for ktd users [2]. Adding more checks to it will help us to catch inconsistencies from the beginning. To summarize, writing code is cheap, maintaining code is way more expensive! It is easier to get the attention of developers before the patches are pushed to master than after, so we could be more ambitious and ask more. For discussion :) Cheers, Jonathan [1] If you are using vim, open ~/vimrc, add vmap :!perltidy -q Reload vim, select code in visual mode [2] https://gitlab.com/koha-community/koha-misc4dev/-/issues/59 From julian.maurice at biblibre.com Fri Dec 2 18:16:24 2022 From: julian.maurice at biblibre.com (Julian Maurice) Date: Fri, 2 Dec 2022 18:16:24 +0100 Subject: [Koha-devel] Good enough? In-Reply-To: References: Message-ID: <85c5041c-a3ee-db1c-b576-97e789193c14@biblibre.com> Hi, It seems to me that several issues could be solved by having the CI run sooner, so authors can have feedback as soon as possible. Something like github's CI integration with pull requests would be amazing to detect common mistakes early and stop wasting human time. We should know if tests pass before pushing to master, not after. We could check tidyness, commit message conventions, code coverage by tests, ... all before another person have to look at it. Also tests are not easy to run locally. They might pass on Jenkins, but they do not on my local setup, so they are basically useless to tell me if I broke something. It also makes writing new tests more difficult than it should. If I wanted to improve Koha developer experience, I would start with that. My 2 cents. Le 02/12/2022 Ă  15:42, Jonathan Druart a Ă©crit : > Hi devs, > > I was wondering... How good is your "good enough"? > It's a question I often ask myself, when writing patches or QAing > yours: is it good enough to be into Koha? It does not have to be > perfect or it may never reach master, but it must meet certain > standards. > > When I was RM I tried to put that limit quite high. Not necessarily by > asking the author for improving the follow-up patches, but also by > adding the missing bits myself. > > There are various types of "good enough", depending on what we look > at: good enough to be reviewed, good enough to be tested, to be put in > production, get feedback, try an idea, etc. > > Most of the time, what is driving the limit is answering the > questions "Is it maintainable?" / "Is it future proof?". Making sure > the code you are writing won't be broken inadvertently is very > important and must be part of the reflection. > > Katrin asked the QA team members what were our plans for 23.05. In my > opinion we should enforce this "be future proof" concept. Writing code > is easy, especially in Koha (yes it is!). Writing maintainable and > robust code, following our guidelines is a bit harder. That's why we > have a QA process that tries to catch inconsistencies or edge cases > you may have missed. > But I think we can be even more rigorous, and aim for better > standards. We can elevate our ambitions and do better. When we see the > number of new enhancements we are releasing every 6 months, it shows > well that writing code is definitely not a problem. However sometimes > developers are tempted to abandon their work once they are pushed to > master. Pushed does not mean "done". Providing bug fixes, following-up > with patches when needed, fixing CI jenkins, etc. is part of job > (/fun) > > As a Koha developer for a long time now, I know how frustrating it can > be to be asked for follow-ups/rewrite/tests to have our patches > stamped with the precious PQA mark. But from the other point of view > (RM, RMaints, QA team), I also know it's very frustrating when you are > in charge of the release and you do not get the appropriate follow-up > work once it's pushed to master. > > There are some easy steps to write/review better patches. All have > been discussed already several times, but that can be enforced even > more: > 1. Perltidy (!) This is really a very trivial step. Please perltidy > your code. There are hundreds of commits that have been pushed in the > last months that are not tidy (alignment, indentation, lines too long, > etc.) This can easily be configured in your IDE! [1] > > 2. Provide clean code. As said it's not necessarily easy, but the QA > team and RM are supposed to know if the code is clean regarding Koha > guidelines. If the code is not clean, don't PQA, don't push. Either > clean yourself, or ask the original author of the patch to do it > (explaining to them how it can be improved ofc). > > 3. Squash! I have been away for a couple of months and had to read the > git history to know what I missed. And it was really hard to follow > what was going on. First of all, we are not consistent: the commit > message must tell what the patch is doing, not what the bug was (if > you are writing a bug fix). Then, there are way too many follow-ups: > tidiness, indentation fix, typo, spelling, etc. All those tiny > follow-ups could be squashed into the original patch. We don't need > unnecessary tons of entries in our git log for that. For instance, I > usually add a "JD Amended patch: perltidy" for instance when I tidy > the original patch, to keep track of the modification. Squash can be > done by the original author, the QAer, the RM. So yes, you are losing > one commit in the stats but the git log is easy to read! > We could have an "Amended-by" marker if we really want to add credit > on the dashboard (and/or release notes). > > 4. Run tests. Don't wait for Jenkins to fail. This is valid for the > author and QA. Anticipate the failures by running more tests. If you > are modifying C4::Circulation, then run prove on > t/db_dependent/Circulation*, not only Circulation.t. It will help you > catch edge cases. > When something is pushed, track down jenkins failures that could be > caused by your patches. > > 6. Be strict if you are QAing. Each QA member has their own "good > enough", and the RM as well (either relying on the QAer or providing a > full review). But QA must fail if the code is old Koha style code, or > not "good enough". > > 7. Provide support for failing tests, fix things you broke. The QA > team will be more comfortable with your patches if you show them you > are providing support for your stuff. > It's not because it's pushed that you don't have any more efforts to > make. Provide follow-up patches you promised, provide bug fixes, etc. > We don't have a good way to keep track of such demands, which does not > make tracking easier for devs, QA and RM. Any suggestions? > > 8. QA team MUST NEVER* pass QA a change that is not covered by tests, > never. You should not provide change to modules without tests! > * almost never... > > 9. Stick to existing patterns. We should not have different ways to do > the same thing. We should not have different places where a code is > doing the same thing. Ask for help or advice on the list or IRC before > you start coding. We will be happy to guide you. Even if you are a > regular Koha developer it's not always easy to be aware of the latest > master changes. > We will tell you what's the current good practice, or point you to > examples you could reuse for what you want to implement. > > 10. CI should drive the pushes. No more push if CI is not green. The > more we wait the harder it is to track down the origin of the problem. > Last cycle some jobs have been red for months, and we released > 22.11.00 with D10, D11, D12 marked unstable... > > What will I do next cycle? > All of that, and more. I will track down jenkins failures and > responsibilize developers telling them when they break tests (and > won't fix them anymore as I have been doing for years). > I will raise on the bug reports what could have been improved. Yes, > read that I will be even more annoying (to put it politely) than > before. > > I've noticed that the pre-commit git hook on the wiki has been broken > for more than 3 years. And also caught some core developers that do > not have it in place. I am relying on it to keep Vue files tidy so > it's important to have it set up properly. I am planning to force its > usage for ktd users [2]. Adding more checks to it will help us to > catch inconsistencies from the beginning. > > To summarize, writing code is cheap, maintaining code is way more > expensive! It is easier to get the attention of developers before the > patches are pushed to master than after, so we could be more ambitious > and ask more. > > For discussion :) > > Cheers, > Jonathan > > [1] If you are using vim, open ~/vimrc, add > vmap :!perltidy -q > Reload vim, select code in visual mode > [2] https://gitlab.com/koha-community/koha-misc4dev/-/issues/59 > _______________________________________________ > Koha-devel mailing list > Koha-devel at lists.koha-community.org > https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-devel > website : https://www.koha-community.org/ > git : https://git.koha-community.org/ > bugs : https://bugs.koha-community.org/ -- Julian Maurice BibLibre From chris at bigballofwax.co.nz Sun Dec 4 21:23:37 2022 From: chris at bigballofwax.co.nz (Chris Cormack) Date: Mon, 5 Dec 2022 09:23:37 +1300 Subject: [Koha-devel] OS upgrade for my linode (running bugs, splinter, the dashboard etc) Message-ID: Hi All I am about to do any OS upgrade for the server that runs a bunch of our services. I figure now is the best time post the major release. It will be going from 18.04 to 20.04 (maybe to 22.04 later also) Chris From chris at bigballofwax.co.nz Sun Dec 4 22:30:07 2022 From: chris at bigballofwax.co.nz (Chris Cormack) Date: Mon, 5 Dec 2022 10:30:07 +1300 Subject: [Koha-devel] OS upgrade for my linode (running bugs, splinter, the dashboard etc) In-Reply-To: References: Message-ID: Ok, maybe not. It needs to change from i386 to amd64. So I will have to block out some actual time to do that. Will try again another day Chris On Mon, 5 Dec 2022 at 09:23, Chris Cormack wrote: > > Hi All > > I am about to do any OS upgrade for the server that runs a bunch of > our services. I figure now is the best time post the major release. It > will be going from 18.04 to 20.04 (maybe to 22.04 later also) > > Chris From dcook at prosentient.com.au Mon Dec 5 04:32:24 2022 From: dcook at prosentient.com.au (David Cook) Date: Mon, 5 Dec 2022 14:32:24 +1100 Subject: [Koha-devel] x-koha-query header doesn't support UTF-8 data Message-ID: <073b01d9085a$3f885b40$be9911c0$@prosentient.com.au> Hi all, It turns out the “x-koha-query” header can’t support any value that isn’t ISO-8859-1 encoded. That really limits the number of languages that can use that header, so I think it has to be removed from Koha. See https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=32401 for more details. I bumped into the problem via the DataTables in the “Pending Orders” table when I was trying to search through the pending orders using an Arabic string. It just silently failed, so I started digging. When I opened the F12 tools, clicked “Sources”, and clicked “Pause on caught exceptions”, and then ran my search
 I got a nice “Paused in debugger” message along with the caught exception: Failed to execute 'setRequestHeader' on 'XMLHttpRequest': String contains non ISO-8859-1 code point. Feel free to try it yourself using Arabic (ŰźÙ…ŰłŰ©), Chinese (äș”), or whatever language you like that requires UTF-8. I find the kohaTable code in koha-tmpl/intranet-tmpl/prog/js/datatables.js to be unwieldy so hesitant to touch it myself. I think Agustin, Tomas, and Jonathan have the best understanding of it, so they might be best positioned to fix this one. Not sure if everything will need to be refactored to POSTs or if the queries can be put into the “q” parameter, but anyway
 just raising awareness about it here for now. Curious to hear what people think. Cheers, David Cook Senior Software Engineer Prosentient Systems Suite 7.03 6a Glen St Milsons Point NSW 2061 Australia Office: 02 9212 0899 Online: 02 8005 0595 -------------- next part -------------- An HTML attachment was scrubbed... URL: From M.de.Rooy at rijksmuseum.nl Mon Dec 5 11:34:39 2022 From: M.de.Rooy at rijksmuseum.nl (Marcel de Rooy) Date: Mon, 5 Dec 2022 10:34:39 +0000 Subject: [Koha-devel] Good enough? In-Reply-To: References: Message-ID: Could you provide more info on those git hooks? A wiki URL? Marcel -----Original Message----- From: Koha-devel On Behalf Of Jonathan Druart Sent: Friday, December 2, 2022 3:43 PM To: koha-devel Subject: [Koha-devel] Good enough? Hi devs, I was wondering... How good is your "good enough"? It's a question I often ask myself, when writing patches or QAing yours: is it good enough to be into Koha? It does not have to be perfect or it may never reach master, but it must meet certain standards. When I was RM I tried to put that limit quite high. Not necessarily by asking the author for improving the follow-up patches, but also by adding the missing bits myself. There are various types of "good enough", depending on what we look at: good enough to be reviewed, good enough to be tested, to be put in production, get feedback, try an idea, etc. Most of the time, what is driving the limit is answering the questions "Is it maintainable?" / "Is it future proof?". Making sure the code you are writing won't be broken inadvertently is very important and must be part of the reflection. Katrin asked the QA team members what were our plans for 23.05. In my opinion we should enforce this "be future proof" concept. Writing code is easy, especially in Koha (yes it is!). Writing maintainable and robust code, following our guidelines is a bit harder. That's why we have a QA process that tries to catch inconsistencies or edge cases you may have missed. But I think we can be even more rigorous, and aim for better standards. We can elevate our ambitions and do better. When we see the number of new enhancements we are releasing every 6 months, it shows well that writing code is definitely not a problem. However sometimes developers are tempted to abandon their work once they are pushed to master. Pushed does not mean "done". Providing bug fixes, following-up with patches when needed, fixing CI jenkins, etc. is part of job (/fun) As a Koha developer for a long time now, I know how frustrating it can be to be asked for follow-ups/rewrite/tests to have our patches stamped with the precious PQA mark. But from the other point of view (RM, RMaints, QA team), I also know it's very frustrating when you are in charge of the release and you do not get the appropriate follow-up work once it's pushed to master. There are some easy steps to write/review better patches. All have been discussed already several times, but that can be enforced even more: 1. Perltidy (!) This is really a very trivial step. Please perltidy your code. There are hundreds of commits that have been pushed in the last months that are not tidy (alignment, indentation, lines too long, etc.) This can easily be configured in your IDE! [1] 2. Provide clean code. As said it's not necessarily easy, but the QA team and RM are supposed to know if the code is clean regarding Koha guidelines. If the code is not clean, don't PQA, don't push. Either clean yourself, or ask the original author of the patch to do it (explaining to them how it can be improved ofc). 3. Squash! I have been away for a couple of months and had to read the git history to know what I missed. And it was really hard to follow what was going on. First of all, we are not consistent: the commit message must tell what the patch is doing, not what the bug was (if you are writing a bug fix). Then, there are way too many follow-ups: tidiness, indentation fix, typo, spelling, etc. All those tiny follow-ups could be squashed into the original patch. We don't need unnecessary tons of entries in our git log for that. For instance, I usually add a "JD Amended patch: perltidy" for instance when I tidy the original patch, to keep track of the modification. Squash can be done by the original author, the QAer, the RM. So yes, you are losing one commit in the stats but the git log is easy to read! We could have an "Amended-by" marker if we really want to add credit on the dashboard (and/or release notes). 4. Run tests. Don't wait for Jenkins to fail. This is valid for the author and QA. Anticipate the failures by running more tests. If you are modifying C4::Circulation, then run prove on t/db_dependent/Circulation*, not only Circulation.t. It will help you catch edge cases. When something is pushed, track down jenkins failures that could be caused by your patches. 6. Be strict if you are QAing. Each QA member has their own "good enough", and the RM as well (either relying on the QAer or providing a full review). But QA must fail if the code is old Koha style code, or not "good enough". 7. Provide support for failing tests, fix things you broke. The QA team will be more comfortable with your patches if you show them you are providing support for your stuff. It's not because it's pushed that you don't have any more efforts to make. Provide follow-up patches you promised, provide bug fixes, etc. We don't have a good way to keep track of such demands, which does not make tracking easier for devs, QA and RM. Any suggestions? 8. QA team MUST NEVER* pass QA a change that is not covered by tests, never. You should not provide change to modules without tests! * almost never... 9. Stick to existing patterns. We should not have different ways to do the same thing. We should not have different places where a code is doing the same thing. Ask for help or advice on the list or IRC before you start coding. We will be happy to guide you. Even if you are a regular Koha developer it's not always easy to be aware of the latest master changes. We will tell you what's the current good practice, or point you to examples you could reuse for what you want to implement. 10. CI should drive the pushes. No more push if CI is not green. The more we wait the harder it is to track down the origin of the problem. Last cycle some jobs have been red for months, and we released 22.11.00 with D10, D11, D12 marked unstable... What will I do next cycle? All of that, and more. I will track down jenkins failures and responsibilize developers telling them when they break tests (and won't fix them anymore as I have been doing for years). I will raise on the bug reports what could have been improved. Yes, read that I will be even more annoying (to put it politely) than before. I've noticed that the pre-commit git hook on the wiki has been broken for more than 3 years. And also caught some core developers that do not have it in place. I am relying on it to keep Vue files tidy so it's important to have it set up properly. I am planning to force its usage for ktd users [2]. Adding more checks to it will help us to catch inconsistencies from the beginning. To summarize, writing code is cheap, maintaining code is way more expensive! It is easier to get the attention of developers before the patches are pushed to master than after, so we could be more ambitious and ask more. For discussion :) Cheers, Jonathan [1] If you are using vim, open ~/vimrc, add vmap :!perltidy -q Reload vim, select code in visual mode [2] https://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgitlab.com%2Fkoha-community%2Fkoha-misc4dev%2F-%2Fissues%2F59&data=05%7C01%7Cm.de.rooy%40rijksmuseum.nl%7Cbbb48c7719e24d59d93e08dad4737fd0%7C635b05eb66c748e1a94fb4b05a1b058b%7C0%7C0%7C638055889782743169%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=SJ9ji43M2xQ2hMwPO5cugb0%2FPgA7ruPq955zxsS8OAg%3D&reserved=0 _______________________________________________ Koha-devel mailing list Koha-devel at lists.koha-community.org https://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Flists.koha-community.org%2Fcgi-bin%2Fmailman%2Flistinfo%2Fkoha-devel&data=05%7C01%7Cm.de.rooy%40rijksmuseum.nl%7Cbbb48c7719e24d59d93e08dad4737fd0%7C635b05eb66c748e1a94fb4b05a1b058b%7C0%7C0%7C638055889782743169%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=iKPUZ%2FWFXZd4pJ5bE0bn4AQQ%2FT3Qls7cp3zIrfAJZrU%3D&reserved=0 website : https://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.koha-community.org%2F&data=05%7C01%7Cm.de.rooy%40rijksmuseum.nl%7Cbbb48c7719e24d59d93e08dad4737fd0%7C635b05eb66c748e1a94fb4b05a1b058b%7C0%7C0%7C638055889782743169%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=oMko%2ByjqtsCELrgK8HMYQaOs0tb6z11QFkITTN18eXw%3D&reserved=0 git : https://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgit.koha-community.org%2F&data=05%7C01%7Cm.de.rooy%40rijksmuseum.nl%7Cbbb48c7719e24d59d93e08dad4737fd0%7C635b05eb66c748e1a94fb4b05a1b058b%7C0%7C0%7C638055889782743169%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=3DIkt154oi3HFJOmTnal6Kf4JvDZtLJZKF6HLxOjfcs%3D&reserved=0 bugs : https://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fbugs.koha-community.org%2F&data=05%7C01%7Cm.de.rooy%40rijksmuseum.nl%7Cbbb48c7719e24d59d93e08dad4737fd0%7C635b05eb66c748e1a94fb4b05a1b058b%7C0%7C0%7C638055889782743169%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=2mBEKSFfDbgfQ0kb1I1%2BFn4DkLddzAy5C99mhlH5Opg%3D&reserved=0 From jonathan.druart at bugs.koha-community.org Mon Dec 5 14:55:01 2022 From: jonathan.druart at bugs.koha-community.org (Jonathan Druart) Date: Mon, 5 Dec 2022 14:55:01 +0100 Subject: [Koha-devel] Good enough? In-Reply-To: <85c5041c-a3ee-db1c-b576-97e789193c14@biblibre.com> References: <85c5041c-a3ee-db1c-b576-97e789193c14@biblibre.com> Message-ID: I don't think we should run the whole test suite everytime we attach patches, that would be very expensive in terms of resources. However it would be interesting to have a temporary 'master' branch that would become 'master' only if jenkins is happy. "master" would never be broken :D Using koha-testing-docker it's very easy to run tests locally. It will be even more easier with https://gitlab.com/koha-community/koha-misc4dev/-/issues/58 Le ven. 2 dĂ©c. 2022 Ă  18:16, Julian Maurice a Ă©crit : > > Hi, > > It seems to me that several issues could be solved by having the CI run > sooner, so authors can have feedback as soon as possible. Something like > github's CI integration with pull requests would be amazing to detect > common mistakes early and stop wasting human time. We should know if > tests pass before pushing to master, not after. > We could check tidyness, commit message conventions, code coverage by > tests, ... all before another person have to look at it. > > Also tests are not easy to run locally. They might pass on Jenkins, but > they do not on my local setup, so they are basically useless to tell me > if I broke something. It also makes writing new tests more difficult > than it should. > > If I wanted to improve Koha developer experience, I would start with that. > > My 2 cents. > > Le 02/12/2022 Ă  15:42, Jonathan Druart a Ă©crit : > > Hi devs, > > > > I was wondering... How good is your "good enough"? > > It's a question I often ask myself, when writing patches or QAing > > yours: is it good enough to be into Koha? It does not have to be > > perfect or it may never reach master, but it must meet certain > > standards. > > > > When I was RM I tried to put that limit quite high. Not necessarily by > > asking the author for improving the follow-up patches, but also by > > adding the missing bits myself. > > > > There are various types of "good enough", depending on what we look > > at: good enough to be reviewed, good enough to be tested, to be put in > > production, get feedback, try an idea, etc. > > > > Most of the time, what is driving the limit is answering the > > questions "Is it maintainable?" / "Is it future proof?". Making sure > > the code you are writing won't be broken inadvertently is very > > important and must be part of the reflection. > > > > Katrin asked the QA team members what were our plans for 23.05. In my > > opinion we should enforce this "be future proof" concept. Writing code > > is easy, especially in Koha (yes it is!). Writing maintainable and > > robust code, following our guidelines is a bit harder. That's why we > > have a QA process that tries to catch inconsistencies or edge cases > > you may have missed. > > But I think we can be even more rigorous, and aim for better > > standards. We can elevate our ambitions and do better. When we see the > > number of new enhancements we are releasing every 6 months, it shows > > well that writing code is definitely not a problem. However sometimes > > developers are tempted to abandon their work once they are pushed to > > master. Pushed does not mean "done". Providing bug fixes, following-up > > with patches when needed, fixing CI jenkins, etc. is part of job > > (/fun) > > > > As a Koha developer for a long time now, I know how frustrating it can > > be to be asked for follow-ups/rewrite/tests to have our patches > > stamped with the precious PQA mark. But from the other point of view > > (RM, RMaints, QA team), I also know it's very frustrating when you are > > in charge of the release and you do not get the appropriate follow-up > > work once it's pushed to master. > > > > There are some easy steps to write/review better patches. All have > > been discussed already several times, but that can be enforced even > > more: > > 1. Perltidy (!) This is really a very trivial step. Please perltidy > > your code. There are hundreds of commits that have been pushed in the > > last months that are not tidy (alignment, indentation, lines too long, > > etc.) This can easily be configured in your IDE! [1] > > > > 2. Provide clean code. As said it's not necessarily easy, but the QA > > team and RM are supposed to know if the code is clean regarding Koha > > guidelines. If the code is not clean, don't PQA, don't push. Either > > clean yourself, or ask the original author of the patch to do it > > (explaining to them how it can be improved ofc). > > > > 3. Squash! I have been away for a couple of months and had to read the > > git history to know what I missed. And it was really hard to follow > > what was going on. First of all, we are not consistent: the commit > > message must tell what the patch is doing, not what the bug was (if > > you are writing a bug fix). Then, there are way too many follow-ups: > > tidiness, indentation fix, typo, spelling, etc. All those tiny > > follow-ups could be squashed into the original patch. We don't need > > unnecessary tons of entries in our git log for that. For instance, I > > usually add a "JD Amended patch: perltidy" for instance when I tidy > > the original patch, to keep track of the modification. Squash can be > > done by the original author, the QAer, the RM. So yes, you are losing > > one commit in the stats but the git log is easy to read! > > We could have an "Amended-by" marker if we really want to add credit > > on the dashboard (and/or release notes). > > > > 4. Run tests. Don't wait for Jenkins to fail. This is valid for the > > author and QA. Anticipate the failures by running more tests. If you > > are modifying C4::Circulation, then run prove on > > t/db_dependent/Circulation*, not only Circulation.t. It will help you > > catch edge cases. > > When something is pushed, track down jenkins failures that could be > > caused by your patches. > > > > 6. Be strict if you are QAing. Each QA member has their own "good > > enough", and the RM as well (either relying on the QAer or providing a > > full review). But QA must fail if the code is old Koha style code, or > > not "good enough". > > > > 7. Provide support for failing tests, fix things you broke. The QA > > team will be more comfortable with your patches if you show them you > > are providing support for your stuff. > > It's not because it's pushed that you don't have any more efforts to > > make. Provide follow-up patches you promised, provide bug fixes, etc. > > We don't have a good way to keep track of such demands, which does not > > make tracking easier for devs, QA and RM. Any suggestions? > > > > 8. QA team MUST NEVER* pass QA a change that is not covered by tests, > > never. You should not provide change to modules without tests! > > * almost never... > > > > 9. Stick to existing patterns. We should not have different ways to do > > the same thing. We should not have different places where a code is > > doing the same thing. Ask for help or advice on the list or IRC before > > you start coding. We will be happy to guide you. Even if you are a > > regular Koha developer it's not always easy to be aware of the latest > > master changes. > > We will tell you what's the current good practice, or point you to > > examples you could reuse for what you want to implement. > > > > 10. CI should drive the pushes. No more push if CI is not green. The > > more we wait the harder it is to track down the origin of the problem. > > Last cycle some jobs have been red for months, and we released > > 22.11.00 with D10, D11, D12 marked unstable... > > > > What will I do next cycle? > > All of that, and more. I will track down jenkins failures and > > responsibilize developers telling them when they break tests (and > > won't fix them anymore as I have been doing for years). > > I will raise on the bug reports what could have been improved. Yes, > > read that I will be even more annoying (to put it politely) than > > before. > > > > I've noticed that the pre-commit git hook on the wiki has been broken > > for more than 3 years. And also caught some core developers that do > > not have it in place. I am relying on it to keep Vue files tidy so > > it's important to have it set up properly. I am planning to force its > > usage for ktd users [2]. Adding more checks to it will help us to > > catch inconsistencies from the beginning. > > > > To summarize, writing code is cheap, maintaining code is way more > > expensive! It is easier to get the attention of developers before the > > patches are pushed to master than after, so we could be more ambitious > > and ask more. > > > > For discussion :) > > > > Cheers, > > Jonathan > > > > [1] If you are using vim, open ~/vimrc, add > > vmap :!perltidy -q > > Reload vim, select code in visual mode > > [2] https://gitlab.com/koha-community/koha-misc4dev/-/issues/59 > > _______________________________________________ > > Koha-devel mailing list > > Koha-devel at lists.koha-community.org > > https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-devel > > website : https://www.koha-community.org/ > > git : https://git.koha-community.org/ > > bugs : https://bugs.koha-community.org/ > > -- > Julian Maurice > BibLibre > _______________________________________________ > Koha-devel mailing list > Koha-devel at lists.koha-community.org > https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-devel > website : https://www.koha-community.org/ > git : https://git.koha-community.org/ > bugs : https://bugs.koha-community.org/ From jonathan.druart at bugs.koha-community.org Mon Dec 5 14:56:07 2022 From: jonathan.druart at bugs.koha-community.org (Jonathan Druart) Date: Mon, 5 Dec 2022 14:56:07 +0100 Subject: [Koha-devel] Good enough? In-Reply-To: References: Message-ID: Existing hooks can be found at https://wiki.koha-community.org/wiki/Tips_and_tricks#Hooks Le lun. 5 dĂ©c. 2022 Ă  11:34, Marcel de Rooy a Ă©crit : > > Could you provide more info on those git hooks? A wiki URL? > > Marcel > > -----Original Message----- > From: Koha-devel On Behalf Of Jonathan Druart > Sent: Friday, December 2, 2022 3:43 PM > To: koha-devel > Subject: [Koha-devel] Good enough? > > Hi devs, > > I was wondering... How good is your "good enough"? > It's a question I often ask myself, when writing patches or QAing > yours: is it good enough to be into Koha? It does not have to be perfect or it may never reach master, but it must meet certain standards. > > When I was RM I tried to put that limit quite high. Not necessarily by asking the author for improving the follow-up patches, but also by adding the missing bits myself. > > There are various types of "good enough", depending on what we look > at: good enough to be reviewed, good enough to be tested, to be put in production, get feedback, try an idea, etc. > > Most of the time, what is driving the limit is answering the questions "Is it maintainable?" / "Is it future proof?". Making sure the code you are writing won't be broken inadvertently is very important and must be part of the reflection. > > Katrin asked the QA team members what were our plans for 23.05. In my opinion we should enforce this "be future proof" concept. Writing code is easy, especially in Koha (yes it is!). Writing maintainable and robust code, following our guidelines is a bit harder. That's why we have a QA process that tries to catch inconsistencies or edge cases you may have missed. > But I think we can be even more rigorous, and aim for better standards. We can elevate our ambitions and do better. When we see the number of new enhancements we are releasing every 6 months, it shows well that writing code is definitely not a problem. However sometimes developers are tempted to abandon their work once they are pushed to master. Pushed does not mean "done". Providing bug fixes, following-up with patches when needed, fixing CI jenkins, etc. is part of job > (/fun) > > As a Koha developer for a long time now, I know how frustrating it can be to be asked for follow-ups/rewrite/tests to have our patches stamped with the precious PQA mark. But from the other point of view (RM, RMaints, QA team), I also know it's very frustrating when you are in charge of the release and you do not get the appropriate follow-up work once it's pushed to master. > > There are some easy steps to write/review better patches. All have been discussed already several times, but that can be enforced even > more: > 1. Perltidy (!) This is really a very trivial step. Please perltidy your code. There are hundreds of commits that have been pushed in the last months that are not tidy (alignment, indentation, lines too long, > etc.) This can easily be configured in your IDE! [1] > > 2. Provide clean code. As said it's not necessarily easy, but the QA team and RM are supposed to know if the code is clean regarding Koha guidelines. If the code is not clean, don't PQA, don't push. Either clean yourself, or ask the original author of the patch to do it (explaining to them how it can be improved ofc). > > 3. Squash! I have been away for a couple of months and had to read the git history to know what I missed. And it was really hard to follow what was going on. First of all, we are not consistent: the commit message must tell what the patch is doing, not what the bug was (if you are writing a bug fix). Then, there are way too many follow-ups: > tidiness, indentation fix, typo, spelling, etc. All those tiny follow-ups could be squashed into the original patch. We don't need unnecessary tons of entries in our git log for that. For instance, I usually add a "JD Amended patch: perltidy" for instance when I tidy the original patch, to keep track of the modification. Squash can be done by the original author, the QAer, the RM. So yes, you are losing one commit in the stats but the git log is easy to read! > We could have an "Amended-by" marker if we really want to add credit on the dashboard (and/or release notes). > > 4. Run tests. Don't wait for Jenkins to fail. This is valid for the author and QA. Anticipate the failures by running more tests. If you are modifying C4::Circulation, then run prove on t/db_dependent/Circulation*, not only Circulation.t. It will help you catch edge cases. > When something is pushed, track down jenkins failures that could be caused by your patches. > > 6. Be strict if you are QAing. Each QA member has their own "good enough", and the RM as well (either relying on the QAer or providing a full review). But QA must fail if the code is old Koha style code, or not "good enough". > > 7. Provide support for failing tests, fix things you broke. The QA team will be more comfortable with your patches if you show them you are providing support for your stuff. > It's not because it's pushed that you don't have any more efforts to make. Provide follow-up patches you promised, provide bug fixes, etc. > We don't have a good way to keep track of such demands, which does not make tracking easier for devs, QA and RM. Any suggestions? > > 8. QA team MUST NEVER* pass QA a change that is not covered by tests, never. You should not provide change to modules without tests! > * almost never... > > 9. Stick to existing patterns. We should not have different ways to do the same thing. We should not have different places where a code is doing the same thing. Ask for help or advice on the list or IRC before you start coding. We will be happy to guide you. Even if you are a regular Koha developer it's not always easy to be aware of the latest master changes. > We will tell you what's the current good practice, or point you to examples you could reuse for what you want to implement. > > 10. CI should drive the pushes. No more push if CI is not green. The more we wait the harder it is to track down the origin of the problem. > Last cycle some jobs have been red for months, and we released > 22.11.00 with D10, D11, D12 marked unstable... > > What will I do next cycle? > All of that, and more. I will track down jenkins failures and responsibilize developers telling them when they break tests (and won't fix them anymore as I have been doing for years). > I will raise on the bug reports what could have been improved. Yes, read that I will be even more annoying (to put it politely) than before. > > I've noticed that the pre-commit git hook on the wiki has been broken for more than 3 years. And also caught some core developers that do not have it in place. I am relying on it to keep Vue files tidy so it's important to have it set up properly. I am planning to force its usage for ktd users [2]. Adding more checks to it will help us to catch inconsistencies from the beginning. > > To summarize, writing code is cheap, maintaining code is way more expensive! It is easier to get the attention of developers before the patches are pushed to master than after, so we could be more ambitious and ask more. > > For discussion :) > > Cheers, > Jonathan > > [1] If you are using vim, open ~/vimrc, add > vmap :!perltidy -q > Reload vim, select code in visual mode > [2] https://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgitlab.com%2Fkoha-community%2Fkoha-misc4dev%2F-%2Fissues%2F59&data=05%7C01%7Cm.de.rooy%40rijksmuseum.nl%7Cbbb48c7719e24d59d93e08dad4737fd0%7C635b05eb66c748e1a94fb4b05a1b058b%7C0%7C0%7C638055889782743169%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=SJ9ji43M2xQ2hMwPO5cugb0%2FPgA7ruPq955zxsS8OAg%3D&reserved=0 > _______________________________________________ > Koha-devel mailing list > Koha-devel at lists.koha-community.org > https://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Flists.koha-community.org%2Fcgi-bin%2Fmailman%2Flistinfo%2Fkoha-devel&data=05%7C01%7Cm.de.rooy%40rijksmuseum.nl%7Cbbb48c7719e24d59d93e08dad4737fd0%7C635b05eb66c748e1a94fb4b05a1b058b%7C0%7C0%7C638055889782743169%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=iKPUZ%2FWFXZd4pJ5bE0bn4AQQ%2FT3Qls7cp3zIrfAJZrU%3D&reserved=0 > website : https://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.koha-community.org%2F&data=05%7C01%7Cm.de.rooy%40rijksmuseum.nl%7Cbbb48c7719e24d59d93e08dad4737fd0%7C635b05eb66c748e1a94fb4b05a1b058b%7C0%7C0%7C638055889782743169%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=oMko%2ByjqtsCELrgK8HMYQaOs0tb6z11QFkITTN18eXw%3D&reserved=0 > git : https://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgit.koha-community.org%2F&data=05%7C01%7Cm.de.rooy%40rijksmuseum.nl%7Cbbb48c7719e24d59d93e08dad4737fd0%7C635b05eb66c748e1a94fb4b05a1b058b%7C0%7C0%7C638055889782743169%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=3DIkt154oi3HFJOmTnal6Kf4JvDZtLJZKF6HLxOjfcs%3D&reserved=0 > bugs : https://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fbugs.koha-community.org%2F&data=05%7C01%7Cm.de.rooy%40rijksmuseum.nl%7Cbbb48c7719e24d59d93e08dad4737fd0%7C635b05eb66c748e1a94fb4b05a1b058b%7C0%7C0%7C638055889782743169%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=2mBEKSFfDbgfQ0kb1I1%2BFn4DkLddzAy5C99mhlH5Opg%3D&reserved=0 From M.de.Rooy at rijksmuseum.nl Mon Dec 5 15:17:14 2022 From: M.de.Rooy at rijksmuseum.nl (Marcel de Rooy) Date: Mon, 5 Dec 2022 14:17:14 +0000 Subject: [Koha-devel] Good enough? In-Reply-To: References: Message-ID: Thanks for sending this link. Could we merge that code with QA code we already pushed into our codebase somehow ? -----Original Message----- From: Koha-devel On Behalf Of Jonathan Druart Sent: Monday, December 5, 2022 2:56 PM To: koha-devel Subject: Re: [Koha-devel] Good enough? Existing hooks can be found at https://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwiki.koha-community.org%2Fwiki%2FTips_and_tricks%23Hooks&data=05%7C01%7Cm.de.rooy%40rijksmuseum.nl%7C5d76e33c09514528c67808dad6c87f27%7C635b05eb66c748e1a94fb4b05a1b058b%7C0%7C0%7C638058453872270550%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=bxdii78avAYEIcecc8o0tQZ4v7atwVTkmbJMYPe1Dek%3D&reserved=0 Le lun. 5 dĂ©c. 2022 Ă  11:34, Marcel de Rooy a Ă©crit : > > Could you provide more info on those git hooks? A wiki URL? > > Marcel > > -----Original Message----- > From: Koha-devel On > Behalf Of Jonathan Druart > Sent: Friday, December 2, 2022 3:43 PM > To: koha-devel > Subject: [Koha-devel] Good enough? > > Hi devs, > > I was wondering... How good is your "good enough"? > It's a question I often ask myself, when writing patches or QAing > yours: is it good enough to be into Koha? It does not have to be perfect or it may never reach master, but it must meet certain standards. > > When I was RM I tried to put that limit quite high. Not necessarily by asking the author for improving the follow-up patches, but also by adding the missing bits myself. > > There are various types of "good enough", depending on what we look > at: good enough to be reviewed, good enough to be tested, to be put in production, get feedback, try an idea, etc. > > Most of the time, what is driving the limit is answering the questions "Is it maintainable?" / "Is it future proof?". Making sure the code you are writing won't be broken inadvertently is very important and must be part of the reflection. > > Katrin asked the QA team members what were our plans for 23.05. In my opinion we should enforce this "be future proof" concept. Writing code is easy, especially in Koha (yes it is!). Writing maintainable and robust code, following our guidelines is a bit harder. That's why we have a QA process that tries to catch inconsistencies or edge cases you may have missed. > But I think we can be even more rigorous, and aim for better > standards. We can elevate our ambitions and do better. When we see the > number of new enhancements we are releasing every 6 months, it shows > well that writing code is definitely not a problem. However sometimes > developers are tempted to abandon their work once they are pushed to > master. Pushed does not mean "done". Providing bug fixes, following-up > with patches when needed, fixing CI jenkins, etc. is part of job > (/fun) > > As a Koha developer for a long time now, I know how frustrating it can be to be asked for follow-ups/rewrite/tests to have our patches stamped with the precious PQA mark. But from the other point of view (RM, RMaints, QA team), I also know it's very frustrating when you are in charge of the release and you do not get the appropriate follow-up work once it's pushed to master. > > There are some easy steps to write/review better patches. All have > been discussed already several times, but that can be enforced even > more: > 1. Perltidy (!) This is really a very trivial step. Please perltidy > your code. There are hundreds of commits that have been pushed in the > last months that are not tidy (alignment, indentation, lines too long, > etc.) This can easily be configured in your IDE! [1] > > 2. Provide clean code. As said it's not necessarily easy, but the QA team and RM are supposed to know if the code is clean regarding Koha guidelines. If the code is not clean, don't PQA, don't push. Either clean yourself, or ask the original author of the patch to do it (explaining to them how it can be improved ofc). > > 3. Squash! I have been away for a couple of months and had to read the git history to know what I missed. And it was really hard to follow what was going on. First of all, we are not consistent: the commit message must tell what the patch is doing, not what the bug was (if you are writing a bug fix). Then, there are way too many follow-ups: > tidiness, indentation fix, typo, spelling, etc. All those tiny follow-ups could be squashed into the original patch. We don't need unnecessary tons of entries in our git log for that. For instance, I usually add a "JD Amended patch: perltidy" for instance when I tidy the original patch, to keep track of the modification. Squash can be done by the original author, the QAer, the RM. So yes, you are losing one commit in the stats but the git log is easy to read! > We could have an "Amended-by" marker if we really want to add credit on the dashboard (and/or release notes). > > 4. Run tests. Don't wait for Jenkins to fail. This is valid for the author and QA. Anticipate the failures by running more tests. If you are modifying C4::Circulation, then run prove on t/db_dependent/Circulation*, not only Circulation.t. It will help you catch edge cases. > When something is pushed, track down jenkins failures that could be caused by your patches. > > 6. Be strict if you are QAing. Each QA member has their own "good enough", and the RM as well (either relying on the QAer or providing a full review). But QA must fail if the code is old Koha style code, or not "good enough". > > 7. Provide support for failing tests, fix things you broke. The QA team will be more comfortable with your patches if you show them you are providing support for your stuff. > It's not because it's pushed that you don't have any more efforts to make. Provide follow-up patches you promised, provide bug fixes, etc. > We don't have a good way to keep track of such demands, which does not make tracking easier for devs, QA and RM. Any suggestions? > > 8. QA team MUST NEVER* pass QA a change that is not covered by tests, never. You should not provide change to modules without tests! > * almost never... > > 9. Stick to existing patterns. We should not have different ways to do the same thing. We should not have different places where a code is doing the same thing. Ask for help or advice on the list or IRC before you start coding. We will be happy to guide you. Even if you are a regular Koha developer it's not always easy to be aware of the latest master changes. > We will tell you what's the current good practice, or point you to examples you could reuse for what you want to implement. > > 10. CI should drive the pushes. No more push if CI is not green. The more we wait the harder it is to track down the origin of the problem. > Last cycle some jobs have been red for months, and we released > 22.11.00 with D10, D11, D12 marked unstable... > > What will I do next cycle? > All of that, and more. I will track down jenkins failures and responsibilize developers telling them when they break tests (and won't fix them anymore as I have been doing for years). > I will raise on the bug reports what could have been improved. Yes, read that I will be even more annoying (to put it politely) than before. > > I've noticed that the pre-commit git hook on the wiki has been broken for more than 3 years. And also caught some core developers that do not have it in place. I am relying on it to keep Vue files tidy so it's important to have it set up properly. I am planning to force its usage for ktd users [2]. Adding more checks to it will help us to catch inconsistencies from the beginning. > > To summarize, writing code is cheap, maintaining code is way more expensive! It is easier to get the attention of developers before the patches are pushed to master than after, so we could be more ambitious and ask more. > > For discussion :) > > Cheers, > Jonathan > > [1] If you are using vim, open ~/vimrc, add > vmap :!perltidy -q > Reload vim, select code in visual mode [2] > https://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgitl > ab.com%2Fkoha-community%2Fkoha-misc4dev%2F-%2Fissues%2F59&data=05% > 7C01%7Cm.de.rooy%40rijksmuseum.nl%7C5d76e33c09514528c67808dad6c87f27%7 > C635b05eb66c748e1a94fb4b05a1b058b%7C0%7C0%7C638058453872270550%7CUnkno > wn%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiL > CJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=AkLM4JATxwG4L8CcLpZ7z34oM6GCEN > 49uAo3dt4Sw18%3D&reserved=0 > _______________________________________________ > Koha-devel mailing list > Koha-devel at lists.koha-community.org > https://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Flist > s.koha-community.org%2Fcgi-bin%2Fmailman%2Flistinfo%2Fkoha-devel&d > ata=05%7C01%7Cm.de.rooy%40rijksmuseum.nl%7C5d76e33c09514528c67808dad6c > 87f27%7C635b05eb66c748e1a94fb4b05a1b058b%7C0%7C0%7C638058453872270550% > 7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik > 1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=aqN6n2Z2kvsodCb1MlLsNiA > AomgcvKr6fB7tCNNdBrg%3D&reserved=0 > website : > https://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww. > koha-community.org%2F&data=05%7C01%7Cm.de.rooy%40rijksmuseum.nl%7C > 5d76e33c09514528c67808dad6c87f27%7C635b05eb66c748e1a94fb4b05a1b058b%7C > 0%7C0%7C638058453872270550%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDA > iLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sd > ata=3vEx8LIuZAkid78L%2FF15H58ECkQQu4m4PO7R4Tnbot0%3D&reserved=0 > git : > https://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgit. > koha-community.org%2F&data=05%7C01%7Cm.de.rooy%40rijksmuseum.nl%7C > 5d76e33c09514528c67808dad6c87f27%7C635b05eb66c748e1a94fb4b05a1b058b%7C > 0%7C0%7C638058453872270550%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDA > iLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sd > ata=5bMenLRZpyj4HYRx%2FExY83Mwe%2F8bpaLvlJvG3jLcBCg%3D&reserved=0 > bugs : > https://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fbugs > .koha-community.org%2F&data=05%7C01%7Cm.de.rooy%40rijksmuseum.nl%7 > C5d76e33c09514528c67808dad6c87f27%7C635b05eb66c748e1a94fb4b05a1b058b%7 > C0%7C0%7C638058453872270550%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMD > AiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&s > data=iT5Hl4dNO8NQXuQZmJmkdLr1pXv0k3TBGK8NceA82iE%3D&reserved=0 _______________________________________________ Koha-devel mailing list Koha-devel at lists.koha-community.org https://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Flists.koha-community.org%2Fcgi-bin%2Fmailman%2Flistinfo%2Fkoha-devel&data=05%7C01%7Cm.de.rooy%40rijksmuseum.nl%7C5d76e33c09514528c67808dad6c87f27%7C635b05eb66c748e1a94fb4b05a1b058b%7C0%7C0%7C638058453872270550%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=aqN6n2Z2kvsodCb1MlLsNiAAomgcvKr6fB7tCNNdBrg%3D&reserved=0 website : https://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.koha-community.org%2F&data=05%7C01%7Cm.de.rooy%40rijksmuseum.nl%7C5d76e33c09514528c67808dad6c87f27%7C635b05eb66c748e1a94fb4b05a1b058b%7C0%7C0%7C638058453872270550%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=3vEx8LIuZAkid78L%2FF15H58ECkQQu4m4PO7R4Tnbot0%3D&reserved=0 git : https://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgit.koha-community.org%2F&data=05%7C01%7Cm.de.rooy%40rijksmuseum.nl%7C5d76e33c09514528c67808dad6c87f27%7C635b05eb66c748e1a94fb4b05a1b058b%7C0%7C0%7C638058453872270550%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=5bMenLRZpyj4HYRx%2FExY83Mwe%2F8bpaLvlJvG3jLcBCg%3D&reserved=0 bugs : https://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fbugs.koha-community.org%2F&data=05%7C01%7Cm.de.rooy%40rijksmuseum.nl%7C5d76e33c09514528c67808dad6c87f27%7C635b05eb66c748e1a94fb4b05a1b058b%7C0%7C0%7C638058453872270550%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=iT5Hl4dNO8NQXuQZmJmkdLr1pXv0k3TBGK8NceA82iE%3D&reserved=0 From paul.poulain at biblibre.com Mon Dec 5 15:35:02 2022 From: paul.poulain at biblibre.com (Paul Poulain) Date: Mon, 5 Dec 2022 15:35:02 +0100 Subject: [Koha-devel] Good enough? In-Reply-To: References: <85c5041c-a3ee-db1c-b576-97e789193c14@biblibre.com> Message-ID: From what Julian write, I don't think he's suggesting running jenkins every type a patch is attached, but more something where a developper could send a message "hey, bot, can you check if this patch passes all tests ?". that would be manual (and I have no idea how to do that ;) My 1 cent (not worth 2 ;) ) Le 05/12/2022 Ă  14:55, Jonathan Druart a Ă©crit : > I don't think we should run the whole test suite everytime we attach > patches, that would be very expensive in terms of resources. > However it would be interesting to have a temporary 'master' branch > that would become 'master' only if jenkins is happy. > "master" would never be broken :D > > Using koha-testing-docker it's very easy to run tests locally. > It will be even more easier with > https://gitlab.com/koha-community/koha-misc4dev/-/issues/58 > > Le ven. 2 dĂ©c. 2022 Ă  18:16, Julian Maurice > a Ă©crit : >> Hi, >> >> It seems to me that several issues could be solved by having the CI run >> sooner, so authors can have feedback as soon as possible. Something like >> github's CI integration with pull requests would be amazing to detect >> common mistakes early and stop wasting human time. We should know if >> tests pass before pushing to master, not after. >> We could check tidyness, commit message conventions, code coverage by >> tests, ... all before another person have to look at it. >> >> Also tests are not easy to run locally. They might pass on Jenkins, but >> they do not on my local setup, so they are basically useless to tell me >> if I broke something. It also makes writing new tests more difficult >> than it should. >> >> If I wanted to improve Koha developer experience, I would start with that. >> >> My 2 cents. >> >> Le 02/12/2022 Ă  15:42, Jonathan Druart a Ă©crit : >>> Hi devs, >>> >>> I was wondering... How good is your "good enough"? >>> It's a question I often ask myself, when writing patches or QAing >>> yours: is it good enough to be into Koha? It does not have to be >>> perfect or it may never reach master, but it must meet certain >>> standards. >>> >>> When I was RM I tried to put that limit quite high. Not necessarily by >>> asking the author for improving the follow-up patches, but also by >>> adding the missing bits myself. >>> >>> There are various types of "good enough", depending on what we look >>> at: good enough to be reviewed, good enough to be tested, to be put in >>> production, get feedback, try an idea, etc. >>> >>> Most of the time, what is driving the limit is answering the >>> questions "Is it maintainable?" / "Is it future proof?". Making sure >>> the code you are writing won't be broken inadvertently is very >>> important and must be part of the reflection. >>> >>> Katrin asked the QA team members what were our plans for 23.05. In my >>> opinion we should enforce this "be future proof" concept. Writing code >>> is easy, especially in Koha (yes it is!). Writing maintainable and >>> robust code, following our guidelines is a bit harder. That's why we >>> have a QA process that tries to catch inconsistencies or edge cases >>> you may have missed. >>> But I think we can be even more rigorous, and aim for better >>> standards. We can elevate our ambitions and do better. When we see the >>> number of new enhancements we are releasing every 6 months, it shows >>> well that writing code is definitely not a problem. However sometimes >>> developers are tempted to abandon their work once they are pushed to >>> master. Pushed does not mean "done". Providing bug fixes, following-up >>> with patches when needed, fixing CI jenkins, etc. is part of job >>> (/fun) >>> >>> As a Koha developer for a long time now, I know how frustrating it can >>> be to be asked for follow-ups/rewrite/tests to have our patches >>> stamped with the precious PQA mark. But from the other point of view >>> (RM, RMaints, QA team), I also know it's very frustrating when you are >>> in charge of the release and you do not get the appropriate follow-up >>> work once it's pushed to master. >>> >>> There are some easy steps to write/review better patches. All have >>> been discussed already several times, but that can be enforced even >>> more: >>> 1. Perltidy (!) This is really a very trivial step. Please perltidy >>> your code. There are hundreds of commits that have been pushed in the >>> last months that are not tidy (alignment, indentation, lines too long, >>> etc.) This can easily be configured in your IDE! [1] >>> >>> 2. Provide clean code. As said it's not necessarily easy, but the QA >>> team and RM are supposed to know if the code is clean regarding Koha >>> guidelines. If the code is not clean, don't PQA, don't push. Either >>> clean yourself, or ask the original author of the patch to do it >>> (explaining to them how it can be improved ofc). >>> >>> 3. Squash! I have been away for a couple of months and had to read the >>> git history to know what I missed. And it was really hard to follow >>> what was going on. First of all, we are not consistent: the commit >>> message must tell what the patch is doing, not what the bug was (if >>> you are writing a bug fix). Then, there are way too many follow-ups: >>> tidiness, indentation fix, typo, spelling, etc. All those tiny >>> follow-ups could be squashed into the original patch. We don't need >>> unnecessary tons of entries in our git log for that. For instance, I >>> usually add a "JD Amended patch: perltidy" for instance when I tidy >>> the original patch, to keep track of the modification. Squash can be >>> done by the original author, the QAer, the RM. So yes, you are losing >>> one commit in the stats but the git log is easy to read! >>> We could have an "Amended-by" marker if we really want to add credit >>> on the dashboard (and/or release notes). >>> >>> 4. Run tests. Don't wait for Jenkins to fail. This is valid for the >>> author and QA. Anticipate the failures by running more tests. If you >>> are modifying C4::Circulation, then run prove on >>> t/db_dependent/Circulation*, not only Circulation.t. It will help you >>> catch edge cases. >>> When something is pushed, track down jenkins failures that could be >>> caused by your patches. >>> >>> 6. Be strict if you are QAing. Each QA member has their own "good >>> enough", and the RM as well (either relying on the QAer or providing a >>> full review). But QA must fail if the code is old Koha style code, or >>> not "good enough". >>> >>> 7. Provide support for failing tests, fix things you broke. The QA >>> team will be more comfortable with your patches if you show them you >>> are providing support for your stuff. >>> It's not because it's pushed that you don't have any more efforts to >>> make. Provide follow-up patches you promised, provide bug fixes, etc. >>> We don't have a good way to keep track of such demands, which does not >>> make tracking easier for devs, QA and RM. Any suggestions? >>> >>> 8. QA team MUST NEVER* pass QA a change that is not covered by tests, >>> never. You should not provide change to modules without tests! >>> * almost never... >>> >>> 9. Stick to existing patterns. We should not have different ways to do >>> the same thing. We should not have different places where a code is >>> doing the same thing. Ask for help or advice on the list or IRC before >>> you start coding. We will be happy to guide you. Even if you are a >>> regular Koha developer it's not always easy to be aware of the latest >>> master changes. >>> We will tell you what's the current good practice, or point you to >>> examples you could reuse for what you want to implement. >>> >>> 10. CI should drive the pushes. No more push if CI is not green. The >>> more we wait the harder it is to track down the origin of the problem. >>> Last cycle some jobs have been red for months, and we released >>> 22.11.00 with D10, D11, D12 marked unstable... >>> >>> What will I do next cycle? >>> All of that, and more. I will track down jenkins failures and >>> responsibilize developers telling them when they break tests (and >>> won't fix them anymore as I have been doing for years). >>> I will raise on the bug reports what could have been improved. Yes, >>> read that I will be even more annoying (to put it politely) than >>> before. >>> >>> I've noticed that the pre-commit git hook on the wiki has been broken >>> for more than 3 years. And also caught some core developers that do >>> not have it in place. I am relying on it to keep Vue files tidy so >>> it's important to have it set up properly. I am planning to force its >>> usage for ktd users [2]. Adding more checks to it will help us to >>> catch inconsistencies from the beginning. >>> >>> To summarize, writing code is cheap, maintaining code is way more >>> expensive! It is easier to get the attention of developers before the >>> patches are pushed to master than after, so we could be more ambitious >>> and ask more. >>> >>> For discussion :) >>> >>> Cheers, >>> Jonathan >>> >>> [1] If you are using vim, open ~/vimrc, add >>> vmap :!perltidy -q >>> Reload vim, select code in visual mode >>> [2] https://gitlab.com/koha-community/koha-misc4dev/-/issues/59 >>> _______________________________________________ >>> Koha-devel mailing list >>> Koha-devel at lists.koha-community.org >>> https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-devel >>> website : https://www.koha-community.org/ >>> git : https://git.koha-community.org/ >>> bugs : https://bugs.koha-community.org/ >> -- >> Julian Maurice >> BibLibre >> _______________________________________________ >> Koha-devel mailing list >> Koha-devel at lists.koha-community.org >> https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-devel >> website : https://www.koha-community.org/ >> git : https://git.koha-community.org/ >> bugs : https://bugs.koha-community.org/ > _______________________________________________ > Koha-devel mailing list > Koha-devel at lists.koha-community.org > https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-devel > website : https://www.koha-community.org/ > git : https://git.koha-community.org/ > bugs : https://bugs.koha-community.org/ -- Paul Poulain, AssociĂ©-gĂ©rant / co-owner BibLibre, Services en logiciels libres pour les bibliothĂšques BibLibre, Open Source software and services for libraries From tomascohen at gmail.com Mon Dec 5 16:27:26 2022 From: tomascohen at gmail.com (Tomas Cohen Arazi) Date: Mon, 5 Dec 2022 12:27:26 -0300 Subject: [Koha-devel] Good enough? In-Reply-To: References: Message-ID: El vie, 2 dic 2022 a las 11:43, Jonathan Druart (< jonathan.druart at bugs.koha-community.org>) escribiĂł: > 10. CI should drive the pushes. No more push if CI is not green. The > more we wait the harder it is to track down the origin of the problem. > Last cycle some jobs have been red for months, and we released > 22.11.00 with D10, D11, D12 marked unstable... > That assertion is not that precise. We've had lots of failures during the cycle which were addressed sooner or later. The only long standing one was a selenium test failure we couldn't reproduce locally, and was more about the test suite setup than a Koh failure. And we only released once things were green :-D I will answer about the rest of the email later. But I will mention that everyone can: $ ktd --selenium --es7 --sso up -d and be able to test everything locally [1]. And I'm available to chat or troubleshoot that. [1] With the exception of macOS + ARM users. -- TomĂĄs Cohen Arazi Theke Solutions (http://theke.io) ✆ +54 9351 3513384 GPG: B2F3C15F -------------- next part -------------- An HTML attachment was scrubbed... URL: From Emily.Lamancusa at montgomerycountymd.gov Mon Dec 5 17:06:29 2022 From: Emily.Lamancusa at montgomerycountymd.gov (Lamancusa, Emily) Date: Mon, 5 Dec 2022 16:06:29 +0000 Subject: [Koha-devel] Setting up Koha Testing Docker Message-ID: Hello all, I'm trying to set up a Koha Testing Docker on my local machine (running Pop!_OS 22.04). I haven't used Docker before, and I have a few questions about how to get it set up. 1. It looks like Linux users have the option to install the whole Docker Desktop VM, or install just Docker Engine, Docker CLI, and the Compose plugin. Do I need Docker Desktop to run KTD, or will the latter installation option work on its own? (Aside from not wanting to install a bunch of extra stuff that I don't need, I would also need to request that my library system purchase a license for Docker Desktop, so I want to check if that is actually necessary.) 2. Is the idea to do all of the code editing, git commands, etc, from inside the container shell? Or would all of that be done outside the container, and the container be used only to run code/tests? 3. From what I've read in the wiki/getting started resources, I need Docker, the KTD itself, git-bz, and the git hooks from https://wiki.koha-community.org/wiki/Tips_and_tricks#git_tips_and_tricks (other than the pre-commit hook that is currently broken?) And then there's a tutorial on creating and submitting patches here (https://gitlab.com/koha-community/koha-howto/blob/master/README.md) that I can run once I have the KTD working. Am I missing anything? Hopefully this will make more sense once I have everything installed and can actually start playing with it, but I'll admit I'm feeling a little overwhelmed right now. Thank you so much to anyone who can give this newbie a little extra guidance on how to get started! Emily Lamancusa (she/her) IT Specialist III Montgomery County Public Libraries 21 Maryland Ave, Suite 310 Rockville, MD 20850 (240) 777-0676 [https://www.montgomerycountymd.gov/mcg/Resources/Images/Cybersecurity-footer.png] For more helpful Cybersecurity Resources, visit: https://www.montgomerycountymd.gov/cybersecurity -------------- next part -------------- An HTML attachment was scrubbed... URL: From jonathan.druart at bugs.koha-community.org Mon Dec 5 17:08:20 2022 From: jonathan.druart at bugs.koha-community.org (Jonathan Druart) Date: Mon, 5 Dec 2022 17:08:20 +0100 Subject: [Koha-devel] Good enough? In-Reply-To: References: Message-ID: As said in the email I am planning to integrate them into our dev env (via misc4dev), see https://gitlab.com/koha-community/koha-misc4dev/-/issues/59 Le lun. 5 dĂ©c. 2022 Ă  15:17, Marcel de Rooy a Ă©crit : > > Thanks for sending this link. > Could we merge that code with QA code we already pushed into our codebase somehow ? > > -----Original Message----- > From: Koha-devel On Behalf Of Jonathan Druart > Sent: Monday, December 5, 2022 2:56 PM > To: koha-devel > Subject: Re: [Koha-devel] Good enough? > > Existing hooks can be found at > https://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwiki.koha-community.org%2Fwiki%2FTips_and_tricks%23Hooks&data=05%7C01%7Cm.de.rooy%40rijksmuseum.nl%7C5d76e33c09514528c67808dad6c87f27%7C635b05eb66c748e1a94fb4b05a1b058b%7C0%7C0%7C638058453872270550%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=bxdii78avAYEIcecc8o0tQZ4v7atwVTkmbJMYPe1Dek%3D&reserved=0 > > Le lun. 5 dĂ©c. 2022 Ă  11:34, Marcel de Rooy a Ă©crit : > > > > Could you provide more info on those git hooks? A wiki URL? > > > > Marcel > > > > -----Original Message----- > > From: Koha-devel On > > Behalf Of Jonathan Druart > > Sent: Friday, December 2, 2022 3:43 PM > > To: koha-devel > > Subject: [Koha-devel] Good enough? > > > > Hi devs, > > > > I was wondering... How good is your "good enough"? > > It's a question I often ask myself, when writing patches or QAing > > yours: is it good enough to be into Koha? It does not have to be perfect or it may never reach master, but it must meet certain standards. > > > > When I was RM I tried to put that limit quite high. Not necessarily by asking the author for improving the follow-up patches, but also by adding the missing bits myself. > > > > There are various types of "good enough", depending on what we look > > at: good enough to be reviewed, good enough to be tested, to be put in production, get feedback, try an idea, etc. > > > > Most of the time, what is driving the limit is answering the questions "Is it maintainable?" / "Is it future proof?". Making sure the code you are writing won't be broken inadvertently is very important and must be part of the reflection. > > > > Katrin asked the QA team members what were our plans for 23.05. In my opinion we should enforce this "be future proof" concept. Writing code is easy, especially in Koha (yes it is!). Writing maintainable and robust code, following our guidelines is a bit harder. That's why we have a QA process that tries to catch inconsistencies or edge cases you may have missed. > > But I think we can be even more rigorous, and aim for better > > standards. We can elevate our ambitions and do better. When we see the > > number of new enhancements we are releasing every 6 months, it shows > > well that writing code is definitely not a problem. However sometimes > > developers are tempted to abandon their work once they are pushed to > > master. Pushed does not mean "done". Providing bug fixes, following-up > > with patches when needed, fixing CI jenkins, etc. is part of job > > (/fun) > > > > As a Koha developer for a long time now, I know how frustrating it can be to be asked for follow-ups/rewrite/tests to have our patches stamped with the precious PQA mark. But from the other point of view (RM, RMaints, QA team), I also know it's very frustrating when you are in charge of the release and you do not get the appropriate follow-up work once it's pushed to master. > > > > There are some easy steps to write/review better patches. All have > > been discussed already several times, but that can be enforced even > > more: > > 1. Perltidy (!) This is really a very trivial step. Please perltidy > > your code. There are hundreds of commits that have been pushed in the > > last months that are not tidy (alignment, indentation, lines too long, > > etc.) This can easily be configured in your IDE! [1] > > > > 2. Provide clean code. As said it's not necessarily easy, but the QA team and RM are supposed to know if the code is clean regarding Koha guidelines. If the code is not clean, don't PQA, don't push. Either clean yourself, or ask the original author of the patch to do it (explaining to them how it can be improved ofc). > > > > 3. Squash! I have been away for a couple of months and had to read the git history to know what I missed. And it was really hard to follow what was going on. First of all, we are not consistent: the commit message must tell what the patch is doing, not what the bug was (if you are writing a bug fix). Then, there are way too many follow-ups: > > tidiness, indentation fix, typo, spelling, etc. All those tiny follow-ups could be squashed into the original patch. We don't need unnecessary tons of entries in our git log for that. For instance, I usually add a "JD Amended patch: perltidy" for instance when I tidy the original patch, to keep track of the modification. Squash can be done by the original author, the QAer, the RM. So yes, you are losing one commit in the stats but the git log is easy to read! > > We could have an "Amended-by" marker if we really want to add credit on the dashboard (and/or release notes). > > > > 4. Run tests. Don't wait for Jenkins to fail. This is valid for the author and QA. Anticipate the failures by running more tests. If you are modifying C4::Circulation, then run prove on t/db_dependent/Circulation*, not only Circulation.t. It will help you catch edge cases. > > When something is pushed, track down jenkins failures that could be caused by your patches. > > > > 6. Be strict if you are QAing. Each QA member has their own "good enough", and the RM as well (either relying on the QAer or providing a full review). But QA must fail if the code is old Koha style code, or not "good enough". > > > > 7. Provide support for failing tests, fix things you broke. The QA team will be more comfortable with your patches if you show them you are providing support for your stuff. > > It's not because it's pushed that you don't have any more efforts to make. Provide follow-up patches you promised, provide bug fixes, etc. > > We don't have a good way to keep track of such demands, which does not make tracking easier for devs, QA and RM. Any suggestions? > > > > 8. QA team MUST NEVER* pass QA a change that is not covered by tests, never. You should not provide change to modules without tests! > > * almost never... > > > > 9. Stick to existing patterns. We should not have different ways to do the same thing. We should not have different places where a code is doing the same thing. Ask for help or advice on the list or IRC before you start coding. We will be happy to guide you. Even if you are a regular Koha developer it's not always easy to be aware of the latest master changes. > > We will tell you what's the current good practice, or point you to examples you could reuse for what you want to implement. > > > > 10. CI should drive the pushes. No more push if CI is not green. The more we wait the harder it is to track down the origin of the problem. > > Last cycle some jobs have been red for months, and we released > > 22.11.00 with D10, D11, D12 marked unstable... > > > > What will I do next cycle? > > All of that, and more. I will track down jenkins failures and responsibilize developers telling them when they break tests (and won't fix them anymore as I have been doing for years). > > I will raise on the bug reports what could have been improved. Yes, read that I will be even more annoying (to put it politely) than before. > > > > I've noticed that the pre-commit git hook on the wiki has been broken for more than 3 years. And also caught some core developers that do not have it in place. I am relying on it to keep Vue files tidy so it's important to have it set up properly. I am planning to force its usage for ktd users [2]. Adding more checks to it will help us to catch inconsistencies from the beginning. > > > > To summarize, writing code is cheap, maintaining code is way more expensive! It is easier to get the attention of developers before the patches are pushed to master than after, so we could be more ambitious and ask more. > > > > For discussion :) > > > > Cheers, > > Jonathan > > > > [1] If you are using vim, open ~/vimrc, add > > vmap :!perltidy -q > > Reload vim, select code in visual mode [2] > > https://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgitl > > ab.com%2Fkoha-community%2Fkoha-misc4dev%2F-%2Fissues%2F59&data=05% > > 7C01%7Cm.de.rooy%40rijksmuseum.nl%7C5d76e33c09514528c67808dad6c87f27%7 > > C635b05eb66c748e1a94fb4b05a1b058b%7C0%7C0%7C638058453872270550%7CUnkno > > wn%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiL > > CJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=AkLM4JATxwG4L8CcLpZ7z34oM6GCEN > > 49uAo3dt4Sw18%3D&reserved=0 > > _______________________________________________ > > Koha-devel mailing list > > Koha-devel at lists.koha-community.org > > https://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Flist > > s.koha-community.org%2Fcgi-bin%2Fmailman%2Flistinfo%2Fkoha-devel&d > > ata=05%7C01%7Cm.de.rooy%40rijksmuseum.nl%7C5d76e33c09514528c67808dad6c > > 87f27%7C635b05eb66c748e1a94fb4b05a1b058b%7C0%7C0%7C638058453872270550% > > 7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik > > 1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=aqN6n2Z2kvsodCb1MlLsNiA > > AomgcvKr6fB7tCNNdBrg%3D&reserved=0 > > website : > > https://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww. > > koha-community.org%2F&data=05%7C01%7Cm.de.rooy%40rijksmuseum.nl%7C > > 5d76e33c09514528c67808dad6c87f27%7C635b05eb66c748e1a94fb4b05a1b058b%7C > > 0%7C0%7C638058453872270550%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDA > > iLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sd > > ata=3vEx8LIuZAkid78L%2FF15H58ECkQQu4m4PO7R4Tnbot0%3D&reserved=0 > > git : > > https://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgit. > > koha-community.org%2F&data=05%7C01%7Cm.de.rooy%40rijksmuseum.nl%7C > > 5d76e33c09514528c67808dad6c87f27%7C635b05eb66c748e1a94fb4b05a1b058b%7C > > 0%7C0%7C638058453872270550%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDA > > iLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sd > > ata=5bMenLRZpyj4HYRx%2FExY83Mwe%2F8bpaLvlJvG3jLcBCg%3D&reserved=0 > > bugs : > > https://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fbugs > > .koha-community.org%2F&data=05%7C01%7Cm.de.rooy%40rijksmuseum.nl%7 > > C5d76e33c09514528c67808dad6c87f27%7C635b05eb66c748e1a94fb4b05a1b058b%7 > > C0%7C0%7C638058453872270550%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMD > > AiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&s > > data=iT5Hl4dNO8NQXuQZmJmkdLr1pXv0k3TBGK8NceA82iE%3D&reserved=0 > _______________________________________________ > Koha-devel mailing list > Koha-devel at lists.koha-community.org > https://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Flists.koha-community.org%2Fcgi-bin%2Fmailman%2Flistinfo%2Fkoha-devel&data=05%7C01%7Cm.de.rooy%40rijksmuseum.nl%7C5d76e33c09514528c67808dad6c87f27%7C635b05eb66c748e1a94fb4b05a1b058b%7C0%7C0%7C638058453872270550%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=aqN6n2Z2kvsodCb1MlLsNiAAomgcvKr6fB7tCNNdBrg%3D&reserved=0 > website : https://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.koha-community.org%2F&data=05%7C01%7Cm.de.rooy%40rijksmuseum.nl%7C5d76e33c09514528c67808dad6c87f27%7C635b05eb66c748e1a94fb4b05a1b058b%7C0%7C0%7C638058453872270550%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=3vEx8LIuZAkid78L%2FF15H58ECkQQu4m4PO7R4Tnbot0%3D&reserved=0 > git : https://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgit.koha-community.org%2F&data=05%7C01%7Cm.de.rooy%40rijksmuseum.nl%7C5d76e33c09514528c67808dad6c87f27%7C635b05eb66c748e1a94fb4b05a1b058b%7C0%7C0%7C638058453872270550%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=5bMenLRZpyj4HYRx%2FExY83Mwe%2F8bpaLvlJvG3jLcBCg%3D&reserved=0 > bugs : https://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fbugs.koha-community.org%2F&data=05%7C01%7Cm.de.rooy%40rijksmuseum.nl%7C5d76e33c09514528c67808dad6c87f27%7C635b05eb66c748e1a94fb4b05a1b058b%7C0%7C0%7C638058453872270550%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=iT5Hl4dNO8NQXuQZmJmkdLr1pXv0k3TBGK8NceA82iE%3D&reserved=0 > _______________________________________________ > Koha-devel mailing list > Koha-devel at lists.koha-community.org > https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-devel > website : https://www.koha-community.org/ > git : https://git.koha-community.org/ > bugs : https://bugs.koha-community.org/ From nick at bywatersolutions.com Mon Dec 5 17:22:48 2022 From: nick at bywatersolutions.com (Nick Clemens) Date: Mon, 5 Dec 2022 11:22:48 -0500 Subject: [Koha-devel] Setting up Koha Testing Docker In-Reply-To: References: Message-ID: 1 - On Linux you do not need Docker Desktop - I have docker.io and docker-compose - I set this up long ago, so don't remember the steps, but you should not require a licence 2 - That is up to you. For editing code I think most developers use an IDE or something outside the docker container - I personally tend to work from within the container, but it is not necessary. Tests etc should be run in the container 3 - If you work inside the container then git-bz and all should be installed - it looks like work is being done to get the hools automatically installed (https://gitlab.com/koha-community/koha-misc4dev/-/issues/59) - if you work outside the container, then yes, installing git bz and pre-commit hook should get you well started On Mon, Dec 5, 2022 at 11:06 AM Lamancusa, Emily < Emily.Lamancusa at montgomerycountymd.gov> wrote: > Hello all, > > I'm trying to set up a Koha Testing Docker on my local machine (running > Pop!_OS 22.04). I haven't used Docker before, and I have a few questions > about how to get it set up. > > 1. It looks like Linux users have the option to install the whole Docker > Desktop VM, or install just Docker Engine, Docker CLI, and the Compose > plugin. Do I need Docker Desktop to run KTD, or will the latter > installation option work on its own? (Aside from not wanting to install a > bunch of extra stuff that I don't need, I would also need to request that > my library system purchase a license for Docker Desktop, so I want to check > if that is actually necessary.) > > 2. Is the idea to do all of the code editing, git commands, etc, from > inside the container shell? Or would all of that be done outside the > container, and the container be used only to run code/tests? > > 3. From what I've read in the wiki/getting started resources, I need > Docker, the KTD itself, git-bz, and the git hooks from > https://wiki.koha-community.org/wiki/Tips_and_tricks#git_tips_and_tricks (other > than the pre-commit hook that is currently broken?) And then there's a > tutorial on creating and submitting patches here ( > https://gitlab.com/koha-community/koha-howto/blob/master/README.md) that > I can run once I have the KTD working. Am I missing anything? Hopefully > this will make more sense once I have everything installed and can actually > start playing with it, but I'll admit I'm feeling a little overwhelmed > right now. > > Thank you so much to anyone who can give this newbie a little extra > guidance on how to get started! > > Emily Lamancusa *(she/her)* > IT Specialist III > Montgomery County Public Libraries > 21 Maryland Ave, Suite 310 > Rockville, MD 20850 > (240) 777-0676 > > > > *For more helpful Cybersecurity Resources, visit: > https://www.montgomerycountymd.gov/cybersecurity > * > _______________________________________________ > Koha-devel mailing list > Koha-devel at lists.koha-community.org > https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-devel > website : https://www.koha-community.org/ > git : https://git.koha-community.org/ > bugs : https://bugs.koha-community.org/ > -- Nick Clemens ByWater Solutions bywatersolutions.com Phone: (888) 900-8944 Pronouns: (he/him/his) Timezone: Eastern Follow us: -------------- next part -------------- An HTML attachment was scrubbed... URL: From pedro.amorim at ptfs-europe.com Mon Dec 5 17:31:02 2022 From: pedro.amorim at ptfs-europe.com (Amorim, Pedro) Date: Mon, 5 Dec 2022 15:31:02 -0100 Subject: [Koha-devel] Setting up Koha Testing Docker In-Reply-To: References: Message-ID: Hi Emily, Nick already laid it all out pretty well but just to add to it: My suggestion would be to follow the instructions at koha-testing-docker https://gitlab.com/koha-community/koha-testing-docker which also include docker instructions links. You will end up with 2 directories, one for k-t-d and one for the Koha code you're working with. This Koha code directory is mapped as a volume to the container when launched, so whatever changes to the code you make in this directory will reflect inside the container in real time (and vice-versa). Another good place to ask questions and find valuable help is the community irc https://koha-community.org/get-involved/irc/ Pedro Amorim Software Engineer E: pedro.amorim at ptfs-europe.com www.ptfs-europe.com On Mon, 5 Dec 2022 at 15:23, Nick Clemens wrote: > 1 - On Linux you do not need Docker Desktop - I have docker.io and > docker-compose - I set this up long ago, so don't remember the steps, but > you should not require a licence > 2 - That is up to you. For editing code I think most developers use an IDE > or something outside the docker container - I personally tend to work from > within the container, but it is not necessary. Tests etc should be run in > the container > 3 - If you work inside the container then git-bz and all should be > installed - it looks like work is being done to get the hools automatically > installed (https://gitlab.com/koha-community/koha-misc4dev/-/issues/59) - > if you work outside the container, then yes, installing git bz and > pre-commit hook should get you well started > > On Mon, Dec 5, 2022 at 11:06 AM Lamancusa, Emily < > Emily.Lamancusa at montgomerycountymd.gov> wrote: > >> Hello all, >> >> I'm trying to set up a Koha Testing Docker on my local machine (running >> Pop!_OS 22.04). I haven't used Docker before, and I have a few questions >> about how to get it set up. >> >> 1. It looks like Linux users have the option to install the whole Docker >> Desktop VM, or install just Docker Engine, Docker CLI, and the Compose >> plugin. Do I need Docker Desktop to run KTD, or will the latter >> installation option work on its own? (Aside from not wanting to install a >> bunch of extra stuff that I don't need, I would also need to request that >> my library system purchase a license for Docker Desktop, so I want to check >> if that is actually necessary.) >> >> 2. Is the idea to do all of the code editing, git commands, etc, from >> inside the container shell? Or would all of that be done outside the >> container, and the container be used only to run code/tests? >> >> 3. From what I've read in the wiki/getting started resources, I need >> Docker, the KTD itself, git-bz, and the git hooks from >> https://wiki.koha-community.org/wiki/Tips_and_tricks#git_tips_and_tricks (other >> than the pre-commit hook that is currently broken?) And then there's a >> tutorial on creating and submitting patches here ( >> https://gitlab.com/koha-community/koha-howto/blob/master/README.md) that >> I can run once I have the KTD working. Am I missing anything? Hopefully >> this will make more sense once I have everything installed and can actually >> start playing with it, but I'll admit I'm feeling a little overwhelmed >> right now. >> >> Thank you so much to anyone who can give this newbie a little extra >> guidance on how to get started! >> >> Emily Lamancusa *(she/her)* >> IT Specialist III >> Montgomery County Public Libraries >> 21 Maryland Ave, Suite 310 >> Rockville, MD 20850 >> (240) 777-0676 >> >> >> >> *For more helpful Cybersecurity Resources, visit: >> https://www.montgomerycountymd.gov/cybersecurity >> * >> _______________________________________________ >> Koha-devel mailing list >> Koha-devel at lists.koha-community.org >> https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-devel >> website : https://www.koha-community.org/ >> git : https://git.koha-community.org/ >> bugs : https://bugs.koha-community.org/ >> > > > -- > Nick Clemens > ByWater Solutions > bywatersolutions.com > Phone: (888) 900-8944 > Pronouns: (he/him/his) > Timezone: Eastern > Follow us: > > > > > _______________________________________________ > Koha-devel mailing list > Koha-devel at lists.koha-community.org > https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-devel > website : https://www.koha-community.org/ > git : https://git.koha-community.org/ > bugs : https://bugs.koha-community.org/ > -------------- next part -------------- An HTML attachment was scrubbed... URL: From julian.maurice at biblibre.com Mon Dec 5 18:24:00 2022 From: julian.maurice at biblibre.com (Julian Maurice) Date: Mon, 5 Dec 2022 18:24:00 +0100 Subject: [Koha-devel] Good enough? In-Reply-To: References: <85c5041c-a3ee-db1c-b576-97e789193c14@biblibre.com> Message-ID: Yes it is expensive. But manual testing is much more expensive. I'd rather pay for some CPU time than do manually what a bot can do better and faster than me. Free services exist too (CircleCI, github actions, ...) Pushing to a temporary master branch is not a bad idea (a never-broken master branch sounds nice), but I think it would happen too late. Patches would have already been rebased multiple times, tested and reviewed before we notice a test failure. koha-testing-docker feels more like a symptom of the difficulty to run tests than a satisfying solution. It is probably necessary in order to run complicated end-to-end tests, but it should not be mandatory to run simple unit tests. And ktd is not that easy to use. It can break, and it's not easy to debug for someone not familiar with docker. Don't you think it would be a lot easier if we could run `prove t` (or `npm test` or whatever) on any dev setup and have the same failures/successes as Jenkins (minus the complicated tests like those that require selenium) ? Of course making that happen is not an easy task, but it should be a long term goal IMO. Also I think no one wants to be the person that refuse a patch because a comment is misaligned, so if that kind of thing is not automated, "not good enough code" will continue to be pushed. Le 05/12/2022 Ă  14:55, Jonathan Druart a Ă©crit : > I don't think we should run the whole test suite everytime we attach > patches, that would be very expensive in terms of resources. > However it would be interesting to have a temporary 'master' branch > that would become 'master' only if jenkins is happy. > "master" would never be broken :D > > Using koha-testing-docker it's very easy to run tests locally. > It will be even more easier with > https://gitlab.com/koha-community/koha-misc4dev/-/issues/58 > > Le ven. 2 dĂ©c. 2022 Ă  18:16, Julian Maurice > a Ă©crit : >> >> Hi, >> >> It seems to me that several issues could be solved by having the CI run >> sooner, so authors can have feedback as soon as possible. Something like >> github's CI integration with pull requests would be amazing to detect >> common mistakes early and stop wasting human time. We should know if >> tests pass before pushing to master, not after. >> We could check tidyness, commit message conventions, code coverage by >> tests, ... all before another person have to look at it. >> >> Also tests are not easy to run locally. They might pass on Jenkins, but >> they do not on my local setup, so they are basically useless to tell me >> if I broke something. It also makes writing new tests more difficult >> than it should. >> >> If I wanted to improve Koha developer experience, I would start with that. >> >> My 2 cents. >> >> Le 02/12/2022 Ă  15:42, Jonathan Druart a Ă©crit : >>> Hi devs, >>> >>> I was wondering... How good is your "good enough"? >>> It's a question I often ask myself, when writing patches or QAing >>> yours: is it good enough to be into Koha? It does not have to be >>> perfect or it may never reach master, but it must meet certain >>> standards. >>> >>> When I was RM I tried to put that limit quite high. Not necessarily by >>> asking the author for improving the follow-up patches, but also by >>> adding the missing bits myself. >>> >>> There are various types of "good enough", depending on what we look >>> at: good enough to be reviewed, good enough to be tested, to be put in >>> production, get feedback, try an idea, etc. >>> >>> Most of the time, what is driving the limit is answering the >>> questions "Is it maintainable?" / "Is it future proof?". Making sure >>> the code you are writing won't be broken inadvertently is very >>> important and must be part of the reflection. >>> >>> Katrin asked the QA team members what were our plans for 23.05. In my >>> opinion we should enforce this "be future proof" concept. Writing code >>> is easy, especially in Koha (yes it is!). Writing maintainable and >>> robust code, following our guidelines is a bit harder. That's why we >>> have a QA process that tries to catch inconsistencies or edge cases >>> you may have missed. >>> But I think we can be even more rigorous, and aim for better >>> standards. We can elevate our ambitions and do better. When we see the >>> number of new enhancements we are releasing every 6 months, it shows >>> well that writing code is definitely not a problem. However sometimes >>> developers are tempted to abandon their work once they are pushed to >>> master. Pushed does not mean "done". Providing bug fixes, following-up >>> with patches when needed, fixing CI jenkins, etc. is part of job >>> (/fun) >>> >>> As a Koha developer for a long time now, I know how frustrating it can >>> be to be asked for follow-ups/rewrite/tests to have our patches >>> stamped with the precious PQA mark. But from the other point of view >>> (RM, RMaints, QA team), I also know it's very frustrating when you are >>> in charge of the release and you do not get the appropriate follow-up >>> work once it's pushed to master. >>> >>> There are some easy steps to write/review better patches. All have >>> been discussed already several times, but that can be enforced even >>> more: >>> 1. Perltidy (!) This is really a very trivial step. Please perltidy >>> your code. There are hundreds of commits that have been pushed in the >>> last months that are not tidy (alignment, indentation, lines too long, >>> etc.) This can easily be configured in your IDE! [1] >>> >>> 2. Provide clean code. As said it's not necessarily easy, but the QA >>> team and RM are supposed to know if the code is clean regarding Koha >>> guidelines. If the code is not clean, don't PQA, don't push. Either >>> clean yourself, or ask the original author of the patch to do it >>> (explaining to them how it can be improved ofc). >>> >>> 3. Squash! I have been away for a couple of months and had to read the >>> git history to know what I missed. And it was really hard to follow >>> what was going on. First of all, we are not consistent: the commit >>> message must tell what the patch is doing, not what the bug was (if >>> you are writing a bug fix). Then, there are way too many follow-ups: >>> tidiness, indentation fix, typo, spelling, etc. All those tiny >>> follow-ups could be squashed into the original patch. We don't need >>> unnecessary tons of entries in our git log for that. For instance, I >>> usually add a "JD Amended patch: perltidy" for instance when I tidy >>> the original patch, to keep track of the modification. Squash can be >>> done by the original author, the QAer, the RM. So yes, you are losing >>> one commit in the stats but the git log is easy to read! >>> We could have an "Amended-by" marker if we really want to add credit >>> on the dashboard (and/or release notes). >>> >>> 4. Run tests. Don't wait for Jenkins to fail. This is valid for the >>> author and QA. Anticipate the failures by running more tests. If you >>> are modifying C4::Circulation, then run prove on >>> t/db_dependent/Circulation*, not only Circulation.t. It will help you >>> catch edge cases. >>> When something is pushed, track down jenkins failures that could be >>> caused by your patches. >>> >>> 6. Be strict if you are QAing. Each QA member has their own "good >>> enough", and the RM as well (either relying on the QAer or providing a >>> full review). But QA must fail if the code is old Koha style code, or >>> not "good enough". >>> >>> 7. Provide support for failing tests, fix things you broke. The QA >>> team will be more comfortable with your patches if you show them you >>> are providing support for your stuff. >>> It's not because it's pushed that you don't have any more efforts to >>> make. Provide follow-up patches you promised, provide bug fixes, etc. >>> We don't have a good way to keep track of such demands, which does not >>> make tracking easier for devs, QA and RM. Any suggestions? >>> >>> 8. QA team MUST NEVER* pass QA a change that is not covered by tests, >>> never. You should not provide change to modules without tests! >>> * almost never... >>> >>> 9. Stick to existing patterns. We should not have different ways to do >>> the same thing. We should not have different places where a code is >>> doing the same thing. Ask for help or advice on the list or IRC before >>> you start coding. We will be happy to guide you. Even if you are a >>> regular Koha developer it's not always easy to be aware of the latest >>> master changes. >>> We will tell you what's the current good practice, or point you to >>> examples you could reuse for what you want to implement. >>> >>> 10. CI should drive the pushes. No more push if CI is not green. The >>> more we wait the harder it is to track down the origin of the problem. >>> Last cycle some jobs have been red for months, and we released >>> 22.11.00 with D10, D11, D12 marked unstable... >>> >>> What will I do next cycle? >>> All of that, and more. I will track down jenkins failures and >>> responsibilize developers telling them when they break tests (and >>> won't fix them anymore as I have been doing for years). >>> I will raise on the bug reports what could have been improved. Yes, >>> read that I will be even more annoying (to put it politely) than >>> before. >>> >>> I've noticed that the pre-commit git hook on the wiki has been broken >>> for more than 3 years. And also caught some core developers that do >>> not have it in place. I am relying on it to keep Vue files tidy so >>> it's important to have it set up properly. I am planning to force its >>> usage for ktd users [2]. Adding more checks to it will help us to >>> catch inconsistencies from the beginning. >>> >>> To summarize, writing code is cheap, maintaining code is way more >>> expensive! It is easier to get the attention of developers before the >>> patches are pushed to master than after, so we could be more ambitious >>> and ask more. >>> >>> For discussion :) >>> >>> Cheers, >>> Jonathan >>> >>> [1] If you are using vim, open ~/vimrc, add >>> vmap :!perltidy -q >>> Reload vim, select code in visual mode >>> [2] https://gitlab.com/koha-community/koha-misc4dev/-/issues/59 >>> _______________________________________________ >>> Koha-devel mailing list >>> Koha-devel at lists.koha-community.org >>> https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-devel >>> website : https://www.koha-community.org/ >>> git : https://git.koha-community.org/ >>> bugs : https://bugs.koha-community.org/ >> >> -- >> Julian Maurice >> BibLibre >> _______________________________________________ >> Koha-devel mailing list >> Koha-devel at lists.koha-community.org >> https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-devel >> website : https://www.koha-community.org/ >> git : https://git.koha-community.org/ >> bugs : https://bugs.koha-community.org/ > _______________________________________________ > Koha-devel mailing list > Koha-devel at lists.koha-community.org > https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-devel > website : https://www.koha-community.org/ > git : https://git.koha-community.org/ > bugs : https://bugs.koha-community.org/ -- Julian Maurice BibLibre From jonathan.druart at bugs.koha-community.org Mon Dec 5 18:40:20 2022 From: jonathan.druart at bugs.koha-community.org (Jonathan Druart) Date: Mon, 5 Dec 2022 18:40:20 +0100 Subject: [Koha-devel] Good enough? In-Reply-To: References: <85c5041c-a3ee-db1c-b576-97e789193c14@biblibre.com> Message-ID: Having the hooks installed automatically will help devs catch small inconsistencies, that's a first step for the easy-to-catch things. The run_tests.pl script in misc4dev will hopefully help to run the tests by anybody. Running the tests on a DB without the sample data or with existing data is already a goal, it's something we are trying to catch during QA. TestBuilder helped us a lot for that. Create the data the tests need and the cleanup/rollback should be the way to write tests. Le lun. 5 dĂ©c. 2022 Ă  18:24, Julian Maurice a Ă©crit : > Yes it is expensive. But manual testing is much more expensive. I'd > rather pay for some CPU time than do manually what a bot can do better > and faster than me. Free services exist too (CircleCI, github actions, ...) > > Pushing to a temporary master branch is not a bad idea (a never-broken > master branch sounds nice), but I think it would happen too late. > Patches would have already been rebased multiple times, tested and > reviewed before we notice a test failure. > > koha-testing-docker feels more like a symptom of the difficulty to run > tests than a satisfying solution. It is probably necessary in order to > run complicated end-to-end tests, but it should not be mandatory to run > simple unit tests. > And ktd is not that easy to use. It can break, and it's not easy to > debug for someone not familiar with docker. > > Don't you think it would be a lot easier if we could run `prove t` (or > `npm test` or whatever) on any dev setup and have the same > failures/successes as Jenkins (minus the complicated tests like those > that require selenium) ? > Of course making that happen is not an easy task, but it should be a > long term goal IMO. > > Also I think no one wants to be the person that refuse a patch because a > comment is misaligned, so if that kind of thing is not automated, "not > good enough code" will continue to be pushed. > > > Le 05/12/2022 Ă  14:55, Jonathan Druart a Ă©crit : > > I don't think we should run the whole test suite everytime we attach > > patches, that would be very expensive in terms of resources. > > However it would be interesting to have a temporary 'master' branch > > that would become 'master' only if jenkins is happy. > > "master" would never be broken :D > > > > Using koha-testing-docker it's very easy to run tests locally. > > It will be even more easier with > > https://gitlab.com/koha-community/koha-misc4dev/-/issues/58 > > > > Le ven. 2 dĂ©c. 2022 Ă  18:16, Julian Maurice > > a Ă©crit : > >> > >> Hi, > >> > >> It seems to me that several issues could be solved by having the CI run > >> sooner, so authors can have feedback as soon as possible. Something like > >> github's CI integration with pull requests would be amazing to detect > >> common mistakes early and stop wasting human time. We should know if > >> tests pass before pushing to master, not after. > >> We could check tidyness, commit message conventions, code coverage by > >> tests, ... all before another person have to look at it. > >> > >> Also tests are not easy to run locally. They might pass on Jenkins, but > >> they do not on my local setup, so they are basically useless to tell me > >> if I broke something. It also makes writing new tests more difficult > >> than it should. > >> > >> If I wanted to improve Koha developer experience, I would start with > that. > >> > >> My 2 cents. > >> > >> Le 02/12/2022 Ă  15:42, Jonathan Druart a Ă©crit : > >>> Hi devs, > >>> > >>> I was wondering... How good is your "good enough"? > >>> It's a question I often ask myself, when writing patches or QAing > >>> yours: is it good enough to be into Koha? It does not have to be > >>> perfect or it may never reach master, but it must meet certain > >>> standards. > >>> > >>> When I was RM I tried to put that limit quite high. Not necessarily by > >>> asking the author for improving the follow-up patches, but also by > >>> adding the missing bits myself. > >>> > >>> There are various types of "good enough", depending on what we look > >>> at: good enough to be reviewed, good enough to be tested, to be put in > >>> production, get feedback, try an idea, etc. > >>> > >>> Most of the time, what is driving the limit is answering the > >>> questions "Is it maintainable?" / "Is it future proof?". Making sure > >>> the code you are writing won't be broken inadvertently is very > >>> important and must be part of the reflection. > >>> > >>> Katrin asked the QA team members what were our plans for 23.05. In my > >>> opinion we should enforce this "be future proof" concept. Writing code > >>> is easy, especially in Koha (yes it is!). Writing maintainable and > >>> robust code, following our guidelines is a bit harder. That's why we > >>> have a QA process that tries to catch inconsistencies or edge cases > >>> you may have missed. > >>> But I think we can be even more rigorous, and aim for better > >>> standards. We can elevate our ambitions and do better. When we see the > >>> number of new enhancements we are releasing every 6 months, it shows > >>> well that writing code is definitely not a problem. However sometimes > >>> developers are tempted to abandon their work once they are pushed to > >>> master. Pushed does not mean "done". Providing bug fixes, following-up > >>> with patches when needed, fixing CI jenkins, etc. is part of job > >>> (/fun) > >>> > >>> As a Koha developer for a long time now, I know how frustrating it can > >>> be to be asked for follow-ups/rewrite/tests to have our patches > >>> stamped with the precious PQA mark. But from the other point of view > >>> (RM, RMaints, QA team), I also know it's very frustrating when you are > >>> in charge of the release and you do not get the appropriate follow-up > >>> work once it's pushed to master. > >>> > >>> There are some easy steps to write/review better patches. All have > >>> been discussed already several times, but that can be enforced even > >>> more: > >>> 1. Perltidy (!) This is really a very trivial step. Please perltidy > >>> your code. There are hundreds of commits that have been pushed in the > >>> last months that are not tidy (alignment, indentation, lines too long, > >>> etc.) This can easily be configured in your IDE! [1] > >>> > >>> 2. Provide clean code. As said it's not necessarily easy, but the QA > >>> team and RM are supposed to know if the code is clean regarding Koha > >>> guidelines. If the code is not clean, don't PQA, don't push. Either > >>> clean yourself, or ask the original author of the patch to do it > >>> (explaining to them how it can be improved ofc). > >>> > >>> 3. Squash! I have been away for a couple of months and had to read the > >>> git history to know what I missed. And it was really hard to follow > >>> what was going on. First of all, we are not consistent: the commit > >>> message must tell what the patch is doing, not what the bug was (if > >>> you are writing a bug fix). Then, there are way too many follow-ups: > >>> tidiness, indentation fix, typo, spelling, etc. All those tiny > >>> follow-ups could be squashed into the original patch. We don't need > >>> unnecessary tons of entries in our git log for that. For instance, I > >>> usually add a "JD Amended patch: perltidy" for instance when I tidy > >>> the original patch, to keep track of the modification. Squash can be > >>> done by the original author, the QAer, the RM. So yes, you are losing > >>> one commit in the stats but the git log is easy to read! > >>> We could have an "Amended-by" marker if we really want to add credit > >>> on the dashboard (and/or release notes). > >>> > >>> 4. Run tests. Don't wait for Jenkins to fail. This is valid for the > >>> author and QA. Anticipate the failures by running more tests. If you > >>> are modifying C4::Circulation, then run prove on > >>> t/db_dependent/Circulation*, not only Circulation.t. It will help you > >>> catch edge cases. > >>> When something is pushed, track down jenkins failures that could be > >>> caused by your patches. > >>> > >>> 6. Be strict if you are QAing. Each QA member has their own "good > >>> enough", and the RM as well (either relying on the QAer or providing a > >>> full review). But QA must fail if the code is old Koha style code, or > >>> not "good enough". > >>> > >>> 7. Provide support for failing tests, fix things you broke. The QA > >>> team will be more comfortable with your patches if you show them you > >>> are providing support for your stuff. > >>> It's not because it's pushed that you don't have any more efforts to > >>> make. Provide follow-up patches you promised, provide bug fixes, etc. > >>> We don't have a good way to keep track of such demands, which does not > >>> make tracking easier for devs, QA and RM. Any suggestions? > >>> > >>> 8. QA team MUST NEVER* pass QA a change that is not covered by tests, > >>> never. You should not provide change to modules without tests! > >>> * almost never... > >>> > >>> 9. Stick to existing patterns. We should not have different ways to do > >>> the same thing. We should not have different places where a code is > >>> doing the same thing. Ask for help or advice on the list or IRC before > >>> you start coding. We will be happy to guide you. Even if you are a > >>> regular Koha developer it's not always easy to be aware of the latest > >>> master changes. > >>> We will tell you what's the current good practice, or point you to > >>> examples you could reuse for what you want to implement. > >>> > >>> 10. CI should drive the pushes. No more push if CI is not green. The > >>> more we wait the harder it is to track down the origin of the problem. > >>> Last cycle some jobs have been red for months, and we released > >>> 22.11.00 with D10, D11, D12 marked unstable... > >>> > >>> What will I do next cycle? > >>> All of that, and more. I will track down jenkins failures and > >>> responsibilize developers telling them when they break tests (and > >>> won't fix them anymore as I have been doing for years). > >>> I will raise on the bug reports what could have been improved. Yes, > >>> read that I will be even more annoying (to put it politely) than > >>> before. > >>> > >>> I've noticed that the pre-commit git hook on the wiki has been broken > >>> for more than 3 years. And also caught some core developers that do > >>> not have it in place. I am relying on it to keep Vue files tidy so > >>> it's important to have it set up properly. I am planning to force its > >>> usage for ktd users [2]. Adding more checks to it will help us to > >>> catch inconsistencies from the beginning. > >>> > >>> To summarize, writing code is cheap, maintaining code is way more > >>> expensive! It is easier to get the attention of developers before the > >>> patches are pushed to master than after, so we could be more ambitious > >>> and ask more. > >>> > >>> For discussion :) > >>> > >>> Cheers, > >>> Jonathan > >>> > >>> [1] If you are using vim, open ~/vimrc, add > >>> vmap :!perltidy -q > >>> Reload vim, select code in visual mode > >>> [2] https://gitlab.com/koha-community/koha-misc4dev/-/issues/59 > >>> _______________________________________________ > >>> Koha-devel mailing list > >>> Koha-devel at lists.koha-community.org > >>> https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-devel > >>> website : https://www.koha-community.org/ > >>> git : https://git.koha-community.org/ > >>> bugs : https://bugs.koha-community.org/ > >> > >> -- > >> Julian Maurice > >> BibLibre > >> _______________________________________________ > >> Koha-devel mailing list > >> Koha-devel at lists.koha-community.org > >> https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-devel > >> website : https://www.koha-community.org/ > >> git : https://git.koha-community.org/ > >> bugs : https://bugs.koha-community.org/ > > _______________________________________________ > > Koha-devel mailing list > > Koha-devel at lists.koha-community.org > > https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-devel > > website : https://www.koha-community.org/ > > git : https://git.koha-community.org/ > > bugs : https://bugs.koha-community.org/ > > -- > Julian Maurice > BibLibre > _______________________________________________ > Koha-devel mailing list > Koha-devel at lists.koha-community.org > https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-devel > website : https://www.koha-community.org/ > git : https://git.koha-community.org/ > bugs : https://bugs.koha-community.org/ > -------------- next part -------------- An HTML attachment was scrubbed... URL: From tomascohen at gmail.com Mon Dec 5 21:15:22 2022 From: tomascohen at gmail.com (Tomas Cohen Arazi) Date: Mon, 5 Dec 2022 17:15:22 -0300 Subject: [Koha-devel] Good enough? In-Reply-To: References: <85c5041c-a3ee-db1c-b576-97e789193c14@biblibre.com> Message-ID: El lun, 5 dic 2022 a las 14:24, Julian Maurice () escribiĂł: > Yes it is expensive. But manual testing is much more expensive. I'd > rather pay for some CPU time than do manually what a bot can do better > and faster than me. Free services exist too (CircleCI, github actions, ...) > When you develop, you usually add some tests and run them, along with related ones that could be broken because of the changes. Running the full test suite is not mandatory. Pushing to a branch somewhere there's CI setup to run the full test suite sounds like a good idea. > > Pushing to a temporary master branch is not a bad idea (a never-broken > master branch sounds nice), but I think it would happen too late. > Patches would have already been rebased multiple times, tested and > reviewed before we notice a test failure. > Agreed! > koha-testing-docker feels more like a symptom of the difficulty to run > tests than a satisfying solution. It is probably necessary in order to > run complicated end-to-end tests, but it should not be mandatory to run > simple unit tests. > And ktd is not that easy to use. It can break, and it's not easy to > debug for someone not familiar with docker. > I don't think it is mandatory, but it is the most convenient way to run in a controlled environment that matches as much as possible the conditions in production. That said, I'm pretty sure we could rely on carton or local::lib to generate an environment to work locally. > Don't you think it would be a lot easier if we could run `prove t` (or > `npm test` or whatever) on any dev setup and have the same > failures/successes as Jenkins (minus the complicated tests like those > that require selenium) ? > Of course making that happen is not an easy task, but it should be a > long term goal IMO. > > Also I think no one wants to be the person that refuse a patch because a > comment is misaligned, so if that kind of thing is not automated, "not > good enough code" will continue to be pushed. > > > Le 05/12/2022 Ă  14:55, Jonathan Druart a Ă©crit : > > I don't think we should run the whole test suite everytime we attach > > patches, that would be very expensive in terms of resources. > > However it would be interesting to have a temporary 'master' branch > > that would become 'master' only if jenkins is happy. > > "master" would never be broken :D > That would only save us some reverts if the authors fail to provide fixes. But I don't think reverts happen so often we need to change our workflow to avoid them. And jenkins to fail is the goal of it. We don't need it green all day, we need it to catch the errors (lots of errors are not covered by tests, and that's something we need to invest time on) and devs to take responsibility of fixing their breakages. I will take the blame there, because Jenkins notifications are not reaching the devs inboxes, and we need it fixed ASAP. -- TomĂĄs Cohen Arazi Theke Solutions (http://theke.io) ✆ +54 9351 3513384 GPG: B2F3C15F -------------- next part -------------- An HTML attachment was scrubbed... URL: From dcook at prosentient.com.au Mon Dec 5 23:40:00 2022 From: dcook at prosentient.com.au (David Cook) Date: Tue, 6 Dec 2022 09:40:00 +1100 Subject: [Koha-devel] Fundamental flaw in Koha REST API Message-ID: <07f401d908fa$85ab1bd0$91015370$@prosentient.com.au> Hi all, Just wondering if my email titled “x-koha-query header doesn't support UTF-8 data” actually made it to the koha-devel list as I didn’t get any replies to what I perceive as a significant issue. Basically, the x-koha-query header can only support ISO 8859-1/Latin-1 encoded characters. If you try to send UTF-8 encoded data to the API via Javascript, it will silently fail. See Bug 32401 for a practical example of this. I encountered this issue when using Arabic, but it is also obvious when using Hebrew, Chinese, Devanagari, or any non-Latin based script. It is less obvious – but still a problem – for some languages like Dutch, Finnish, French, and German. For instance, a x-koha-query header containing the French word “cƓur” will fail, because Ɠ is not a ISO 8859-1/Latin-1 character. (See the Wikipedia entry for ISO 8859-1 for more examples of problems with languages mostly covered by Latin-1.) Koha devs need to discuss this and possibly remove “x-koha-query” from Koha. At the moment, it’s not widely used by Koha itself, so I don’t think it will be hard to remove from Koha, but any third-party integrations would need to refactor to use a different option. Anyway, it would be great if someone could acknowledge receipt of this email. I have figured out how to get DataTables to stop using x-koha-query, so I’m going to do that for the particular acquisitions problem mentioned in Bug 32401, but we do need to talk about this issue. David Cook Senior Software Engineer Prosentient Systems Suite 7.03 6a Glen St Milsons Point NSW 2061 Australia Office: 02 9212 0899 Online: 02 8005 0595 -------------- next part -------------- An HTML attachment was scrubbed... URL: From gmc at equinoxoli.org Tue Dec 6 00:08:00 2022 From: gmc at equinoxoli.org (Galen Charlton) Date: Mon, 5 Dec 2022 18:08:00 -0500 Subject: [Koha-devel] Fundamental flaw in Koha REST API In-Reply-To: <07f401d908fa$85ab1bd0$91015370$@prosentient.com.au> References: <07f401d908fa$85ab1bd0$91015370$@prosentient.com.au> Message-ID: Hi, On Mon, Dec 5, 2022 at 5:40 PM David Cook wrote: > At the moment, it’s not widely used by Koha itself, so I don’t think > it will be hard to remove from Koha, but any third-party integrations > would need to refactor to use a different option. This might not be a huge factor, though of course removing that header should go through a deprecation procedure. Specifically, upon skimming the results of a GitHub search of "x-koha-query", the only uses I found outside of Koha itself were in plugins published by a couple active community members. Regards, Galen -- Galen Charlton Implementation and IT Manager Equinox Open Library Initiative gmc at equinoxOLI.org https://www.equinoxOLI.org phone: 877-OPEN-ILS (673-6457) direct: 770-709-5581 From tomascohen at gmail.com Tue Dec 6 01:24:27 2022 From: tomascohen at gmail.com (Tomas Cohen Arazi) Date: Mon, 5 Dec 2022 21:24:27 -0300 Subject: [Koha-devel] Fundamental flaw in Koha REST API In-Reply-To: References: <07f401d908fa$85ab1bd0$91015370$@prosentient.com.au> Message-ID: I recall we used the header at some point, to keep the base query clean when adding new filters. The only alternative is forcing the use of some form of ascii encoding (base64?) But I agree we should better deprecate it. It's use in the codebase on DT is tied to an option switch and probably not used El lun, 5 dic 2022 20:08, Galen Charlton escribiĂł: > Hi, > > On Mon, Dec 5, 2022 at 5:40 PM David Cook > wrote: > > At the moment, it’s not widely used by Koha itself, so I don’t think > > it will be hard to remove from Koha, but any third-party integrations > > would need to refactor to use a different option. > > This might not be a huge factor, though of course removing that header > should go through a deprecation procedure. > > Specifically, upon skimming the results of a GitHub search of > "x-koha-query", the only uses I found outside of Koha itself were in > plugins published by a couple active community members. > > Regards, > > Galen > -- > Galen Charlton > Implementation and IT Manager > Equinox Open Library Initiative > gmc at equinoxOLI.org > https://www.equinoxOLI.org > phone: 877-OPEN-ILS (673-6457) > direct: 770-709-5581 > _______________________________________________ > Koha-devel mailing list > Koha-devel at lists.koha-community.org > https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-devel > website : https://www.koha-community.org/ > git : https://git.koha-community.org/ > bugs : https://bugs.koha-community.org/ > -------------- next part -------------- An HTML attachment was scrubbed... URL: From dcook at prosentient.com.au Tue Dec 6 01:30:46 2022 From: dcook at prosentient.com.au (David Cook) Date: Tue, 6 Dec 2022 11:30:46 +1100 Subject: [Koha-devel] Fundamental flaw in Koha REST API In-Reply-To: References: <07f401d908fa$85ab1bd0$91015370$@prosentient.com.au> Message-ID: <082501d9090a$00e0b670$02a22350$@prosentient.com.au> Perhaps Bug 30165 allowing multiple “q” parameters would provide an alternative as well? In the DataTables, it looks like it’s used for receiving acquisitions and something with POS registers. In master, the acquisitions issue is solved by turning off that option switch. (Although in older versions of Koha the Orders route doesn’t seem able to handle the UTF-8 encoded characters.) David Cook Senior Software Engineer Prosentient Systems Suite 7.03 6a Glen St Milsons Point NSW 2061 Australia Office: 02 9212 0899 Online: 02 8005 0595 From: Tomas Cohen Arazi Sent: Tuesday, 6 December 2022 11:24 AM To: Galen Charlton Cc: David Cook ; koha-devel Subject: Re: [Koha-devel] Fundamental flaw in Koha REST API I recall we used the header at some point, to keep the base query clean when adding new filters. The only alternative is forcing the use of some form of ascii encoding (base64?) But I agree we should better deprecate it. It's use in the codebase on DT is tied to an option switch and probably not used El lun, 5 dic 2022 20:08, Galen Charlton > escribiĂł: Hi, On Mon, Dec 5, 2022 at 5:40 PM David Cook > wrote: > At the moment, it’s not widely used by Koha itself, so I don’t think > it will be hard to remove from Koha, but any third-party integrations > would need to refactor to use a different option. This might not be a huge factor, though of course removing that header should go through a deprecation procedure. Specifically, upon skimming the results of a GitHub search of "x-koha-query", the only uses I found outside of Koha itself were in plugins published by a couple active community members. Regards, Galen -- Galen Charlton Implementation and IT Manager Equinox Open Library Initiative gmc at equinoxOLI.org https://www.equinoxOLI.org phone: 877-OPEN-ILS (673-6457) direct: 770-709-5581 _______________________________________________ Koha-devel mailing list Koha-devel at lists.koha-community.org https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-devel website : https://www.koha-community.org/ git : https://git.koha-community.org/ bugs : https://bugs.koha-community.org/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From dcook at prosentient.com.au Tue Dec 6 01:33:03 2022 From: dcook at prosentient.com.au (David Cook) Date: Tue, 6 Dec 2022 11:33:03 +1100 Subject: [Koha-devel] Fundamental flaw in Koha REST API References: <07f401d908fa$85ab1bd0$91015370$@prosentient.com.au> Message-ID: <082a01d9090a$4d79a140$e86ce3c0$@prosentient.com.au> Btw looks like Bug 30165 is what resolves the UTF-8 issue for the orders route
 David Cook Senior Software Engineer Prosentient Systems Suite 7.03 6a Glen St Milsons Point NSW 2061 Australia Office: 02 9212 0899 Online: 02 8005 0595 From: David Cook Sent: Tuesday, 6 December 2022 11:31 AM To: 'Tomas Cohen Arazi' ; 'Galen Charlton' Cc: 'koha-devel' Subject: RE: [Koha-devel] Fundamental flaw in Koha REST API Perhaps Bug 30165 allowing multiple “q” parameters would provide an alternative as well? In the DataTables, it looks like it’s used for receiving acquisitions and something with POS registers. In master, the acquisitions issue is solved by turning off that option switch. (Although in older versions of Koha the Orders route doesn’t seem able to handle the UTF-8 encoded characters.) David Cook Senior Software Engineer Prosentient Systems Suite 7.03 6a Glen St Milsons Point NSW 2061 Australia Office: 02 9212 0899 Online: 02 8005 0595 From: Tomas Cohen Arazi > Sent: Tuesday, 6 December 2022 11:24 AM To: Galen Charlton > Cc: David Cook >; koha-devel > Subject: Re: [Koha-devel] Fundamental flaw in Koha REST API I recall we used the header at some point, to keep the base query clean when adding new filters. The only alternative is forcing the use of some form of ascii encoding (base64?) But I agree we should better deprecate it. It's use in the codebase on DT is tied to an option switch and probably not used El lun, 5 dic 2022 20:08, Galen Charlton > escribiĂł: Hi, On Mon, Dec 5, 2022 at 5:40 PM David Cook > wrote: > At the moment, it’s not widely used by Koha itself, so I don’t think > it will be hard to remove from Koha, but any third-party integrations > would need to refactor to use a different option. This might not be a huge factor, though of course removing that header should go through a deprecation procedure. Specifically, upon skimming the results of a GitHub search of "x-koha-query", the only uses I found outside of Koha itself were in plugins published by a couple active community members. Regards, Galen -- Galen Charlton Implementation and IT Manager Equinox Open Library Initiative gmc at equinoxOLI.org https://www.equinoxOLI.org phone: 877-OPEN-ILS (673-6457) direct: 770-709-5581 _______________________________________________ Koha-devel mailing list Koha-devel at lists.koha-community.org https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-devel website : https://www.koha-community.org/ git : https://git.koha-community.org/ bugs : https://bugs.koha-community.org/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From dcook at prosentient.com.au Tue Dec 6 02:27:58 2022 From: dcook at prosentient.com.au (David Cook) Date: Tue, 6 Dec 2022 12:27:58 +1100 Subject: [Koha-devel] Maximum execution time for background jobs? Message-ID: <083d01d90911$faa6cdf0$eff469d0$@prosentient.com.au> Hi all, I was closing old Bugzilla bugs when I bumped into "Bug 10460 - way to stop a run away report". It got me thinking about maximum execution time for background jobs. What do people think about background jobs workers killing their child processes if they exceed a maximum execution time? I don't have a particular use case for it at the moment, so not a priority by any means, but thoughts on the concept? David Cook Senior Software Engineer Prosentient Systems Suite 7.03 6a Glen St Milsons Point NSW 2061 Australia Office: 02 9212 0899 Online: 02 8005 0595 -------------- next part -------------- An HTML attachment was scrubbed... URL: From dcook at prosentient.com.au Tue Dec 6 02:49:25 2022 From: dcook at prosentient.com.au (David Cook) Date: Tue, 6 Dec 2022 12:49:25 +1100 Subject: [Koha-devel] Formally give up on trying to provide PostgreSQL DB support? Message-ID: <084201d90914$f9818070$ec848150$@prosentient.com.au> Hi all, Should we formally give up on trying to provide PostgreSQL support? I find it highly unlikely that we will ever provide it. We may as well close issues like the following: https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=7365 https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=7802 https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=11422 Over the last year or so, the Keycloak community has undergone a lot of conversations on database support. Historically, it's provided support for MariaDB, MSSQL, MySQL, Oracle, PostgreSQL, and more. However, going forward, Keycloak plans to only provide first class support for PostgreSQL and CockroachDB databases. CockroachDB uses the same wire protocol as PostgreSQL, so really they're supporting 1 database type. You can read more about it all at the following links: https://www.keycloak.org/2022/02/dbs.html https://www.keycloak.org/2022/07/storage-map.html Personally, I really like PostgreSQL. I use it for a number of other projects. But I think we're pretty wedded to MySQL/MariaDB with Koha, and maybe we just need to own that? David Cook Senior Software Engineer Prosentient Systems Suite 7.03 6a Glen St Milsons Point NSW 2061 Australia Office: 02 9212 0899 Online: 02 8005 0595 -------------- next part -------------- An HTML attachment was scrubbed... URL: From M.de.Rooy at rijksmuseum.nl Tue Dec 6 13:17:17 2022 From: M.de.Rooy at rijksmuseum.nl (Marcel de Rooy) Date: Tue, 6 Dec 2022 12:17:17 +0000 Subject: [Koha-devel] Good enough? In-Reply-To: References: Message-ID: > As said in the email I am planning to integrate them into our dev env (via misc4dev), see Great but please note that 'our dev env' is not per definition KTD for everybody. What we do share is the git repo 🙂 Marcel -------------- next part -------------- An HTML attachment was scrubbed... URL: From tomascohen at gmail.com Tue Dec 6 18:50:00 2022 From: tomascohen at gmail.com (Tomas Cohen Arazi) Date: Tue, 6 Dec 2022 14:50:00 -0300 Subject: [Koha-devel] Good enough? In-Reply-To: References: Message-ID: What is your dev environment like? a dev install? on the host OS or a VM? We tried hard to make the misc4dev tools agnostic of KTD. Feedback is welcome. El mar, 6 dic 2022 a las 9:17, Marcel de Rooy () escribiĂł: > > As said in the email I am planning to integrate them into our dev > env (via misc4dev), see > > Great but please note that 'our dev env' is not per definition KTD for > everybody. > What we do share is the git repo 🙂 > > Marcel > _______________________________________________ > Koha-devel mailing list > Koha-devel at lists.koha-community.org > https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-devel > website : https://www.koha-community.org/ > git : https://git.koha-community.org/ > bugs : https://bugs.koha-community.org/ > -- TomĂĄs Cohen Arazi Theke Solutions (http://theke.io) ✆ +54 9351 3513384 GPG: B2F3C15F -------------- next part -------------- An HTML attachment was scrubbed... URL: From fridolin.somers at biblibre.com Tue Dec 6 22:37:36 2022 From: fridolin.somers at biblibre.com (Fridolin SOMERS) Date: Tue, 6 Dec 2022 11:37:36 -1000 Subject: [Koha-devel] Formally give up on trying to provide PostgreSQL DB support? In-Reply-To: <084201d90914$f9818070$ec848150$@prosentient.com.au> References: <084201d90914$f9818070$ec848150$@prosentient.com.au> Message-ID: +1 for giving up. We can focus on other cool features ;) Le 05/12/2022 Ă  15:49, David Cook a Ă©crit : > Hi all, > > Should we formally give up on trying to provide PostgreSQL support? > > I find it highly unlikely that we will ever provide it. > > We may as well close issues like the following: > > https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=7365 > > > https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=7802 > > > https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=11422 > > > Over the last year or so, the Keycloak community has undergone a lot of > conversations on database support. > > Historically, it’s provided support for MariaDB, MSSQL, MySQL, Oracle, > PostgreSQL, and more. > > However, going forward, Keycloak plans to only provide first class > support for PostgreSQL and CockroachDB databases. CockroachDB uses the > same wire protocol as PostgreSQL, so really they’re supporting 1 > database type. You can read more about it all at the following links: > > https://www.keycloak.org/2022/02/dbs.html > > > https://www.keycloak.org/2022/07/storage-map.html > > > Personally, I really like PostgreSQL. I use it for a number of other > projects. But I think we’re pretty wedded to MySQL/MariaDB with Koha, > and maybe we just need to own that? > > David Cook > > Senior Software Engineer > > Prosentient Systems > > Suite 7.03 > > 6a Glen St > > Milsons Point NSW 2061 > > Australia > > Office: 02 9212 0899 > > Online: 02 8005 0595 > > > _______________________________________________ > Koha-devel mailing list > Koha-devel at lists.koha-community.org > https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-devel > website : https://www.koha-community.org/ > git : https://git.koha-community.org/ > bugs : https://bugs.koha-community.org/ -- Fridolin SOMERS Software and system maintainer 🩄 BibLibre, France From M.de.Rooy at rijksmuseum.nl Wed Dec 7 13:10:22 2022 From: M.de.Rooy at rijksmuseum.nl (Marcel de Rooy) Date: Wed, 7 Dec 2022 12:10:22 +0000 Subject: [Koha-devel] Formally give up on trying to provide PostgreSQL DB support? In-Reply-To: References: <084201d90914$f9818070$ec848150$@prosentient.com.au> Message-ID: Yes, we already gave up years ago. But we never said so 🙂 ________________________________ Van: Koha-devel namens Fridolin SOMERS Verzonden: dinsdag 6 december 2022 22:37 Aan: koha-devel at lists.koha-community.org Onderwerp: Re: [Koha-devel] Formally give up on trying to provide PostgreSQL DB support? +1 for giving up. We can focus on other cool features ;) Le 05/12/2022 Ă  15:49, David Cook a Ă©crit : > Hi all, > > Should we formally give up on trying to provide PostgreSQL support? > > I find it highly unlikely that we will ever provide it. > > We may as well close issues like the following: > > https://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fbugs.koha-community.org%2Fbugzilla3%2Fshow_bug.cgi%3Fid%3D7365&data=05%7C01%7Cm.de.rooy%40rijksmuseum.nl%7C0def9e5a25da444295ba08dad7d3135c%7C635b05eb66c748e1a94fb4b05a1b058b%7C0%7C0%7C638059598833271295%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=LYnWdxJxT9NzlyTYxL7YGRfAyUqn5XDSQoUzm0rK91c%3D&reserved=0 > > > > https://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fbugs.koha-community.org%2Fbugzilla3%2Fshow_bug.cgi%3Fid%3D7802&data=05%7C01%7Cm.de.rooy%40rijksmuseum.nl%7C0def9e5a25da444295ba08dad7d3135c%7C635b05eb66c748e1a94fb4b05a1b058b%7C0%7C0%7C638059598833427540%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=HTW18FpqsJA4wuqlef63rFnAMKqEkZ38LG15e3QE8Rc%3D&reserved=0 > > > > https://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fbugs.koha-community.org%2Fbugzilla3%2Fshow_bug.cgi%3Fid%3D11422&data=05%7C01%7Cm.de.rooy%40rijksmuseum.nl%7C0def9e5a25da444295ba08dad7d3135c%7C635b05eb66c748e1a94fb4b05a1b058b%7C0%7C0%7C638059598833427540%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=%2F5lNH3THCpISrrptFEKwWVl8RFQ1T9m%2BPkUihQE%2B9tk%3D&reserved=0 > > > > Over the last year or so, the Keycloak community has undergone a lot of > conversations on database support. > > Historically, it’s provided support for MariaDB, MSSQL, MySQL, Oracle, > PostgreSQL, and more. > > However, going forward, Keycloak plans to only provide first class > support for PostgreSQL and CockroachDB databases. CockroachDB uses the > same wire protocol as PostgreSQL, so really they’re supporting 1 > database type. You can read more about it all at the following links: > > https://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.keycloak.org%2F2022%2F02%2Fdbs.html&data=05%7C01%7Cm.de.rooy%40rijksmuseum.nl%7C0def9e5a25da444295ba08dad7d3135c%7C635b05eb66c748e1a94fb4b05a1b058b%7C0%7C0%7C638059598833427540%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=yTqxQLhGj4yCTQ%2Fnl6ridH7pB9%2F8wc2d%2BTA8cCu0w7U%3D&reserved=0 > > > > https://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.keycloak.org%2F2022%2F07%2Fstorage-map.html&data=05%7C01%7Cm.de.rooy%40rijksmuseum.nl%7C0def9e5a25da444295ba08dad7d3135c%7C635b05eb66c748e1a94fb4b05a1b058b%7C0%7C0%7C638059598833427540%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=q1%2BMldQLmrWd38KXiLlxviiIMaOsDiXXcDGslbY55io%3D&reserved=0 > > > > Personally, I really like PostgreSQL. I use it for a number of other > projects. But I think we’re pretty wedded to MySQL/MariaDB with Koha, > and maybe we just need to own that? > > David Cook > > Senior Software Engineer > > Prosentient Systems > > Suite 7.03 > > 6a Glen St > > Milsons Point NSW 2061 > > Australia > > Office: 02 9212 0899 > > Online: 02 8005 0595 > > > _______________________________________________ > Koha-devel mailing list > Koha-devel at lists.koha-community.org > https://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Flists.koha-community.org%2Fcgi-bin%2Fmailman%2Flistinfo%2Fkoha-devel&data=05%7C01%7Cm.de.rooy%40rijksmuseum.nl%7C0def9e5a25da444295ba08dad7d3135c%7C635b05eb66c748e1a94fb4b05a1b058b%7C0%7C0%7C638059598833427540%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=3h6jayfnHhuYlU2XHaIgQuj5PvC4fR11SI58xe4Z2Ag%3D&reserved=0 > website : https://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.koha-community.org%2F&data=05%7C01%7Cm.de.rooy%40rijksmuseum.nl%7C0def9e5a25da444295ba08dad7d3135c%7C635b05eb66c748e1a94fb4b05a1b058b%7C0%7C0%7C638059598833427540%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=7oxWxYoeWacmmMf15vANzMVtLuevbWBkgmReOtKhY%2FA%3D&reserved=0 > git : https://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgit.koha-community.org%2F&data=05%7C01%7Cm.de.rooy%40rijksmuseum.nl%7C0def9e5a25da444295ba08dad7d3135c%7C635b05eb66c748e1a94fb4b05a1b058b%7C0%7C0%7C638059598833427540%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=XHqzAuAVzGWJ8rD0bYqPp5eqTMvCuJcuN1P1f%2FrgrPI%3D&reserved=0 > bugs : https://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fbugs.koha-community.org%2F&data=05%7C01%7Cm.de.rooy%40rijksmuseum.nl%7C0def9e5a25da444295ba08dad7d3135c%7C635b05eb66c748e1a94fb4b05a1b058b%7C0%7C0%7C638059598833427540%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=FhKRY4cBmXAbpMK9ZfTvmWKePfIOEEHeetknV8YwGJY%3D&reserved=0 -- Fridolin SOMERS Software and system maintainer 🩄 BibLibre, France _______________________________________________ Koha-devel mailing list Koha-devel at lists.koha-community.org https://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Flists.koha-community.org%2Fcgi-bin%2Fmailman%2Flistinfo%2Fkoha-devel&data=05%7C01%7Cm.de.rooy%40rijksmuseum.nl%7C0def9e5a25da444295ba08dad7d3135c%7C635b05eb66c748e1a94fb4b05a1b058b%7C0%7C0%7C638059598833427540%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=3h6jayfnHhuYlU2XHaIgQuj5PvC4fR11SI58xe4Z2Ag%3D&reserved=0 website : https://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.koha-community.org%2F&data=05%7C01%7Cm.de.rooy%40rijksmuseum.nl%7C0def9e5a25da444295ba08dad7d3135c%7C635b05eb66c748e1a94fb4b05a1b058b%7C0%7C0%7C638059598833427540%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=7oxWxYoeWacmmMf15vANzMVtLuevbWBkgmReOtKhY%2FA%3D&reserved=0 git : https://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgit.koha-community.org%2F&data=05%7C01%7Cm.de.rooy%40rijksmuseum.nl%7C0def9e5a25da444295ba08dad7d3135c%7C635b05eb66c748e1a94fb4b05a1b058b%7C0%7C0%7C638059598833427540%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=XHqzAuAVzGWJ8rD0bYqPp5eqTMvCuJcuN1P1f%2FrgrPI%3D&reserved=0 bugs : https://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fbugs.koha-community.org%2F&data=05%7C01%7Cm.de.rooy%40rijksmuseum.nl%7C0def9e5a25da444295ba08dad7d3135c%7C635b05eb66c748e1a94fb4b05a1b058b%7C0%7C0%7C638059598833427540%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=FhKRY4cBmXAbpMK9ZfTvmWKePfIOEEHeetknV8YwGJY%3D&reserved=0 -------------- next part -------------- An HTML attachment was scrubbed... URL: From jonathan.druart at bugs.koha-community.org Wed Dec 7 21:59:08 2022 From: jonathan.druart at bugs.koha-community.org (Jonathan Druart) Date: Wed, 7 Dec 2022 21:59:08 +0100 Subject: [Koha-devel] Formally give up on trying to provide PostgreSQL DB support? In-Reply-To: <084201d90914$f9818070$ec848150$@prosentient.com.au> References: <084201d90914$f9818070$ec848150$@prosentient.com.au> Message-ID: We are not giving up, we never really tried. It could stay a long term goal, waiting for someone to work on it. I don't think we should start again adding MySQLisms. Le mar. 6 dĂ©c. 2022 Ă  02:49, David Cook a Ă©crit : > Hi all, > > > > Should we formally give up on trying to provide PostgreSQL support? > > > > I find it highly unlikely that we will ever provide it. > > > > We may as well close issues like the following: > > https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=7365 > > https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=7802 > > https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=11422 > > > > Over the last year or so, the Keycloak community has undergone a lot of > conversations on database support. > > > > Historically, it’s provided support for MariaDB, MSSQL, MySQL, Oracle, > PostgreSQL, and more. > > > > However, going forward, Keycloak plans to only provide first class support > for PostgreSQL and CockroachDB databases. CockroachDB uses the same wire > protocol as PostgreSQL, so really they’re supporting 1 database type. You > can read more about it all at the following links: > > https://www.keycloak.org/2022/02/dbs.html > > https://www.keycloak.org/2022/07/storage-map.html > > > > Personally, I really like PostgreSQL. I use it for a number of other > projects. But I think we’re pretty wedded to MySQL/MariaDB with Koha, and > maybe we just need to own that? > > > > David Cook > > Senior Software Engineer > > Prosentient Systems > > Suite 7.03 > > 6a Glen St > > Milsons Point NSW 2061 > > Australia > > > > Office: 02 9212 0899 > > Online: 02 8005 0595 > > > _______________________________________________ > Koha-devel mailing list > Koha-devel at lists.koha-community.org > https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-devel > website : https://www.koha-community.org/ > git : https://git.koha-community.org/ > bugs : https://bugs.koha-community.org/ > -------------- next part -------------- An HTML attachment was scrubbed... URL: From wainuiwitikapark at catalyst.net.nz Wed Dec 7 23:24:31 2022 From: wainuiwitikapark at catalyst.net.nz (Wainui Witika-Park) Date: Thu, 8 Dec 2022 11:24:31 +1300 Subject: [Koha-devel] Error - browsing Patrons by Last Name Message-ID: <0fa35144-e615-70c6-4ad0-7d1654ffc1a7@catalyst.net.nz> Hi everyone, I was wondering if anyone can help me please, I am getting an error when Browsing Patrons by Last Name. When clicking on some letters, the search gets stuck on "Processing". The letters that it seems to get stuck on are: A, H, N, T, V and W It also gives a pop up error when clicking on S that sometimes says "Something went wrong when loading the table. 500: Internal Server Error" and sometimes says "Something went wrong. The note has not been saved". The plack-error.log says: [2022-12-06 14:47:41.21998] [19070] [warn] OpenAPI >>> GET api/v1/patrons [{"message":"Expected string - got null.","path":"\/body\/4\/extended_attributes\/0\/value"}] Any help would very much appreciated :) Wainui From katrin.fischer.83 at web.de Wed Dec 7 23:46:02 2022 From: katrin.fischer.83 at web.de (Katrin Fischer) Date: Wed, 7 Dec 2022 23:46:02 +0100 Subject: [Koha-devel] Error - browsing Patrons by Last Name In-Reply-To: <0fa35144-e615-70c6-4ad0-7d1654ffc1a7@catalyst.net.nz> References: <0fa35144-e615-70c6-4ad0-7d1654ffc1a7@catalyst.net.nz> Message-ID: Hi Wainui, could it be this bug? *Bug 31381* - Searching patrons by letter broken when using non-mandatory extended attributes Hope this helps, Katrin On 07.12.22 23:24, Wainui Witika-Park wrote: > Hi everyone, > > I was wondering if anyone can help me please, I am getting an error > when Browsing Patrons by Last Name. > > When clicking on some letters, the search gets stuck on "Processing". > > The letters that it seems to get stuck on are: A, H, N, T, V and W > > It also gives a pop up error when clicking on S that sometimes says > "Something went wrong when loading the table. 500: Internal Server > Error" and sometimes says "Something went wrong. The note has not been > saved". The plack-error.log says: > > [2022-12-06 14:47:41.21998] [19070] [warn] OpenAPI >>> GET > api/v1/patrons [{"message":"Expected string - got > null.","path":"\/body\/4\/extended_attributes\/0\/value"}] > > Any help would very much appreciated :) > > Wainui > _______________________________________________ > Koha-devel mailing list > Koha-devel at lists.koha-community.org > https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-devel > website : https://www.koha-community.org/ > git : https://git.koha-community.org/ > bugs : https://bugs.koha-community.org/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From dcook at prosentient.com.au Fri Dec 9 04:26:54 2022 From: dcook at prosentient.com.au (David Cook) Date: Fri, 9 Dec 2022 14:26:54 +1100 Subject: [Koha-devel] What's the definitive way to use Perl tidy in koha-testing-docker? Message-ID: <001a01d90b7e$17e4dd80$47ae9880$@prosentient.com.au> Hi all, I've heard a few people saying lately that they wish people would use Perl tidy more. So I used the ":Tidy" command in koha-testing-docker on my visual selection and I got the following: + $this_item->{shelves} = + Koha::Virtualshelves->get_shelves_containing_record( + { + biblionumber => $biblionumber, + borrowernumber => $loggedinuser, + } + ); But that goes against the Indentation coding guidelines from https://wiki.koha-community.org/wiki/Coding_Guidelines#Indentation So. what are we supposed to be doing these days? David Cook Senior Software Engineer Prosentient Systems Suite 7.03 6a Glen St Milsons Point NSW 2061 Australia Office: 02 9212 0899 Online: 02 8005 0595 -------------- next part -------------- An HTML attachment was scrubbed... URL: From dcook at prosentient.com.au Fri Dec 9 07:25:41 2022 From: dcook at prosentient.com.au (David Cook) Date: Fri, 9 Dec 2022 17:25:41 +1100 Subject: [Koha-devel] MySQL/MariaDB index prefix length Message-ID: <002701d90b97$11303700$3390a500$@prosentient.com.au> Hi all, We often create indexes with a prefix length of 191 because we use 4 byte encoded fields. 191 * 4 = 764, which comes in under the prefix index length threshold of 767. But that looks like a limit that was imposed due to the COMPACT row format. Most newer MySQL/MariaDB databases will be using the DYNAMIC row format, which can support an index key prefix limit of 3072 bytes (or 768 4-byte encoded characters). On Bugzilla, there have been times where I've speculated that maybe it's the sysadmin's job to update the database row format, but perhaps we should take a more proactive effort to move row format from COMPACT to DYNAMIC, so that we can more systematically take advantage of longer index prefixes. I often think about how to index URLs, since the maximum length of URLs are very large. Web browsers hover around the 2000 character mark while web servers can be much higher than that. Of course, even with 3072 bytes, that still wouldn't be enough for a full URL. But it would get a person a lot farther than 191 characters. David Cook Senior Software Engineer Prosentient Systems Suite 7.03 6a Glen St Milsons Point NSW 2061 Australia Office: 02 9212 0899 Online: 02 8005 0595 -------------- next part -------------- An HTML attachment was scrubbed... URL: From katrin.fischer.83 at web.de Sun Dec 11 14:54:18 2022 From: katrin.fischer.83 at web.de (Katrin Fischer) Date: Sun, 11 Dec 2022 14:54:18 +0100 Subject: [Koha-devel] What's the definitive way to use Perl tidy in koha-testing-docker? In-Reply-To: <001a01d90b7e$17e4dd80$47ae9880$@prosentient.com.au> References: <001a01d90b7e$17e4dd80$47ae9880$@prosentient.com.au> Message-ID: Hi David, when using perltidy in my editor with the standard configuration, the indentation appears to be correct. But I am not sure what you mean with :Tidy in ktd? It looks like our coding guidelines matches the defaults for PerlTidy: n=4 and space. Could it be that some local config file is used instead of the default? There is a wiki page for how to use PerlTidy that we might want to update with any findings: https://wiki.koha-community.org/wiki/Perltidy Katrin On 09.12.22 04:26, David Cook wrote: > > Hi all, > > I’ve heard a few people saying lately that they wish people would use > Perl tidy more. > > So I used the “:Tidy” command in koha-testing-docker on my visual > selection and I got the following: > > +                $this_item->{shelves} = > > + Koha::Virtualshelves->get_shelves_containing_record( > > +                    { > > +                        biblionumber => $biblionumber, > > +                        borrowernumber => $loggedinuser, > > +                    } > > +                  ); > > But that goes against the Indentation coding guidelines from > https://wiki.koha-community.org/wiki/Coding_Guidelines#Indentation > > So
 what are we supposed to be doing these days? > > David Cook > > Senior Software Engineer > > Prosentient Systems > > Suite 7.03 > > 6a Glen St > > Milsons Point NSW 2061 > > Australia > > Office: 02 9212 0899 > > Online: 02 8005 0595 > > > _______________________________________________ > Koha-devel mailing list > Koha-devel at lists.koha-community.org > https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-devel > website :https://www.koha-community.org/ > git :https://git.koha-community.org/ > bugs :https://bugs.koha-community.org/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From katrin.fischer.83 at web.de Sun Dec 11 14:58:07 2022 From: katrin.fischer.83 at web.de (Katrin Fischer) Date: Sun, 11 Dec 2022 14:58:07 +0100 Subject: [Koha-devel] Formally give up on trying to provide PostgreSQL DB support? In-Reply-To: References: <084201d90914$f9818070$ec848150$@prosentient.com.au> Message-ID: <6de10759-4be4-2e92-434c-6e9010772ee1@web.de> On 07.12.22 21:59, Jonathan Druart wrote: > We are not giving up, we never really tried. > It could stay a long term goal, waiting for someone to work on it. > I don't think we should start again adding MySQLisms. +1 My question would also be how this decision would impact our coding guidelines. > > Le mar. 6 dĂ©c. 2022 Ă  02:49, David Cook a > Ă©crit : > > Hi all, > > Should we formally give up on trying to provide PostgreSQL support? > > I find it highly unlikely that we will ever provide it. > > We may as well close issues like the following: > > https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=7365 > > https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=7802 > > https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=11422 > > Over the last year or so, the Keycloak community has undergone a > lot of conversations on database support. > > Historically, it’s provided support for MariaDB, MSSQL, MySQL, > Oracle, PostgreSQL, and more. > > However, going forward, Keycloak plans to only provide first class > support for PostgreSQL and CockroachDB databases. CockroachDB uses > the same wire protocol as PostgreSQL, so really they’re supporting > 1 database type. You can read more about it all at the following > links: > > https://www.keycloak.org/2022/02/dbs.html > > https://www.keycloak.org/2022/07/storage-map.html > > Personally, I really like PostgreSQL. I use it for a number of > other projects. But I think we’re pretty wedded to MySQL/MariaDB > with Koha, and maybe we just need to own that? > > David Cook > > Senior Software Engineer > > Prosentient Systems > > Suite 7.03 > > 6a Glen St > > Milsons Point NSW 2061 > > Australia > > Office: 02 9212 0899 > > Online: 02 8005 0595 > > _______________________________________________ > Koha-devel mailing list > Koha-devel at lists.koha-community.org > https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-devel > website : https://www.koha-community.org/ > git : https://git.koha-community.org/ > bugs : https://bugs.koha-community.org/ > > > _______________________________________________ > Koha-devel mailing list > Koha-devel at lists.koha-community.org > https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-devel > website :https://www.koha-community.org/ > git :https://git.koha-community.org/ > bugs :https://bugs.koha-community.org/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From tomascohen at gmail.com Sun Dec 11 16:00:04 2022 From: tomascohen at gmail.com (Tomas Cohen Arazi) Date: Sun, 11 Dec 2022 12:00:04 -0300 Subject: [Koha-devel] What's the definitive way to use Perl tidy in koha-testing-docker? In-Reply-To: <001a01d90b7e$17e4dd80$47ae9880$@prosentient.com.au> References: <001a01d90b7e$17e4dd80$47ae9880$@prosentient.com.au> Message-ID: I normally point vscode's perltidy extension to the shipped perltidyrc: https://git.koha-community.org/Koha-community/Koha/src/branch/master/xt/perltidyrc and do Ctrl+k-f on the region I'm editing. El vie, 9 dic 2022 0:27, David Cook escribiĂł: > Hi all, > > > > I’ve heard a few people saying lately that they wish people would use Perl > tidy more. > > > > So I used the “:Tidy” command in koha-testing-docker on my visual > selection and I got the following: > > > > + $this_item->{shelves} = > > + Koha::Virtualshelves->get_shelves_containing_record( > > + { > > + biblionumber => $biblionumber, > > + borrowernumber => $loggedinuser, > > + } > > + ); > > > > But that goes against the Indentation coding guidelines from > https://wiki.koha-community.org/wiki/Coding_Guidelines#Indentation > > > > So
 what are we supposed to be doing these days? > > > > David Cook > > Senior Software Engineer > > Prosentient Systems > > Suite 7.03 > > 6a Glen St > > Milsons Point NSW 2061 > > Australia > > > > Office: 02 9212 0899 > > Online: 02 8005 0595 > > > _______________________________________________ > Koha-devel mailing list > Koha-devel at lists.koha-community.org > https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-devel > website : https://www.koha-community.org/ > git : https://git.koha-community.org/ > bugs : https://bugs.koha-community.org/ > -------------- next part -------------- An HTML attachment was scrubbed... URL: From katrin.fischer.83 at web.de Sun Dec 11 16:49:27 2022 From: katrin.fischer.83 at web.de (Katrin Fischer) Date: Sun, 11 Dec 2022 16:49:27 +0100 Subject: [Koha-devel] What's the definitive way to use Perl tidy in koha-testing-docker? In-Reply-To: References: <001a01d90b7e$17e4dd80$47ae9880$@prosentient.com.au> Message-ID: <336872db-a098-bc50-7888-7f0bdc9202fa@web.de> Hm, so we have a mismatch between documentation and real life use? Wiki: "The offical perltidy style for perl code in Koha is the perltidy 'default' or 'larry-wall' style. There was a customised perltidy style in Koha for a while, but it was never official or enforced and later removed." There is also: *Bug 30002* - Remove unused xt/perltidyrc On 11.12.22 16:00, Tomas Cohen Arazi wrote: > I normally point vscode's perltidy extension to the shipped perltidyrc: > > https://git.koha-community.org/Koha-community/Koha/src/branch/master/xt/perltidyrc > > and do Ctrl+k-f on the region I'm editing. > > > El vie, 9 dic 2022 0:27, David Cook escribiĂł: > > Hi all, > > I’ve heard a few people saying lately that they wish people would > use Perl tidy more. > > So I used the “:Tidy” command in koha-testing-docker on my visual > selection and I got the following: > > + $this_item->{shelves} = > > + Koha::Virtualshelves->get_shelves_containing_record( > > +                    { > > + biblionumber   => $biblionumber, > > + borrowernumber => $loggedinuser, > > +                    } > > +                  ); > > But that goes against the Indentation coding guidelines from > https://wiki.koha-community.org/wiki/Coding_Guidelines#Indentation > > So
 what are we supposed to be doing these days? > > David Cook > > Senior Software Engineer > > Prosentient Systems > > Suite 7.03 > > 6a Glen St > > Milsons Point NSW 2061 > > Australia > > Office: 02 9212 0899 > > Online: 02 8005 0595 > > _______________________________________________ > Koha-devel mailing list > Koha-devel at lists.koha-community.org > https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-devel > website : https://www.koha-community.org/ > git : https://git.koha-community.org/ > bugs : https://bugs.koha-community.org/ > > > _______________________________________________ > Koha-devel mailing list > Koha-devel at lists.koha-community.org > https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-devel > website :https://www.koha-community.org/ > git :https://git.koha-community.org/ > bugs :https://bugs.koha-community.org/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From kohanews at gmail.com Tue Dec 13 09:48:54 2022 From: kohanews at gmail.com (Koha Newsletter) Date: Tue, 13 Dec 2022 09:48:54 +0100 Subject: [Koha-devel] Call for news - Newsletter December 2022 Message-ID: Hi I'm collecting news for the December 2022 Koha Community Newsletter. Please send anything noteworthy to: kohanews (at) gmail (dot) com News criteria: * News items can be of any length. * Images are fine. * Anything and everything Koha. * Submit by the 26th of the month. Text format criteria: * Just structured plain text, or * HTML text to include tables or similar For events: * Consider adding your event to the Koha Community calendar at https://koha-community.org/calendar/ Thank you! Michael Kuhn Editor, Koha Community Newsletter -------------- next part -------------- An HTML attachment was scrubbed... URL: From tomascohen at gmail.com Wed Dec 14 14:51:57 2022 From: tomascohen at gmail.com (Tomas Cohen Arazi) Date: Wed, 14 Dec 2022 10:51:57 -0300 Subject: [Koha-devel] Jenkins notifications Message-ID: Hi all, I've (finally) restored jenkins notifications, and I plan to send failure notifications to: - Release maintainers - QA team Developers whose commits are suspected to have broken the tests will still be notified. I created two forwarding addresses to simplify notifications as well. -- TomĂĄs Cohen Arazi Theke Solutions (http://theke.io) ✆ +54 9351 3513384 GPG: B2F3C15F -------------- next part -------------- An HTML attachment was scrubbed... URL: From fridolin.somers at biblibre.com Thu Dec 15 07:46:54 2022 From: fridolin.somers at biblibre.com (Fridolin SOMERS) Date: Wed, 14 Dec 2022 20:46:54 -1000 Subject: [Koha-devel] Jenkins notifications In-Reply-To: References: Message-ID: <9a6e229a-bc40-6547-5e24-64147b368ff7@biblibre.com> OK great ;) Le 14/12/2022 Ă  03:51, Tomas Cohen Arazi a Ă©crit : > Hi all, I've (finally) restored jenkins notifications, and I plan to > send failure notifications to: > > - Release maintainers > - QA team > > Developers whose commits are suspected to have broken the tests will > still be notified. > > I created two forwarding addresses to simplify notifications as well. > > -- > TomĂĄs Cohen Arazi > Theke Solutions (http://theke.io ) > ✆ +54 9351 3513384 > GPG: B2F3C15F > > _______________________________________________ > Koha-devel mailing list > Koha-devel at lists.koha-community.org > https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-devel > website : https://www.koha-community.org/ > git : https://git.koha-community.org/ > bugs : https://bugs.koha-community.org/ -- Fridolin SOMERS Software and system maintainer 🩄 BibLibre, France From tomascohen at gmail.com Mon Dec 19 18:22:22 2022 From: tomascohen at gmail.com (Tomas Cohen Arazi) Date: Mon, 19 Dec 2022 14:22:22 -0300 Subject: [Koha-devel] ktd - enhancements Message-ID: Hi all, just wanted to let you know that we solved a long standing issue in KTD [1] that made it awkward to switch from using ES or not with KTD (required tweaking the .env file and would break if things didn't match). The `ktd` command now takes advantage of this enhancement so, as a reminder, if you want to run KTD with ES 7 [2] you just need to run: $ ktd --es7 up Any questions about it, just ask on IRC or on this thread. Best regards [1] https://gitlab.com/koha-community/koha-testing-docker/-/issues/306 [2] ktd --help will show you all the available options -- TomĂĄs Cohen Arazi Theke Solutions (http://theke.io) ✆ +54 9351 3513384 GPG: B2F3C15F -------------- next part -------------- An HTML attachment was scrubbed... URL: From dcook at prosentient.com.au Mon Dec 19 23:58:30 2022 From: dcook at prosentient.com.au (David Cook) Date: Tue, 20 Dec 2022 09:58:30 +1100 Subject: [Koha-devel] HOLDPLACED should be emailed to item branch rather than patron branch Message-ID: <008101d913fd$69ed18e0$3dc74aa0$@prosentient.com.au> Hi all, I had some librarians raise to me that the HOLDPLACED email goes to the patron branch rather than the item branch. I've raised https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=32499 to address this, but can anyone think of a reason why it should stay the patron branch and not the item branch? David Cook Senior Software Engineer Prosentient Systems Suite 7.03 6a Glen St Milsons Point NSW 2061 Australia Office: 02 9212 0899 Online: 02 8005 0595 -------------- next part -------------- An HTML attachment was scrubbed... URL: From dcook at prosentient.com.au Tue Dec 20 02:35:16 2022 From: dcook at prosentient.com.au (David Cook) Date: Tue, 20 Dec 2022 12:35:16 +1100 Subject: [Koha-devel] Added Testing SSO guide to wiki Message-ID: <009e01d91413$55e119d0$01a34d70$@prosentient.com.au> Hi all, At Martin's request at https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=31651#c3 I've put together a wiki page for testing single sign on (SSO) using Keycloak in koha-testing-docker: https://wiki.koha-community.org/wiki/Testing_SSO#SAML It's a first draft, so it might not be perfect, but I was able to use Keycloak to log into Koha using OpenID Connect and SAML while writing the guides, so it should work. Note that the instructions are just for testing. They're not production instructions. The intention is just to provide this for Koha developers, so that everyone can easily work on Koha SSO patches without being a SSO expert or without having commercial 3rd party SSO services available. I hope people find it useful! Let me know if you have any feedback on how it can improved. (Note: I already know the formatting is awful, but I was in a hurry and I'm not familiar enough with Mediawiki to format it better. Maybe someone can give a hand there?) David Cook Senior Software Engineer Prosentient Systems Suite 7.03 6a Glen St Milsons Point NSW 2061 Australia Office: 02 9212 0899 Online: 02 8005 0595 -------------- next part -------------- An HTML attachment was scrubbed... URL: From philippe.blouin at inlibro.com Tue Dec 20 20:13:04 2022 From: philippe.blouin at inlibro.com (Philippe Blouin) Date: Tue, 20 Dec 2022 14:13:04 -0500 Subject: [Koha-devel] The many failings of background_jobs_worker.pl Message-ID: Howdy! Since moving a lot of our users to 22.05.06, we've installed the worker everywhere.  But the number of issues encountered is staggering. The first one was Can't call method "process" on an undefined value where the id received from MQ was not found in the DB, and the process is going straight to process_job and failing.  Absolutely no idea how that occurs, seems completely counterintuitive (the ID comes from the DB after all), but here it is.  Hacked the code to add a "sleep 1" to fix most of that one. Then came the fact that stored events were not checked if the connection to MQ was successful at startup.  Bug 30654 refers it. Hacked a little "$init" in there to clear that up at startup. Then came the malformed UTF-8 character in JSON string, at character offset 296 (before "\x{e9}serv\x{e9} au ...") at decode_json that crashes the whole process.  And for some reason, it never gets over it, gets the same problem at every restart, like the event is never "eaten" from the queue.  Hacked an eval then a try-catch over it... After coding a monitor to alert when a background_jobs has been "new" over 5 minutes in the DB, I was inundated by messages. There's alway one elasticsearch_update that escapes among the flurry, and they slowly add up. At this point, the only viable solution is to run the workers but disable RabbitMQ everywhere.  Are we really the only ones experiencing that? Regards, PS Our servers are well-above-average Debian 11 machines with lot of firepower (ram, cpu, i/o...). -- Philippe Blouin, Directeur de la technologie TĂ©l.  : (833) 465-4276, poste 230 philippe.blouin at inLibro.com inLibro | pour esprit libre | www.inLibro.com -------------- next part -------------- An HTML attachment was scrubbed... URL: From michael.hafen at washk12.org Tue Dec 20 21:59:03 2022 From: michael.hafen at washk12.org (Michael Hafen) Date: Tue, 20 Dec 2022 13:59:03 -0700 Subject: [Koha-devel] The many failings of background_jobs_worker.pl In-Reply-To: References: Message-ID: I've also been bitten by the malformed JSON string error. On Tue, Dec 20, 2022 at 12:13 PM Philippe Blouin < philippe.blouin at inlibro.com> wrote: > Howdy! > > Since moving a lot of our users to 22.05.06, we've installed the worker > everywhere. But the number of issues encountered is staggering. > > The first one was > > Can't call method "process" on an undefined value > > where the id received from MQ was not found in the DB, and the process is > going straight to process_job and failing. Absolutely no idea how that > occurs, seems completely counterintuitive (the ID comes from the DB after > all), but here it is. Hacked the code to add a "sleep 1" to fix most of > that one. > > Then came the fact that stored events were not checked if the connection > to MQ was successful at startup. Bug 30654 refers it. Hacked a little > "$init" in there to clear that up at startup. > > Then came the > > malformed UTF-8 character in JSON string, at character offset 296 (before > "\x{e9}serv\x{e9} au ...") > > at decode_json that crashes the whole process. And for some reason, it > never gets over it, gets the same problem at every restart, like the event > is never "eaten" from the queue. Hacked an eval then a try-catch over it... > > After coding a monitor to alert when a background_jobs has been "new" over > 5 minutes in the DB, I was inundated by messages. There's alway one > elasticsearch_update that escapes among the flurry, and they slowly add up. > > At this point, the only viable solution is to run the workers but disable > RabbitMQ everywhere. Are we really the only ones experiencing that? > > Regards, > > PS Our servers are well-above-average Debian 11 machines with lot of > firepower (ram, cpu, i/o...). > -- > Philippe Blouin, > Directeur de la technologie > > TĂ©l. : (833) 465-4276, poste 230 > philippe.blouin at inLibro.com > inLibro | pour esprit libre | www.inLibro.com > _______________________________________________ > Koha-devel mailing list > Koha-devel at lists.koha-community.org > https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-devel > website : https://www.koha-community.org/ > git : https://git.koha-community.org/ > bugs : https://bugs.koha-community.org/ > -- Michael Hafen Washington County School District Technology Department Systems Analyst -------------- next part -------------- An HTML attachment was scrubbed... URL: From dcook at prosentient.com.au Tue Dec 20 23:55:56 2022 From: dcook at prosentient.com.au (David Cook) Date: Wed, 21 Dec 2022 09:55:56 +1100 Subject: [Koha-devel] The many failings of background_jobs_worker.pl In-Reply-To: References: Message-ID: <010c01d914c6$3b1b3820$b151a860$@prosentient.com.au> Salut Philippe, That first issue should’ve been resolved in 22.05.00 by https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=30172. I haven’t had any problems like that since applying that patch. Are you running Koha with or without customizations? As you say, bug 30654 discusses that second issue. And I obviously have my own opinion on that one 😉. That JSON issue should be fixed by Bug 31351 in Koha 22.05.06 as well I believe: https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=31351 -- The only issue I’ve had with the background jobs has been the one covered by Bug 30172. Otherwise, it’s been all fine for me, although I use Zebra rather than Elasticsearch. I think part of the reason I haven’t had issues is that I haven’t had many people using the background jobs either though. I’m actually planning on writing a background job system based on RabbitMQ for a different non-Koha system. The main difference is that I’ll reject or fail tasks where messages aren’t sent to RabbitMQ. I think that’ll make my system a bit more robust than Koha’s. The problem with the background jobs at the moment is that we haven’t fully committed to RabbitMQ. We’re trying to do this weird hybrid with the database fallback which is not the right direction in my mind. We should do one or the other but not try to do both. But that’s just my 2 cents. David Cook Senior Software Engineer Prosentient Systems Suite 7.03 6a Glen St Milsons Point NSW 2061 Australia Office: 02 9212 0899 Online: 02 8005 0595 From: Koha-devel On Behalf Of Philippe Blouin Sent: Wednesday, 21 December 2022 6:13 AM To: koha-devel at lists.koha-community.org Subject: [Koha-devel] The many failings of background_jobs_worker.pl Howdy! Since moving a lot of our users to 22.05.06, we've installed the worker everywhere. But the number of issues encountered is staggering. The first one was Can't call method "process" on an undefined value where the id received from MQ was not found in the DB, and the process is going straight to process_job and failing. Absolutely no idea how that occurs, seems completely counterintuitive (the ID comes from the DB after all), but here it is. Hacked the code to add a "sleep 1" to fix most of that one. Then came the fact that stored events were not checked if the connection to MQ was successful at startup. Bug 30654 refers it. Hacked a little "$init" in there to clear that up at startup. Then came the malformed UTF-8 character in JSON string, at character offset 296 (before "\x{e9}serv\x{e9} au ...") at decode_json that crashes the whole process. And for some reason, it never gets over it, gets the same problem at every restart, like the event is never "eaten" from the queue. Hacked an eval then a try-catch over it... After coding a monitor to alert when a background_jobs has been "new" over 5 minutes in the DB, I was inundated by messages. There's alway one elasticsearch_update that escapes among the flurry, and they slowly add up. At this point, the only viable solution is to run the workers but disable RabbitMQ everywhere. Are we really the only ones experiencing that? Regards, PS Our servers are well-above-average Debian 11 machines with lot of firepower (ram, cpu, i/o...). -- Philippe Blouin, Directeur de la technologie TĂ©l. : (833) 465-4276, poste 230 philippe.blouin at inLibro.com inLibro | pour esprit libre | www.inLibro.com -------------- next part -------------- An HTML attachment was scrubbed... URL: From mail at davidschmidt.at Wed Dec 21 09:48:10 2022 From: mail at davidschmidt.at (David Schmidt) Date: Wed, 21 Dec 2022 09:48:10 +0100 Subject: [Koha-devel] The many failings of background_jobs_worker.pl In-Reply-To: References: Message-ID: jup, the JSON bug hit us too we also encountered some problems regarding rabbitmq/workers debugging wasn't straigthforward because I had a hard time to find meaningful debug output. luckily the folks on IRC were very helpful as usual. the latest bug (not sure if it is a bug as of yet) was that the indexing job for manually created biblios wasnt triggered unless zebra was running (we use elasticsearch) overall my confidence in the implementation quality of the rabbitmq/worker feature isnt overwhelming. that's not a complaint and I dont mean to step on anyones toes, I know that it's OSS and thus up to me to put in the work if im unhappy. and as always there is the possibility that I did something wrong and the software is perfectly fine. :) cheers david On Tue, 20 Dec 2022, at 8:13 PM, Philippe Blouin wrote: > Howdy! > > Since moving a lot of our users to 22.05.06, we've installed the worker everywhere. But the number of issues encountered is staggering. > > The first one was > > Can't call method "process" on an undefined value > > where the id received from MQ was not found in the DB, and the process is going straight to process_job and failing. Absolutely no idea how that occurs, seems completely counterintuitive (the ID comes from the DB after all), but here it is. Hacked the code to add a "sleep 1" to fix most of that one. > > Then came the fact that stored events were not checked if the connection to MQ was successful at startup. Bug 30654 refers it. Hacked a little "$init" in there to clear that up at startup. > > Then came the > > malformed UTF-8 character in JSON string, at character offset 296 (before "\x{e9}serv\x{e9} au ...") > > at decode_json that crashes the whole process. And for some reason, it never gets over it, gets the same problem at every restart, like the event is never "eaten" from the queue. Hacked an eval then a try-catch over it... > > After coding a monitor to alert when a background_jobs has been "new" over 5 minutes in the DB, I was inundated by messages. There's alway one elasticsearch_update that escapes among the flurry, and they slowly add up. > > At this point, the only viable solution is to run the workers but disable RabbitMQ everywhere. Are we really the only ones experiencing that? > > Regards, > > PS Our servers are well-above-average Debian 11 machines with lot of firepower (ram, cpu, i/o...). > > -- > > Philippe Blouin, > Directeur de la technologie > > > TĂ©l. : (833) 465-4276, poste 230 > philippe.blouin at inLibro.com > > inLibro | pour esprit libre | www.inLibro.com > _______________________________________________ > Koha-devel mailing list > Koha-devel at lists.koha-community.org > https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-devel > website : https://www.koha-community.org/ > git : https://git.koha-community.org/ > bugs : https://bugs.koha-community.org/ > -------------- next part -------------- An HTML attachment was scrubbed... URL: From philippe.blouin at inlibro.com Wed Dec 21 14:58:40 2022 From: philippe.blouin at inlibro.com (Philippe Blouin) Date: Wed, 21 Dec 2022 08:58:40 -0500 Subject: [Koha-devel] The many failings of background_jobs_worker.pl In-Reply-To: <010c01d914c6$3b1b3820$b151a860$@prosentient.com.au> References: <010c01d914c6$3b1b3820$b151a860$@prosentient.com.au> Message-ID: <7565bc5a-666e-8f43-9cbc-3f5003c59fcc@inlibro.com> Good evening, David, Thanks for the response.  Yours and David's and Michael's.  I feel less alone... I validated, and yes all the patches you refer are in our pile. And until the problems arose, there were no customizations around that code. So yeah, even at 22.05.06, I get the JSON error and the race condition (we use ES).  And the _abandonned_ children.  So I surmise, or dare I say postulate, that those issues are not as resolved as some would presume. I will revert background_jobs_worker.pl to its default, and shutdown MQ everywhere, for now.  :( Philippe Blouin, Directeur de la technologie TĂ©l.  : (833) 465-4276, poste 230 philippe.blouin at inLibro.com inLibro | pour esprit libre | www.inLibro.com On 2022-12-20 17:55, David Cook wrote: > > Salut Philippe, > > That first issue should’ve been resolved in 22.05.00 by > https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=30172. I > haven’t had any problems like that since applying that patch. Are you > running Koha with or without customizations? > > As you say, bug 30654 discusses that second issue. And I obviously > have my own opinion on that one 😉. > > That JSON issue should be fixed by Bug 31351 in Koha 22.05.06 as well > I believe: https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=31351 > > -- > > The only issue I’ve had with the background jobs has been the one > covered by Bug 30172. Otherwise, it’s been all fine for me, although I > use Zebra rather than Elasticsearch. I think part of the reason I > haven’t had issues is that I haven’t had many people using the > background jobs either though. > > I’m actually planning on writing a background job system based on > RabbitMQ for a different non-Koha system. The main difference is that > I’ll reject or fail tasks where messages aren’t sent to RabbitMQ. I > think that’ll make my system a bit more robust than Koha’s. > > The problem with the background jobs at the moment is that we haven’t > fully committed to RabbitMQ. We’re trying to do this weird hybrid with > the database fallback which is not the right direction in my mind. We > should do one or the other but not try to do both. > > But that’s just my 2 cents. > > David Cook > > Senior Software Engineer > > Prosentient Systems > > Suite 7.03 > > 6a Glen St > > Milsons Point NSW 2061 > > Australia > > Office: 02 9212 0899 > > Online: 02 8005 0595 > > *From:*Koha-devel *On > Behalf Of *Philippe Blouin > *Sent:* Wednesday, 21 December 2022 6:13 AM > *To:* koha-devel at lists.koha-community.org > *Subject:* [Koha-devel] The many failings of background_jobs_worker.pl > > Howdy! > > Since moving a lot of our users to 22.05.06, we've installed the > worker everywhere.  But the number of issues encountered is staggering. > > The first one was > > Can't call method "process" on an undefined value > > where the id received from MQ was not found in the DB, and the process > is going straight to process_job and failing. Absolutely no idea how > that occurs, seems completely counterintuitive (the ID comes from the > DB after all), but here it is.  Hacked the code to add a "sleep 1" to > fix most of that one. > > Then came the fact that stored events were not checked if the > connection to MQ was successful at startup.  Bug 30654 refers it.  > Hacked a little "$init" in there to clear that up at startup. > > Then came the > > malformed UTF-8 character in JSON string, at character offset 296 > (before "\x{e9}serv\x{e9} au ...") > > at decode_json that crashes the whole process.  And for some reason, > it never gets over it, gets the same problem at every restart, like > the event is never "eaten" from the queue. Hacked an eval then a > try-catch over it... > > After coding a monitor to alert when a background_jobs has been "new" > over 5 minutes in the DB, I was inundated by messages.  There's alway > one elasticsearch_update that escapes among the flurry, and they > slowly add up. > > At this point, the only viable solution is to run the workers but > disable RabbitMQ everywhere.  Are we really the only ones experiencing > that? > > Regards, > > PS Our servers are well-above-average Debian 11 machines with lot of > firepower (ram, cpu, i/o...). > > -- > > Philippe Blouin, > Directeur de la technologie > > TĂ©l.  : (833) 465-4276, poste 230 > philippe.blouin at inLibro.com > > inLibro| pour esprit libre |www.inLibro.com > -------------- next part -------------- An HTML attachment was scrubbed... URL: From philippe.blouin at inlibro.com Wed Dec 21 15:04:16 2022 From: philippe.blouin at inlibro.com (Philippe Blouin) Date: Wed, 21 Dec 2022 09:04:16 -0500 Subject: [Koha-devel] The many failings of background_jobs_worker.pl In-Reply-To: <7565bc5a-666e-8f43-9cbc-3f5003c59fcc@inlibro.com> References: <010c01d914c6$3b1b3820$b151a860$@prosentient.com.au> <7565bc5a-666e-8f43-9cbc-3f5003c59fcc@inlibro.com> Message-ID: <739868d8-0f6f-4827-3599-a113860ce68c@inlibro.com> Although I precise, Cannot connect to broker Failed to connect: Error connecting to localhost:61613: Connection refused at /usr/share/perl5/Net/Stomp.pm line 27.; giving up at /usr/share/perl5/Net/Stomp.pm line 27. So shutting down MQ has its own issues.... Philippe Blouin, Directeur de la technologie TĂ©l.  : (833) 465-4276, poste 230 philippe.blouin at inLibro.com inLibro | pour esprit libre | www.inLibro.com On 2022-12-21 08:58, Philippe Blouin wrote: > > Good evening, David, > > Thanks for the response.  Yours and David's and Michael's.  I feel > less alone... > > I validated, and yes all the patches you refer are in our pile.  And > until the problems arose, there were no customizations around that code. > > So yeah, even at 22.05.06, I get the JSON error and the race condition > (we use ES).  And the _abandonned_ children.  So I surmise, or dare I > say postulate, that those issues are not as resolved as some would > presume. > > I will revert background_jobs_worker.pl to its default, and shutdown > MQ everywhere, for now.  :( > > Philippe Blouin, > Directeur de la technologie > > TĂ©l.  : (833) 465-4276, poste 230 > philippe.blouin at inLibro.com > > inLibro | pour esprit libre | www.inLibro.com > On 2022-12-20 17:55, David Cook wrote: >> >> Salut Philippe, >> >> That first issue should’ve been resolved in 22.05.00 by >> https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=30172. I >> haven’t had any problems like that since applying that patch. Are you >> running Koha with or without customizations? >> >> As you say, bug 30654 discusses that second issue. And I obviously >> have my own opinion on that one 😉. >> >> That JSON issue should be fixed by Bug 31351 in Koha 22.05.06 as well >> I believe: >> https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=31351 >> >> -- >> >> The only issue I’ve had with the background jobs has been the one >> covered by Bug 30172. Otherwise, it’s been all fine for me, although >> I use Zebra rather than Elasticsearch. I think part of the reason I >> haven’t had issues is that I haven’t had many people using the >> background jobs either though. >> >> I’m actually planning on writing a background job system based on >> RabbitMQ for a different non-Koha system. The main difference is that >> I’ll reject or fail tasks where messages aren’t sent to RabbitMQ. I >> think that’ll make my system a bit more robust than Koha’s. >> >> The problem with the background jobs at the moment is that we haven’t >> fully committed to RabbitMQ. We’re trying to do this weird hybrid >> with the database fallback which is not the right direction in my >> mind. We should do one or the other but not try to do both. >> >> But that’s just my 2 cents. >> >> David Cook >> >> Senior Software Engineer >> >> Prosentient Systems >> >> Suite 7.03 >> >> 6a Glen St >> >> Milsons Point NSW 2061 >> >> Australia >> >> Office: 02 9212 0899 >> >> Online: 02 8005 0595 >> >> *From:*Koha-devel *On >> Behalf Of *Philippe Blouin >> *Sent:* Wednesday, 21 December 2022 6:13 AM >> *To:* koha-devel at lists.koha-community.org >> *Subject:* [Koha-devel] The many failings of background_jobs_worker.pl >> >> Howdy! >> >> Since moving a lot of our users to 22.05.06, we've installed the >> worker everywhere.  But the number of issues encountered is staggering. >> >> The first one was >> >> Can't call method "process" on an undefined value >> >> where the id received from MQ was not found in the DB, and the >> process is going straight to process_job and failing. Absolutely no >> idea how that occurs, seems completely counterintuitive (the ID comes >> from the DB after all), but here it is.  Hacked the code to add a >> "sleep 1" to fix most of that one. >> >> Then came the fact that stored events were not checked if the >> connection to MQ was successful at startup.  Bug 30654 refers it.  >> Hacked a little "$init" in there to clear that up at startup. >> >> Then came the >> >> malformed UTF-8 character in JSON string, at character offset 296 >> (before "\x{e9}serv\x{e9} au ...") >> >> at decode_json that crashes the whole process.  And for some reason, >> it never gets over it, gets the same problem at every restart, like >> the event is never "eaten" from the queue.  Hacked an eval then a >> try-catch over it... >> >> After coding a monitor to alert when a background_jobs has been "new" >> over 5 minutes in the DB, I was inundated by messages.  There's alway >> one elasticsearch_update that escapes among the flurry, and they >> slowly add up. >> >> At this point, the only viable solution is to run the workers but >> disable RabbitMQ everywhere.  Are we really the only ones >> experiencing that? >> >> Regards, >> >> PS Our servers are well-above-average Debian 11 machines with lot of >> firepower (ram, cpu, i/o...). >> >> -- >> >> Philippe Blouin, >> Directeur de la technologie >> >> TĂ©l.  : (833) 465-4276, poste 230 >> philippe.blouin at inLibro.com >> >> inLibro| pour esprit libre |www.inLibro.com >> -------------- next part -------------- An HTML attachment was scrubbed... URL: From mail at davidschmidt.at Wed Dec 21 15:15:26 2022 From: mail at davidschmidt.at (David Schmidt) Date: Wed, 21 Dec 2022 15:15:26 +0100 Subject: [Koha-devel] The many failings of background_jobs_worker.pl In-Reply-To: <739868d8-0f6f-4827-3599-a113860ce68c@inlibro.com> References: <010c01d914c6$3b1b3820$b151a860$@prosentient.com.au> <7565bc5a-666e-8f43-9cbc-3f5003c59fcc@inlibro.com> <739868d8-0f6f-4827-3599-a113860ce68c@inlibro.com> Message-ID: <62e09dba-3aef-41db-b9b8-bd0909df6acf@app.fastmail.com> I dont think you can shut down rabbitmq and expect things to work. rabbitmq needs to be running and this command should return 4 processes (at least it does on our systems) ``ps ax|grep 'background_jobs_worker\.pl.*--queue.*\(default\|long_tasks\)'`` 1016012 ? S 0:00 daemon --name=inlibro-koha-worker --errlog=/var/log/koha/inlibro/worker-error.log --stdout=/var/log/koha/inlibro/worker.log --output=/var/log/koha/inlibro/worker-output.log --pidfiles=/var/run/koha/inlibro/ --verbose=1 --respawn --delay=30 --user=inlibro-koha.inlibro-koha -- /usr/share/koha/bin/background_jobs_worker.pl --queue default 1016014 ? S 0:37 /usr/bin/perl /usr/share/koha/bin/background_jobs_worker.pl --queue default 1016040 ? S 0:00 daemon --name=inlibro-koha-worker-long_tasks --errlog=/var/log/koha/inlibro/worker-error.log --stdout=/var/log/koha/inlibro/worker.log --output=/var/log/koha/inlibro/worker-output.log --pidfiles=/var/run/koha/inlibro/ --verbose=1 --respawn --delay=30 --user=inlibro-koha.inlibro-koha -- /usr/share/koha/bin/background_jobs_worker.pl --queue long_tasks 1016042 ? S 0:01 /usr/bin/perl /usr/share/koha/bin/background_jobs_worker.pl --queue long_tasks `` regards david On Wed, 21 Dec 2022, at 3:04 PM, Philippe Blouin wrote: > Although I precise, > > Cannot connect to broker Failed to connect: Error connecting to localhost:61613: Connection refused at /usr/share/perl5/Net/Stomp.pm line 27.; giving up at /usr/share/perl5/Net/Stomp.pm line 27. > > > So shutting down MQ has its own issues.... > > Philippe Blouin, > Directeur de la technologie > > > TĂ©l. : (833) 465-4276, poste 230 > philippe.blouin at inLibro.com > > inLibro | pour esprit libre | www.inLibro.com > On 2022-12-21 08:58, Philippe Blouin wrote: >> Good evening, David, >> >> Thanks for the response. Yours and David's and Michael's. I feel less alone... >> >> I validated, and yes all the patches you refer are in our pile. And until the problems arose, there were no customizations around that code. >> >> So yeah, even at 22.05.06, I get the JSON error and the race condition (we use ES). And the _abandonned_ children. So I surmise, or dare I say postulate, that those issues are not as resolved as some would presume. >> >> I will revert background_jobs_worker.pl to its default, and shutdown MQ everywhere, for now. :( >> >> Philippe Blouin, >> Directeur de la technologie >> >> >> TĂ©l. : (833) 465-4276, poste 230 >> philippe.blouin at inLibro.com >> >> inLibro | pour esprit libre | www.inLibro.com >> On 2022-12-20 17:55, David Cook wrote: >>> Salut Philippe, >>> >>> That first issue should’ve been resolved in 22.05.00 by https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=30172. I haven’t had any problems like that since applying that patch. Are you running Koha with or without customizations? >>> >>> As you say, bug 30654 discusses that second issue. And I obviously have my own opinion on that one 😉. >>> >>> That JSON issue should be fixed by Bug 31351 in Koha 22.05.06 as well I believe: https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=31351 >>> >>> -- >>> >>> The only issue I’ve had with the background jobs has been the one covered by Bug 30172. Otherwise, it’s been all fine for me, although I use Zebra rather than Elasticsearch. I think part of the reason I haven’t had issues is that I haven’t had many people using the background jobs either though. >>> >>> I’m actually planning on writing a background job system based on RabbitMQ for a different non-Koha system. The main difference is that I’ll reject or fail tasks where messages aren’t sent to RabbitMQ. I think that’ll make my system a bit more robust than Koha’s. >>> >>> The problem with the background jobs at the moment is that we haven’t fully committed to RabbitMQ. We’re trying to do this weird hybrid with the database fallback which is not the right direction in my mind. We should do one or the other but not try to do both. >>> >>> But that’s just my 2 cents. >>> >>> David Cook >>> Senior Software Engineer >>> Prosentient Systems >>> Suite 7.03 >>> 6a Glen St >>> Milsons Point NSW 2061 >>> Australia >>> >>> Office: 02 9212 0899 >>> Online: 02 8005 0595 >>> >>> *From:* Koha-devel *On Behalf Of *Philippe Blouin >>> *Sent:* Wednesday, 21 December 2022 6:13 AM >>> *To:* koha-devel at lists.koha-community.org >>> *Subject:* [Koha-devel] The many failings of background_jobs_worker.pl >>> >>> Howdy! >>> >>> Since moving a lot of our users to 22.05.06, we've installed the worker everywhere. But the number of issues encountered is staggering. >>> >>> The first one was >>> >>> Can't call method "process" on an undefined value >>> >>> where the id received from MQ was not found in the DB, and the process is going straight to process_job and failing. Absolutely no idea how that occurs, seems completely counterintuitive (the ID comes from the DB after all), but here it is. Hacked the code to add a "sleep 1" to fix most of that one. >>> >>> Then came the fact that stored events were not checked if the connection to MQ was successful at startup. Bug 30654 refers it. Hacked a little "$init" in there to clear that up at startup. >>> >>> Then came the >>> >>> malformed UTF-8 character in JSON string, at character offset 296 (before "\x{e9}serv\x{e9} au ...") >>> >>> at decode_json that crashes the whole process. And for some reason, it never gets over it, gets the same problem at every restart, like the event is never "eaten" from the queue. Hacked an eval then a try-catch over it... >>> >>> After coding a monitor to alert when a background_jobs has been "new" over 5 minutes in the DB, I was inundated by messages. There's alway one elasticsearch_update that escapes among the flurry, and they slowly add up. >>> >>> At this point, the only viable solution is to run the workers but disable RabbitMQ everywhere. Are we really the only ones experiencing that? >>> >>> Regards, >>> >>> PS Our servers are well-above-average Debian 11 machines with lot of firepower (ram, cpu, i/o...). >>> >>> -- >>> Philippe Blouin, >>> Directeur de la technologie >>> TĂ©l. : (833) 465-4276, poste 230 >>> philippe.blouin at inLibro.com >>> >>> inLibro | pour esprit libre | www.inLibro.com > _______________________________________________ > Koha-devel mailing list > Koha-devel at lists.koha-community.org > https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-devel > website : https://www.koha-community.org/ > git : https://git.koha-community.org/ > bugs : https://bugs.koha-community.org/ > -------------- next part -------------- An HTML attachment was scrubbed... URL: From philippe.blouin at inlibro.com Wed Dec 21 17:53:35 2022 From: philippe.blouin at inlibro.com (Philippe Blouin) Date: Wed, 21 Dec 2022 11:53:35 -0500 Subject: [Koha-devel] The many failings of background_jobs_worker.pl In-Reply-To: <62e09dba-3aef-41db-b9b8-bd0909df6acf@app.fastmail.com> References: <010c01d914c6$3b1b3820$b151a860$@prosentient.com.au> <7565bc5a-666e-8f43-9cbc-3f5003c59fcc@inlibro.com> <739868d8-0f6f-4827-3599-a113860ce68c@inlibro.com> <62e09dba-3aef-41db-b9b8-bd0909df6acf@app.fastmail.com> Message-ID: <4ff9a7fc-3516-69f4-c45a-2a83ec46e3f7@inlibro.com> My understanding is that all tasks go to the DB right now, so shutting down MQ doesn't lose anything. Right now. But having MQ running _does_ lose me some things, since not all tasks are processed, and the process can even get stuck in limbo with the UTF8 issue. With tens of installations to monitor, some putting thousands of jobs on the queue each day, I'm really looking for a quick fix before the Holidays... Philippe Blouin, Directeur de la technologie TĂ©l.  : (833) 465-4276, poste 230 philippe.blouin at inLibro.com inLibro | pour esprit libre | www.inLibro.com On 2022-12-21 09:15, David Schmidt wrote: > I dont think you can shut down rabbitmq and expect things to work. > > rabbitmq needs to be running and this command should return 4 > processes (at least it does on our systems) > > `|ps ax|grep > 'background_jobs_worker\.pl.*--queue.*\(default\|long_tasks\)'`| > 1016012 ?        S      0:00 daemon --name=inlibro-koha-worker > --errlog=/var/log/koha/inlibro/worker-error.log > --stdout=/var/log/koha/inlibro/worker.log > --output=/var/log/koha/inlibro/worker-output.log > --pidfiles=/var/run/koha/inlibro/ --verbose=1 --respawn --delay=30 > --user=inlibro-koha.inlibro-koha -- > /usr/share/koha/bin/background_jobs_worker.pl --queue default > 1016014 ?        S      0:37 /usr/bin/perl > /usr/share/koha/bin/background_jobs_worker.pl --queue default > 1016040 ?        S      0:00 daemon > --name=inlibro-koha-worker-long_tasks > --errlog=/var/log/koha/inlibro/worker-error.log > --stdout=/var/log/koha/inlibro/worker.log > --output=/var/log/koha/inlibro/worker-output.log > --pidfiles=/var/run/koha/inlibro/ --verbose=1 --respawn --delay=30 > --user=inlibro-koha.inlibro-koha -- > /usr/share/koha/bin/background_jobs_worker.pl --queue long_tasks > 1016042 ?        S      0:01 /usr/bin/perl > /usr/share/koha/bin/background_jobs_worker.pl --queue long_tasks > || > > > regards > david > > On Wed, 21 Dec 2022, at 3:04 PM, Philippe Blouin wrote: >> >> Although I precise, >> >> Cannot connect to broker Failed to connect: Error connecting to localhost:61613: Connection refused at /usr/share/perl5/Net/Stomp.pm line 27.; giving up at /usr/share/perl5/Net/Stomp.pm line 27. >> >> >> So shutting down MQ has its own issues.... >> >> Philippe Blouin, >> Directeur de la technologie >> >> TĂ©l.  : (833) 465-4276, poste 230 >> philippe.blouin at inLibro.com >> inLibro | pour esprit libre | www.inLibro.com >> On 2022-12-21 08:58, Philippe Blouin wrote: >>> >>> Good evening, David, >>> >>> Thanks for the response.  Yours and David's and Michael's. I feel >>> less alone... >>> >>> I validated, and yes all the patches you refer are in our pile.  And >>> until the problems arose, there were no customizations around that >>> code. >>> >>> So yeah, even at 22.05.06, I get the JSON error and the race >>> condition (we use ES).  And the _abandonned_ children.  So I >>> surmise, or dare I say postulate, that those issues are not as >>> resolved as some would presume. >>> >>> I will revert background_jobs_worker.pl to its default, and shutdown >>> MQ everywhere, for now.  :( >>> >>> Philippe Blouin, >>> Directeur de la technologie >>> >>> TĂ©l.  : (833) 465-4276, poste 230 >>> philippe.blouin at inLibro.com >>> inLibro | pour esprit libre | www.inLibro.com >>> On 2022-12-20 17:55, David Cook wrote: >>>> >>>> Salut Philippe, >>>> >>>> >>>> That first issue should’ve been resolved in 22.05.00 by >>>> https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=30172 >>>> . >>>> I haven’t had any problems like that since applying that patch. Are >>>> you running Koha with or without customizations? >>>> >>>> >>>> As you say, bug 30654 discusses that second issue. And I obviously >>>> have my own opinion on that one 😉. >>>> >>>> >>>> That JSON issue should be fixed by Bug 31351 in Koha 22.05.06 as >>>> well I believe: >>>> https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=31351 >>>> >>>> >>>> >>>> -- >>>> >>>> >>>> The only issue I’ve had with the background jobs has been the one >>>> covered by Bug 30172. Otherwise, it’s been all fine for me, >>>> although I use Zebra rather than Elasticsearch. I think part of the >>>> reason I haven’t had issues is that I haven’t had many people using >>>> the background jobs either though. >>>> >>>> >>>> I’m actually planning on writing a background job system based on >>>> RabbitMQ for a different non-Koha system. The main difference is >>>> that I’ll reject or fail tasks where messages aren’t sent to >>>> RabbitMQ. I think that’ll make my system a bit more robust than Koha’s. >>>> >>>> >>>> The problem with the background jobs at the moment is that we >>>> haven’t fully committed to RabbitMQ. We’re trying to do this weird >>>> hybrid with the database fallback which is not the right direction >>>> in my mind. We should do one or the other but not try to do both. >>>> >>>> >>>> But that’s just my 2 cents. >>>> >>>> >>>> David Cook >>>> >>>> Senior Software Engineer >>>> >>>> Prosentient Systems >>>> >>>> Suite 7.03 >>>> >>>> 6a Glen St >>>> >>>> Milsons Point NSW 2061 >>>> >>>> Australia >>>> >>>> >>>> Office: 02 9212 0899 >>>> >>>> Online: 02 8005 0595 >>>> >>>> >>>> *From:*Koha-devel >>>> *On Behalf Of >>>> *Philippe Blouin >>>> *Sent:* Wednesday, 21 December 2022 6:13 AM >>>> *To:* koha-devel at lists.koha-community.org >>>> >>>> *Subject:* [Koha-devel] The many failings of background_jobs_worker.pl >>>> >>>> >>>> Howdy! >>>> >>>> Since moving a lot of our users to 22.05.06, we've installed the >>>> worker everywhere.  But the number of issues encountered is staggering. >>>> >>>> The first one was >>>> >>>> Can't call method "process" on an undefined value >>>> >>>> where the id received from MQ was not found in the DB, and the >>>> process is going straight to process_job and failing.  Absolutely >>>> no idea how that occurs, seems completely counterintuitive (the ID >>>> comes from the DB after all), but here it is.  Hacked the code to >>>> add a "sleep 1" to fix most of that one. >>>> >>>> Then came the fact that stored events were not checked if the >>>> connection to MQ was successful at startup.  Bug 30654 refers it.  >>>> Hacked a little "$init" in there to clear that up at startup. >>>> >>>> Then came the >>>> >>>> malformed UTF-8 character in JSON string, at character offset 296 >>>> (before "\x{e9}serv\x{e9} au ...") >>>> >>>> at decode_json that crashes the whole process.  And for some >>>> reason, it never gets over it, gets the same problem at every >>>> restart, like the event is never "eaten" from the queue.  Hacked an >>>> eval then a try-catch over it... >>>> >>>> After coding a monitor to alert when a background_jobs has been >>>> "new" over 5 minutes in the DB, I was inundated by messages.  >>>> There's alway one elasticsearch_update that escapes among the >>>> flurry, and they slowly add up. >>>> >>>> At this point, the only viable solution is to run the workers but >>>> disable RabbitMQ everywhere.  Are we really the only ones >>>> experiencing that? >>>> >>>> Regards, >>>> >>>> PS Our servers are well-above-average Debian 11 machines with lot >>>> of firepower (ram, cpu, i/o...). >>>> >>>> -- >>>> >>>> Philippe Blouin, >>>> Directeur de la technologie >>>> >>>> TĂ©l.  : (833) 465-4276, poste 230 >>>> philippe.blouin at inLibro.com >>>> >>>> inLibro| pour esprit libre |www.inLibro.com >>>> >> _______________________________________________ >> Koha-devel mailing list >> Koha-devel at lists.koha-community.org >> https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-devel >> website : https://www.koha-community.org/ >> git : https://git.koha-community.org/ >> bugs : https://bugs.koha-community.org/ >> > > > _______________________________________________ > Koha-devel mailing list > Koha-devel at lists.koha-community.org > https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-devel > website :https://www.koha-community.org/ > git :https://git.koha-community.org/ > bugs :https://bugs.koha-community.org/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From fridolin.somers at biblibre.com Wed Dec 21 19:06:09 2022 From: fridolin.somers at biblibre.com (Fridolin SOMERS) Date: Wed, 21 Dec 2022 08:06:09 -1000 Subject: [Koha-devel] The many failings of background_jobs_worker.pl In-Reply-To: References: Message-ID: <332dbcf5-cd97-885a-fec5-6d71f9803f5c@biblibre.com> Hi, I think network performance is really important for RabbitMQ. We at Biblibre have a virtual machine in each server, to share between each virtual machine (one Koha per machine) but keep a good network performance. Looks to work well, but we are still in 21.11. Best regards, Le 20/12/2022 Ă  09:13, Philippe Blouin a Ă©crit : > Howdy! > > Since moving a lot of our users to 22.05.06, we've installed the worker > everywhere.  But the number of issues encountered is staggering. > > The first one was > > Can't call method "process" on an undefined value > > where the id received from MQ was not found in the DB, and the process > is going straight to process_job and failing.  Absolutely no idea how > that occurs, seems completely counterintuitive (the ID comes from the DB > after all), but here it is.  Hacked the code to add a "sleep 1" to fix > most of that one. > > Then came the fact that stored events were not checked if the connection > to MQ was successful at startup.  Bug 30654 refers it. Hacked a little > "$init" in there to clear that up at startup. > > Then came the > > malformed UTF-8 character in JSON string, at character offset 296 > (before "\x{e9}serv\x{e9} au ...") > > at decode_json that crashes the whole process.  And for some reason, it > never gets over it, gets the same problem at every restart, like the > event is never "eaten" from the queue.  Hacked an eval then a try-catch > over it... > > After coding a monitor to alert when a background_jobs has been "new" > over 5 minutes in the DB, I was inundated by messages. There's alway one > elasticsearch_update that escapes among the flurry, and they slowly add up. > > At this point, the only viable solution is to run the workers but > disable RabbitMQ everywhere.  Are we really the only ones experiencing that? > > Regards, > > PS Our servers are well-above-average Debian 11 machines with lot of > firepower (ram, cpu, i/o...). > > -- > Philippe Blouin, > Directeur de la technologie > > TĂ©l.  : (833) 465-4276, poste 230 > philippe.blouin at inLibro.com > > inLibro | pour esprit libre | www.inLibro.com > > _______________________________________________ > Koha-devel mailing list > Koha-devel at lists.koha-community.org > https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-devel > website : https://www.koha-community.org/ > git : https://git.koha-community.org/ > bugs : https://bugs.koha-community.org/ -- Fridolin SOMERS Software and system maintainer 🩄 BibLibre, France From tomascohen at gmail.com Wed Dec 21 20:52:34 2022 From: tomascohen at gmail.com (Tomas Cohen Arazi) Date: Wed, 21 Dec 2022 16:52:34 -0300 Subject: [Koha-devel] The many failings of background_jobs_worker.pl In-Reply-To: <7565bc5a-666e-8f43-9cbc-3f5003c59fcc@inlibro.com> References: <010c01d914c6$3b1b3820$b151a860$@prosentient.com.au> <7565bc5a-666e-8f43-9cbc-3f5003c59fcc@inlibro.com> Message-ID: You need to understand that you have dangling messages on your 🐇. Old messages that have invalid JSON, and ids that were picked by the hybrid fallback mechanism so fail (which I agree it's not cool) Stop Koha, stop the worker, stop rabbit, clean rabbit, start the worker, let it exhaust already injected tasks. Stop the worker, start rabbit, start the worker, start Koha. Sounds too much, doesn't it? El miĂ©, 21 dic 2022 10:59, Philippe Blouin escribiĂł: > Good evening, David, > > Thanks for the response. Yours and David's and Michael's. I feel less > alone... > > I validated, and yes all the patches you refer are in our pile. And until > the problems arose, there were no customizations around that code. > > So yeah, even at 22.05.06, I get the JSON error and the race condition (we > use ES). And the *abandonned* children. So I surmise, or dare I say > postulate, that those issues are not as resolved as some would presume. > > I will revert background_jobs_worker.pl to its default, and shutdown MQ > everywhere, for now. :( > Philippe Blouin, > Directeur de la technologie > > TĂ©l. : (833) 465-4276, poste 230 > philippe.blouin at inLibro.com > inLibro | pour esprit libre | www.inLibro.com > On 2022-12-20 17:55, David Cook wrote: > > Salut Philippe, > > > > That first issue should’ve been resolved in 22.05.00 by > https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=30172. I > haven’t had any problems like that since applying that patch. Are you > running Koha with or without customizations? > > > > As you say, bug 30654 discusses that second issue. And I obviously have my > own opinion on that one 😉. > > > > That JSON issue should be fixed by Bug 31351 in Koha 22.05.06 as well I > believe: https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=31351 > > > > -- > > > > The only issue I’ve had with the background jobs has been the one covered > by Bug 30172. Otherwise, it’s been all fine for me, although I use Zebra > rather than Elasticsearch. I think part of the reason I haven’t had issues > is that I haven’t had many people using the background jobs either though. > > > > I’m actually planning on writing a background job system based on RabbitMQ > for a different non-Koha system. The main difference is that I’ll reject or > fail tasks where messages aren’t sent to RabbitMQ. I think that’ll make my > system a bit more robust than Koha’s. > > > > The problem with the background jobs at the moment is that we haven’t > fully committed to RabbitMQ. We’re trying to do this weird hybrid with the > database fallback which is not the right direction in my mind. We should do > one or the other but not try to do both. > > > > But that’s just my 2 cents. > > > > David Cook > > Senior Software Engineer > > Prosentient Systems > > Suite 7.03 > > 6a Glen St > > Milsons Point NSW 2061 > > Australia > > > > Office: 02 9212 0899 > > Online: 02 8005 0595 > > > > *From:* Koha-devel > *On Behalf Of *Philippe > Blouin > *Sent:* Wednesday, 21 December 2022 6:13 AM > *To:* koha-devel at lists.koha-community.org > *Subject:* [Koha-devel] The many failings of background_jobs_worker.pl > > > > Howdy! > > Since moving a lot of our users to 22.05.06, we've installed the worker > everywhere. But the number of issues encountered is staggering. > > The first one was > > Can't call method "process" on an undefined value > > where the id received from MQ was not found in the DB, and the process is > going straight to process_job and failing. Absolutely no idea how that > occurs, seems completely counterintuitive (the ID comes from the DB after > all), but here it is. Hacked the code to add a "sleep 1" to fix most of > that one. > > Then came the fact that stored events were not checked if the connection > to MQ was successful at startup. Bug 30654 refers it. Hacked a little > "$init" in there to clear that up at startup. > > Then came the > > malformed UTF-8 character in JSON string, at character offset 296 (before > "\x{e9}serv\x{e9} au ...") > > at decode_json that crashes the whole process. And for some reason, it > never gets over it, gets the same problem at every restart, like the event > is never "eaten" from the queue. Hacked an eval then a try-catch over it... > > After coding a monitor to alert when a background_jobs has been "new" over > 5 minutes in the DB, I was inundated by messages. There's alway one > elasticsearch_update that escapes among the flurry, and they slowly add up. > > At this point, the only viable solution is to run the workers but disable > RabbitMQ everywhere. Are we really the only ones experiencing that? > > Regards, > > PS Our servers are well-above-average Debian 11 machines with lot of > firepower (ram, cpu, i/o...). > > -- > > Philippe Blouin, > Directeur de la technologie > > TĂ©l. : (833) 465-4276, poste 230 > philippe.blouin at inLibro.com > > inLibro | pour esprit libre | www.inLibro.com > > _______________________________________________ > Koha-devel mailing list > Koha-devel at lists.koha-community.org > https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-devel > website : https://www.koha-community.org/ > git : https://git.koha-community.org/ > bugs : https://bugs.koha-community.org/ > -------------- next part -------------- An HTML attachment was scrubbed... URL: From philippe.blouin at inlibro.com Wed Dec 21 21:33:54 2022 From: philippe.blouin at inlibro.com (Philippe Blouin) Date: Wed, 21 Dec 2022 15:33:54 -0500 Subject: [Koha-devel] The many failings of background_jobs_worker.pl In-Reply-To: References: <010c01d914c6$3b1b3820$b151a860$@prosentient.com.au> <7565bc5a-666e-8f43-9cbc-3f5003c59fcc@inlibro.com> Message-ID: <1b3d9ff0-600d-16ca-886c-8ab932fb8de5@inlibro.com> Could you define "the hybrid fallback" ? I'm not sure where those old messages would come from, on a clean install.  And why would they occur randomly every few hundred or thousand messages, or not at all. This would explain the JSON issue (msg put there in 21.05, collected in 22.05 ?). But I can't see how this scenario would occur on a freshly installed MQ. Philippe Blouin, Directeur de la technologie TĂ©l.  : (833) 465-4276, poste 230 philippe.blouin at inLibro.com inLibro | pour esprit libre | www.inLibro.com On 2022-12-21 14:52, Tomas Cohen Arazi wrote: > You need to understand that you have dangling messages on your 🐇. > > Old messages that have invalid JSON, and ids that were picked by the > hybrid fallback mechanism so fail (which I agree it's not cool) > > Stop Koha, stop the worker, stop rabbit, clean rabbit, start the > worker, let it exhaust already injected tasks. Stop the worker, start > rabbit, start the worker, start Koha. > > Sounds too much, doesn't it? > > > > El miĂ©, 21 dic 2022 10:59, Philippe Blouin > escribiĂł: > > Good evening, David, > > Thanks for the response.  Yours and David's and Michael's.  I feel > less alone... > > I validated, and yes all the patches you refer are in our pile.  > And until the problems arose, there were no customizations around > that code. > > So yeah, even at 22.05.06, I get the JSON error and the race > condition (we use ES).  And the _abandonned_ children.  So I > surmise, or dare I say postulate, that those issues are not as > resolved as some would presume. > > I will revert background_jobs_worker.pl > to its default, and shutdown MQ > everywhere, for now.  :( > > Philippe Blouin, > Directeur de la technologie > > TĂ©l.  : (833) 465-4276, poste 230 > philippe.blouin at inLibro.com > > inLibro | pour esprit libre | www.inLibro.com > > On 2022-12-20 17:55, David Cook wrote: >> >> Salut Philippe, >> >> That first issue should’ve been resolved in 22.05.00 by >> https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=30172. >> I haven’t had any problems like that since applying that patch. >> Are you running Koha with or without customizations? >> >> As you say, bug 30654 discusses that second issue. And I >> obviously have my own opinion on that one 😉. >> >> That JSON issue should be fixed by Bug 31351 in Koha 22.05.06 as >> well I believe: >> https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=31351 >> >> -- >> >> The only issue I’ve had with the background jobs has been the one >> covered by Bug 30172. Otherwise, it’s been all fine for me, >> although I use Zebra rather than Elasticsearch. I think part of >> the reason I haven’t had issues is that I haven’t had many people >> using the background jobs either though. >> >> I’m actually planning on writing a background job system based on >> RabbitMQ for a different non-Koha system. The main difference is >> that I’ll reject or fail tasks where messages aren’t sent to >> RabbitMQ. I think that’ll make my system a bit more robust than >> Koha’s. >> >> The problem with the background jobs at the moment is that we >> haven’t fully committed to RabbitMQ. We’re trying to do this >> weird hybrid with the database fallback which is not the right >> direction in my mind. We should do one or the other but not try >> to do both. >> >> But that’s just my 2 cents. >> >> David Cook >> >> Senior Software Engineer >> >> Prosentient Systems >> >> Suite 7.03 >> >> 6a Glen St >> >> Milsons Point NSW 2061 >> >> Australia >> >> Office: 02 9212 0899 >> >> Online: 02 8005 0595 >> >> *From:*Koha-devel >> *On Behalf >> Of *Philippe Blouin >> *Sent:* Wednesday, 21 December 2022 6:13 AM >> *To:* koha-devel at lists.koha-community.org >> *Subject:* [Koha-devel] The many failings of >> background_jobs_worker.pl >> >> Howdy! >> >> Since moving a lot of our users to 22.05.06, we've installed the >> worker everywhere.  But the number of issues encountered is >> staggering. >> >> The first one was >> >> Can't call method "process" on an undefined value >> >> where the id received from MQ was not found in the DB, and the >> process is going straight to process_job and failing.  Absolutely >> no idea how that occurs, seems completely counterintuitive (the >> ID comes from the DB after all), but here it is.  Hacked the code >> to add a "sleep 1" to fix most of that one. >> >> Then came the fact that stored events were not checked if the >> connection to MQ was successful at startup.  Bug 30654 refers >> it.  Hacked a little "$init" in there to clear that up at startup. >> >> Then came the >> >> malformed UTF-8 character in JSON string, at character offset 296 >> (before "\x{e9}serv\x{e9} au ...") >> >> at decode_json that crashes the whole process.  And for some >> reason, it never gets over it, gets the same problem at every >> restart, like the event is never "eaten" from the queue.  Hacked >> an eval then a try-catch over it... >> >> After coding a monitor to alert when a background_jobs has been >> "new" over 5 minutes in the DB, I was inundated by messages.  >> There's alway one elasticsearch_update that escapes among the >> flurry, and they slowly add up. >> >> At this point, the only viable solution is to run the workers but >> disable RabbitMQ everywhere.  Are we really the only ones >> experiencing that? >> >> Regards, >> >> PS Our servers are well-above-average Debian 11 machines with lot >> of firepower (ram, cpu, i/o...). >> >> -- >> >> Philippe Blouin, >> Directeur de la technologie >> >> TĂ©l.  : (833) 465-4276, poste 230 >> philippe.blouin at inLibro.com >> >> inLibro| pour esprit libre |www.inLibro.com >> > _______________________________________________ > Koha-devel mailing list > Koha-devel at lists.koha-community.org > https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-devel > website : https://www.koha-community.org/ > git : https://git.koha-community.org/ > bugs : https://bugs.koha-community.org/ > -------------- next part -------------- An HTML attachment was scrubbed... URL: From philippe.blouin at inlibro.com Wed Dec 21 21:48:47 2022 From: philippe.blouin at inlibro.com (Philippe Blouin) Date: Wed, 21 Dec 2022 15:48:47 -0500 Subject: [Koha-devel] The many failings of background_jobs_worker.pl In-Reply-To: <332dbcf5-cd97-885a-fec5-6d71f9803f5c@biblibre.com> References: <332dbcf5-cd97-885a-fec5-6d71f9803f5c@biblibre.com> Message-ID: <19edeea7-54f6-1323-577d-86fc2bcf0a6a@inlibro.com> Hello Fridolin, Yes, performance is probably important for MQ, it seems to eat a disproportionate amount of resources sometime.  :) But like I said, our server is a beast.  In every aspect of its build, it has the best components for 2022. Let's see about the race condition: * An update is done on a biblio * An update_elastic_index job is created in the db. * Its ID is pushed onto MQ * background_jobs_worker.pl picks up the ID from MQ o it goes to the DB, and finds nothing with that ID.  We get a pointer error (yeah, I come from C) o This is NOT an old forgotten floating job, since we can see the job in the database when looking manually. o The jobs stays there forever, with status 'new'. * If I add a "sleep 1", this issue _mostly_ disappear. There's no server performance that could explain this.  Maybe some DB caching ? Philippe Blouin, Directeur de la technologie TĂ©l.  : (833) 465-4276, poste 230 philippe.blouin at inLibro.com inLibro | pour esprit libre | www.inLibro.com On 2022-12-21 13:06, Fridolin SOMERS wrote: > Hi, > > I think network performance is really important for RabbitMQ. > We at Biblibre have a virtual machine in each server, to share between > each virtual machine (one Koha per machine) but keep a good network > performance. > Looks to work well, but we are still in 21.11. > > Best regards, > > Le 20/12/2022 Ă  09:13, Philippe Blouin a Ă©crit : >> Howdy! >> >> Since moving a lot of our users to 22.05.06, we've installed the >> worker everywhere.  But the number of issues encountered is staggering. >> >> The first one was >> >> Can't call method "process" on an undefined value >> >> where the id received from MQ was not found in the DB, and the >> process is going straight to process_job and failing. Absolutely no >> idea how that occurs, seems completely counterintuitive (the ID comes >> from the DB after all), but here it is.  Hacked the code to add a >> "sleep 1" to fix most of that one. >> >> Then came the fact that stored events were not checked if the >> connection to MQ was successful at startup.  Bug 30654 refers it. >> Hacked a little "$init" in there to clear that up at startup. >> >> Then came the >> >> malformed UTF-8 character in JSON string, at character offset 296 >> (before "\x{e9}serv\x{e9} au ...") >> >> at decode_json that crashes the whole process.  And for some reason, >> it never gets over it, gets the same problem at every restart, like >> the event is never "eaten" from the queue.  Hacked an eval then a >> try-catch over it... >> >> After coding a monitor to alert when a background_jobs has been "new" >> over 5 minutes in the DB, I was inundated by messages. There's alway >> one elasticsearch_update that escapes among the flurry, and they >> slowly add up. >> >> At this point, the only viable solution is to run the workers but >> disable RabbitMQ everywhere.  Are we really the only ones >> experiencing that? >> >> Regards, >> >> PS Our servers are well-above-average Debian 11 machines with lot of >> firepower (ram, cpu, i/o...). >> >> -- >> Philippe Blouin, >> Directeur de la technologie >> >> TĂ©l.  : (833) 465-4276, poste 230 >> philippe.blouin at inLibro.com >> >> inLibro | pour esprit libre | www.inLibro.com >> >> _______________________________________________ >> Koha-devel mailing list >> Koha-devel at lists.koha-community.org >> https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-devel >> website : https://www.koha-community.org/ >> git : https://git.koha-community.org/ >> bugs : https://bugs.koha-community.org/ > -------------- next part -------------- An HTML attachment was scrubbed... URL: From dcook at prosentient.com.au Thu Dec 22 01:13:28 2022 From: dcook at prosentient.com.au (David Cook) Date: Thu, 22 Dec 2022 11:13:28 +1100 Subject: [Koha-devel] The many failings of background_jobs_worker.pl In-Reply-To: <739868d8-0f6f-4827-3599-a113860ce68c@inlibro.com> References: <010c01d914c6$3b1b3820$b151a860$@prosentient.com.au> <7565bc5a-666e-8f43-9cbc-3f5003c59fcc@inlibro.com> <739868d8-0f6f-4827-3599-a113860ce68c@inlibro.com> Message-ID: <01e401d9159a$38783850$a968a8f0$@prosentient.com.au> Something to investigate. I tend to stick to the *.11 releases, so I’ll see with 22.11 if we have any message queue issues. David Cook Senior Software Engineer Prosentient Systems Suite 7.03 6a Glen St Milsons Point NSW 2061 Australia Office: 02 9212 0899 Online: 02 8005 0595 From: Philippe Blouin Sent: Thursday, 22 December 2022 1:04 AM To: David Cook ; koha-devel at lists.koha-community.org Subject: Re: [Koha-devel] The many failings of background_jobs_worker.pl Although I precise, Cannot connect to broker Failed to connect: Error connecting to localhost:61613: Connection refused at /usr/share/perl5/Net/Stomp.pm line 27.; giving up at /usr/share/perl5/Net/Stomp.pm line 27. So shutting down MQ has its own issues.... Philippe Blouin, Directeur de la technologie TĂ©l. : (833) 465-4276, poste 230 philippe.blouin at inLibro.com inLibro | pour esprit libre | www.inLibro.com On 2022-12-21 08:58, Philippe Blouin wrote: Good evening, David, Thanks for the response. Yours and David's and Michael's. I feel less alone... I validated, and yes all the patches you refer are in our pile. And until the problems arose, there were no customizations around that code. So yeah, even at 22.05.06, I get the JSON error and the race condition (we use ES). And the abandonned children. So I surmise, or dare I say postulate, that those issues are not as resolved as some would presume. I will revert background_jobs_worker.pl to its default, and shutdown MQ everywhere, for now. :( Philippe Blouin, Directeur de la technologie TĂ©l. : (833) 465-4276, poste 230 philippe.blouin at inLibro.com inLibro | pour esprit libre | www.inLibro.com On 2022-12-20 17:55, David Cook wrote: Salut Philippe, That first issue should’ve been resolved in 22.05.00 by https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=30172. I haven’t had any problems like that since applying that patch. Are you running Koha with or without customizations? As you say, bug 30654 discusses that second issue. And I obviously have my own opinion on that one 😉. That JSON issue should be fixed by Bug 31351 in Koha 22.05.06 as well I believe: https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=31351 -- The only issue I’ve had with the background jobs has been the one covered by Bug 30172. Otherwise, it’s been all fine for me, although I use Zebra rather than Elasticsearch. I think part of the reason I haven’t had issues is that I haven’t had many people using the background jobs either though. I’m actually planning on writing a background job system based on RabbitMQ for a different non-Koha system. The main difference is that I’ll reject or fail tasks where messages aren’t sent to RabbitMQ. I think that’ll make my system a bit more robust than Koha’s. The problem with the background jobs at the moment is that we haven’t fully committed to RabbitMQ. We’re trying to do this weird hybrid with the database fallback which is not the right direction in my mind. We should do one or the other but not try to do both. But that’s just my 2 cents. David Cook Senior Software Engineer Prosentient Systems Suite 7.03 6a Glen St Milsons Point NSW 2061 Australia Office: 02 9212 0899 Online: 02 8005 0595 From: Koha-devel On Behalf Of Philippe Blouin Sent: Wednesday, 21 December 2022 6:13 AM To: koha-devel at lists.koha-community.org Subject: [Koha-devel] The many failings of background_jobs_worker.pl Howdy! Since moving a lot of our users to 22.05.06, we've installed the worker everywhere. But the number of issues encountered is staggering. The first one was Can't call method "process" on an undefined value where the id received from MQ was not found in the DB, and the process is going straight to process_job and failing. Absolutely no idea how that occurs, seems completely counterintuitive (the ID comes from the DB after all), but here it is. Hacked the code to add a "sleep 1" to fix most of that one. Then came the fact that stored events were not checked if the connection to MQ was successful at startup. Bug 30654 refers it. Hacked a little "$init" in there to clear that up at startup. Then came the malformed UTF-8 character in JSON string, at character offset 296 (before "\x{e9}serv\x{e9} au ...") at decode_json that crashes the whole process. And for some reason, it never gets over it, gets the same problem at every restart, like the event is never "eaten" from the queue. Hacked an eval then a try-catch over it... After coding a monitor to alert when a background_jobs has been "new" over 5 minutes in the DB, I was inundated by messages. There's alway one elasticsearch_update that escapes among the flurry, and they slowly add up. At this point, the only viable solution is to run the workers but disable RabbitMQ everywhere. Are we really the only ones experiencing that? Regards, PS Our servers are well-above-average Debian 11 machines with lot of firepower (ram, cpu, i/o...). -- Philippe Blouin, Directeur de la technologie TĂ©l. : (833) 465-4276, poste 230 philippe.blouin at inLibro.com inLibro | pour esprit libre | www.inLibro.com -------------- next part -------------- An HTML attachment was scrubbed... URL: From dcook at prosentient.com.au Thu Dec 22 05:18:51 2022 From: dcook at prosentient.com.au (David Cook) Date: Thu, 22 Dec 2022 15:18:51 +1100 Subject: [Koha-devel] How best to support MARC21 880? Message-ID: <01fb01d915bc$812f3590$838da0b0$@prosentient.com.au> Hi all, The MARC21 880 field allows you to provide alternative graphical representations for pretty much any other MARC field. Typically, in Koha (and perhaps other library systems), we display these alternative graphical representations on the "detail page". You can see the roman script, the Arabic script, the Chinese script, etc. In Koha, we also show some data in the search results, and that's more problematic. because the search results should take up a limited amount of space on the screen, but hypothetically you can include many different scripts via the 880. I used to work in a government library in Canada where many of our bibliographic records contained quite a few different 880s. So it does happen. I have a library that has catalogued records using the roman alphabet (ie using Pinyin) and provided Chinese versions via the 880. They'd like to show both the romanized and Chinese versions throughout Koha (e.g. in the cart, checkout screens, course reserves, etc.) I've coded some of those customizations, since they only have 880s in Chinese, but I'm increasingly reluctant to make these changes. I don't want to carry these customizations locally, but I don't know that the Koha community would accept these changes. What do you all think about this? Should we include every 880 title and author in every screen? Should we maybe include them but hide them by default using CSS and then allow certain libraries to just show particular languages used by their patrons or by checking the current language? I know I can go faster on my own, but I rather go farther with all of you, so please let me know your thoughts : ) David Cook Senior Software Engineer Prosentient Systems Suite 7.03 6a Glen St Milsons Point NSW 2061 Australia Office: 02 9212 0899 Online: 02 8005 0595 -------------- next part -------------- An HTML attachment was scrubbed... URL: From tomascohen at gmail.com Fri Dec 23 14:12:56 2022 From: tomascohen at gmail.com (Tomas Cohen Arazi) Date: Fri, 23 Dec 2022 10:12:56 -0300 Subject: [Koha-devel] KTD - stable branches Message-ID: Hi all, I made some pushes to the stable branches (22.11, 22.05 and 21.11) so the images are (internally) aligned with (master's) `ktd` command. Please let me know if you encounter any issues. Merry xmas everyone! -- TomĂĄs Cohen Arazi Theke Solutions (http://theke.io) ✆ +54 9351 3513384 GPG: B2F3C15F -------------- next part -------------- An HTML attachment was scrubbed... URL: From kohanews at gmail.com Thu Dec 29 22:27:34 2022 From: kohanews at gmail.com (Koha Newsletter) Date: Thu, 29 Dec 2022 22:27:34 +0100 Subject: [Koha-devel] Koha Community Newsletter: December 2022 Message-ID: The Koha Community Newsletter for December 2022 is here: https://koha-community.org/koha-community-newsletter-december-2022/ Many thanks to everyone who submitted articles and news to this newsletter! Please feel free to email me with any corrections or suggestions. -- Michael Kuhn Editor, Koha Community Newsletter -------------- next part -------------- An HTML attachment was scrubbed... URL: