https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=25965 David Cook <dcook@prosentient.com.au> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |dcook@prosentient.com.au --- Comment #2 from David Cook <dcook@prosentient.com.au> --- (In reply to Lari Strand from comment #0)

I am trying to pass messages to a sip server using a new endpoint I created in swagger/openApi. This endpoint uses the guidelines described here:

http://biblstandard.dk/rfid/dk/rfid_sip2_over_https.htm

I find it odd that they say "SIP2 messages are typically exchanged directly over tcp or via telnet, but this prevents SIP2 being used over an internet connection". The Internet is made up of predominately TCP/IP networks. Maybe they meant "the World Wide Web" in regards to HTTP(S) specifically? I don't understand why they'd have to use HTTPS. What are the target SIP2 client and SIP2 server technologies being used? Is this for Koha's SIP2 server? What's going to be acting as the SIP2 client?

My problem is that the sip server closes the connection after this first SC device login message has been passed and the response has been read back in my module:

So I need a way to keep this socket connection open for other future messages.

Koha's SIP2 server will have a TCP socket in a LISTEN state. The SIP2 client will establish a TCP connection and when the SIP2 server accepts that connection, it creates a new socket which lasts for the duration of that connection (you could view this using system utilities like 'netstat' or 'ss', the latter being more modern and useful). If either the server or the client close their end of that socket, that socket is finished.

Without touching the sip server's code itself, the only solution I can come up with is a proxy server that handles and passes on the REST api's incoming messages to sockets and keeps each invidual socket connection open to specific instances of the sip servers (I suppose the sip server itself ends the connections when needed so you wouldn't have to worry about it in the sender's end). Another solution would be to build a new api separate from the swagger/Openapi one. Like an all-in-one solution that deals with the requests and keeps tabs on the open socket connections.

Does it have to be HTTPS? You should be able to use Nginx as a TCP proxy with SSL termination. (I don't know about Apache's abilities as a TCP proxy. I haven't heard of it, but it could be a thing. I only have personal experience using Nginx as a TCP proxy for non-HTTP application protocols.) Client: https://metacpan.org/pod/IO::Socket::SSL Proxy: https://docs.nginx.com/nginx/admin-guide/security-controls/terminating-ssl-t... Remember that HTTP is just an application protocol which uses TCP for its transport, and the TLS/SSL is done at the transport level. But... it sounds like there might be a bit of a sunken cost in terms of doing this SIP2 over HTTPS as per http://biblstandard.dk/rfid/dk/rfid_sip2_over_https.htm. If you have to go HTTPS, I'd say use either Koha or Apache to proxy the HTTPS to a backend service that has a HTTP(S) listener, uses sessions, and can create persistent connections to the SIP2 server. -- You are receiving this mail because: You are the assignee for the bug. You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=25965 --- Comment #4 from David Cook <dcook@prosentient.com.au> --- I'd be curious to hear why Koha Suomi (Finland) are opposed to stunnel as well. I know many Koha libraries use stunnel, and a quick Google suggests other library software does too: https://www.stunnel.org/ https://developers.exlibrisgroup.com/alma/integrations/selfcheck/ https://github.com/folio-org/edge-sip2 -- You are receiving this mail because: You are watching all bug changes. You are the assignee for the bug.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=25965 David Cook <dcook@prosentient.com.au> changed: What |Removed |Added ---------------------------------------------------------------------------- See Also| |https://bugs.koha-community | |.org/bugzilla3/show_bug.cgi | |?id=25972 -- You are receiving this mail because: You are the assignee for the bug. You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=25965 --- Comment #7 from Lari Strand <lmstrand@gmail.com> --- I managed to write a script for communicating with the SIP server through the REST api with XML messages. Attached is a patch and a screenshot from Postman exchanging a SIP message. -- You are receiving this mail because: You are the assignee for the bug. You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=25965 --- Comment #9 from David Cook <dcook@prosentient.com.au> --- If I understand correctly, Lari, there is a vendor that's is creating or has already created a SIP2 client that will use this HTTPS endpoint, right? Is it possible for people to get more details about that? -- You are receiving this mail because: You are watching all bug changes. You are the assignee for the bug.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=25965 --- Comment #11 from Lari Strand <lmstrand@gmail.com> --- (In reply to David Cook from comment #9)

If I understand correctly, Lari, there is a vendor that's is creating or has already created a SIP2 client that will use this HTTPS endpoint, right?

Is it possible for people to get more details about that?

Well there's a finnish self check machine vendor here who agreed to test the new solution with us. It's been tested on a test server successfully. I can't get my hands on their code and they won't share it, but in the discussions we've had they've mentioned that this is the way they communicate with some other systems as well (originally the POST request xml needed to be in escape form, and they requested that the xml could be sent unescaped for a more unified solution, without a need for a Koha specific solution). -- You are receiving this mail because: You are the assignee for the bug. You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=25965 --- Comment #13 from Lari Strand <lmstrand@gmail.com> --- Here a quick how-to: For this module to work you obviously need to apply the patch. Then there's a configuration file called sip2ohttp-conf.xml attached that needs to be put into the same configuration file folder as the environment variable KOHA_CONF points to (where koha-conf.xml file is located). The xml validation requires an xml schema file (sipschema.xml). The schema file should be located in the koha-tmpl folder (the parent directory of koha-tmpl folder gets read like this: C4::Context->config('intranetdir');) When the REST endpoint receives an XML message, it will be validated against this schema file before further processing. In the configuration file sip2ohttp-conf.xml there's a short description on how to use it. You need to add the devices that you want to use sipohttps with here. The module SIPoHTTP.pm will search for device parameters from here when the REST endpoint /sipmessages receives a POST request with an XML in the request body. example config parameters for a device login with an ID "SIPDEVICE" (also configured in SIPdevices.xml by the same ID): <SIPDEVICENAME> <host>127.0.0.1</host> <port>6009</port> </SIPDEVICENAME> Host and port are the sip server's address you want the device to talk with. Obviously you need a sip server running in that address. As mentioned, the device should also be configured to be able to use the sip servers in the SIPconfig.xml file. The device login id's configured in the sip2ohttp-conf.xml file need to match the login id's defined in SIPconfig.xml. That's about it. Now you can communicate with the sip server/servers via the REST endpoint by passing it an XML file like in the Postman attachment I included. -- You are receiving this mail because: You are the assignee for the bug. You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=25965 --- Comment #15 from Lari Strand <lmstrand@gmail.com> --- (In reply to David Cook from comment #10)

I notice a few hard-coded references to koha-suomi.fi, so those would need to be removed and use replaceable placeholders.

You mean just like xsi:schemaLocation="<INSERTYOUROWNXMLSCHEMALOCATIONHERE>"? The values could be anything, once I figure out how to only validate the xml structure, not the insides of the elements as well.

For the config, you might want to put it into koha-conf.xml as well (even if I do think that we overload that 1 configuration file with everything...)

You mean that I should ditch the whole sip2ohttp-conf.xml file altogether and add like an example configuration in the koha-conf.xml file and make the module read the configs from that file instead? Wouldn't it be more easily accessible and simpler, if the sipohttps config would be a separate file? If we added dozens of device configs there the file would become even more bloated. Imagine putting the stuff in SIPconfig.xml into koha-conf.xml, almost the same thing. I would like to keep it separate. -- You are receiving this mail because: You are the assignee for the bug. You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=25965 --- Comment #17 from David Cook <dcook@prosentient.com.au> --- (In reply to Kyle M Hall from comment #16)

This is a very interesting concept, but completely non-standard and I would not expect to see any adoption of it or even much use outside of you specific needs. NCIP has pretty much supplanted SIP for this use case ( NISO is still trying to figure out what to do with SIP & NCIP now that it controls both specifications ).

I think this would be much better suited as an API plugin.

This sounds reasonable to me as well. I think that Lari has already implemented this locally (in testing) at Koha-Suomi in Finland, but I think that they'd probably benefit from doing it as an API plugin too. I suggested sharing the work on Bugzilla anyway though in case anyone else finds the work interesting. It also gives a chance to get some community feedback. Lari, is there a reason the vendor isn't using NCIP? (I admit that I've never used it so didn't even think of it. Cheers, Kyle!) -- You are receiving this mail because: You are watching all bug changes. You are the assignee for the bug.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=25965 --- Comment #19 from Lari Strand <lmstrand@gmail.com> --- Also what comes to the plugin development, we don't have support for api endpoint injection yet. We are way behind in merging community stuff to our local development. My Koha version in my developer container says 17.06.00.001, but there's a lot of sftuff added since. But plugin support is outdated for sure. So I'm struggling on how to set up a community dev environment where I could continue work with the plugin version. -- You are receiving this mail because: You are the assignee for the bug. You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=25965 Christopher Davis <tubaclarinet@protonmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |tubaclarinet@protonmail.com -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=25965 --- Comment #21 from Lari Strand <lmstrand@gmail.com> --- Koha-Suomi SIPoHTTP (SIP2 over HTTPS) communication plugin can be found at: https://github.com/KohaSuomi/koha-plugin-SIPoHTTP It's in full production use. It uses the same old SIP2 communication protocol and has required software updates for machines that we wanted to communicate thourgh it. We still use a couple of tunneled connections to SIP-servers but the plugin works nicely next to those pre-existing SIP2-server configurations. The plugin creates a new endpoint which accepts XML messages with their embedded SIP commands and passes them to Koha's SIP servers. The plugin relays the replies via a response XML in the response body. The documentation on how to configure/use it is in Finnish though. Take a look if you're interested. -- You are receiving this mail because: You are the assignee for the bug. You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=25965 --- Comment #22 from David Cook <dcook@prosentient.com.au> --- (In reply to Lari Strand from comment #21)

Koha-Suomi SIPoHTTP (SIP2 over HTTPS) communication plugin can be found at:

https://github.com/KohaSuomi/koha-plugin-SIPoHTTP

Cool! I'll certainly keep it in mind for the future. -- You are receiving this mail because: You are watching all bug changes. You are the assignee for the bug.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=25965 --- Comment #24 from Lari Strand <lmstrand@gmail.com> --- (In reply to Katrin Fischer from comment #23)

Hi Lari, does you plugin cover the full functionality or are there any changes needed in Koha still?

We could add it to https://wiki.koha-community.org/wiki/Koha_plugins and close the bug maybe.

It covers the full functionality of SIP2 and acts as a wrapper for talking to the Koha SIP2 server. Some changes to the SIP server configuration file is needed, nothing else. So yes it works as a "standalone" plugin. I guess we could close the ticket. -- You are receiving this mail because: You are the assignee for the bug. You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=25965 --- Comment #26 from Marcel de Rooy <m.de.rooy@rijksmuseum.nl> --- I have the idea that the reason behind this report is still valid. People want to protect their SIP2 communication and this is still not available in standard Koha. But it might deserve a report on its own? -- You are receiving this mail because: You are the assignee for the bug. You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=25965 --- Comment #28 from David Cook <dcook@prosentient.com.au> --- (In reply to Kyle M Hall from comment #27)

(In reply to Marcel de Rooy from comment #26)

...

I have the idea that the reason behind this report is still valid. People want to protect their SIP2 communication and this is still not available in standard Koha. But it might deserve a report on its own?

Unfortunately this won't protect or work with standard SIP2. SIP2 over http is not a commonly supported feature as far as I am aware. The standard for protecting regular SIP2 traffic would be stunnel. Many if not most SIP2 devices have stunnel support built in. As far as I can tell, there is no way to bake stunnel into our SIP server.

Yeah this is apples and oranges. Bug 25965 is about adding support for HTTPS which wouldn't be helpful for 99.99% of libraries. We did have bug 25972 for adding SSL/TLS support to the SIP server, but ended up closing it, since stunnel is an already working industry standard. That said, if someone did want to build SSL/TLS support into our SIP server, I don't think it would be very hard at all. -- You are receiving this mail because: You are watching all bug changes. You are the assignee for the bug.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=25965 Lari Strand <lmstrand@gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|NEW |RESOLVED Resolution|--- |FIXED -- You are receiving this mail because: You are the assignee for the bug. You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=25965 --- Comment #31 from Kyle M Hall <kyle@bywatersolutions.com> --- (In reply to Lari Strand from comment #29)

Lyngsoe systems and Mikroväylä added support for their self-check machines for our SIPoHTTPS plugin system here in Finland. We have hundreds of machines here that talk with our Koha SIP servers, including door systems, smart book cabinets, etc. The need for this came because of the near impossibility to set up and manage stunnels for all devices without problems. Now we can maintain and reconfigure our server side freely without affecting the self check machines or need for reconfiguration for the stunnels. The self-check machines also work with any available network, even with sometimes spotty wifi or mobile networks (library buses/bookmobiles). Nowadays we only have a couple of old sorting devices that still use stunnels because they were really expensive, hard to replace and they don't have an active software development team anymore to make necessary changes to them for SIPoHTTPS support. This is fine, since no changes to the SIP server logic was needed, but we need to remember to mind them when we make server side changes that might affect them.

💯 -- You are receiving this mail because: You are watching all bug changes. You are the assignee for the bug.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=25965 --- Comment #33 from Lari Strand <lmstrand@gmail.com> ---

When you say "manage stunnels for all devices", do you mean stunnel for the SIP client (e.g. client software in self-check machine)? Or do you mean stunnel for the Koha SIP server? Or both?

Setting up stunnel for the Koha SIP server should be very simple, but I could see how setting up stunnel for all the SIP clients could be challenging. But then if the self-check provider added support for SIP over HTTPS, then why couldn't they just add support for SIP over TLS? Surely that would've been easier as well.

In any case, it's cool that you've got it all working! I'm just keen to understand is all :)

I haven't worked with the stunnels myself on our systems or even asked one to be set up, but as far as I understand it needs work from both our server's host/provider's support personnel (which is not us) + the municipalities' IT personnel for each library and they have to work together to set up a tunnel. We can only ask them to enable an stunnel to be set up between a self check machine's network and out server host. So it's out of our hands how they manage to set them up and in what time (+upkeep). So naturally we wanted to get rid of all that hassle. I haven't looked at SIP over TLS so I have no experience of that. Does Koha's SIP server support it natively? -- You are receiving this mail because: You are watching all bug changes. You are the assignee for the bug.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=25965 --- Comment #35 from Lari Strand <lmstrand@gmail.com> --- Seems like a valid alternative approach if the self check machines could support/could add support to their software. -- You are receiving this mail because: You are the assignee for the bug. You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=25965 --- Comment #37 from David Cook <dcook@prosentient.com.au> --- But in the end... whatever works :D -- You are receiving this mail because: You are watching all bug changes. You are the assignee for the bug.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=25965 --- Comment #39 from Lari Strand <lmstrand@gmail.com> --- But as I said SIPoHTTP has been working extremely well for us and has been tried and tested with almost a thousand devices in our production environment for almost two years now and we have been very pleased with the performance, stability + ease of configuration (all aspects of the system can be set up and managed by us developers and librarians basically, without need for 3rd parties in between, although librarians usually send support tickets to the SIP devices' manufacturers support personnel for big batch changes for device configurations or when a librarian isn't tech-savvy :D ). -- You are receiving this mail because: You are watching all bug changes. You are the assignee for the bug.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=25965 --- Comment #41 from David Cook <dcook@prosentient.com.au> --- (In reply to Lari Strand from comment #40)

We have also been thinking of another enhancemet for SIPoHTTP. Getting rid of the need to configure the SIP server configs for new/changed device accounts and instead using koha objects + some extra code for the SIPoHTTP endpoint for authenticating the accounts and thus getting rid of the extra work us developers have to do, managing them server side in SIPconfigs. That way the work of managing the device accounts would be left to just the librarians to do in Koha.

That does sound like a useful enhancement. (In reply to Lari Strand from comment #38)

Oh ok thanks for the explanation.

These three major device manufactureres Mikroväylä, Lyngsoe systems and Bibliotheca all support SIPoHTTP and it has almost become a standard in that sense at least here. A parallel implementation that aims to do the same thing would be a bit overkill. If we ever intergated some of this stuff in Koha (not as a plugin but native support), we should pick one or the other? Or just leave both SIP over TLS (if doable) and SIPoHTTP implementations as plugins to cover local needs?

In theory, there would be model code that handles all the core SIP logic, and the SIP server and SIPoHTTP endpoint would just be different controllers. Easier said than done though of course. That's cool about it becoming a standard over there. I don't think the rest of the world is ready for SIPoHTTP, but maybe they will be one day. -- You are receiving this mail because: You are the assignee for the bug. You are watching all bug changes.