We also unified our licensing to GPLv3. The most important changes in it were in relation to software patents, free software license compatibility, the definition of “source code”, and hardware restrictions on software modification (“tivoization”). More info at GNU project website.

Next novelty concerns about SZARP under Windows! We replaced maintenance of separate code for Microsoft’s operating system users by new functionality that appeared on Windows 10 Professional. Users of that system can use latest version of our software by using Ubuntu Shell (here is how to do it). Thanks to that functionality Windows users can now use Linux version of SZARP, and we, programmers, can focus on developing one code which significantly simplifies its maintanance.

During this work, the need for a new Danfoss FC protocol was born. This protocol specifies the master-slave access technique for serial communication. We will not dwell on specifics here, but it is worth mentioning the quality and amount of documentation. Just be careful – the English version of the documentation is the only recommendable.

After a brief discussion on whether it should be a separate program or whether it can use all the good offered by metadaemon, more commonly known as borutadmn, the answer was clear. Our good borough will get another protocol to handle. Where to start?

We started by reviewing the documentation (once again we recommend English version!), wherein we define the critical moments of this protocol. We promised not to get into the detailed specifications, but the protocol required that. From the documentation we learned that in order to obtain data we have to create a 16 byte telegram

• STX is always 0x02 (we do not like magic numbers),
• LGE in case of 16 byte telegram will always be 0x0E (hic!) ¹,
• This ADR is the address of our slave, enlarged by 0x80 (the 7th bit is always 1),
• Databytes are the contents of a telegram (described below),
• BCC is a logical sum of XOR logic on all remaining bytes

Databytes in our case contains 6 of 2 bytes positions – PKE, IND, PWE_HIGH, PWE_LOW, PCD₁, PCD₂. There is a parameter number in the PKE and a function that the inverter is requested to do with the parameter. IND is used to query array parameters. Other items are used by the slave to provide information.

Backing to borutadmn, it is a metadaemon that already has plugins for supporting protocols such as modbus (tcp and serial, client and server), zet (tcp and serial, client), fp210 (serial, client), lumel (serial, client) wmtp (tcp, client). It is based on libevent event handling, which ensures continuity of tasks on many devices. This ensures efficient work with minimal hardware effort. By writing a new module for our metadata we tried to keep the C ++ 11 standard, which was not always in line with the spirit of our boruta.

After a few days of uneven battle, the borutadmn was abducted and our contributor received the first data from the Danfoss VLT 6000 inverter. There was a so-called test phase on the table that detected a plenty of abnormalities in maintaining communication stability. These errors were quickly localized and corrected.

Configuration of the Danfoss inverters using the FC protocol is very similar to the daemon configuration for other protocols. For our new module we had to introduce a new attribute to param, called parameter-number, located in the ipk-extra namespace. It corresponds to the parameter number we want to read. For configuration details, please visit our Documentation section.

In the near future, the plugin should appear in the newer metadata borutadmn_z based on libzmq, which offers even more possibilities, including read and write 2/4 byte unsigned data.

__________________
¹ – for our needs we only use 16 byte telegrams (LGE = 0x0E), but 8 byte telegrams are allowed – then LGE = 0x06

In typical SZARP deployment we have one process that collects values of all parameters, another process that grabs those values and shovels them off into persistent storage, bunch of processes, so called demons, that (typically) communicate with physical devices (usually one process per device), yet another process that make parameters’ values available over HTTP protocol and so on…

All those processes employ the old school UNIX IPC (SYS V) mechanisms to communicate. The parameters’ values are stored in shared memory regions that are accessed using readers-writers locking scheme. Data sent to devices is being passed via queues (again – SYS V IPC primitive).

Here’s an artist’s impression of some of the components comprising SZARP system and the way they are interconnected:



Such design and the way it is presently implemented in SZARP comes with several very nice features, like decoupling and non-fragility to failure of individual processes.

However, there are also some drawbacks to the implementation, which with time become irritating enough, that the need for changes become apparent.

One of the limitations of current approach is that the data in the collection/propagation pipeline (the thing depicted above) have no timestamps associated. Everything sitting in there is considered to be a live value. Data in the shared memory regions is accessed periodically by respective components at fixed time intervals. The exact moment when it happens is determined by each process individual clock (modulo the locking constraints), and those clocks are not synchronized. So, while storing a value to the database we cannot be exactly sure when this value comes from – only that it was acquired sometime in between 1 to 4 clock ticks ago. Which in most of the cases is just fine, as the clocks are ticking fast enough, for this to not matter. But sometime it would be nice to be more precise and also faster.

Another annoyance is that the data collection pipeline works basically with only one data type (and pretty limited at that) which 16 bit integer. In fact that’s exactly what shared memory regions are holding – raw arrays of short integers. Due to certain hacks introduced here and there, this is not exactly correct and SZARP is equally fine with collecting data of larger size, but the truth is the problem of acquiring values of different types and sizes is not solved in a systematic way.

So, as the title of this post suggests, we are in the process of introducing changes that are supposed to address those 2, and some others, lesser problems. As it comes, it is pretty big change, but the good news is that we should be able to make this incrementally so the risk of breaking too many things, is relatively low.

So without further ado, ladies and gentlemen, lo and behold, the new (planned) SZARP data acquisition pipeline (artistic impression of another artist):



There are a few differences between new and old design. First of all, the inter-process communication primitive is no longer shared memory regions, but publisher-subscriber ZeroMQ sockets.

There is a new, main central component in the system, param hub, which is tasked with broadcasting everything it gets on sub socket onto the pub socket. That is the only part of the system that other components need to be aware of and connect to.

The other significant departure from the existing design is that all the messages exchanged between processes are timestamped, so no matter how much time the message travels along the pipeline, it is always clear when that particular datum was collected/generated. Also, those msgs are transmitted asynchronously, which means that the receiving component is not bound by any particular clock or set of clocks, and is free to act on new data as soon as they arrive.

Needless to say, we are also giving up on the assumption that everything is a short int.

And last but not least, the important feature of this architecture is that we can plug in good-old parcook as a yet another piece of the puzzle here. That means that there will be no big-bang needed and we can incrementally work our way towards new design by gradually adapting existing daemons/components.

Let’s see how it goes!

Well, not exactly. It appeared that in many parts of code Latin-2 (ISO/IEC 8859-2) were still considered as default or even hard-coded! (it seemed appropriate 10 years ago) It also applied to SZARP’s encoding conversion module and that’s why it bit from time to time. The issue had to be put straight.

Introduced changes did not take a lot of code-lines (not more than a few hundreds), but possibly may affect almost every part of SZARP. For that reason they had been carefully tested. Nevertheless a few bugs spilled out month or two later.

Today, after a half year of our effort and decent testing, we announce that SZARP is fully capable of UTF-8 character encoding standard.

The application was implemented using WebSockets, offering the new web applications paradigm – asynchronous browser-server communication, where the server may update the browser state at any time. Thanks to WebSockets, new parameter values are transmitted from the server and immediately displayed on all connected client apps. Besides SZARP Draw, another example of such application is the control panel application created by Newterm (screenshots below).

In my opinion the most important, but not yet complete, is change of a database format. The goal is to let SZARP store data with (nearly) arbitrary time resolution, effectively. In addition to the extra flexibility, this also brings other advantages over existing format:

• database occupies less disk space
• (as a consequence of previous point) database sync takes less time
• there is no longer distinction between high-resolution probes that sit on a server and can only be accessed trough TCP connection and archival data of 10 minute resolution that can be synchronized to local machine, with new db format all data can be synchronized to local machine and viewed with no connection to a server.

While working on new database format, we make our new db engine to support multiple data types (our old db supports just one data type). This is also a change that will make our life easier, but for SZARP to be able to take advantage of that improvement, other components of the system will need to be adapted. Even more goodies will be coming with new db format/engine but we’ll talk about them more once we have the code ready.

Of course this is not the only thing that we have been busy with. Here follows the list of changes that are not as major as the db switch, but still are worth highlighting:

• draw3 has a data extraction feature, this feature somewhat duplicates ekstraktor3 features, but with draw3 you can extract data from user defined/network parameters, which cannot be done with ekstraktor3
• szarp logging library “liblog” has been reworked to support multiple, configurable at runtime, backends, for now there exits 3 backends:
  • classic – old-style (i.e. used by SZARP so far) backend directly writing to files
  • syslog – backend using syslog library
  • async_syslog – our home-brew, libevent based implementation of syslog protool, aimed for apps using libevent
• New feature for draw3. Some background info – on main draw3 graph widget each point (for periods decade, year, month, week and season) representes average value of parameter from corresponding time span (e.g. for period year, each point represented average param value for a month). For some types of params (especially counters) it isn’t necessarily the most interesting piece of information user would like to have. Because of this draw3 has been extended to allow user to choose among 3 different kind of values to be displayed for decade, year, month, week and season period:
  • average, default and only option available so far, each point represents param average value
  • last, last 10-minute probe for a point time span is drawn
  • diff, difference between value of last 10-minute probe and first 10-minute of time span covered by point is drawn

  New option is accessible from context menu of graphs’ list
  widget.

• and as usual, tons of minor changes and improvements
]]>