Evolving SZARP architecture!

SZARP, since the very beginning (if we exclude short period of SZARP infancy, when it was running on multiple computers under DOS operating system) possesses architecture which is very UNIXy in its spirit. SZARP basically is a collection of processes/applications, where each of the processes is tasked with performing (mostly) one job – and (obviously) performing it well :).

In typical SZARP deployment we have one process that collects values of all parameters, another process that grabs those values and shovels them off into persistent storage, bunch of processes, so called demons, that (typically) communicate with physical devices (usually one process per device), yet another process that make parameters’ values available over HTTP protocol and so on…

All those processes employ the old school UNIX IPC (SYS V) mechanisms to communicate. The parameters’ values are stored in shared memory regions that are accessed using readers-writers locking scheme. Data sent to devices is being passed via queues (again – SYS V IPC primitive).

Here’s an artist’s impression of some of the components comprising SZARP system and the way they are interconnected:

Such design and the way it is presently implemented in SZARP comes with several very nice features, like decoupling and non-fragility to failure of individual processes.

However, there are also some drawbacks to the implementation, which with time become irritating enough, that the need for changes become apparent.

One of the limitations of current approach is that the data in the collection/propagation pipeline (the thing depicted above) have no timestamps associated. Everything sitting in there is considered to be a live value. Data in the shared memory regions is accessed periodically by respective components at fixed time intervals. The exact moment when it happens is determined by each process individual clock (modulo the locking constraints), and those clocks are not synchronized. So, while storing a value to the database we cannot be exactly sure when this value comes from – only that it was acquired sometime in between 1 to 4 clock ticks ago. Which in most of the cases is just fine, as the clocks are ticking fast enough, for this to not matter. But sometime it would be nice to be more precise and also faster.

Another annoyance is that the data collection pipeline works basically with only one data type (and pretty limited at that) which 16 bit integer. In fact that’s exactly what shared memory regions are holding – raw arrays of short integers. Due to certain hacks introduced here and there, this is not exactly correct and SZARP is equally fine with collecting data of larger size, but the truth is the problem of acquiring values of different types and sizes is not solved in a systematic way.

So, as the title of this post suggests, we are in the process of introducing changes that are supposed to address those 2, and some others, lesser problems. As it comes, it is pretty big change, but the good news is that we should be able to make this incrementally so the risk of breaking too many things, is relatively low.

So without further ado, ladies and gentlemen, lo and behold, the new (planned) SZARP data acquisition pipeline (artistic impression of another artist):

There are a few differences between new and old design. First of all, the inter-process communication primitive is no longer shared memory regions, but publisher-subscriber ZeroMQ sockets.

There is a new, main central component in the system, param hub, which is tasked with broadcasting everything it gets on sub socket onto the pub socket. That is the only part of the system that other components need to be aware of and connect to.

The other significant departure from the existing design is that all the messages exchanged between processes are timestamped, so no matter how much time the message travels along the pipeline, it is always clear when that particular datum was collected/generated. Also, those msgs are transmitted asynchronously, which means that the receiving component is not bound by any particular clock or set of clocks, and is free to act on new data as soon as they arrive.

Needless to say, we are also giving up on the assumption that everything is a short int.

And last but not least, the important feature of this architecture is that we can plug in good-old parcook as a yet another piece of the puzzle here. That means that there will be no big-bang needed and we can incrementally work our way towards new design by gradually adapting existing daemons/components.

Let’s see how it goes!

Character encoding cleanup in SZARP

In the past half-year we did some profound cleanup in character encoding in SZARP. Many users may consider this as negligible, but in fact it could be a bit of hassle – especially for Polish users. What’s the point? SZARP software has been developed continuously for over 20 years, so for sure one will find in it parts of code written a dozen or so years ago. In that time character encoding were not as straight forward as it is today. A series of standards, gathered in ISO/IEC 8859, were commonly used. Year 2006 brought a change – a character encoding UTF-8 (which implements Unicode standard) started gaining popularity and in a few years it has become the dominant for the World Wide Web. Again, what’s the point? SZARP is using UTF-8 for some time.

Well, not exactly. It appeared that in many parts of code Latin-2 (ISO/IEC 8859-2) were still considered as default or even hard-coded! (it seemed appropriate 10 years ago) It also applied to SZARP’s encoding conversion module and that’s why it bit from time to time. The issue had to be put straight.

Introduced changes did not take a lot of code-lines (not more than a few hundreds), but possibly may affect almost every part of SZARP. For that reason they had been carefully tested. Nevertheless a few bugs spilled out month or two later.

Today, after a half year of our effort and decent testing, we announce that SZARP is fully capable of UTF-8 character encoding standard.

SZARP for mobile devices

From now on SZARP supports mobile devices. The mobile version of SZARP Draw offers all the basic functionality: viewing graphs, split cursor mode, mean and summaric values, finding parameters and sets by name. We chose HTML5+JS as the core technology to ensure that the application would run on all mobile devices, on every OS (in particular, Android and Windows Phone are supported). The user interface has been redesigned to provide comfortable navigation even on small smartphone screens, using touch events. The mobile version of the SZARP system complements the base SZARP system, which remains the core system, offering the widest range of functionality.

The application was implemented using WebSockets, offering the new web applications paradigm – asynchronous browser-server communication, where the server may update the browser state at any time. Thanks to WebSockets, new parameter values are transmitted from the server and immediately displayed on all connected client apps. Besides SZARP Draw, another example of such application is the control panel application created by Newterm (screenshots below).

The taste of things to come…

We’ve been severly neglecting readers of szarp.org over last serveral months and we’re very sorry about that. To compensate for this, we decided to give you, our beloved readers, a little visual seak peek at what we’ve been up to recently.
So, lo and behold – draw3 and sz4 engine in action!


What you can see on the picture is good ol’ draw3 hooked to brand new, almost completed sz4 database engine. What’s all the fuss about, you may ask, after all it’s just draw3 drawing some graph. And yes, this is indeed true. However if you take a closer look you may notice one novel thing – the time resolution of the graph being displayed. New database engine is able to store data with subsecond time resolution and now draw3 is capable of (at least partially for now) taking adantage of this fact.
Theoretically new database format lets us store data with up to 1ns precision, however other factors are limiting this value somewhat – e.g. draw3 due to usage of wxWidgtes date and time classes is only able to draw trends with up to 1 ms resolution, but if the need arises we could go definitely go lower than that.
So, when it’s going to be production ready – I hear you asking. Soon, we hope. There are still some bits and pieces missing and we’re also have some bugs to squash. So, stay tuned!

SZARP moves to github

Following latest trends, SZARP project is moving its source code repository to github.
The new project’s URL is https://github.com/Newterm/szarp. In order to fetch SZARP’s source code use command:
git clone https://github.com/Newterm/szarp.git
We’ll keep our sourceforge repository for some time, however we do not plan to commit any new stuff in there.

Alive and kickin’

Hello.
We’ve been little sluggish on updating you with news from SZARP world for a last few months. It’s not because nothing of interest was happening with our system. Quite the contrary – we’ve been doing some serious groundwork on core SZARP components. It’s just that updating szarp.org occupies low position on our priority list, there are always things to do that are more important or fun ;).
But now that moment has finally come – on a gentle nudge from the guy who is paying our salaries, we had no choice but to give you some update on what is going on with SZARP 😉

In my opinion the most important, but not yet complete, is change of a database format. The goal is to let SZARP store data with (nearly) arbitrary time resolution, effectively. In addition to the extra flexibility, this also brings other advantages over existing format:

  • database occupies less disk space
  • (as a consequence of previous point) database sync takes less time
  • there is no longer distinction between high-resolution probes that sit on a server and can only be accessed trough TCP connection and archival data of 10 minute resolution that can be synchronized to local machine, with new db format all data can be synchronized to local machine and viewed with no connection to a server.

While working on new database format, we make our new db engine to support multiple data types (our old db supports just one data type). This is also a change that will make our life easier, but for SZARP to be able to take advantage of that improvement, other components of the system will need to be adapted. Even more goodies will be coming with new db format/engine but we’ll talk about them more once we have the code ready.

Of course this is not the only thing that we have been busy with. Here follows the list of changes that are not as major as the db switch, but still are worth highlighting:

  • draw3 has a data extraction feature, this feature somewhat duplicates ekstraktor3 features, but with draw3 you can extract data from user defined/network parameters, which cannot be done with ekstraktor3
  • szarp logging library “liblog” has been reworked to support multiple, configurable at runtime, backends, for now there exits 3 backends:
    • classic – old-style (i.e. used by SZARP so far) backend directly writing to files
    • syslog – backend using syslog library
    • async_syslog – our home-brew, libevent based implementation of syslog protool, aimed for apps using libevent
  • New feature for draw3. Some background info – on main draw3 graph widget each point (for periods decade, year, month, week and season) representes average value of parameter from corresponding time span (e.g. for period year, each point represented average param value for a month). For some types of params (especially counters) it isn’t necessarily the most interesting piece of information user would like to have. Because of this draw3 has been extended to allow user to choose among 3 different kind of values to be displayed for decade, year, month, week and season period:
    • average, default and only option available so far, each point represents param average value
    • last, last 10-minute probe for a point time span is drawn
    • diff, difference between value of last 10-minute probe and first 10-minute of time span covered by point is drawn

    New option is accessible from context menu of graphs’ list
    widget.

  • and as usual, tons of minor changes and improvements

Spring cleaning and constant march forward

Hello SZARP users.
Spring has come, we’ll be eating Ester eggs in couple of days, so we though that it would be a good time to give you some update on what has been happening with SZARP over last couple of weeks:

  • new version of SZARP for windows was made available – this version comes with much improved (over a factor of 2.5) data synchronization speed – it is much quicker now and on par with its linux counterpart
  • one of our colleagues finished his work on new component in our system: pyipk, a plugin-based extensible application for manipulating SZARP configuration files – params.xml. This is first SZARP application that sports both console and graphical (Qt based) user interface.
  • with the coming of the spring SZARP dev team have made some refactoring of parts of our code base that got a little bit clunky over time, with cleaner code we will be able to provide new features and improvements more quickly
  • a new major feature was added to SZARP – support for network params and sets (but we covered that already on http://szarp.org)

But thats by no means all what has happened with SZARP – since our last general update 2 months ago, there were over 700 commits to SZARP repository, with lots of small improvements and bug-fixes that are just impossible to list here.

See you next time and we wish you a Happy Easter!

SZARP on Oneiric!

SZARP packages for Ubuntu Oneiric were just uploaded to our packages repository. SZARP Oneiric packages can be accessed at :
http://newterm.pl/debian oneiric main. From now on expect regular updates of SZARP packages for Ubuntu Oneiric distro.

SZARP in autumn

Hello dear SZARP users.
Autumn was a busy period for SZARP developers who have been involved in a couple of big projects within SZARP system. Those projects probably won’t see the light of the day until beginning of the next year. In the meantime, however, we were able to bring some small and not small changes and improvements to SZARP codebase, more to come – stay tuned:

  • Documentation of SZARP data aggregation daemon – meaner3 – is more complete now
  • Fix: button ‘SEASON’ was brought back to draw3 application
  • Support for ‘ISO-8859-2’ encoding in raporter.py app was imporved
  • k601dmn now properly handles the case when serial port gets disconnected (which may happen when e.g. usb2rs converter is unplugged) – in this case daemon will periodically make attempts to reopen the port
  • Two minor issues with modbus support in boruta’s modbus plugin were fixed
  • Draw3 now optionally can be set up to monitor user activity – gathered data is completely anonymous and is collected to allow us understand better how users interact with the application, which in turn we hope, we will be able to use to make the app even more user friendly
  • draw3 activity logger was fixed to not block the GUI thread when server ip address cannot be resolved
  • Regression fix – draw3 was crashing when attempt was made to edit user sets and params when remarks server address was not configured
  • Mechanism to inform user that more recent version of the draw3 app is already installed was improved to accommodate for systems with lacking short form alias in locale settings
  • Support for python 2.x version was improved in szarp python scripts
  • Introduction of new utility – pyipk. the purpose of pyipk is to ease editing of szarp configurations
  • szarp-xsltd package was fixed to not overwrite configuration files on package update
  • Modbus ASCII protocol support fixes in boruta daemon
  • Draw3: handling of sets with more than 12 graphs improved
  • Filler app now orrectly behaves on Debian Squeeze
  • Implementation of szarp ipk parser was switched from DOM to xmlTextReader based, thus reducing memory consumption of configuration parsing process