With average over one change per day (400 commits) previous year 2010 was another year of continuous SZARP evolution. Time for a short summary of past 12 months.

In April we moved from Subversion to git, while still staying on SourceForge. It was quite a big change for us, but git is a better tool – more robust, with better merging/branching functionality, and a lot faster. Also, even whole git repository needs less disk space, then single working copy of Subversion repository. This change is also partially visible for users – previously SZARP minor version number was upgraded with every Subversion commit. Git does not use numerical version identifiers, so every version change is now initiated manually by developers and results in releasing new version – either for Debian stable or unstable or Windows. It means less frequent version changes and every new version is actually delivered to users and brings some new functionality or bug fixes.

New important SZARP functionality is availability to acquire and present parameters historical values with 10 seconds frequency, in addition to previously available 10 minutes frequency. It requires some additional configuration and needs working LAN or VPN connection to SZARP server, but it’s completely integrated with existing tools, allowing users to “dig in deeper” into historical data.

Draw3 application – the most important SZARP data analyse tool – has been enhanced with several other new functionalities. These included 3-dimensional (XYZ) parameters’ graphs, ability to jump to or follow newest data, tree presentation of draws’ sets or searching for time based on condition described by Lua formula.

At the same time, we still worked on improving Draw3 performance. For example, there where some fixes in data caching. Also, the completely new module tries to parse and precompile Lua formulas of defined parameters. If the formula is “simple enough” (contains mainly arithmetical operations and conditions) – which is true for most of them – it’s converted to internal representation and calculated by SZARP itself, without running Lua interpreter. As a result, Lua becomes the ultimate tool for creating defined parameters – simple formulas can be processed with uncompromising performance while still allowing to construct more complicated formulas, using full power of Lua programming language.

One of the other most important “under hood” changes is introducing a new device drivers architecture. Previously, each device driver (daemon) was a separate program, using IPC to communicate with other parts of system. It fits traditional Unix “do one thing” philosophy, but has some limitations. For example, it was not possible to share one serial port among different devices using different protocols – and this situation is common, for example for RS485 serial line. In case of hundreds connected devices (yes, we do it) it also meant running hundreds processes, which lead to performance problems. New solution is to use one meta-daemon called “boruta” (kind of joke – see http://en.wikipedia.org/wiki/Devil_Boruta). Boruta provides common framework for handling configuration and serial or network connections. Specific protocols handling is provided as boruta modules. We’re using libevent library, which allows for fast, even-driving architecture, capable of handling great numbers of connected devices. Modules for most commonly used protocols, including Modbus, are currently available.

We counted 6 new SZARP implementations in previous year, summing up to about 60 installations starting from 1992, 58 (!) of these 60 systems still active. For users, the most important SZARP feature seems to be the ability of remote data access. Installations with commercial support (provided by Newterm company) receive regular software updates and help with configuration/maintenance problems.

We’d like to thank all SZARP users for another year with us, and wish us all more successes in new 2011 year!

SZARP is developed mainly with Debian as a target distribution. Compiling on other systems, even so close to Debian as Ubuntu, can be a little tricky. Anyway, we try to release Ubuntu packages from time to time. Packages for the last SZARP version (including support for recording 10-seconds probes) are now available for Ubuntu Lucid Lynx 10.04 LTS. Installation requires just adding line deb http://newterm.pl/debian lucid main non-free to your repositories list. Then you can install SZARP packages using your favourite package managing software.

The best things about SZARP are it’s historical data analyse possibilities. Until know, these possibilities where limited to 10-minutes resolution. It’s usually sufficient for main SZARP applications, which are slow-reactintg heating systems, but it became an important gap between functionality of SZARP and other SCADA systems. But last unstable versions contain working (usually…) implementation of saving and displaying probes with 10-seconds resolution.

SZARP was always capable of collecting and displaying data with few-seconds resolution, but only 10-minutes average values where recorded in database and available for displaying and analysing historical trends. Currently a new process (prober) for saving 10-seconds average values to disk is introduced. Data is saved to disk in specialized format, similar to current szbase format used for 10-minutes probes. New data is kept outside synchronized config/data pool – there’s to many data to fit into current Internet synchronization/distribution scheme.

Another process, probes_server allows to access locally saved data through TCP protocol. Main SZARP data-access library was extended to allow for real-time querying of probes_server for data requested by application. It brings 10-seconds probes to Draw3 program (along with Lua scripting and old-style defined parameters) and possibly to other applications, as Extractor (but it currently requires a little more work).

10-seconds probes are available anywhere within network connectivity to source SZARP server. It works out of box on traditional server and local terminals installations, but it’s also possible to tunnel data through VPN for remote access.

There’s still some work going on (for example, fixes in Windows version of draw3 program), but new feature should soon become enough polished to be included in the next stable version.

New stable version of SZARP – 3.1.406 – was released. New in this version:

• Important bug fixes, including fixing problems with running line daemons (device drivers) by parcook process.
• Automatically generated documentation for all SZARP device drivers.
• New setdmn driver for setting value of parameter from network, along with graphical client – setter. This funcionality could be obtained by “glueing” previously existing SZARP modules, but new implementation is simpler and more robust.
• Import/export of defined draws set in draw3 program. User-defined sets of parameters can be exported to file and exchanged with other users/installations.
• New edit as new function in draw3 program, allowing to easly create defined draws set based on existing one.
• Optimizer for Lua parameters. Lua gives great flexibility in creating defined parameters, but Lua-based parameters can be slow to calulate – we mean slow by SZARP standards, so the user can sometimes even notice process of calculating/drawing parameters. The new code understands quite a large subset of Lua language, including calls for SZAR-specific functions. Most parameters can be parsed to “compiled” form and then calculated even faster then traditional SZARP RPN-based parameters. If parameter is to complicated to being understood by optimizer, it is passed to Lua interpreter to calculate. This new feature makes SZARP even faster in analyzing historical data.

This is the last stable version released from Subversion repository.

You can also try new unstable version 3.1.409, than contains new cool feature – 3D graphs for analyzing relations between 3 selected parameters. This is not complete yet and some bugs are expected…

Starting from today SZARP uses Git as a source version control. The reasons for change are likely similar as for other projects that move from centralized SCM systems like CVS/SVN to distributed and more feature-rich like Git/Baazar/Mercurial. Git was chosen for it’s stability, maturity, speed and extensive branching/merging capabilities.

Project still uses SourceForge, you can get source code using command like:

git clone git://szarp.git.sourceforge.net/gitroot/szarp/szarp

The side-effect of this transition can be a less frequent updating of Szarp version – previous scheme was using Subversion revision numbers as version numbers. Since Git does not use numeric revision identifiers, versions will be upgraded manually and will more closely reflect changes in functionality. | Starting from today SZARP uses Git as a source version control. The reasons for change are likely similar as for other projects that move from centralized SCM systems like CVS/SVN to distributed and more feature-rich like Git/Baazar/Mercurial. Git was chosen for it’s stability, maturity, speed and extensive branching/merging capabilities.

Project still uses SourceForge, you can get source code using command like:

git clone git://szarp.git.sourceforge.net/gitroot/szarp/szarp

There were over 30 commits into SZARP repository in January. In previous month large parts of draw3 program code was rewritten in order to make code simpler to manage, which introduced quite a few bugs, so we are holding ‘stable’ release for a moment, to make sure all of them are removed. The major functional change was improving communication with DDE-enabled Windows applications.

DDE (Dynamic Data Exchange) is an ancient message-queue-like protocol introduced by Microsoft in Windows 2.0 in 1987 (!). It enables applications to exchange any kind of data, for example to automatically update some value in word processor when it is changed in Excel. Wonderware, producer of popular InTouch SCADA software, introduced DDE into world of automation. It even created network-enabled version of DDE – NetDDE, which was licensed by Microsoft in some versions of Windows. It was finally removed from Windows for security reasons in Windows XP service pack 2 and Windows Vista. But old, ordinal DDE is currently available even in new shiny Windows 7 (BTW – szarp4win works under Windows 7) and you can always install NetDDE from Wonderware. DDE is insecure by nature, there’s no way to identify origin of request. There are newer technologies like COM, OLE and OPC that intended to replace DDE, but it is still popular in some applications, including exchanging data with SCADA systems.

There is no easy way to connect from Linux application (like SZARP Scada) to Windows application using DDE. DDE (and NetDDE) is based on proprietary Microsoft technologies, there aren’t any independent implementations – Windows applications just use win32 API. For the need of exchanging data with Windows-based SCADA (InTouch, actually) SZARP was using small Python script called ddespy which transferred XML-RPC request from Linux box (SZARP ddedmn driver) to DDE request on Windows box. Initialy it was working only for one special case – InTouch Modbus driver (MBENET). Recent changes (including script name changed from ddespy to ddeproxy) allow SZARP to easily fetch all data from InTouch or other DDE-enabled application – you just need to know ‘items’ names (in DDE nomenclature), which are really ‘tagnames’ (in InTouch nomenclature).

Thanks to Python ‘batteries included’ philosophy and Python Win32 extensions writing XML-RPC server working as DDE client is easy and requires just a few lines of code. When started, it just silently run in background. With py2exe (http://www.py2exe.org) tool you can easily deploy script as a single exe plus a bunch of dll files, without the need of independent Python + Win32 extensions installation. Because ddeproxy uses standard XML-RPC protocol, you can also use it independent of SZARP – to fetch data from virtually any DDE Windows application (like Excel) to any XML-RPC client.

Installing ddeproxy does not really exposes security of your Windows box – at least far less then NetDDE service. It serves only ‘Request’ DDE messages (it means read-only access) and runs on a single, configurable TCP port, that can be easily filtered on firewall.

Easy ddeproxy deployment is a great way to give SZARP a try if you are using some Windows-based SCADA – you just need to get an old PC box, install Debian on it, fetch SZARP packages and enjoy a new level of historical data analyses availability.

Other January changes include refining of drivers launching process. Until now we were using shell (sh -c) to start drivers (line daemons), which lead to some problems with signal distributing and to rather unexpected results if special shell constructions (like pipeline) where used in drivers command lines.

Other change mentioned in NEWS file is a start of long process toward handling of shorter time intervals in historical data base – for example 10 seconds instead of current 10 minutes, which can also make SZARP suitable for more applications. At this moment we plan to narrow down availability of these data to local installation (without distributed propagation), but almost every technical detail can still change. | There were over 30 commits into SZARP repository in January. In previous month large parts of draw3 program code was rewritten in order to make code simpler to manage, which introduced quite a few bugs, so we are holding ‘stable’ release for a moment, to make sure all of them are removed. The major functional change was improving communication with DDE-enabled Windows applications.

Major SZARP changes in December include code refactoring of draw3 program and few new applications added. Details follows:

• Major code refactoring of draw3 program, changes introduced quite a few bugs, most of them was hopefully resolved. Separate thread for saving cache to disk was removed. Correct icons where added to multiple dialog windows within application.
• Parameter value on dark blue background in draw3 program was pourly visible on some displays, background color is a little lighter now.
• Added new viszio application for displaying values of selected parameters directly on screen. Viszio is still ‘work in progress’.
• Added szbndremover – tool for “removal” of no-data holes in database – missing values are replaced with averages computed from surrounding values.
• Fix in HTML links generated by paramd program.
• Added M function for fast filtering too big values in defined parameters.
• New universal driver for Modbus protocol, able to handle RTU/ASCII encoding and both serial and TCP connections.

SZARP stable and unstable repositories have been upgraded to version 3.1.214 and 3.1.212. Most important changes include new way of handling user reports in Reporter3 application.

All changes and fixes from version 3.1.163 include:

• New apt repository branch avaiable in szarp.org repository – karmic, containing version of szarp compiled for karmic koala Ubuntu distribution.
• draw3: it now possible to set margins, orientation and page size of printouts
• Buggy wxSingleInstanceChecker (class resposible for detecting if other copy of application is running) replaced with our own implementation.
• mbrtudmn: slave mode was fixed
• ipkedit: update, min max ranges and color attributes can be changed, also documentation of ipkedit was updated
• reporter3: program was expaned – user report templates are now saved automaticaly and are available directly from menu – this is more compliant with draw3 behaviour, lots of other small fixes and improvements, code review.
• reporter3: numerous fixes.
• reporter3: initial version of documentation is available.
• Help fix: searching in documentation no longer causes applications to crash.

Most important changes from version 3.1.97 include:

• Bug fixes and improvements in kontroler3 program, added sound playing on alarm.
• Multiple bug fixes in Modbus RTU driver (timing synchronization); added support for BCD encoded values.
• Added support for different word order in float values in Modbus TCP driver.
• Completed “antialiasing style” draws displaying; it is more suitable for high resolution displays; it also should have less memory usage, but can be slower on some computers. This mode is now default under both Linux and Windows.
• New driver for Aquameter Calec MCP-300 heatmeter.
• Added RPN ‘m’ function for fast filtering of values outside meaningful range.
• Bug fixes in draw3: X/Y drawing, draws colors assigments, opening new window, drawing new points while filter > 0, info about currently selected graphs style, race condition in remarks server address resolution.
• Added scripts for fetching weather forecast from meteoprog.ua website.
• Sender fix: it now ignores empty elements in configuration.
• Compilation/building: fixes for latest mingw32, “critical” lintian bugs fixed, removed boost-1.34 specific code, building for latest Debian unstable.
• Refreshing of isledit (Inkscape plugin for creating SVG visualisation schemas), it now works under both
  Linux and Windows.
• More examples of ISL documents added – complete heating plant with 4 boilers.
• All applications have now icons under Windows.

To connect to Linux SZARP server from Windows desktop in the same local network, X-terminal emulator can be used. The most popular choice is free X.Org server from Cygwin package. xorg-setup.exe file available for download from SZARP project page contains GUI installer for setting up X server IP address and is stripped from all unnecessary files, resulting in only 6MB installer size.
It can be used also for other purposes, not only related with SZARP.