Configuring a Data Receiver

This article contains the following sections:

Introduction

Each data receiver protocol has different requirements. Consider the following:

Protocol

 Requirements

SQL

  • Each stream has a query, which runs against the SQL database.
  • The receiver should also include a server connection, which specifies which SQL server to connect to, and how to authenticate.

OPC UA

  • Each stream has a set of tags to fetch and a poll rate, which models data from a specific asset.
  • The receiver should also include a server connection, which specifies which OPC UA server to connect to, and how to authenticate.

File-based

  • Each stream has a directory, instructing FTX from where on the device to grab files for that stream.
  • Each stream can optionally include a file filter, in case FTX should only grab files that include a particular string in the file name.
  • The receiver should also have a parser, which is used for converting a file (e.g., .csv, .xls, .xml) into a Pandas DataFrame.

    NOTE: Parsers can include a stream filter. For more information, see Transforms in Configurations in FactoryTX.

Configuring a SQL Receiver

The customer may have machine data that is already being aggregated into a SQL database. The preferred handoff to Sight Machine is through a FactoryTX receiver that receives the SQL data, parses it into the JSON format, and saves it so it can be transmitted properly.


SQL FTX Configuration File Sample


Creating a Polling Query and Adding Keywords

You will want the design of the SQL query to be as efficient as possible.

For example:

Configuring an OPC UA Receiver

The customer may have data from OPC UA sources.

OPC Unified Architecture (OPC UA) is a machine-to-machine communication protocol for industrial automation that provides a secure and reliable mechanism for moving data between enterprise-type systems and the kinds of controls, monitoring devices, and sensors that interact with real-world data.

The most important enhancement from classic OPC to the newer OPC UA is that OPC UA does not rely on Microsoft technology (OLE or DCOM) so it can be implemented on any platform (Apple, Linux, Windows). In addition, UA makes it possible to use structures or models so data tags or points can be grouped and given context to simplify system management and maintenance.

For more information about the OPC UA paradigm, go here.


OPC UA FTX Configuration File Sample


Configuring a File-Based Receiver

The customer may have data from local machines. A file-based receiver could be a CSV file, logfiles, Excel file, etc.

File-Based FTX Configuration File Sample

Calling pandas.read_csv

The PARSE function allows for all of the functionality for Pandas. This is a powerful toolset that should handle many of the cases of differences in the CSV file format.

For more information, refer to:
https://pandas.pydata.org/pandas-docs/stable/generated/pandas.read_csv.html

Managing the Completed/Incoming Folder

Before pandas.read_csv is called, the file is copied locally to a temporary file.

You can see which files have been completed in the completed folder. By default, this is located in: /var/spool/sightmachine/factorytx/databuffer/completed

If you do not have delete_completed enabled, you need to keep all files in the completed folder so that FTX knows which ones have already been processed.

If you have delete_completed enabled, you can rotate files using a tool such as Logrotate.