Deep Dive into Record Files
This article contains the following sections:
- Introduction
- Sections of the Record File
- Converting Data into the Record Format
- Understanding the Schema
Introduction
FactoryTX encrypts data at the edge device before transmitting it to Sight Machine. We transmit the data in a consistent, human-readable record format that contains a JSON document per point in time. In other words, the records are customer data that have gone through a first layer of pre-processing in order to turn them into the JSON format.
Sections of the Record File
The record file consists of two sections:
- Metadata: The top-level reserved keywords that work to transmit the data payload to the correct place. In the sample below, from timestamp through sslog_type.
- Field values: The data payload from the original data source, translated into field names and values. In the sample below, from OuterBodyWidth to Size.
Sample Record File
Converting Data into the Record Format
The conversion process involves formatting and renaming fields. The core of FactoryTX’s functionality is taking in data in many different formats and producing a standardized, human-readable output. It is possible to take existing fields and map them to reserved keywords, which will make Extract, Transform, and Load (ETL) configuration easier in the Sight Machine platform. Most of these can be set by simply naming the field in the receiver configuration or renaming it as part of a transform operation.
Understanding the Schema
The following lists the schema that we use to transmit data from plant floor assets:
Schema |
Description |
capturetime |
The timestamp indicating the time when the record has been received on the destination machine. This is used to determine the order of message arrival. |
capturetime_src |
An array of objects in the form of host: capturetime, which are used to trace records that are transmitted through multiple endpoints. |
counter |
An integer value that increments/changes at the machine level with each cycle. This is used for cycle bundling during Extract, Transform, and Load (ETL). This is an optional field and can be computed in the AI Data Pipeline. |
running |
The status of the machine: running, idle, or in an off state. This is an optional field and can be computed in ETL. |
source |
The ID of the machine that transmitted the information. This is a required field. |
source_customer |
The customer ID of the machine. This is an optional field. |
source_location |
The plant-level location information. This is an optional field. |
source_machine_type |
The type of machine from which the data came. This is an optional field. |
source_logtype |
The type of log being generated. This information is used to create rules for modeling when there are multiple sources that describe a single asset. Typically, something like “cycle,” "PLC," or "CSV." |
timestamp |
The time at which the event was logged at the point of origin. For best results, this should be an NTP-synchronized source. This is a required field. |
alarm_codes |
A key value set of alarm codes from the machine. Typically, non-zero means alarm state. This is an optional field. |
defect_codes |
A key value set of codes that describe any inline inspections done by the machine. Typically, non-zero means inspection failure. |
serial |
An array of serial numbers used to associate data in the process. Currently, this can handle either systems that have multiple serialization schemes OR multiple serials being produced at the same time. This is an optional field and can be assigned in ETL. |
batch |
An array of batch_type and batch_id that can handle different levels of group serialization (e.g., carton/pallet/raw material). This is an optional field. |
poll_rate |
The target rate of polling for this asset. This is an optional field. |
encryption |
Any bookkeeping fields used for encryption. This is an optional field. |
attachments |
An array of attachments that includes full path references on the local file system and MIME types for each. This is an optional field. |
fieldvalues |
A free-form document that includes all the data extracted from the asset (PLC or equivalent). Typically, this is in the form of FIELDNAME { units "mm", value 12345 }, but it can be altered to include arrays, key value pairs, etc. This is intended to be human-readable values (i.e., in units, not voltages, etc.). This is a required field. |