---
title: "General Modeling Principles"
slug: "general-modeling-principles"
updated: 2026-04-01T18:55:57Z
published: 2026-04-01T18:55:57Z
canonical: "docs.sightmachine.com/general-modeling-principles"
---

> ## Documentation Index
> Fetch the complete documentation index at: https://docs.sightmachine.com/llms.txt
> Use this file to discover all available pages before exploring further.

# General Modeling Principles

### **Tag Data**

Pipeline designs often combine high-volume, near real-time sources (tags) with other data sources. For tag data, consider the following:

- **Upstream aggregation:** Determine whether time-based aggregation can be performed earlier in the pipeline to reduce record volume and improve performance.
- **Topic segmentation by machine:**
  - *Pros:* Simplifies modeling and can improve processing efficiency
  - *Cons:* Increases operational overhead, especially if tags are frequently reassigned
- **Cycle boundary definition:** Identify whether the cycle boundary is event-based or time-window-based. The optimal boundary should reflect natural process delineations.
- **Aggregation timing:** Evaluate how early aggregation can be pushed in the pipeline. Earlier aggregation, combined with a smaller late-arriving data window, can improve simplicity and performance.

### Tag Data Modeling Checklist:

![](https://cdn.us.document360.io/c65001f1-30e3-4f1d-b335-57abf64ebd31/Images/Documentation/Screenshot 2026-04-01 at 2.54.52 PM.png)

### **Transactional (Database) Data**

For transactional data sources, consider the following:

- **Late-arriving data window:** Assess the expected lateness of records. If the window is large, delay merging with high-volume tag data as late in the modeling process as possible.
- **Time semantics:** Determine whether each record represents a point in time, a time range, or has no time component. If it represents a range, consider decomposing it into point-in-time events (e.g., downtime start/stop). Refer to ETL best practices for examples.
- **Data mutability:** Identify whether records may change over time (e.g., upserts or deletes). If so, ensure each record has a unique ID and an operation indicator (U/D).
- **Joins and enrichment:** Evaluate whether “left join” logic is required. If possible, push joins upstream into the source query. If not, consider the Lookup table design pattern within the pipeline.
- **Multi-machine associations:** When attaching data to multiple machines, assess trade-offs between duplicating records per machine (higher message volume) versus using forward fill on a non-machine partition (increased stateful processing).