Custom File Receiver

Creating a custom file receiver

In FactoryTX, a FileReceiver fetches files from a Connection and parses the files with any configured Parser. FactoryTX has a few built-in file receivers:

• Local File Data Receiver

• SMB Data Receiver

• Amazon S3 Data Receiver

• Azure Data Lake Data Receiver

If none of the available file receivers meet your needs, you’ll need to create your own.

Defining a Connection

A Connection connects to a data source (e.g. local system, SMB) and can list available files, fetch files, and optionally delete files.

Your custom connection should inherit from the base Connection class and define the following methods:

• __init__(config, root_config): This method initializes the connection instance. It accepts two parameters:

  • config: The configuration dictionary of the connection from the FactoryTX config. The clean method is used to validate and normalize this configuration, and the connection will only be initialized if there is no ValidationError.

  • root_config: The root configuration dictionary, which can be used to access global settings (e.g. Sight Machine cloud credentials). This is the FactoryTX config.

• clean(config, root_config): This method validates and normalizes a configuration dictionary. It validates that the configuration it receives is well-formed and will be accepted by the constructor. The method may also modify the configuration, such as inserting default values, and changes made to the config will persist in the configuration passed to the constructor. This method returns a list of ValidationMessage objects.

• close(): Closes the connection and frees up any associated resources (e.g. sockets).

• copy(file_entry, local_path): Copies a file’s contents from a remote server to a specified temporary file on the local disk. It accepts two parameters:

  • file_entry: metadata about the file, such as the path to the file on the remote server.

  • local_path: path to write the file to. It will always point to an existing zero-byte file.

• delete(file_entry): Deletes a file from the remote server. Connection classes that do not support deletion may raise a NotImplementedError instead. The file_entry parameter is metadata about the file, such as the path to the file on the remote server.

• list(): Lists the files (FileEntry) on the remote server.

As mentioned in the Creating a custom transform tutorial, you’ll need to include a schema that can be applied to your connection. This schema defines what properties can be configured for your connection and is what should be used in the clean method for validation.

When a Connection  object is initialized, its connection will be established until the close method is called to disconnect it. Since FactoryTX initializes all Receiver instances at runtime and polls for new data on an interval, there isn’t a need to track the status of a connection separately from the state of FactoryTX. If FactoryTX is active, the connection should be active as well.

Catching Connection errors

If any errors or exceptions are encountered while using the connection, a ConnectionError should be thrown. For example, to catch and surface an error when deleting a file, our delete method would look something like this:

def delete(self, file_entry: FileEntry) -> None:
    try:
        del file_entry.path
    except Exception as e:
        raise ConnectionError("Unable to delete the file!")

Using the Connection

Your custom connection helps connect to a data source, but it doesn’t retrieve any files. You’ll have to associate it with a FileReceiver, which polls the connection and collects new files for FactoryTX to process.

Create a subclass of the base FileReceiver and assign the connection_class attribute to your custom connection.

class MyFirstFileReceiver(FileReceiver):
    connection_class = MyCustomConnection

In effect, you’ve created a new Receiver. To use your data receiver, you’ll need to add it to FactoryTX’s Registry.

@receivers.register('my_first_file_receiver')
class MyFirstFileReceiver(FileReceiver):
    connection_class = MyCustomConnection
    ...

Please refer to the LocalFileReceiver for an example of a custom file receiver.