Data loaders

A DataLoader is an object used by an Optimiser for loading data into a Graph. Usually this data is a pair of Tensor types (data and label).

Declare and initialise a DataLoader giving a string parameter which defines the type of input data. The common generic DataLoader takes Tensor objects.

function main()

    var dataLoader = DataLoader("tensor");

endfunction

There are currently three flavours of DataLoader in etch.

Fetch.ai developers have built specific DataLoader types for research and example purposes that take MNIST files, and csv files for a commodity application.

Tensor data loader

Use the addData() function for loading Tensor objects.

var dataloader = DataLoader("tensor");
dataloader.addData(data_tensor, label_tensor);

The following code builds two Tensor objects containing data and label values respectively.

These are then loaded into a DataLoader() with the addData() function.

function main()

    var data_shape = Array<UInt64>(2);
    data_shape[0] = 2u64;
    data_shape[1] = 4u64;

    var label_shape = Array<UInt64>(2);
    label_shape[0] = 2u64;
    label_shape[1] = 4u64;

    var data_tensor = Tensor(data_shape);
    var label_tensor = Tensor(label_shape);

    var dataloader = DataLoader("tensor");
    dataloader.addData(data_tensor, label_tensor);

endfunction

isDone() and getNext()

DataLoader function isDone() returns a boolean if a read has reached the end of a training epoch.

The getNext() function iterates through the TrainingPair types contained within a DataLoader.

Typically, use isDone() and getNext() together.

function main()

    var data_shape = Array<UInt64>(2);
    data_shape[0] = 2u64;
    data_shape[1] = 4u64;

    var label_shape = Array<UInt64>(2);
    label_shape[0] = 2u64;
    label_shape[1] = 4u64;

    var data_tensor = Tensor(data_shape);
    var label_tensor = Tensor(label_shape);

    var dataloader = DataLoader("tensor");
    dataloader.addData(data_tensor, label_tensor);

    while (!dataloader.isDone())
      dataloader.getNext();
      // do stuff here
    endwhile

endfunction

Serialisation

A DataLoader is serialisable and deserialisable.

The following code stores a DataLoader in a State object. It then creates a new DataLoader object and retrieves the DataLoader from the State.

function main()

    var dataloader = DataLoader("tensor");

    var dataloader_state = State<DataLoader>("dataloader");
    dataloader_state.set(dataloader);

    var retrieved_dataloader = DataLoader("tensor");
    retrieved_dataloader = dataloader_state.get();

endfunction