Data loaders
A DataLoader
is an object used by an Optimiser
for loading data into a Graph
. Usually this data is a pair of Tensor
types (data and label).
Declare and initialise a DataLoader
giving a string parameter which defines the type of input data. The common generic DataLoader
takes Tensor
objects.
function main()
var dataLoader = DataLoader("tensor");
endfunction
There are currently three flavours of DataLoader
in etch
.
Fetch.ai developers have built specific DataLoader
types for research and example purposes that take MNIST
files, and csv
files for a commodity application.
Tensor
data loader
Use the addData()
function for loading Tensor
objects.
var dataloader = DataLoader("tensor");
dataloader.addData(data_tensor, label_tensor);
The following code builds two Tensor
objects containing data and label values respectively.
These are then loaded into a DataLoader()
with the addData()
function.
function main()
var data_shape = Array<UInt64>(2);
data_shape[0] = 2u64;
data_shape[1] = 4u64;
var label_shape = Array<UInt64>(2);
label_shape[0] = 2u64;
label_shape[1] = 4u64;
var data_tensor = Tensor(data_shape);
var label_tensor = Tensor(label_shape);
var dataloader = DataLoader("tensor");
dataloader.addData(data_tensor, label_tensor);
endfunction
isDone()
and getNext()
DataLoader
function isDone()
returns a boolean if a read has reached the end of a training epoch.
The getNext()
function iterates through the TrainingPair
types contained within a DataLoader
.
Typically, use isDone()
and getNext()
together.
function main()
var data_shape = Array<UInt64>(2);
data_shape[0] = 2u64;
data_shape[1] = 4u64;
var label_shape = Array<UInt64>(2);
label_shape[0] = 2u64;
label_shape[1] = 4u64;
var data_tensor = Tensor(data_shape);
var label_tensor = Tensor(label_shape);
var dataloader = DataLoader("tensor");
dataloader.addData(data_tensor, label_tensor);
while (!dataloader.isDone())
dataloader.getNext();
// do stuff here
endwhile
endfunction
Serialisation
A DataLoader
is serialisable and deserialisable.
The following code stores a DataLoader
in a State
object. It then creates a new DataLoader
object and retrieves the DataLoader
from the State
.
function main()
var dataloader = DataLoader("tensor");
var dataloader_state = State<DataLoader>("dataloader");
dataloader_state.set(dataloader);
var retrieved_dataloader = DataLoader("tensor");
retrieved_dataloader = dataloader_state.get();
endfunction