Step-by-step Guide to Indexing Data

Introduction

Once you have the Data Cube software installed and connected to a database, you can start to load in some data. This step is performed using the datacube command line tool.

Note

class

important

When you load data into the Data Cube using indexing, all you are doing is recording the existence of and detailed metadata about the data into the index. None of the data itself is copied, moved or transformed. This is therefore a relatively safe and fast process.

Steps Overview

• Create a new product definition

  Before the data itself can be added, a product describing the data must be created. Requires creation of a product-definitions document (yaml)

• Ensure the data is prepared

  The data to be indexed requires datacube friendly dataset-documents for data which is to be indexed

• Index the data

  Run the actual indexing process

• (OPTIONAL) Ingest the data

  After indexing the data you can choose to ingest. This provides the ability to tile the original data into a faster storage format or a new projection system. Requires creation of an ingestion configuration file (yaml). This is not recommended.

Step 1. Creating a Product Definition

The ODC can handle many different types of data, and requires a bit of information up front to know what to do with them. This is the task of a Product Definition.

More detailed information on the structure of a product definition document can be found here

Some example Product definitions are supplied here. Other examples include the Digital Earth Africa product definitions.

Loading Product Definitions

To load Products into your ODC run:

datacube product add <path-to-product-definition-yml>

If you made a mistake, you can update them with:

datacube product update <path-to-product-definition-yml>

Step 2. Ensure Dataset Documents are complete

Every dataset that you intend to index requires a metadata document describing what the data represents and where it has come from, as well has what format it is stored in. At a minimum, you need the dimensions or fields your want to search by, such as lat, lon and time, but you can include any information you deem useful.

It is typically stored in YAML documents, but JSON is also supported. It is stored in the index for searching, querying and accessing the data.

The data from Geoscience Australia already comes with relevant files (named ga-metadata.yaml), so no further steps are required for indexing them.

For third party datasets, see the examples detailed here. For common distribution formats, data can be indexed using one of the tools from odc-apps-dc-tools. In other cases, the metadata may need to be mapped to an ODC-compatible format. You can find examples of data preparation scripts here.

Step 3. Run the Indexing process

Everything is now ready, and we can use the datacube tool to add one or more datasets into our Cube

datacube dataset add <path-to-dataset-document-yaml>

Note that this path can be a URI, such as the path to a document on S3.