Nunaliit Tutorial: Importing Data

Although the Nunaliit mapping framework is often used for collecting data, it can also import existing data-sets. By using an import profile JSON and GeoJSON documents can be imported into Nunaliit. This allows large data-sets to be quickly included into an atlas and can be re-synchronized when the data-set is updated and re-imported.

To import a JSON or GeoJSON file, you will need to create an import profile. An import profile provides the structure needed by Nunaliit to process external data-sets. It defines instructions on how and what type of data will be imported into an atlas, and what schema will be used for defining the structure of each newly created document that’s imported.

To begin this process we should create a new import profile directory inside the docs directory which is where we will store the import profile. Typically a directory follows the naming conventions of the other directories in the docs directory. In this example we will be importing spatial data about cities and therefore will create a directory called import_profile.testatlas_cities (Note: you don’t need to follow this naming pattern, but it has proven useful for organizing one’s atlas work).

Within the import profile directory you will need to have two files. An _id.txt file for providing a unique id for the import profile, and a nunaliit_import_profile.json file used to define how data will be imported using this profile. See below for the structure of an import profile and examples of each file.

Structure of an atlas import profile:
atlas/
└── docs
└── import_profile.testatlas_cities
├── _id.txt
└── nunaliit_import_profile.json

Example _id.txt:

import_profile.testatlas_cities

Example nunaliit_import_profile.json

{
"id":"cities",
"nunaliit_type":"import_profile",
"label":{"nunaliit_type":"localized","en":"Cities"},
"type":"geojson",
"options":{"idAttribute":"id"},
"operations":["copyAllAndFixNames(testatlas_city)"],
"schemaName":"testatlas_city",
"layerName":"public"
}

The nunaliit_import_profile.json file contains a number of items which need to be defined for an import profile to function correctly. Provided below is an explanation of each item;

  1. id: provides a unique id for the profile
  2. nunaliit_type: defines the type of document, which in this case is an “import_profile”
  3. label: the label field defines the name of the import profile shown by the drop down list in the Import tool window.
  4. type: defines the type of data being imported (either json or geojson).
  5. options: since the data being imported can be re-imported and the previous imported documents can be updated with changes (re-synchronized), you will need an idAttribute to compare changes between imports. In the example provided above, this field was given the value “id” but this could be any field in the data-set which has unique values. Note: if you’re data doesn’t contain a unique id field, you may need to add an id column to your table data and populated it with unique values prior to converting it to either JSON or GeoJSON format.
  6. operations: the operations field allows the user to define how the data should be imported. Available options include; CopyAll, CopyAllAndFixNames, assign, reference, and importReference. For greater detail on each, please consult the documentation (https://github.com/GCRC/nunaliit/wiki/Atlas-Builder-Documentation#importing_data_profile)
  7. schemaName: the schema name is simply the name of the schema you want used when creating each of these documents. Note: make sure to spell the schema name correctly or the import profile will not be shown as a valid choice in the import tool window.
  8. layerName: the layer name field is an optional field, but can be used to associate a layer to each of the documents being imported.

Importing Data with the Import Tool: To import data you will need to visit the Nunaliit tools page (<your atlas domain>/tools/index.html) and then click on the import tool link. If the import profile was created successfully, you should see the option for your import profile available in the drop down list. If your new import profile is not showing up after your atlas was updated, you may need to review the profile and ensure no errors exist.

Nunaliit Import Tool

Importing geojson data with the Nunaliit Import Tool (screenshot)

Select the import profile you created from the drop down list and then copy and paste your JSON or GeoJSON data into the input text box (as shown above). Once successfully pasted, the data will need to be verified by clicking the Verify button.

Assuming no errors existed in the data-set, a series of new proposed documents or changes to existing documents will be listed below, which you can automatically process by clicking the “Proceed” button or by addressing each document individually. Note: make sure you’re logged in before you proceed with creating/making changes to existing documents with the import tool.

Import verification with Nunaliit

Verification of imported documents processed by the Nunaliit Import Tool.

 

Tools for converting data into JSON or GeoJSON format:
Although data-sets can be imported into Nunaliit, your data may be in a different format than JSON or GeoJSON. For example you may have collected your data in a spread sheet or are working with different spatial data format in a GIS. Although there are numerous ways to convert your existing data-sets into JSON or GeoJSON formats, here are two free software packages which can help with data conversion.

csv2json ( https://www.npmjs.com/package/csv2json): Frequently when collecting data you will be working with tables. Regardless of the spreadsheet/database program you’re working with or the file type, you’ll likely be able to export the table data into a comma separated values (csv) file format. Helpful Tip: if your csv data contains strings commas, try exporting your data with each value wrapped in quotes.

If you’re able to perform this conversion, you can then easily convert your csv file into a json file using the command line program csv2json. The command to perform this conversion is simply csv2json [input csv file] [output json file]. Note: if you’re not a fan of working with the command line, there are also numerous websites which can also perform this csv to json conversion on your data.

QGIS ( http://www.qgis.org): QGIS is a leading GIS package which is freely available. For those working with spatial data, you will likely be familiar with a number of different spatial applications, but QGIS provides a simply way of converting your vector data (frequently provided as shape files) into geojson. Simply load the vector data into QGIS, right click on the layer, click Save As and lastly save a copy of the data in GeoJSON format. It’s that simple, and provides a very easy way of getting your data ready for importing into your Nunaliit atlas.

Tips when using import data:

  • When re-importing data, make sure to include all data you want to appear in the atlas (even if no changes occurred to previously imported data). If you only try to import new data and exclude existing data in the atlas, the import tool will assume you will want to delete previous data if it’s excluded from the import process.
  • A copy of the imported data is stored in every document under the nunaliit_import key. If you make a change to the document which you don’t want overwritten with a future re-import, make sure to update the corresponding data in the nunaliit_import portion of the document.
  • When importing large data-sets, it can be very taxing on the browser. Be prepared to be patient when verifying large quantities of data in the atlas.
  • Make sure to match the column names of your imported data with the field names used in the schema. This will allow the imported data to be viewed/edited more easily in the atlas.