Nunaliit Tutorial: Importing Data

Although the Nunaliit mapping framework is often used for collecting data, it can also import existing data-sets. By using an import profile JSON and GeoJSON documents can be imported into Nunaliit. This allows large data-sets to be quickly included into an atlas and can be re-synchronized when the data-set is updated and re-imported.

To import a JSON or GeoJSON file, you will need to create an import profile. An import profile provides the structure needed by Nunaliit to process external data-sets. It defines instructions on how and what type of data will be imported into an atlas, and what schema will be used for defining the structure of each newly created document that’s imported.

To begin this process we should create a new import profile directory inside the docs directory which is where we will store the import profile. Typically a directory follows the naming conventions of the other directories in the docs directory. In this example we will be importing spatial data about cities and therefore will create a directory called import_profile.testatlas_cities (Note: you don’t need to follow this naming pattern, but it has proven useful for organizing one’s atlas work).

Within the import profile directory you will need to have two files. An _id.txt file for providing a unique id for the import profile, and a nunaliit_import_profile.json file used to define how data will be imported using this profile. See below for the structure of an import profile and examples of each file.

Structure of an atlas import profile:
atlas/
└── docs
└── import_profile.testatlas_cities
├── _id.txt
└── nunaliit_import_profile.json

Example _id.txt:

import_profile.testatlas_cities

Example nunaliit_import_profile.json

{
"id":"cities",
"nunaliit_type":"import_profile",
"label":{"nunaliit_type":"localized","en":"Cities"},
"type":"geojson",
"options":{"idAttribute":"id"},
"operations":["copyAllAndFixNames(testatlas_city)"],
"schemaName":"testatlas_city",
"layerName":"public"
}

The nunaliit_import_profile.json file contains a number of items which need to be defined for an import profile to function correctly. Provided below is an explanation of each item;

  1. id: provides a unique id for the profile
  2. nunaliit_type: defines the type of document, which in this case is an “import_profile”
  3. label: the label field defines the name of the import profile shown by the drop down list in the Import tool window.
  4. type: defines the type of data being imported (either json or geojson).
  5. options: since the data being imported can be re-imported and the previous imported documents can be updated with changes (re-synchronized), you will need an idAttribute to compare changes between imports. In the example provided above, this field was given the value “id” but this could be any field in the data-set which has unique values. Note: if you’re data doesn’t contain a unique id field, you may need to add an id column to your table data and populated it with unique values prior to converting it to either JSON or GeoJSON format.
  6. operations: the operations field allows the user to define how the data should be imported. Available options include; CopyAll, CopyAllAndFixNames, assign, reference, and importReference. For greater detail on each, please consult the documentation (https://github.com/GCRC/nunaliit/wiki/Atlas-Builder-Documentation#importing_data_profile)
  7. schemaName: the schema name is simply the name of the schema you want used when creating each of these documents. Note: make sure to spell the schema name correctly or the import profile will not be shown as a valid choice in the import tool window.
  8. layerName: the layer name field is an optional field, but can be used to associate a layer to each of the documents being imported.

Importing Data with the Import Tool: To import data you will need to visit the Nunaliit tools page (<your atlas domain>/tools/index.html) and then click on the import tool link. If the import profile was created successfully, you should see the option for your import profile available in the drop down list. If your new import profile is not showing up after your atlas was updated, you may need to review the profile and ensure no errors exist.

Nunaliit Import Tool

Importing geojson data with the Nunaliit Import Tool (screenshot)

Select the import profile you created from the drop down list and then copy and paste your JSON or GeoJSON data into the input text box (as shown above). Once successfully pasted, the data will need to be verified by clicking the Verify button.

Assuming no errors existed in the data-set, a series of new proposed documents or changes to existing documents will be listed below, which you can automatically process by clicking the “Proceed” button or by addressing each document individually. Note: make sure you’re logged in before you proceed with creating/making changes to existing documents with the import tool.

Import verification with Nunaliit

Verification of imported documents processed by the Nunaliit Import Tool.

 

Tools for converting data into JSON or GeoJSON format:
Although data-sets can be imported into Nunaliit, your data may be in a different format than JSON or GeoJSON. For example you may have collected your data in a spread sheet or are working with different spatial data format in a GIS. Although there are numerous ways to convert your existing data-sets into JSON or GeoJSON formats, here are two free software packages which can help with data conversion.

csv2json ( https://www.npmjs.com/package/csv2json): Frequently when collecting data you will be working with tables. Regardless of the spreadsheet/database program you’re working with or the file type, you’ll likely be able to export the table data into a comma separated values (csv) file format. Helpful Tip: if your csv data contains strings commas, try exporting your data with each value wrapped in quotes.

If you’re able to perform this conversion, you can then easily convert your csv file into a json file using the command line program csv2json. The command to perform this conversion is simply csv2json [input csv file] [output json file]. Note: if you’re not a fan of working with the command line, there are also numerous websites which can also perform this csv to json conversion on your data.

QGIS ( http://www.qgis.org): QGIS is a leading GIS package which is freely available. For those working with spatial data, you will likely be familiar with a number of different spatial applications, but QGIS provides a simply way of converting your vector data (frequently provided as shape files) into geojson. Simply load the vector data into QGIS, right click on the layer, click Save As and lastly save a copy of the data in GeoJSON format. It’s that simple, and provides a very easy way of getting your data ready for importing into your Nunaliit atlas.

Tips when using import data:

  • When re-importing data, make sure to include all data you want to appear in the atlas (even if no changes occurred to previously imported data). If you only try to import new data and exclude existing data in the atlas, the import tool will assume you will want to delete previous data if it’s excluded from the import process.
  • A copy of the imported data is stored in every document under the nunaliit_import key. If you make a change to the document which you don’t want overwritten with a future re-import, make sure to update the corresponding data in the nunaliit_import portion of the document.
  • When importing large data-sets, it can be very taxing on the browser. Be prepared to be patient when verifying large quantities of data in the atlas.
  • Make sure to match the column names of your imported data with the field names used in the schema. This will allow the imported data to be viewed/edited more easily in the atlas.
Advertisements

Nunaliit Tutorial: Switching from classic to tiled display format

By default atlas documents are shown in the side panel using the classic display format. However, the Nunaliit framework can also display documents in a tile format which can be easily adjusted in an atlas with a single line of code.

The tile display can be applied to a single module, or applied to all modules throughout the atlas. To display documents in a tile display format for a specific module, you can edit the display.json file, located in the nunaliit_module directory. Within the display.json file you will need to add the key “type” and the value “tiled”.

Where to find the display.json file:

module name (e.g. module.demo)/
└── nunaliit_module
 –└── display.json

Example display.json:

{
"defaultSchemaName":"demo_doc",
"type":"tiled"
}

Alternatively, if you prefer the tile display over the classic format you can apply this preference to all modules in the atlas. This option can be applied by adding the displayFormat custom service option within the custom service portion of the window.nunaliit_custom.configuration function (located in the nunaliit_custom.js file).

Location of the nunaliit_custom.js file in your atlas:

atlas/
└── htdocs
 –└── nunaliit_custom.js

Example nunaliit_custom.js code addition:

customService.setOption('displayFormat','tiled');
Tiled Display

Screenshot of a Nunaliit atlas using the tiled display (showing four related documents as tiles)

Quick Tip: How to unfreeze your terminal after typing CTRL + s

Every once in a while (usually when my morning coffee hasn’t kicked in yet), I’ll type Ctrl+s to save a document while using vi. Obviously this isn’t how documents are saved in vi, and the result causes your terminal to freeze, becoming completely unresponsive to keyboard commands.

Solution: To unfreeze you terminal after accidentally typing CTRL + s, you simply need to type CTRL + q, and you should now have control of you terminal again.

 

Nunaliit Tutorial: Update the Module Title

The title of the module (seen in the thin bar beneath the larger navigation bar in the atlas), can easily be updated to any string of text. Within the module’s directory you will find the nunaliit_module directory, which will contains the title.json file that you will need to  update.

Where to find the title.json file:
module.demo/
└──  nunaliit_module
 –└── title.json

Example title.json:

{
"nunaliit_type":"localized"
,"en":"Demo Atlas"
,"fr":"Atlas Demo"
}

Update the title of the module:

As seen in the example above, the title.json file can contain multiple versions of the module title for different languages. Each version of the title is identified with the language code key (e.g. “en” for English) and corresponding value of the title in that language (e.g. “Demo Atlas”).  If you wish to update the module’s title, simple update the title’s text value for each required language used in your atlas (e.g. “en”:”Demo Atlas” could be updated to “en”:”Test Atlas Map”).

Note: If your atlas uses other languages in addition to the default English (en) and French (fr), simply include the language code and corresponding translated title, in the same manner as the en and fr examples above.

Nunaliit Tutorial: Updating the atlas’ default module

When accessing your atlas from the root domain (e.g. http://127.0.0.1:8080) it should load a default module. Initially this is set to the demo module but can be updated to a different module within the nunaliit_custom.js file, located in the htdocs directory.

Location of nunaliit_custom.js file in your atlas:
atlas/
└── htdocs
­└── nunaliit_custom.js

Updating nunaliit_custom.js:

Within the custom service configuration portion of the nunaliit_custom.js file, you should see a line of code which looks like the following;

customService.setOption('defaultModuleIdentifier','module.demo');

You will notice that the final parameter being passed to the setOption method, states the id of the demo module ‘module.demo’. To change the default module, simply replace the ‘module.demo’ id with the id of the new module you wish to have as the default.

Nunaliit Tutorial: Creating a Module

Modules provide a means to organize the atlas into related content. They can contain a variety of data, and be presented in numerous formats including the use of dynamic maps, canvases, or if you prefer a simple html page containing traditional content like text and media.

The easiest method to create a new module, is to simply copy an existing module and update the id file. The following tutorial will provide instructions on how to accomplish this task.

Basic Structure of a Module:
module.demo/
├── _id.txt
├── nunaliit_module
│ ├── display.json
│ ├── edit.json
│ ├── help.json
│ ├── introduction
│ │ ├── content
│ │ │ ├── en.html
│ │ │ ├── fr.html
│ │ │ └── nunaliit_type.txt
│ │ └── type.txt
│ ├── map.json
│ └── title.json
└── nunaliit_schema.txt

Copying an existing module:

Within the docs directory of your atlas, first locate an existing module which you can copy.  Note: If you’ve just created an atlas, a template for a module called module.demo should be available for copying.

To copy an existing module via the terminal, type the following command (Note: the Unix copy command ‘cp’ requires you to add the ‘-r’ recursive option in order to copy the directory and its contents):

cp -r <the name of the original module directory>
 <the name of the new module directory>

e.g. cp -r module.demo module.testatlas_map

Note: If you’re working in a GUI, you can alternatively copy the directory and paste it with a new name in the docs directory, as you would copy any directory on your OS.

Updating the ID of the new module:

Following the copying of the module, you should now have two modules with different directory names. However you still need to update the id of the new module, so that its different from the original module it was copied from.

Using a text editor, update the _id.txt file (located within the new module directory) with the new module id. The file will consist of a single line of text containing the id of the module. Using the example above the _id.txt file will contain the text ‘module.demo’ and will need to be updated to something else, such as ‘module.testatlas_map’.

Quick Tip: Subversion Change Logs For Files

Yesterday I needed to determine who last edited portions of a file with subversion, and came across a couple great suggestions on stack overflow which I’d like to share here.

Subversion offer a few ways to perform this task, including the following two methods I learned yesterday;

1. The first method returns the log of previous revisions on the file and is ideal when you need to review a more detailed change log.

 svn log --diff filename 

2. The second method is more concise, and only provides the revision number and author name for the last change to a specific file’s line. This approach is performed with the aptly named subcommand blame.

 svn blame filename