Nunaliit Tutorial: Importing Data

Although the Nunaliit mapping framework is often used for collecting data, it can also import existing data-sets. By using an import profile JSON and GeoJSON documents can be imported into Nunaliit. This allows large data-sets to be quickly included into an atlas and can be re-synchronized when the data-set is updated and re-imported.

To import a JSON or GeoJSON file, you will need to create an import profile. An import profile provides the structure needed by Nunaliit to process external data-sets. It defines instructions on how and what type of data will be imported into an atlas, and what schema will be used for defining the structure of each newly created document that’s imported.

To begin this process we should create a new import profile directory inside the docs directory which is where we will store the import profile. Typically a directory follows the naming conventions of the other directories in the docs directory. In this example we will be importing spatial data about cities and therefore will create a directory called import_profile.testatlas_cities (Note: you don’t need to follow this naming pattern, but it has proven useful for organizing one’s atlas work).

Within the import profile directory you will need to have two files. An _id.txt file for providing a unique id for the import profile, and a nunaliit_import_profile.json file used to define how data will be imported using this profile. See below for the structure of an import profile and examples of each file.

Structure of an atlas import profile:
atlas/
└── docs
└── import_profile.testatlas_cities
├── _id.txt
└── nunaliit_import_profile.json

Example _id.txt:

import_profile.testatlas_cities

Example nunaliit_import_profile.json

{
"id":"cities",
"nunaliit_type":"import_profile",
"label":{"nunaliit_type":"localized","en":"Cities"},
"type":"geojson",
"options":{"idAttribute":"id"},
"operations":["copyAllAndFixNames(testatlas_city)"],
"schemaName":"testatlas_city",
"layerName":"public"
}

The nunaliit_import_profile.json file contains a number of items which need to be defined for an import profile to function correctly. Provided below is an explanation of each item;

  1. id: provides a unique id for the profile
  2. nunaliit_type: defines the type of document, which in this case is an “import_profile”
  3. label: the label field defines the name of the import profile shown by the drop down list in the Import tool window.
  4. type: defines the type of data being imported (either json or geojson).
  5. options: since the data being imported can be re-imported and the previous imported documents can be updated with changes (re-synchronized), you will need an idAttribute to compare changes between imports. In the example provided above, this field was given the value “id” but this could be any field in the data-set which has unique values. Note: if you’re data doesn’t contain a unique id field, you may need to add an id column to your table data and populated it with unique values prior to converting it to either JSON or GeoJSON format.
  6. operations: the operations field allows the user to define how the data should be imported. Available options include; CopyAll, CopyAllAndFixNames, assign, reference, and importReference. For greater detail on each, please consult the documentation (https://github.com/GCRC/nunaliit/wiki/Atlas-Builder-Documentation#importing_data_profile)
  7. schemaName: the schema name is simply the name of the schema you want used when creating each of these documents. Note: make sure to spell the schema name correctly or the import profile will not be shown as a valid choice in the import tool window.
  8. layerName: the layer name field is an optional field, but can be used to associate a layer to each of the documents being imported.

Importing Data with the Import Tool: To import data you will need to visit the Nunaliit tools page (<your atlas domain>/tools/index.html) and then click on the import tool link. If the import profile was created successfully, you should see the option for your import profile available in the drop down list. If your new import profile is not showing up after your atlas was updated, you may need to review the profile and ensure no errors exist.

Nunaliit Import Tool

Importing geojson data with the Nunaliit Import Tool (screenshot)

Select the import profile you created from the drop down list and then copy and paste your JSON or GeoJSON data into the input text box (as shown above). Once successfully pasted, the data will need to be verified by clicking the Verify button.

Assuming no errors existed in the data-set, a series of new proposed documents or changes to existing documents will be listed below, which you can automatically process by clicking the “Proceed” button or by addressing each document individually. Note: make sure you’re logged in before you proceed with creating/making changes to existing documents with the import tool.

Import verification with Nunaliit

Verification of imported documents processed by the Nunaliit Import Tool.

 

Tools for converting data into JSON or GeoJSON format:
Although data-sets can be imported into Nunaliit, your data may be in a different format than JSON or GeoJSON. For example you may have collected your data in a spread sheet or are working with different spatial data format in a GIS. Although there are numerous ways to convert your existing data-sets into JSON or GeoJSON formats, here are two free software packages which can help with data conversion.

csv2json ( https://www.npmjs.com/package/csv2json): Frequently when collecting data you will be working with tables. Regardless of the spreadsheet/database program you’re working with or the file type, you’ll likely be able to export the table data into a comma separated values (csv) file format. Helpful Tip: if your csv data contains strings commas, try exporting your data with each value wrapped in quotes.

If you’re able to perform this conversion, you can then easily convert your csv file into a json file using the command line program csv2json. The command to perform this conversion is simply csv2json [input csv file] [output json file]. Note: if you’re not a fan of working with the command line, there are also numerous websites which can also perform this csv to json conversion on your data.

QGIS ( http://www.qgis.org): QGIS is a leading GIS package which is freely available. For those working with spatial data, you will likely be familiar with a number of different spatial applications, but QGIS provides a simply way of converting your vector data (frequently provided as shape files) into geojson. Simply load the vector data into QGIS, right click on the layer, click Save As and lastly save a copy of the data in GeoJSON format. It’s that simple, and provides a very easy way of getting your data ready for importing into your Nunaliit atlas.

Tips when using import data:

  • When re-importing data, make sure to include all data you want to appear in the atlas (even if no changes occurred to previously imported data). If you only try to import new data and exclude existing data in the atlas, the import tool will assume you will want to delete previous data if it’s excluded from the import process.
  • A copy of the imported data is stored in every document under the nunaliit_import key. If you make a change to the document which you don’t want overwritten with a future re-import, make sure to update the corresponding data in the nunaliit_import portion of the document.
  • When importing large data-sets, it can be very taxing on the browser. Be prepared to be patient when verifying large quantities of data in the atlas.
  • Make sure to match the column names of your imported data with the field names used in the schema. This will allow the imported data to be viewed/edited more easily in the atlas.
Advertisements

Nunaliit Tutorial: Switching from classic to tiled display format

By default atlas documents are shown in the side panel using the classic display format. However, the Nunaliit framework can also display documents in a tile format which can be easily adjusted in an atlas with a single line of code.

The tile display can be applied to a single module, or applied to all modules throughout the atlas. To display documents in a tile display format for a specific module, you can edit the display.json file, located in the nunaliit_module directory. Within the display.json file you will need to add the key “type” and the value “tiled”.

Where to find the display.json file:

module name (e.g. module.demo)/
└── nunaliit_module
 –└── display.json

Example display.json:

{
"defaultSchemaName":"demo_doc",
"type":"tiled"
}

Alternatively, if you prefer the tile display over the classic format you can apply this preference to all modules in the atlas. This option can be applied by adding the displayFormat custom service option within the custom service portion of the window.nunaliit_custom.configuration function (located in the nunaliit_custom.js file).

Location of the nunaliit_custom.js file in your atlas:

atlas/
└── htdocs
 –└── nunaliit_custom.js

Example nunaliit_custom.js code addition:

customService.setOption('displayFormat','tiled');
Tiled Display

Screenshot of a Nunaliit atlas using the tiled display (showing four related documents as tiles)

Nunaliit Tutorial: Update the Module Title

The title of the module (seen in the thin bar beneath the larger navigation bar in the atlas), can easily be updated to any string of text. Within the module’s directory you will find the nunaliit_module directory, which will contains the title.json file that you will need to  update.

Where to find the title.json file:
module.demo/
└──  nunaliit_module
 –└── title.json

Example title.json:

{
"nunaliit_type":"localized"
,"en":"Demo Atlas"
,"fr":"Atlas Demo"
}

Update the title of the module:

As seen in the example above, the title.json file can contain multiple versions of the module title for different languages. Each version of the title is identified with the language code key (e.g. “en” for English) and corresponding value of the title in that language (e.g. “Demo Atlas”).  If you wish to update the module’s title, simple update the title’s text value for each required language used in your atlas (e.g. “en”:”Demo Atlas” could be updated to “en”:”Test Atlas Map”).

Note: If your atlas uses other languages in addition to the default English (en) and French (fr), simply include the language code and corresponding translated title, in the same manner as the en and fr examples above.

Nunaliit Tutorial: Updating the atlas’ default module

When accessing your atlas from the root domain (e.g. http://127.0.0.1:8080) it should load a default module. Initially this is set to the demo module but can be updated to a different module within the nunaliit_custom.js file, located in the htdocs directory.

Location of nunaliit_custom.js file in your atlas:
atlas/
└── htdocs
­└── nunaliit_custom.js

Updating nunaliit_custom.js:

Within the custom service configuration portion of the nunaliit_custom.js file, you should see a line of code which looks like the following;

customService.setOption('defaultModuleIdentifier','module.demo');

You will notice that the final parameter being passed to the setOption method, states the id of the demo module ‘module.demo’. To change the default module, simply replace the ‘module.demo’ id with the id of the new module you wish to have as the default.

Nunaliit Tutorial: Creating a Module

Modules provide a means to organize the atlas into related content. They can contain a variety of data, and be presented in numerous formats including the use of dynamic maps, canvases, or if you prefer a simple html page containing traditional content like text and media.

The easiest method to create a new module, is to simply copy an existing module and update the id file. The following tutorial will provide instructions on how to accomplish this task.

Basic Structure of a Module:
module.demo/
├── _id.txt
├── nunaliit_module
│ ├── display.json
│ ├── edit.json
│ ├── help.json
│ ├── introduction
│ │ ├── content
│ │ │ ├── en.html
│ │ │ ├── fr.html
│ │ │ └── nunaliit_type.txt
│ │ └── type.txt
│ ├── map.json
│ └── title.json
└── nunaliit_schema.txt

Copying an existing module:

Within the docs directory of your atlas, first locate an existing module which you can copy.  Note: If you’ve just created an atlas, a template for a module called module.demo should be available for copying.

To copy an existing module via the terminal, type the following command (Note: the Unix copy command ‘cp’ requires you to add the ‘-r’ recursive option in order to copy the directory and its contents):

cp -r <the name of the original module directory>
 <the name of the new module directory>

e.g. cp -r module.demo module.testatlas_map

Note: If you’re working in a GUI, you can alternatively copy the directory and paste it with a new name in the docs directory, as you would copy any directory on your OS.

Updating the ID of the new module:

Following the copying of the module, you should now have two modules with different directory names. However you still need to update the id of the new module, so that its different from the original module it was copied from.

Using a text editor, update the _id.txt file (located within the new module directory) with the new module id. The file will consist of a single line of text containing the id of the module. Using the example above the _id.txt file will contain the text ‘module.demo’ and will need to be updated to something else, such as ‘module.testatlas_map’.

Nunaliit Tutorial: Atlas Directory Structure

This overview provides details on the structure of a Nunaliit atlas, with a focus on the files which are most relevant to a new user. For additional information on the atlas directory structure, please consult the nunaliit atlas builder wiki page.

Tree view of basic atlas:
atlas/
├── config
│ ├── install.properties
│ ├── mail.properties
│ ├── multimedia.properties
│ ├── nunaliit_manifest.json
│ ├── nunaliit.properties
│ ├── sensitive.properties
│ └── upload.properties
├── docs
│ ├── approved
│ ├── help.browsers
│ ├── help.demo
│ ├── module.demo
│ ├── navigation.demo
│ ├── org.nunaliit.email_template.daily_vetter
│ ├── org.nunaliit.email_template.form_email
│ ├── org.nunaliit.email_template.password_recovery
│ ├── org.nunaliit.email_template.password_reminder
│ ├── org.nunaliit.email_template.submission_approval
│ ├── org.nunaliit.email_template.submission_rejection
│ ├── org.nunaliit.email_template.upload
│ ├── org.nunaliit.email_template.user_creation
│ ├── org.nunaliit.user_agreement
│ ├── public
│ ├── schema.demo_comment
│ ├── schema.demo_doc
│ ├── schema.demo_media
├── extra
│ └── nunaliit.sh
├── dump
│ └── dump_2016-11-25_14:55:07
├── htdocs
│ ├── css
│ ├── favicon.ico
│ ├── index.html
│ ├── js
│ ├── nunaliit_custom.js
│ └── nunaliit_lang.en.js
├── logs
│ └── nunaliit.2016-11-25
├── media
└── site
└── views

As seen in the tree view above, every Nunaliit atlas contains a number of sub-directories which each relate to specific aspects of an atlas. For the purpose of simplicity, this overview will focus on the two directories which new atlas builders spend most of their time in; the docs and htdocs directories. The config, extra, dump, logs, media and site sub-directories will be ignored for the time being but future lessons will provide greater details on these excluded directories.

The Docs Directory:
The docs directory contains documents which are pushed into the CouchDB database used by Nunaliit, and provide much of the structure of how data is viewed, edited and collected in an atlas. When you create a schema, module, layer, import profile, help document, or want to change the appearance of the navigation bar, this is the directory where you will need to work in.

The docs tree view below is of a newly created docs directory, which contains a variety of documents used to created the initial template of an atlas.

Docs Tree Structure:
docs/
├── approved
├── help.browsers
├── help.demo
├── module.demo
├── navigation.demo
├── org.nunaliit.email_template.daily_vetter
├── org.nunaliit.email_template.form_email
├── org.nunaliit.email_template.password_recovery
├── org.nunaliit.email_template.password_reminder
├── org.nunaliit.email_template.submission_approval
├── org.nunaliit.email_template.submission_rejection
├── org.nunaliit.email_template.upload
├── org.nunaliit.email_template.user_creation
├── org.nunaliit.user_agreement
├── public
├── schema.demo_comment
├── schema.demo_doc
└── schema.demo_media

In order to aid the understanding of what each of these files control in an atlas, the atlas screenshot shown below, describes which documents control which aspects of an atlas’s design.

labelled_atlas_window

An overview of which atlas files control which portions of the atlas template (screenshot)

The Htdocs Directory:

The htdocs directory contains the documents typically found in most web pages, including html, javascript and css files. When you need to edit the appearance of the atlas, add additional javascript libraries, or provide custom functionality, these changes typically occur in the htdocs directory.

Htdocs Tree Structure:
htdocs/
├── css
│ └── atlas.css
├── favicon.ico
├── index.html
├── js
│ └── index.js
├── nunaliit_custom.js
└── nunaliit_lang.en.js

Tips for naming atlas files:
When creating new content in an atlas, consistency between directory names can prevent confusion (and in some cases errors) from arising. When creating a new files in the docs directory, the following format has proven useful.

<document_type>.<group_name>_<document_name>

e.g.
module.testatlas_about
module.testatlas_map
schema.testatlas_person
schema.testatlas_bank
navigation.testatlas
help.testatlas_map
layer.testatlas_banks

Although you’re not prevented from using your own directory structure, adopting something consistent in format will make identifying atlas content easier.

Tips for editing files:
When editing atlas files, ensure that your text editor is set to utf-8 character encoding when editing and saving files. In some cases, text editors can add special characters or may leave temp files in the directory you’re editing in, so take care when selecting a text editor to edit files.

Nunaliit Tutorial: Adding a Schema

What is a schema?:

A schema provides guidelines for Nunaliit in the creation, editing and viewing of documents. They allow atlas builders a means to define what data to collect and how it should be viewed by the user. However, it’s important to note that how a schema is defined shouldn’t be viewed as hard set rules, but rather suggestions of the type of content expected in a document.

By default each atlas provides a demo document schema, a demo comment schema and a demo media schema. These schemas provide a new atlas builders a place to start testing out the functionality of the atlas right away. However, if you continue to use Nunaliit, you will likely need to build your own schema for a specific project. This tutorial provides the steps for creating your own schemas.

Designing a schema definition:

Typically when designing a schema, an atlas builder should first consider what data will be collected, and how to segment it into it’s core components. For example, an atlas which focuses on financial institutions, may contain data on banks, bank staff, bank clients, and transactions. This data can be simplified into three schemas: bank, person, and transaction. Notice that both bank staff and bank clients are merged into a single person schema, since their data will likely contain similar information. See below for an example of the fields needed for these three schemas.

schemadesign

Schema designs for Bank, Person and Transaction schemas.

Add a schema to the atlas:

To add a new schema to the atlas, you will need to run the command ‘nunaliit add-schema –id schemaid’ from the root directory of the atlas (similar to how you would run the update or run commands).

 e.g. ../nunaliit2-couch-sdk-2.2.6/bin/nunaliit add-schema --id bank

This command will create a new schema, which is added to the atlas’ docs folder with the name ‘schema.groupname_schemaid’. Within the new schema’s directory, you will see the required files which define the structure of a schema. Initially these files contain the basic structure of the schema, and will need to be updated to represent the schema design mentioned above.

schemadircontents

List of files within a schema directory

schema.testatlas_bank/
├── brief.txt
├── create.json
├── csvExport.json
├── definition.json
├── display.txt
├── export.json
├── form.txt
├── _id.txt
├── isRootSchema.json
├── label.txt
├── name.txt
├── nunaliit_schema.txt
├── nunaliit_type.txt
└── relatedSchemas.json

Updating the schema’s design can be performed by defining the structure of the schema in the definition.json file. Instructions on how to create defintion.json files, including a complete list of field options, can be found on the atlas builder wiki pages.

Using the schema designs shown above, I’ve included three examples of the definition files for the bank, person and transaction schemas.

Bank Schema Definition:

{
    "group": "testatlas"
    ,"id": "bank"
    ,"label": "Bank"
    ,"relatedSchemas": [ ]
    ,"initialLayers": [ "public" ]
    ,"attributes":[
    {
        "type":"string"
        ,"id":"name"
        ,"label":"Name"
        ,"includedInBrief":true
    }
    ,{
        "type":"string"
        ,"id":"branch_number"
        ,"label":"Branch #"
    }
    ,{
        "type":"string"
        ,"id":"address"
        ,"label":"Address"
        ,"textarea": true
    }
    ,{
        "type":"string"
        ,"id":"phone_number"
        ,"label":"Phone #"
    }
    ,{
        "type": "array"
        ,"elementType":"reference"
        ,"id":"employees"
        ,"label":"Employees"
    }
    ,{
        "type":"checkbox"
        ,"id":"atm"
        ,"label":"ATM location?"
    }
    ]
}

Person Schema Definition:

{
    "group": "testatlas"
    ,"id": "person"
    ,"label": "Person"
    ,"relatedSchemas": [ ]
    ,"initialLayers": [ ]
    ,"attributes":[
    {
        "type":"string"
        ,"id":"name"
        ,"label":"Name"
        ,"includedInBrief":true
    }
    ,{
        "type":"string"
        ,"id":"address"
        ,"label":"Address"
        ,"textarea": true
    }
    ,{
        "type":"string"
        ,"id":"phone_number"
        ,"label":"Phone #"
    }
    ]
}

Transaction Schema Definition:

{
    "group": "testatlas"
    ,"id": "transaction"
    ,"label": "Transaction"
    ,"relatedSchemas": [ ]
    ,"initialLayers": [ ]
    ,"attributes":[
    {
        "type": "reference"
        ,"id":"payer"
        ,"label":"Payer"
    }
    ,{
        "type": "reference"
        ,"id":"payee"
        ,"label":"Payee"
    }
    ,{
        "type":"string"
        ,"id":"amount"
        ,"label":"Amount"
    }
    ,{
        "type":"date"
        ,"id":"date"
        ,"label":"Date"
    }
    ,{
        "type":"string"
        ,"id":"details"
        ,"label":"Details"
        ,"textarea": true
    }
    ]
}

Updating Schemas:
Once the definition.json files are updated with the new schema design structures, you can update the schemas in a number of ways. You can either update each schema individually or you can update all of the schemas at once.

Update a specific schema:

 e.g. ../nunaliit2-couch-sdk-2.2.6/bin/nunaliit update-schema --name testatlas_bank

Note: The name of the schema is typically the groupname_schemaid. If you are unsure of the name of the schema, you should consult the name.txt file within the schema’s directory

Update all schemas:

 e.g. ../nunaliit2-couch-sdk-2.2.6/bin/nunaliit update-schema --all

Updating the Atlas:

The update-schema tool updates each of the files in the schema directory with the newly defined schema’s structure. However these changes won’t be seen in the atlas until the atlas it’s self is updated.

 e.g. ../nunaliit2-couch-sdk-2.2.6/bin/nunaliit update

Following the update of the atlas, you should now be able to use these schemas in your atlas. See a screenshot of the schema form view below.

screenshotofschemadisplay

Nunaliit form view of the Bank schema

Customizing your schema:

Using the add-schema and update-schema tools allows for quick creation and updating of schema documents in Nunaliit. However, you are not limited to the use of these tools. You can edit the files in the schema directory, allowing an atlas builder to customize your schema’s design. Note: If you decided to customize a schema, you will likely want to delete your definition.json file from the schema directory to prevent your custom changes being overwritten accidentally with the nunaliit update-schema –all.