Data import definitions require you to define the destination table where the external information should be sent, as well as how the data should be mapped into the table's fields.
Destination tables for a data import definition can be:
- Application tables—Data is mapped directly into the application table. This approach requires that the external data conforms exactly to the structural and data format requirements of the destination table.
- Integration tables—Data is mapped into an integration table. An integration application built over the table enables you to initiate logic blocks to process the data, such as reformatting or mapping the data into Header Detail structures. Once the data has been processed, it can then be inserted into an application table. This is the most common data import configuration.
Once a data import definition has been created, it can be reused to import additional data into the selected destination table. There are also several definition templates available for commonly used data imports. For example, you may use the RI - Employees Directory data import definition to upload employee information into the Employees application.
Ways to insert data
Once a data import definition has been created, data can be inserted with:
- CSV files—Gather your data and save it as a CSV file, attach it to a data import definition, then manually import the data into an application or integration table. Learn more in Import data with a CSV file.
- XML files—Gather your data and save it as an XML file. Upload an XSD file that shapes your data, attach both files to the data import definition, then import the data into an application or integration table. This import type enables you to configure recurrent jobs that retrieve records periodically from the server. Learn more in Import data with an XML file.
- SFTP servers—Create a record in the File Location Setup application with information about the server, reference it in a data import definition, then import the data into an integration table. This import type enables you to configure recurrent jobs that retrieve records periodically from the server. Learn more in Import data with an SFTP server.
For historical data, you must complete additional configuration steps. Learn more in Historical data import configuration.
You can view the status of a data import and any errors that may have occurred by selecting the Import Runs form action. Learn more in Data Import Run Inquiry application.
Performance
You can configure your data imports with specific requirements to enable the COPY command function. This significantly improves performance as it allows the system to copy the entire data import rather than processing each row individually. To enable, your data import must:
- In the Import Mode field, select
Insert. - In the Error Tolerance field, select either
All or nothingorAll or nothing, by file. - Have no attachment mappings or external link mappings specified.
- Have no logic blocks, except for Post-File logic blocks, specified.
The destination table cannot:
- Have workflow enabled.
- Have insert or validation triggers, or insert action blocks.
- Have translatable fields.
- Be used as an active cube or summary table data source.
If the data import has been configured with the appropriate requirements, the Optimized Import Mode checkbox becomes selected once the record has been saved.
Import data with a CSV file
System administrators and business users use the Data Import Definitions application to upload CSV files of external data into Nextworld Platform tables.
Prerequisites
In order to import data from external sources, you must first gather your data and save it as a CSV file.
Data Import Definitions
In the Data Import Definitions application, you must:
- Create a new record and select your destination table. This can either be an integration table, or an application table. Integration tables require additional configuration steps. Learn more in Data import integration configuration.
- In the File Type, select
CSV File. - By default, Data Imports expects comma-delimited CSV files. If your file uses a different delimiter, such as pipe, enter it in the Delimiter field on the General Configuration page. Leave this field empty to use a comma.
- On the Mappings page, in the Data Mappings subtable, map your external data into the destination table's fields. Learn more in Data import mapping configuration.
- On the Processing page, you can configure your error tolerance and dictate what logic blocks you want run at the time of data import.
- On the Import File Details page, select
Attachmentfor your File Location Type. - Once the record is saved, select the Attachments form action to attach your CSV file. Delete older attachments from the definition to avoid importing redundant data.
- Select the Run Import row action to import the data.
How the definition was configured determines where the imported data is sent. Data can be imported into an integration table, or directly into the destination table. If data is imported into an integration table, there are additional steps in the import process.
Integration table import
If the data is imported into an integration table, you must:
- Navigate to the integration application built over the integration table.
- Select the Process Run form action. This processes and modifies the data before it is sent to the appropriate internal application table.
- External data is imported into the integration table selected in the data import definition's Destination Field.
Import data with an XML file
System administrators and business users use the Data Import Definitions application to upload XML files of external data into Nextworld Platform tables.
Prerequisites
In order to import data from external sources, you must first:
- Gather your data and save it as an XML.
- Find the XSD file which provides the data shape for your XML data. The XSD describes the expected elements and data types within the XML, and is required for the mapping tool to correctly identify and display the fields available for import.
Data Import Definitions
In the Data Import Definitions application, you must:
- Create a new record and select your Destination Table. This can either be an integration table, or an application table. Integration tables require additional configuration steps. Learn more in Data import integration configuration.
- In the File Type, select
XML File. - On the Import File Details page, select
Attachmentfor your File Location Type. Then, upload the XSD file that contains the data shape for your XML. - On the Mappings page, map your XML file data into the fields of your destination table in the Visual Data Mapper. The column generated on the left of the Viewer panel holds your XSD data shape. The column on the right holds your internal table fields.
- To add constant values or format multicomponent values, such as currency and addresses, select a value in the Element Palette panel.
- Navigate back to the Import File Details page and upload the XML file that holds your data.
- On the Processing page, you can configure your error tolerance and dictate what logic blocks you want run at the time of data import.
- Save the record then select the Run Import action to import the data. Once the job has finished running, you'll receive a message in the Notifications menu. Optionally, you can configure your data import as a scheduled job. Learn more in Job Scheduler application.
How the definition was configured determines where the imported data is sent. Data can be imported into an integration table, or directly into the destination table. If data is imported into an integration table, there are additional steps in the import process.
Integration table import
If the data is imported into an integration table, you must:
- Navigate to the integration application built over the integration table.
- Select the Process Run form action. This processes and modifies the data before it is sent to the appropriate internal application table.
- External data is imported into the integration table selected in the data import definition's Destination Table.
Import data with an SFTP server
System administrators use the File Location Setup, Data Import Definitions, and Job Scheduler applications to import data, including attachments and images, from an SFTP server.
Prerequisites
In order to import data from external sources, you must access the SFTP server which receives the information you want to import.
File Location Setup
In the File Location Setup application, you must:
- Create a new record.
- Enter the username, authentication method, password, and other identification methods for the server.
Data Import Definitions
In the Data Import Definitions application, you must:
- Create a new record and select the integration table you want to import into. Integration tables require additional configuration steps. Learn more in Data import integration configuration.
- On the Mappings page, in the Data Mappings subtable, map your external data into the destination table's fields. Learn more in Data import mapping configuration.
- On the Processing page, you can configure your error tolerance and dictate what logic blocks you want run at the time of data import.
- On the Import File Details page, select
External Sourcefor the File Location Type. - If you're importing a CSV file and it uses a delimiter other than a comma, select the matching option in the Delimiter field on the General Configuration page. Leave this field empty to use a comma.
- In the File Location field, specify your file location setup record.
- On the Advanced Mappings page, map any pictures, attachments, or external links.
If you are importing images, on the Advanced Mappings page, in the Picture Mappings subtable, you must:
- In the Source Type field, select
Header. Then, enter the Header file name including the file name extension in the Source Field. For example, Image.PDF rather than just Image. If you do not specify the file format it can greatly decrease performance as the system searches for the closest match. - Select the Destination Field. This is the data item in your table where the image should appear.
- Specify the Folder where your image or attachment is located in the SFTP server.
If you are importing attachments, the configuration requirements are the same as images, except you do not need to specify a Source Field since attachments are saved to the record.
Once you have saved the record, select the Run Import form action to import the data, or configure a recurrent job to run in the Job Scheduler application.
Job Scheduler
If your data import should be run recurrently, in the Job Scheduler application, you must:
- Create a new record.
- On the Recurrence page, select your Recurrence Type.
- On the Jobs page, in the Jobs subtable, configure your job.
- In the Data Import Name field, enter the name of your data import definition.
Learn more in Scheduled jobs.
Historical data import configuration
Historical data is the information which a new customer imports into the platform. You can insert historical data directly into internal tables with data imports, or you can use integration tables and logic blocks to format and process the data.
Prerequisites
Historical data may violate filtered table lookups which are configured on internaltables to maintain data integrity. Add the znwHistorical data item in tables where historical data is being inserted to skip filtered table lookup enforcement. Filtered table lookups are still enforced if a table lookup field changes and the record is being updated.
The field behaves like a restricted field when it is added to a table.
Data Import Definitions pathway
To import historical data directly into a table, in the Data Import Definitions application, you must:
- Create a new data import definition with the table specified in the Destination Table, or create a copy of an existing definition with the correct Destination Table and mappings. Learn more about configuring data imports in Data imports.
- In the Transaction Handling section, select the Historical Import checkbox. You must have the
Data Import Is Historical Import Adminrole to select this checkbox. - Save the record and test that your definition successfully imported historical records.
Integration logic block pathway
To import historical data with integration logic blocks, in the Logic Block Builder, in your integration logic block, you must:
- Ensure your logic block is
Public-Secured. - Use the Set Values action to set the value to
Truein theznwHistoricalfield of your table. - Test the logic block changes were successful.
Data Import Definitions application
Use the Data Import Definitions application to incorporate external data into internal tables.
Data imports are a multistep configuration. Learn more in Data imports.
Data import definitions format how inbound data is imported into a specified table. If you are periodically importing the same shape of data into a table, you can reuse the same data import definition. If you import data that does not match the table or formatting of an existing definition, you must create a new one.
Use the header fields to specify a unique name for the data import definition and the table to which the inbound data should imported.
General Configuration page
Specify whether the import mode is an Insert or an Update , and what kind of file currently contains the inbound data.
Mappings page
Specify where and how you want the inbound data to import into a Nextworld table. Optionally, include notes about the purpose of each mapping.
Learn more about the Nextworld standards for data import definition mappings in the Integration table mapping for data imports topic.
Processing page
Configure aspects of the CSV file before you import the data. If there is a workflow over the destination table, configure what states the inbound data should have when it is imported. If the target destination is a Header Detail table, define the header table and specify how the header detail structure reacts to Header Detail import errors.
Specify the tolerance for errors in your data import. Selecting a less restrictive error tolerance makes it less likely for your imports to time out, since error tolerance specifies how many rows are inside a single transaction. Using an error tolerance of All except errors (Default) ensures transaction are small, which means the system is less likely to hit a limit within the allotted time.
You can also perform automatic updates on records that contain a specific field, and define which logic blocks on the destination table that should run when the import runs.
Import File Details page
Define the origin and file type of the source data.
Advanced Mappings page
Configure how pictures, attachments, and links associated with your inbound data are imported.
Data Import Runs page
View a list of data imports initiated within this record.
Data Import Run Inquiry application
Quick Import feature for Header Detail applications
Enable the Quick Import feature in Header Detail applications to allow users to import detail records directly within the application.
Quick Import bypasses the traditional Data Import Definition process. Users can access a CSV template from the form action menu within a Header Detail record, download it, fill it in, and import detail lines directly into the record.
To configure, select the Allow Quick Import checkbox in the Application Properties panel of your application record.
The types of data items which are supported include:
- Number
- TrueFalse
- Date
- DateTime
- Time
- Currency
- Text
- TableLookup
- The Primary Search Field can be of type:
- Text
- Name
- Number
- Other data item types are treated as Text fields internally. This could result in unexpected behavior.
- The Primary Search Field can be of type: