Data is added to FDW through two primary methods:
Users can upload or manually enter data directly to the platform.
New alphabetic, numeric, or symbolic data is added using the Data entry module, described below.
Data pipelines automatically ingest data from web services external to the platform on an established schedule.
This document provides guidance for preparing and adding data using the Data entry module.
Preparing Data for Upload
To ensure a successful upload of data:
Data values must be accurately matched to their column headings.
Column headings must be recognized by FDW; column names may need to be adjusted for ad hoc data.
Metadata values must be recognized by FDW; new metadata must be added through the data series or metadata importing screen.
Use of Templates
IPC Classification Data
The IPC classification data template provides data for four source documents which can be FSO, FSO Internal, FSOU, FSA and KMU select.
Multiple data collections can be created from one upload (Food Security Area Classification, HFA, and IDP).
The source document must align with the date range included in the template or data will not be uploaded. Data collection should be uploaded one month at a time. The collection date should always be the first day of the month.
The first month in the data collection period includes the CS (current situation), ML (near term projection three months), ML2 (medium term projection six months). The outlook collection date is the first date from the current state through four months. A new collection date begins after four months.
FAOB Data
The FAOB template allows multiple data collections to be created from one upload (public and internal population estimates, and public and internal highest household).
Preparing internal and public FAOB data in the template will fall under one of the following three scenarios:
Public and internal data values are the same:
Provide values in the columns for public data and leave the columns for internal data blank. FDW will ascribe the values to both source documents.
Public and internal data values differ:
Provide values in the columns for public and internal data. FDW will separate out the data into the source documents.
Internal data only:
Provide values in the columns for internal data, and either:
Specify NA values in the columns for public data to include the country as not analyzed in extracts of that month’s public data, or
Leave the columns for public data blank if you plan to add public data at a later time. If the values are blank, the country will not be reflected in extracts of that month’s public data.
Data Entry
The Data Entry module provides three methods for adding data.
Add a Data Collection is recommended for uploading monthly data.
Import Data Points is recommended for small bulk uploads of historical data or corrections. Each data collection must be created separately.
Offline Data Uploads is recommended for large bulk uploads of historical data or corrections. When using this method, users will not receive immediate feedback on the success of the upload.
Add a new data collection
To add a new data collection from the Data entry module, click Add a new Data collection. A form will appear with the following fields:
*Source document: Specify the Data Source Document as an Excel worksheet attachment, or manual entry.
*Publication name: Enter the publication name.
*Collection date: Specify the collection date of the data series included within the data source document.
File: Upload an Excel worksheet as an attachment. If no attachment is provided, the user will be redirected to a screen allowing for manual data entry.
(*) represents a required field
Additional fields:
Status: An indicator for the current status of the Data Source Document. Default status during the Data entry phase is Draft.
Status changed: A tracking feature that records the date, time, and action taken regarding changes in status of a Data Source Document.
Import log: For tracking notification warnings or errors, a log file is generated after the creation of every data collection. Warning notifications are sent to the FEWS NET Home Office Markets and Trade team to make the relevant corrections. This action ensures the Alias Market name is specified correctly, and that the appropriate data series are identified.
Collection periods: Specify the collection periods of the data source document.
Start date: The start date of a collection period, usually the first day of a month.
Period date: Refers to the last day of a collection period. If set to monthly, it will be the last day of that same month.
Notes: A text field for notes and comments to improve context.
When ready, click on either the Save, Save and add another, or Save and continue editing buttons to save the data into the system and prepare it for processing. The system will display a message confirming the success of the saving operation.
Save: Save and return to the Data entry homepage.
Save and add another: Implies adding another data collection. Once the current one is saved, the system redirects the user to a similar window to repeat the process as many times as desired.
Save and continue editing: The system allows a user to save the current work and continue editing which serves to minimize the loss of entered data in case of a poor internet connection.
Manual data entry
In cases where an Excel worksheet is not loaded, a user can enter values and set the status manually.
Note: Previous Price and Percentage Change will only be recognized or appear automatically if values were entered for the previous period (the last day of the previous month). This field is not required.
Status options
Collected: Most common status. It means there is a value for the Data-Point in the Data Series for this Data Collection Period.
Not Collected: The Enumerator did not collect the price (a value for the Data Series) this month (for this Data Collection Period), and availability of the product is unknown.
Not Available: The Enumerator attempted to collect the data (a value for the Data Series for this Data Collection Period) but the product was not on sale/available in the Market.
Missing Historic Data: Should not be used by the Field Office because this status does not involve any new data. Concerns missing prices from historical data uploaded in bulk by Home Office: The data is not present in FEWS NET’s historic data; Unclear whether it was not available or not collected.
Bulk uploads
The Import data points and Offline data uploads screens should be used to bulk upload historical data or corrections. It is best to use Import data points when possible as this allows changes to be reviewed before the import is finalized. However, this option can be time consuming when there is large amounts of data so Offline data uploads should be used for large uploads.
Import data points
Data series can be imported in the following formats ( XLS, CSV, TSV, JSON, and YAML ). To import data series, follow the steps below:
Choose an item from the Import data points sub-menu.
Click the Choose File button and choose a file to upload.
Specify the file format that will be imported using the Format drop-down list.
Click Submit to start the import process.
Once the file is uploaded, the system will display the summary including:
New: the data in the file that will be imported.
Updated: the data in the file that are updates to existing data in the system.
Delete: the records in the file that will be deleted from the system.
Skip: the records in the file that will be skipped.
Error: the records in the file that have errors in the data and will not be added to the system.
Total: total number of records in the file.
To continue with the import, select Confirm import. The data series will be imported and a confirmation message will appear when complete.
Offline data uploads
The Offline data uploads screen displays all the data upload records in the system together with their status, data domain, country information, and the user who initiated the upload process.
From this screen users can perform the following activities:
Upload data contained in a file into the system by selecting a file, the format of the file containing the data (XLS, CSV, …), the relevant data domain, and the country
View the data upload process status (in progress, succeeded, failed)
Displays the data upload result for each row of data in the data upload document
Log records in the system while uploading the data for each record
Edit data upload record
Delete one or more data upload records
Reprocess or restart the data upload works for one or more data upload processes.
Search data upload records using a free text search criteria
Filter data upload records using one or more of the following criteria: status, sub type (data domain), user who changed the status, and country.