Data processing is the second step in the data ingestion process. The Data Processing module allows users to review and approve submitted data and source documents in preparation for publishing to the FEWS NET Data Warehouse.

Submitted data can be used to build visualizations in the website CMS. Once published, data becomes available for users with the appropriate permissions to view and export in the Data Explorer system.

From the Data Processing module, the following actions can be performed:

Display or edit an existing data collection
Display or edit the data for an existing collection period

Display or edit an existing data collection

The data collections screen allows you to submit, review, publish, and view an entire data collection. A data collection is a set of data points added in a single import. It may contain multiple data series or data collection periods.

The data collections page lists the currently available collections in a grid format with the following columns:

Source document: The name of the source document as defined in the Data entry process.
Document type: The domain or data type qualification that the data is being submitted under.
Collection date: The collection date of the submitted data collection.
File: Link to the submitted file. Because data can be entered manually, not all collections have a file available.
Status: Status types include Draft, Submitted, Under Review, Published.
Publication name: The name of the publication that the data source document came from.
Status changed: Indicator of when the status was last updated.
Data point count: Indicator of file size as represented by the number of data points submitted in the data collection.

Under the heading, Select data collection to change, there is a search bar. Beneath it, available years are listed from 1960 to 2022. There is an action dropdown list, and a button to add data collection. The grid displays available collections.

Navigating the data collections table

In addition to a free text criteria search bar, filters can also be applied. To set filter criteria, locate the Filter menu. A search bar is available to narrow the filter options. Once a filter is selected from the list, the system will automatically apply the filter and display the results.

The filter menu is located on the right-hand side of the page. At the top is a button to hide the filter pane.

Editing the status information of a data collection

The data collection status can be changed in two ways:

From the table view, use the checkboxes to select one or more data collections and select a status from the Action dropdown:
1. Delete will delete the data collection. If deleted, it cannot be recovered.
2. Submit will update the status to Submitted.
3. Publish will update the status to Published.
4. Review will update the status to Under Review.
Select the data collection from the table to navigate to the Change data collection screen.
1. Modify the data in the Change data collection page as needed and select a Save option.
2. Select Submit (for data collections in Draft or Under Review status) or Review (for data collections that have been Published).

The modifiable fields for the FEWS NET, Afghanistan Key Messages Update, Afghanistan 2023-01-31 data collection includes Geographic Unit, Classification Scale, Scenario, Is Accounting for Assistance, Status, Phase, Description and Previous Value.

Viewing change history of a data collection

When various events are performed against data within a data collection, including the addition of new data, updates, and deletion of data in the system, they are tracked for auditing purposes.

To view the change history of a data collection:

Select the data collections for which the change history is needed.
Locate and select the History button.

The change history screen shows the date and time, user, message and record. Hover over each entry in the Record column for more detail.

Display or edit the data for an existing collection period

The data collection periods screen allows you to view and edit a single collection period.

The data collection periods page lists the currently available collections in a grid format with the following columns:

Collection: The name of the data collection as defined in the Data entry process.
Start date: The start date of the data collection period.
Period date: The last day of the data collection period.
Schedule: The schedule on which the data is collected.
Collection Status: Status types include Draft, Submitted, Under Review, and Published.
Status changed: Indicator of when the status was last updated.
Data point count: The number of data points included in the data collection period.

View and edit a data collection period

To view and edit a data collection period:

Select the data collection period to be updated.
On the Change collection period page, you can:
1. Add notes to a data collection period
2. View the history of the data collection period
3. Edit the status and values of specific data points within the data collection period
Select one of the Save options.