Versions Compared

Old Version 4

changes.mady.by.user Zoe Wingate

Saved on

compared with

New Version Current

changes.mady.by.user Allison Nederveld

Saved on

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

Preparing Data for Upload

To ensure a successful upload of data:

...

There are three categories of information that need to be checked when preparing data for upload.

Info

Spreadsheets for most domains require a straight table with one data point per row. Certain domains, such as Prices, will load cross-tabs (i.e., with markets down the side and products across the top).

Column headings

Column headings must be recognized by FDW. When uploading ad hoc data, column names may need to be adjusted.

Note

Important: If you regularly receive a document from a non-FEWS NET source that contains alternate column headings, do not edit the column headings every time. Instead, submit a help desk ticket so the Hub can add the headings as aliases to the system.

Source files uploaded to FDW must include properly defined column headings, including:

  • Column headings with correctly spelled metadata fields.

  • Column headings that contain metadata fields recognized by FDW.

  • A column heading for each column that contains data.

The examples below describe common errors related to column headings:

Expand
titleExample 1: Incorrect Column Heading

Scenario: Uploading price data using the import screen (https://fdw.fews.net/en/admin/price/marketprice/import/) results in the following error message:

Warning

‘country’ is not in list

Explanation: The country column heading is missing or spelled incorrectly in the source file.

Solution: Ensure the source file contains the corrected column heading.

Expand
titleExample 2: Mismatched or Blank Column Heading

Scenario: Uploading price data from the import screen (https://fdw.fews.net/en/admin/price/marketprice/import/) results in the following error message:

Warning

['Cannot identify a DataCollectionPeriod for every row.']  

Explanation: The error indicates a mismatch in the column headings contained in the file and those recognized by FDW. For example, the data_source_document name or data_source_organization name in the file do not align with those in FDW. This error may also indicate that the file contains a blank column heading.

Solution: Ensure all column headings are titled correctly and there are no blank headings.

Metadata

Metadata values must be recognized by FDW. New metadata must be added through the data series or metadata importing screen.

Use of Templates

IPC Classification Data

  • The IPC classification data template provides data for four source documents which can be FSO, FSO Internal, FSOU, FSA and KMU select.

  • Multiple data collections can be created from one upload (Food Security Area Classification, HFA, and IDP).

  • The source document must align with the date range included in the template or data will not be uploaded. Data collection should be uploaded one month at a time. The collection date should always be the first day of the month. 

  • The first month in the data collection period includes the CS (current situation), ML (near term projection three months), ML2 (medium term projection six months). The outlook collection date is the first date from the current state through four months. A new collection date begins after four months.

FAOB Data

...

The FAOB template allows multiple data collections to be created from one upload (public and internal population estimates, and public and internal highest household).

Preparing internal and public FAOB data in the template will fall under one of the following three scenarios:

...

Public and internal data values are the same:

  • Provide values in the columns for public data and leave the columns for internal data blank. FDW will ascribe the values to both source documents. 

...

Public and internal data values differ:

  • Provide values in the columns for public and internal data. FDW will separate out the data into the source documents. 

Internal data only:

Provide values in the columns for internal data, and either:

...

Specify NA values in the columns for public data to include the country as not analyzed in extracts of that month’s public data, or

...

Source files uploaded to FDW must include appropriate metadata information, including:

  • Metadata values that match those available in FDW.

  • Compatible metadata values for each record.

  • Metadata values free of extra characters or values that cannot be recognized.

The examples below describe common errors related to metadata:

Expand
titleExample 1: Invalid CPI Metadata Values

Scenario: Uploading CPI data (https://fdw.fews.net/en/admin/price/priceindexvalue/import/) results in 1 skipped result.

Explanation: The metadata values for the columns (such as the index_name or geographic_unit) do not match those in FDW.

Solution: Correct the metadata values to match those in FDW.

Expand
titleExample 2: Invalid Exchange Rate Metadata Values

Scenario: Uploading exchange rate data from the import screen (https://fdw.fews.net/en/admin/price/exchangeratevalue/import/) results in the following error message:

Warning

· Cannot identify Data Series for row {'datapoint_status': 'Collected', 'dataseries': None, 'market': None, 'datasourcedocument': <DataSourceDocument: FEWS NET, Democratic Republic of the Congo: Exchange rate, Democratic Republic of the Congo>, 'data_domain': 'ExchangeRate', 'from_currency': 'USD', 'to_currency': 'CDF', 'value': 2028.0, 'rate_type': 'Secondary', 'market_name': 'Uvira, Maendeleo', 'country': 'Congo, The Democratic Republic of the', 'data_source_organization': 'FEWS NET, Democratic Republic of the Congo', 'data_source_document': 'Exchange rate, Democratic Republic of the Congo', 'period_date': 44592.0}

Explanation: The market name in FDW and the file do not match.

Solution: Correct the market name in the file.

Expand
titleExample 3: Incompatible metadata values

Scenario: Uploading cross border trade metadata (https://fdw.fews.net/en/admin/trade/tradeflowquantity/import/) results in 3 invalid results:

Image Added

Explanation: The invalid results are caused by incompatible values across the flow_type, source_country, and destination_country fields. E.g., a flow_type value of export cannot be valid if the source_country and destination_country values are the same.

Solution: The values must be corrected to upload the data successfully.

Expand
titleExample 4: Extra character or value

Scenario: A user uploads IPC data (https://fdw.fews.net/en/admin/warehouse/dataupload/add/) and receives the following error message:

Warning

‘NoneType’ object is not iterable

Explanation: This error is caused by an extra character or value under the metadata columns. Solution: Removing the extra character or value will resolve the issue. You can use the TRIM function in Excel to remove extra spaces from text, leaving only a single space between words and no space characters at the start or end of the text.

Expand
titleExample 5: Metadata not in FDW

Scenario: A user uploads price data from either the import screen (https://fdw.fews.net/en/admin/price/marketprice/import/) or data collection screen (https://fdw.fews.net/en/admin/warehouse/datacollection/add/) and receives a skipped result:

Image AddedImage Added

Explanation: The file contains a new metadata record not in FDW.

Solution: Check for new metadata information that needs to be added through the data series or metadata importing screen before uploading the file again.

Image Added

Data values

Data values must be accurately matched to their column headings.

Source files uploaded to FDW must include appropriate data values, including:

  • Data values that align with the correct metadata field in the column headings.

  • Data values that are within an expected range.

The examples below describe common errors related to data values:

Expand
titleExample 1: Data values do not match metadata

Scenario: A user uploads price data from the data collection screen (https://fdw.fews.net/en/admin/warehouse/datacollection/add/) and receives the following error message:

Warning

Error in 3 market prices from uploaded file warehouse/data collection/filename.xlsxs

Similarly, uploading the same file from the import screen (https://fdw.fews.net/en/admin/price/marketprice/import/) results in the following error message:

Warning

 ‘retail’

Explanation: The file contains interchanged data values for the price_type and product_source columns. As the data values could not be matched to the metadata fields in the column headings, they could not be uploaded.

Image Added

Solution: Correcting the file so the data values align with the correct metadata field will resolve the issue.