The following list of new features, updates and bug fixes are included in the latest Sprint 22.10 release to the FEWS NET Data Platform.

New Features

Introduce IPC HFA data domain (DATA-557)

The FDW is now able to capture Humanitarian Food Assistance Data. This data can be accessed by going to Dataseries Management > Acute Food Insecurity > Humanitarian Food Assistance.

Updates available in the refreshed FDE

The refreshed FDE now has a variety of new features including:

• The ability for users to hide data series with zero data points (DATA-1869)

• A new support widget (DATA-1926)

• All domains now include an info popup for each data series that includes additional information including the data usage policy (DATA-1934, DATA-2097)

• The ability for users to view metadata for selected data series within FDE (DATA-1942, DATA-1969, DATA-1972, DATA-2091)

• A filter design that is useable on a variety of device types (DATA-2116)

Read more about these features.

Updates

Add filtering by keyword for News Article API (DATA-1651)

The News Article API now accepts keywords as a filter parameter.

Updates to KoBo Toolbox

Several updates were made to the FEWS NET KoBoToolbox dashboard and forms including:

• Updates to the pipeline that ingests KoBo cross border trade data into FDW (DATA-1778)

• The addition of a spreadsheet to aid in the validation of KoBo data (DATA-1793)

• Updates to the East Africa cross border trade form to collect the GPS location of enumerator and pre-fill the country and border point (DATA-1811)

• Upgrading to latest version in order to access a new row-based permission feature (DATA-1827)

Faster API renders of endpoints with large drop-downs (DATA-1913)

Some endpoints, particularly ipcphase, were taking a long time to render in API format. Edits have been made which now allow them to render more quickly.

Refreshed domain names (DATA-1922, DATA-1939)

The names of the domains in the Data Platform have been reviewed and several have been updated across the refreshed FDE, FDW, and HDX. Name changes can be reviewed here.

Refreshed FDW Interface

The FDW user interface has been refreshed to reflect the new FEWS NET brand. Changes include:

• A new home page with widgets that make FDW more efficient.

• The ability to change the language in FDW.

• New colors to indicate country categories in the import dependency dashboard

• A variety of updates to colors, fonts, button positions, and other stylistic elements

Read more about the updates to the FDW. These changes included many tickets from prior sprints and several in this sprint (DATA-2035, DATA-2036, DATA-2040, DATA-2042).

Bug Fixes

Trying to open the Semi Structured Data Point page results in 500 Internal Server Error (DATA-1656)

Trying to open https://fdw.fews.net/en/admin/semistructured/semistructureddatapoint/ was resulting in an error. This has been fixed and is now working correctly.

DATA-1929 Unable to update datasourcedocument name

Description

As reported HELP-1291 - Getting issue details... STATUS the web form in the admin doesn’t allow users to update source document name when no license information is selected.

DATA-1935 Investigate issues with refresh materialized views debouncing incorrectly.

Description

The EWT have been having issues with uploaded data collections taking a long time to reflect the updates in FDW. The is because the refresh materialized views have not been running correctly. These can be run manually by ssh-ing into the server and running the commands from psql, like the below;

psql -c "REFRESH MATERIALIZED VIEW CONCURRENTLY price_marketpricefacts_materialized";

It looks as though at least the price_marketpricefacts_materialized is being continually debounced and not actually running. We see a log message like the below;

Task common.tasks.refresh_materialized_view[5d089b85-9811-48ed-a854-1938387de3b9] succeeded in 0.05116157094016671s: 'Request received for refresh_materialized_view_refresh_materialized_view-materialized_view=price_marketpricefacts_materialized while DEBOUNCING so re-scheduled, id 901fd5c6-04fc-4897-be17-7ad3cac3e369 eta 2022-09-30 11:49:53.046322+00:00.'

and tracing the new task that has been spawned we see the below log;

Task common.tasks.refresh_materialized_view[901fd5c6-04fc-4897-be17-7ad3cac3e369] succeeded in 0.03576034493744373s: 'Task refresh_materialized_view_refresh_materialized_view-materialized_view=price_marketpricefacts_materialized triggered to run twice, so cancelling second call 901fd5c6-04fc-4897-be17-7ad3cac3e369.'

The only reason that the code can be reached to execute the last log statement is if the debounce method fails to delete the HERD:{task_key} from the cache. This could happen if the HERD:{task_key} is not set in the first place. Chris Preager Could this also happen if the tasks are on different servers?

The code has the following comment.

# This can be caused by:
#     1. This cache delete occurs just before cache_set(HERD) in apply_async, deletes nothing, or
#        apply_async enqueues a debounce while this is running, and then this method enqueues a
#        duplicate below due to the renewed ETA.
#     2. This cache_delete(HERD) runs immediately after cache_set(HERD) in apply_async, so a
#        second call to apply_async goes through. In this case the latter call won't run after
#        debounce_wait seconds, but will be satisfied by this run anyway. A subsequent call will run fine.

DATA-1938 Fix Comtrade unit test erros

Update FAO pipeline based on changes in remote API (DATA-1943)

Description

The help ticket reported HELP-1278 - Getting issue details... STATUS gave us the chance to review the FAO pipeline, which is affected by the recent changes in the fao remote api. The change is noted below:

1. the datasereis api url is changed from https://fpma.apps.fao.org/giews/food-prices/tool/api/v1/series/ to https://fpma.fao.org/giews/v4/price_module/api/v1/FpmaSerie/

2. The current version also require a two step process to fetch the datapoints.
   First https://fpma.fao.org/giews/v4/price_module/api/v1/FpmaSerie/ returns the list of dataseries. Each dataseries has a guid type identifier as oppose to the previous api's series_id [eg. 8_23_642_2_434_nominal
   Then the datapoints can be fetched from a url like:
   https://fpma.fao.org/giews/v4/price_module/api/v1/FpmaSeriePrice/e23b83a7-7cf7-40b0-87f7-6c03a2d3b2bc/ and there is no datapoint_url now included in each dataseries

3. the seriesId is now has a different datatype - Guid and the column is named uuid

4. fao Country Code of value XXX used to be used to refer international prices, now they replaced that with IPS

5. market is now an id and market_name contains the name, commodity_name contains the product name where commodity used to reference

6. dateRanges field is now periodicity

These changes have side effect on the pipeline and we need to update it accordingly.

DATA-1954 Fix safety warning for protobuf

Description

The below safety error is causing issues in the CI pipelines.

protobuf | 3.20.1 | >=3.20.0rc0,<3.20.2 | 51167 |

A parsing vulnerability for the MessageSet type in the ProtocolBuffers versions prior to and including 3.16.1, 3.17.3, 3.18.2, 3.19.4, 3.20.1 and 3.21.5 for protobuf-cpp, and versions prior to and including 3.16.1, 3.17.3, 3.18.2, 3.19.4, 3.20.1 and 4.21.5 for protobuf-python can lead to out of memory failures. A specially crafted message with multiple key-value per elements creates parsing issues, and can lead to a Denial of Service against services receiving unsanitized input. We recommend upgrading to versions 3.18.3, 3.19.5, 3.20.2, 3.21.6 for protobuf-cpp and 3.18.3, 3.19.5, 3.20.2, 4.21.6 for protobuf-python. Versions for 3.16 and 3.17 are no longer updated.

https://gitlab.com/fewsnet/data/fdw/-/jobs/3126720223

DATA-2010 Bug when uploading XLSForm, both in old fdw KoBo and latest version

DATA-2083 SaveNewArticles fails for articles from multiple sources

Description

SaveNewArticle relies on ReadExistingArticles to avoid inserting duplicates.

However, if an article with the same URL has been downloaded from a different source, then it attempts to save the article using batch_insert, which raises IntegrityError.

This is happening at the moment for the UK FCDO source, which duplicates some (but not all) articles with the UK.gov source.

We will fix this by falling back to saving individual articles if we hit an error, and ignoring the duplicates.