The following list of new features, updates and bug fixes are included in the latest Sprint 22.10 release to the FEWS NET Data Platform.

New Features

Introduce IPC HFA data domain (DATA-557)

The FDW is now able to capture Humanitarian Food Assistance Data. This data can be accessed by going to Dataseries Management > Acute Food Insecurity > Humanitarian Food Assistance.

Updates available in the refreshed FDE

The refreshed FDE now has a variety of new features including:

• The ability for users to hide data series with zero data points (DATA-1869)

• A new support widget (DATA-1926)

• All domains now include an info popup for each data series that includes additional information including the data usage policy (DATA-1934, DATA-2097)

• The ability for users to view metadata for selected data series within FDE (DATA-1942, DATA-1969, DATA-1972, DATA-2091)

• A filter design that is useable on a variety of device types (DATA-2116)

Read more about these features.

Updates

Add filtering by keyword for News Article API (DATA-1651)

The News Article API now accepts keywords as a filter parameter.

Refreshed domain names (DATA-1922, DATA-1939)

The names of the domains in the Data Platform have been reviewed and several have been updated across the refreshed FDE, FDW, and HDX. Name changes can be reviewed here.

Refreshed FDW Interface (DATA-2040,

The FDW user interface has been refreshed to reflect the new FEWS NET brand. Changes include:

Updates to KoBo Toolbox (DATA-1778, DATA-1793, DATA-1811, DATA-1827

Description

Hello Henock and Girum,

Thanks so much for attending the East Africa XBT training and Kobo toolbox rollout. I wanted to recap and list the challenges and the issues we encountered with the backed of kobo toolbox data viewing dashboard. I would like to address those as much as possible as it impacts our users' LOE when they review and download data and how we want to move forward with similar activities if we want to use Kobo.

• Filter records for a specific period of time. In our case, weekly data. The start and end date in the dashboard is an indicator of time spent by the enumerator to complete a record from start to submission. Is there a way to filter data for a range of days? we can only filter by one day now using the "reporting date" column. This is an important feature to have and be able to filter data based on a range of days.

• Validation status: is it possible to change the options to draft, submitted, not approved, and approved? we are trying to set a process where we will use this column for the country offices and regional data assistant to communicate the status of records and their readiness to be submitted to the FDW.

• Filtering products with () in the name such as Rice (Paddy). When we filter to see data for any product that contains () in the names, we don't get any results.

• Instead of submitting by and adding the "username" having the enumerator's name would make it easier to filter by.

• downloading data for a range of days. When we filter data based on some parameters, we weren't able to download the results. Instead, we get an extract of all data collected. this is a huge limitation for our country office who'd want to review weekly data in Excel or other formats on the side or just want a quick extract.

• View submissions from specific users: this was a limitation for country offices who needs to see records for only specific users (their country enumerators only) and be able to edit records if needed. At the moment, if we give "view submissions only from specific users" permission to a user, he won't be able to edit records from those users. This is a big limitation. As of now, when Sudan's country rep logs in to Kobo he will see records for all countries while he only needs to see records for Sundan's enumerators only and edit them if needed.

• Another limitation is contact management. It's hard to manage the enumerators' profiles and contacts list from the backend and see who has access to which form and also see the enumerator profile (name, location, etc). I am also speaking from the point of someone who's managing more than one form.

• When filtering by "group border point/reporting country" and selecting a country, the "group board point/border point" drop-down menu should display border points for the country selected and not all border points.

Additional items we needed to adjust in the form itself (Moved to DATA-1811):

• Add GPS location of the enumerator.

• Adding form submission radius not more than 10km

• When an enumerator logs in, the country and border point will be prefilled to avoid data entry errors.

Upgrade FEWS KoBo to latest version

Description

To be able to use the newer row-based permission feature in KoBo, we need to upgrade the KoBo version currently installed for FEWS to the latest available version.DATA-1913 Speed up api format renders of endpoints with large drop-downs

Bug Fixes

DATA-1656 Trying to open https://fdw.fews.net/en/admin/semistructured/semistructureddatapoint/ results in 500 Internal Server Error

DATA-1929 Unable to update datasourcedocument name

Description

As reported HELP-1291 - Getting issue details... STATUS the web form in the admin doesn’t allow users to update source document name when no license information is selected.

DATA-1935 Investigate issues with refresh materialized views debouncing incorrectly.

Description

The EWT have been having issues with uploaded data collections taking a long time to reflect the updates in FDW. The is because the refresh materialized views have not been running correctly. These can be run manually by ssh-ing into the server and running the commands from psql, like the below;

psql -c "REFRESH MATERIALIZED VIEW CONCURRENTLY price_marketpricefacts_materialized";

It looks as though at least the price_marketpricefacts_materialized is being continually debounced and not actually running. We see a log message like the below;

Task common.tasks.refresh_materialized_view[5d089b85-9811-48ed-a854-1938387de3b9] succeeded in 0.05116157094016671s: 'Request received for refresh_materialized_view_refresh_materialized_view-materialized_view=price_marketpricefacts_materialized while DEBOUNCING so re-scheduled, id 901fd5c6-04fc-4897-be17-7ad3cac3e369 eta 2022-09-30 11:49:53.046322+00:00.'

and tracing the new task that has been spawned we see the below log;

Task common.tasks.refresh_materialized_view[901fd5c6-04fc-4897-be17-7ad3cac3e369] succeeded in 0.03576034493744373s: 'Task refresh_materialized_view_refresh_materialized_view-materialized_view=price_marketpricefacts_materialized triggered to run twice, so cancelling second call 901fd5c6-04fc-4897-be17-7ad3cac3e369.'

The only reason that the code can be reached to execute the last log statement is if the debounce method fails to delete the HERD:{task_key} from the cache. This could happen if the HERD:{task_key} is not set in the first place. Chris Preager Could this also happen if the tasks are on different servers?

The code has the following comment.

# This can be caused by:
#     1. This cache delete occurs just before cache_set(HERD) in apply_async, deletes nothing, or
#        apply_async enqueues a debounce while this is running, and then this method enqueues a
#        duplicate below due to the renewed ETA.
#     2. This cache_delete(HERD) runs immediately after cache_set(HERD) in apply_async, so a
#        second call to apply_async goes through. In this case the latter call won't run after
#        debounce_wait seconds, but will be satisfied by this run anyway. A subsequent call will run fine.

DATA-1938 Fix Comtrade unit test erros

Update FAO pipeline based on changes in remote API (DATA-1943)

Description

The help ticket reported HELP-1278 - Getting issue details... STATUS gave us the chance to review the FAO pipeline, which is affected by the recent changes in the fao remote api. The change is noted below:

1. the datasereis api url is changed from https://fpma.apps.fao.org/giews/food-prices/tool/api/v1/series/ to https://fpma.fao.org/giews/v4/price_module/api/v1/FpmaSerie/

2. The current version also require a two step process to fetch the datapoints.
   First https://fpma.fao.org/giews/v4/price_module/api/v1/FpmaSerie/ returns the list of dataseries. Each dataseries has a guid type identifier as oppose to the previous api's series_id [eg. 8_23_642_2_434_nominal
   Then the datapoints can be fetched from a url like:
   https://fpma.fao.org/giews/v4/price_module/api/v1/FpmaSeriePrice/e23b83a7-7cf7-40b0-87f7-6c03a2d3b2bc/ and there is no datapoint_url now included in each dataseries

3. the seriesId is now has a different datatype - Guid and the column is named uuid

4. fao Country Code of value XXX used to be used to refer international prices, now they replaced that with IPS

5. market is now an id and market_name contains the name, commodity_name contains the product name where commodity used to reference

6. dateRanges field is now periodicity

These changes have side effect on the pipeline and we need to update it accordingly.

DATA-1954 Fix safety warning for protobuf

Description

The below safety error is causing issues in the CI pipelines.

protobuf | 3.20.1 | >=3.20.0rc0,<3.20.2 | 51167 |

A parsing vulnerability for the MessageSet type in the ProtocolBuffers versions prior to and including 3.16.1, 3.17.3, 3.18.2, 3.19.4, 3.20.1 and 3.21.5 for protobuf-cpp, and versions prior to and including 3.16.1, 3.17.3, 3.18.2, 3.19.4, 3.20.1 and 4.21.5 for protobuf-python can lead to out of memory failures. A specially crafted message with multiple key-value per elements creates parsing issues, and can lead to a Denial of Service against services receiving unsanitized input. We recommend upgrading to versions 3.18.3, 3.19.5, 3.20.2, 3.21.6 for protobuf-cpp and 3.18.3, 3.19.5, 3.20.2, 4.21.6 for protobuf-python. Versions for 3.16 and 3.17 are no longer updated.

https://gitlab.com/fewsnet/data/fdw/-/jobs/3126720223

DATA-2010 Bug when uploading XLSForm, both in old fdw KoBo and latest version

DATA-2083 SaveNewArticles fails for articles from multiple sources

Description

SaveNewArticle relies on ReadExistingArticles to avoid inserting duplicates.

However, if an article with the same URL has been downloaded from a different source, then it attempts to save the article using batch_insert, which raises IntegrityError.

This is happening at the moment for the UK FCDO source, which duplicates some (but not all) articles with the UK.gov source.

We will fix this by falling back to saving individual articles if we hit an error, and ignoring the duplicates.