The following list of new features, updates and bug fixes are included in the latest Sprint 22.10 release to the FEWS NET Data Platform.

New Features

Introduce IPC HFA data domain (DATA-557)

The FDW is now able to capture Humanitarian Food Assistance Data. This data can be accessed by going to Dataseries Management > Acute Food Insecurity > Humanitarian Food Assistance.

...

Updates available in the refreshed FDE

The refreshed FDE now has a variety of new features including:

• A new support widget (DATA-1926)

• The ability for users to hide data series with zero data points (DATA-1869, DATA-2045)

• A filter design that is useable on a variety of device types (DATA-2116new support widget (DATA-1926)

• All domains now include an info popup for each data series that includes additional information including the data usage policy (DATA-1934, DATA-2097)

• The ability for users to view metadata for selected data series within FDE (DATA-1942, DATA-1969, DATA-1972, DATA-2091)All domains now include an info popup for each data series that includes additional information including the data usage policy (DATA-2097, DATA-1934

• A new design for the search and filter section that is useable on a variety of device types (DATA-2116)

Read more about these features.

Introduce IPC HFA data domain (DATA-557)

The FDW is now able to capture Humanitarian Food Assistance Data. This data can be accessed by going to Dataseries Management > Acute Food Insecurity > Humanitarian Food Assistance.

...

Updates

...

Refreshed FDW interface

The FDW user interface has been refreshed to reflect the new FEWS NET brand. Changes include:

• A new home page with widgets that make FDW more efficient

• The ability to change the language in FDW

• New colors to indicate country categories in the import dependency dashboard

• A variety of updates to colors, fonts, button positions, and other stylistic elements

Read more about the updates to the FDW. These changes included many tickets from prior sprints and several in this sprint (DATA-2035, DATA-2036, DATA-2040, DATA-2042).

...

Updates

Add filtering by keyword for News Article API (DATA-1651)

The News Article API now accepts keywords as a filter parameter.

Updates to KoBo Toolbox

DATA-1793 Kobo toolbox dashboard

Description

Hello Henock and Girum,

Thanks so much for attending the East Africa XBT training and Kobo toolbox rollout. I wanted to recap and list the challenges and the issues we encountered with the backed of kobo toolbox data viewing dashboard. I would like to address those as much as possible as it impacts our users' LOE when they review and download data and how we want to move forward with similar activities if we want to use Kobo.

• Filter records for a specific period of time. In our case, weekly data. The start and end date in the dashboard is an indicator of time spent by the enumerator to complete a record from start to submission. Is there a way to filter data for a range of days? we can only filter by one day now using the "reporting date" column. This is an important feature to have and be able to filter data based on a range of days.

• Validation status: is it possible to change the options to draft, submitted, not approved, and approved? we are trying to set a process where we will use this column for the country offices and regional data assistant to communicate the status of records and their readiness to be submitted to the FDW.

• Filtering products with () in the name such as Rice (Paddy). When we filter to see data for any product that contains () in the names, we don't get any results.

• Instead of submitting by and adding the "username" having the enumerator's name would make it easier to filter by.

• downloading data for a range of days. When we filter data based on some parameters, we weren't able to download the results. Instead, we get an extract of all data collected. this is a huge limitation for our country office who'd want to review weekly data in Excel or other formats on the side or just want a quick extract.

• View submissions from specific users: this was a limitation for country offices who needs to see records for only specific users (their country enumerators only) and be able to edit records if needed. At the moment, if we give "view submissions only from specific users" permission to a user, he won't be able to edit records from those users. This is a big limitation. As of now, when Sudan's country rep logs in to Kobo he will see records for all countries while he only needs to see records for Sundan's enumerators only and edit them if needed.

• Another limitation is contact management. It's hard to manage the enumerators' profiles and contacts list from the backend and see who has access to which form and also see the enumerator profile (name, location, etc). I am also speaking from the point of someone who's managing more than one form.

• When filtering by "group border point/reporting country" and selecting a country, the "group board point/border point" drop-down menu should display border points for the country selected and not all border points.

Additional items we needed to adjust in the form itself (Moved to DATA-1811):

• Add GPS location of the enumerator.

• Adding form submission radius not more than 10km

• When an enumerator logs in, the country and border point will be prefilled to avoid data entry errors.

DATA-1811 East Africa XBT Kobo Form Modifications

Description

As outlined in

• Add GPS location of enumerators.

• Allow form submission only for a radius of not more than 10km

• Prefill default border-points based on the user using the form.

We opted to handle these changes separately because they are changes that affect the data collection process and relatively urgent than the rest of the Kobo data management and review issues mentioned in

DATA-1827 Upgrade FEWS KoBo to latest version

Description

To be able to use the newer row-based permission feature in KoBo, we need to upgrade the KoBo version currently installed for FEWS to the latest available version.DATA-1913 Speed up api format Several updates were made to the FEWS NET KoBoToolbox dashboard and forms including:

• Updates to the East Africa cross border trade form to collect the GPS location of enumerator and pre-fill the country and border point (DATA-1811)

• Upgrading to latest version in order to access a new row-based permission feature (DATA-1827)

Faster API renders of endpoints with large drop-downs (DATA-1913)

Description

Some endpoints, particularly ipcphase‘IPCphase’, take were taking a long time to render in api format (browsable), because they contain foreign keys to large tables, which are rendered as HTML SELECT OPTION tags.

We have attempted to replace them with auto-corrects, but never made a solid fix.

In most cases these would be more usable simply as TEXT inputs to type an ID in toAPI format. Edits have been made which now allow them to render more quickly.

Refreshed domain names (DATA-1922, DATA-1939)

The names of the domains in the Data Platform have been reviewed and several have been updated across the refreshed FDE, FDW, and HDX. Name changes can be reviewed here.

...

Update FAO pipeline based on changes in remote

...

Description

The help ticket reported

1. the datasereis api url is changed from https://fpma.apps.fao.org/giews/food-prices/tool/api/v1/series/ to https://fpma.fao.org/giews/v4/price_module/api/v1/FpmaSerie/

2. The current version also require a two step process to fetch the datapoints.
   First https://fpma.fao.org/giews/v4/price_module/api/v1/FpmaSerie/ returns the list of dataseries. Each dataseries has a guid type identifier as oppose to the previous api's series_id [eg. 8_23_642_2_434_nominal
   Then the datapoints can be fetched from a url like:
   https://fpma.fao.org/giews/v4/price_module/api/v1/FpmaSeriePrice/e23b83a7-7cf7-40b0-87f7-6c03a2d3b2bc/ and there is no datapoint_url now included in each dataseries

3. the seriesId is now has a different datatype - Guid and the column is named uuid

4. fao Country Code of value XXX used to be used to refer international prices, now they replaced that with IPS

5. market is now an id and market_name contains the name, commodity_name contains the product name where commodity used to reference

6. dateRanges field is now periodicity

These changes have side effect on the pipeline and we need to update it accordingly.

DATA-1994 Add dataset name to cross-domain dataset endpoint

Description

Make the dataset name field available on the cross-domain api/dataset endpoint

Bug Fixes

...

API (DATA-1943)

Recent changes to the FAO remote API effected the FAO pipeline. Updates were made to the pipeline and it is now working as expected.

Bug Fixes

Trying to open the Semi Structured Data Point page results in 500 Internal Server Error (DATA-1656)

Trying to open https://fdw.fews.net/en/admin/semistructured/semistructureddatapoint/

...

was resulting in an error. This has been fixed and is now working correctly.

Unable to update datasourcedocument name

...

Description

...

...

(DATA-1929)

Users were unable to update a source document name when no license information is selected.

DATA-1935 Investigate issues with refresh materialized views debouncing incorrectly.

Description

The EWT have been having issues with uploaded data collections taking a long time to reflect the updates in FDW. The is because the refresh materialized views have not been running correctly. These can be run manually by ssh-ing into the server and running the commands from psql, like the below;

psql -c "REFRESH MATERIALIZED VIEW CONCURRENTLY price_marketpricefacts_materialized";

It looks as though at least the price_marketpricefacts_materialized is being continually debounced and not actually running. We see a log message like the below;

Task common.tasks.refresh_materialized_view[5d089b85-9811-48ed-a854-1938387de3b9] succeeded in 0.05116157094016671s: 'Request received for refresh_materialized_view_refresh_materialized_view-materialized_view=price_marketpricefacts_materialized while DEBOUNCING so re-scheduled, id 901fd5c6-04fc-4897-be17-7ad3cac3e369 eta 2022-09-30 11:49:53.046322+00:00.'

and tracing the new task that has been spawned we see the below log;

Task common.tasks.refresh_materialized_view[901fd5c6-04fc-4897-be17-7ad3cac3e369] succeeded in 0.03576034493744373s: 'Task refresh_materialized_view_refresh_materialized_view-materialized_view=price_marketpricefacts_materialized triggered to run twice, so cancelling second call 901fd5c6-04fc-4897-be17-7ad3cac3e369.'

The only reason that the code can be reached to execute the last log statement is if the debounce method fails to delete the HERD:{task_key} from the cache. This could happen if the HERD:{task_key} is not set in the first place. Chris Preager Could this also happen if the tasks are on different servers?

The code has the following comment.

# This can be caused by:
#     1. This cache delete occurs just before cache_set(HERD) in apply_async, deletes nothing, or
#        apply_async enqueues a debounce while this is running, and then this method enqueues a
#        duplicate below due to the renewed ETA.
#     2. This cache_delete(HERD) runs immediately after cache_set(HERD) in apply_async, so a
#        second call to apply_async goes through. In this case the latter call won't run after
#        debounce_wait seconds, but will be satisfied by this run anyway. A subsequent call will run fine.

DATA-1938 Fix Comtrade unit test erros

DATA-1954 Fix safety warning for protobuf

Description

The below safety error is causing issues in the CI pipelines.

protobuf | 3.20.1 | >=3.20.0rc0,<3.20.2 | 51167 |

A parsing vulnerability for the MessageSet type in the ProtocolBuffers versions prior to and including 3.16.1, 3.17.3, 3.18.2, 3.19.4, 3.20.1 and 3.21.5 for protobuf-cpp, and versions prior to and including 3.16.1, 3.17.3, 3.18.2, 3.19.4, 3.20.1 and 4.21.5 for protobuf-python can lead to out of memory failures. A specially crafted message with multiple key-value per elements creates parsing issues, and can lead to a Denial of Service against services receiving unsanitized input. We recommend upgrading to versions 3.18.3, 3.19.5, 3.20.2, 3.21.6 for protobuf-cpp and 3.18.3, 3.19.5, 3.20.2, 4.21.6 for protobuf-python. Versions for 3.16 and 3.17 are no longer updated.

https://gitlab.com/fewsnet/data/fdw/-/jobs/3126720223

...

was selected. This error has been corrected and users can now update data source document names.

Refreshed FDE Fixes

A variety of fixes and updates were completed for the refreshed FDE based on testing and feedback:

• Exchange rate type filter was added (DATA-1958)

• Fixed scroll arrows in data selection table (DATA-1961)

• Changed Markets from M49 to FN Regions (DATA-1983)

• Changed FDE sidebar domain title to "Acute Food Insecurity Classifications" (DATA-2007)

• Limited filter options based on available data (DATA-2012, DATA-2013)

• Fixed a bug with the start date filter in the Price domain (DATA-2070)

Bug when uploading XLSForm, both in old fdw KoBo and latest version

...

(DATA-2010)

When uploading XLSForms sometimes there is no task registered. This has been fixed and XLSForms can now be successfully uploaded to the deployments.

SaveNewArticles fails for articles from multiple sources

...

Description

SaveNewArticle relies on ReadExistingArticles to avoid inserting duplicates.

...

(DATA-2083)

An error was occurring when an article with the same URL has was been downloaded from a different source, then it attempts to save the article using batch_insert, which raises IntegrityError.

This is happening at the moment for the UK FCDO source, which duplicates some (but not all) articles with the UK.gov source.

We will fix this by falling back to saving individual articles if we hit an error, and ignoring the duplicatestwo different sources. This has been fixed and articles are now being ingested without error.