Skip to end of banner
Go to start of banner

October 2022

Skip to end of metadata
Go to start of metadata

You are viewing an old version of this page. View the current version.

Compare with Current View Page History

Version 1 Next »

The following list of new features, updates and bug fixes are included in the latest Sprint 22.10 release to the FEWS NET Data Platform.

New Features

Updates available in the refreshed FDE

The refreshed FDE now has a variety of new features including:

Read more about these features.

DATA-557 Introduce IPC HFA data domain

Description

FEWSNET captures Humanitarian Food Assistance(HFA data) for some countries. To capture this in FDW, we will add a new Indicator to capture HFA data with the options:
CS - Current Situation(0)
ML1 - Most Likely Scenario (1)
ML2 - Most Likely Scenario(2)
We will also add a new Model class in the IPC domain to store HFA data.
We will add a new resource class to handle HFA data uploads.

Since at the moment most countries capturing HFA data also capture IDP data, we will build the resource class in a way that it accepts a template with both values and extracts HFA data only and ignores the others such that we can use the same file to upload both IDP and HFA data.

DATA-1934 Add Data usage policy in api return for IPC FIPE and FIC domains

Updates

DATA-1651 Add filtering by keyword for News API

Description

The news API should also accept keywords to filter the responses by.

Updates to KoBo Toolbox

DATA-1778 - Getting issue details... STATUS

DATA-1793 Kobo toolbox dashboard

Description

Hello Henock and Girum,

Thanks so much for attending the East Africa XBT training and Kobo toolbox rollout. I wanted to recap and list the challenges and the issues we encountered with the backed of kobo toolbox data viewing dashboard. I would like to address those as much as possible as it impacts our users' LOE when they review and download data and how we want to move forward with similar activities if we want to use Kobo.

  • Filter records for a specific period of time. In our case, weekly data. The start and end date in the dashboard is an indicator of time spent by the enumerator to complete a record from start to submission. Is there a way to filter data for a range of days? we can only filter by one day now using the "reporting date" column. This is an important feature to have and be able to filter data based on a range of days.

  • Validation status: is it possible to change the options to draft, submitted, not approved, and approved? we are trying to set a process where we will use this column for the country offices and regional data assistant to communicate the status of records and their readiness to be submitted to the FDW.

  • Filtering products with () in the name such as Rice (Paddy). When we filter to see data for any product that contains () in the names, we don't get any results.

  • Instead of submitting by and adding the "username" having the enumerator's name would make it easier to filter by.

  • downloading data for a range of days. When we filter data based on some parameters, we weren't able to download the results. Instead, we get an extract of all data collected. this is a huge limitation for our country office who'd want to review weekly data in Excel or other formats on the side or just want a quick extract.

  • View submissions from specific users: this was a limitation for country offices who needs to see records for only specific users (their country enumerators only) and be able to edit records if needed. At the moment, if we give "view submissions only from specific users" permission to a user, he won't be able to edit records from those users. This is a big limitation. As of now, when Sudan's country rep logs in to Kobo he will see records for all countries while he only needs to see records for Sundan's enumerators only and edit them if needed.

  • Another limitation is contact management. It's hard to manage the enumerators' profiles and contacts list from the backend and see who has access to which form and also see the enumerator profile (name, location, etc). I am also speaking from the point of someone who's managing more than one form.

  • When filtering by "group border point/reporting country" and selecting a country, the "group board point/border point" drop-down menu should display border points for the country selected and not all border points.

 

Additional items we needed to adjust in the form itself (Moved to DATA-1811):

  • Add GPS location of the enumerator.

  • Adding form submission radius not more than 10km

  • When an enumerator logs in, the country and border point will be prefilled to avoid data entry errors.

DATA-1811 East Africa XBT Kobo Form Modifications

Description

As outlined in DATA-1793 - Getting issue details... STATUS (final three bullets), we need the following modifications on the East Africa XBT Kobo Form.

  • Add GPS location of enumerators.

  • Allow form submission only for a radius of not more than 10km

  • Prefill default border-points based on the user using the form.

We opted to handle these changes separately because they are changes that affect the data collection process and relatively urgent than the rest of the Kobo data management and review issues mentioned in DATA-1793 - Getting issue details... STATUS .

DATA-1827 Upgrade FEWS KoBo to latest version

Description

To be able to use the newer row-based permission feature in KoBo, we need to upgrade the KoBo version currently installed for FEWS to the latest available version.DATA-1913 Speed up api format renders of endpoints with large drop-downs

Description

Some endpoints, particularly ipcphase, take a long time to render in api format (browsable), because they contain foreign keys to large tables, which are rendered as HTML SELECT OPTION tags.

We have attempted to replace them with auto-corrects, but never made a solid fix.

In most cases these would be more usable simply as TEXT inputs to type an ID in to.

DATA-1922 Rename the domains across FDW as defined by EWT for the sidebar name review

Description

As part of the FDE2 refresh activity the names of the domains shown in the sidebar of FDE2 have been reviewed. These changes need to be propagated throughout the FDW and should be incorporated into the following places;

  • FDW Menu Names

  • HDX names

  • Verbose names on Django models

DATA-1932 Change get_parameter_map so not all forms.ModelChoiceField automatically converted to autocomplete

DATA-1943 Update FAO pipeline based on changes in remote api

Description

The help ticket reported HELP-1278 - Getting issue details... STATUS gave us the chance to review the FAO pipeline, which is affected by the recent changes in the fao remote api. The change is noted below:

  1. the datasereis api url is changed from https://fpma.apps.fao.org/giews/food-prices/tool/api/v1/series/ to https://fpma.fao.org/giews/v4/price_module/api/v1/FpmaSerie/

  2. The current version also require a two step process to fetch the datapoints.
    First https://fpma.fao.org/giews/v4/price_module/api/v1/FpmaSerie/ returns the list of dataseries. Each dataseries has a guid type identifier as oppose to the previous api's series_id [eg. 8_23_642_2_434_nominal
    Then the datapoints can be fetched from a url like:
    https://fpma.fao.org/giews/v4/price_module/api/v1/FpmaSeriePrice/e23b83a7-7cf7-40b0-87f7-6c03a2d3b2bc/ and there is no datapoint_url now included in each dataseries

  3. the seriesId is now has a different datatype - Guid and the column is named uuid

  4. fao Country Code of value XXX used to be used to refer international prices, now they replaced that with IPS

  5. market is now an id and market_name contains the name, commodity_name contains the product name where commodity used to reference

  6. dateRanges field is now periodicity

These changes have side effect on the pipeline and we need to update it accordingly.

DATA-1994 Add dataset name to cross-domain dataset endpoint

Description

Make the dataset name field available on the cross-domain api/dataset endpoint

DATA-2040 Refinements to FDW data entry screen UI

Description

I have some revisions to the FDW data entry screens to improve usability. See example in the linked Figma design—”3.3 Recommended Design - Classified Product Change Screen”

Revisions:

  1. Increase spacing between the entry form title and the record title

  2. Add dividing lines above the record title and below the last input field

  3. Change input field labels from left justified to right justified (except “country classified product aliases”)

  4. Change input field labels from all caps to standard body case, making sure the first word of each field label is capitalized

  5. Change input field labels from regular to semi bold font weight

Bug Fixes

DATA-1656 Trying to open https://fdw.fews.net/en/admin/semistructured/semistructureddatapoint/ results in 500 Internal Server Error

DATA-1929 Unable to update datasourcedocument name

Description

As reported HELP-1291 - Getting issue details... STATUS the web form in the admin doesn’t allow users to update source document name when no license information is selected.

DATA-1935 Investigate issues with refresh materialized views debouncing incorrectly.

Description

The EWT have been having issues with uploaded data collections taking a long time to reflect the updates in FDW. The is because the refresh materialized views have not been running correctly. These can be run manually by ssh-ing into the server and running the commands from psql, like the below;

psql -c "REFRESH MATERIALIZED VIEW CONCURRENTLY price_marketpricefacts_materialized";

It looks as though at least the price_marketpricefacts_materialized is being continually debounced and not actually running. We see a log message like the below;

Task common.tasks.refresh_materialized_view[5d089b85-9811-48ed-a854-1938387de3b9] succeeded in 0.05116157094016671s: 'Request received for refresh_materialized_view_refresh_materialized_view-materialized_view=price_marketpricefacts_materialized while DEBOUNCING so re-scheduled, id 901fd5c6-04fc-4897-be17-7ad3cac3e369 eta 2022-09-30 11:49:53.046322+00:00.'

and tracing the new task that has been spawned we see the below log;

Task common.tasks.refresh_materialized_view[901fd5c6-04fc-4897-be17-7ad3cac3e369] succeeded in 0.03576034493744373s: 'Task refresh_materialized_view_refresh_materialized_view-materialized_view=price_marketpricefacts_materialized triggered to run twice, so cancelling second call 901fd5c6-04fc-4897-be17-7ad3cac3e369.'

The only reason that the code can be reached to execute the last log statement is if the debounce method fails to delete the HERD:{task_key} from the cache. This could happen if the HERD:{task_key} is not set in the first place. Chris Preager Could this also happen if the tasks are on different servers?

The code has the following comment.

# This can be caused by:
#     1. This cache delete occurs just before cache_set(HERD) in apply_async, deletes nothing, or
#        apply_async enqueues a debounce while this is running, and then this method enqueues a
#        duplicate below due to the renewed ETA.
#     2. This cache_delete(HERD) runs immediately after cache_set(HERD) in apply_async, so a
#        second call to apply_async goes through. In this case the latter call won't run after
#        debounce_wait seconds, but will be satisfied by this run anyway. A subsequent call will run fine.

DATA-1938 Fix Comtrade unit test erros

DATA-1954 Fix safety warning for protobuf

Description

The below safety error is causing issues in the CI pipelines.

protobuf | 3.20.1 | >=3.20.0rc0,<3.20.2 | 51167 |

A parsing vulnerability for the MessageSet type in the ProtocolBuffers versions prior to and including 3.16.1, 3.17.3, 3.18.2, 3.19.4, 3.20.1 and 3.21.5 for protobuf-cpp, and versions prior to and including 3.16.1, 3.17.3, 3.18.2, 3.19.4, 3.20.1 and 4.21.5 for protobuf-python can lead to out of memory failures. A specially crafted message with multiple key-value per elements creates parsing issues, and can lead to a Denial of Service against services receiving unsanitized input. We recommend upgrading to versions 3.18.3, 3.19.5, 3.20.2, 3.21.6 for protobuf-cpp and 3.18.3, 3.19.5, 3.20.2, 4.21.6 for protobuf-python. Versions for 3.16 and 3.17 are no longer updated.

https://gitlab.com/fewsnet/data/fdw/-/jobs/3126720223

DATA-2010 Bug when uploading XLSForm, both in old fdw KoBo and latest version

DATA-2083 SaveNewArticles fails for articles from multiple sources

Description

SaveNewArticle relies on ReadExistingArticles to avoid inserting duplicates.

However, if an article with the same URL has been downloaded from a different source, then it attempts to save the article using batch_insert, which raises IntegrityError.

This is happening at the moment for the UK FCDO source, which duplicates some (but not all) articles with the UK.gov source.

We will fix this by falling back to saving individual articles if we hit an error, and ignoring the duplicates.

  • No labels