WRF - Errors in the Precision of Real Numbers
Precision of Real Numbers in WRF - The Issue
An error in the precision of a real number may occur where:
In WRF Version 3.6.1, the code contains 157,120 real numbers. Of these, 26,814 (18%) are of the default precision but are used where a higher precision would probably be appropriate. In WRF Version 3.4.1, there are 140,296 real numbers. 25,355 (18%) are of the default precision but are associated with objects of higher precision.
Real Number Precision in Fortran
A real number in Fortran code may be written, for example, as:
In consequence, a statement like that shown below (WRFV3.4.1, WRFV3/share/module_llxy.F line 1950)
REAL (KIND=HIGH) , PARAMETER :: pi = 3.141592653589793
will set the supposedly high precision parameter pi to the low precision value 3.1415927410125732 instead of 3.1415926535897931.
The Engineering Change
A facility was added to fpt to change the precision of every literal real number in a program according to the following rules:
The exponent character was used to promote the literal numbers for convenience. The addition of a kind tag would conform better to the Fortran standard, but different kind parameters are used in different components of WRF (And other programs) so this complexity was avoided in the first implementations.
An example of the modified code is shown below (WRFV3.4.1, WRFV3/phys/module_ra_cam_support, line 77)
!From radar.F90 module REAL(r8),PARAMETER :: min_tp_h2o = 160.0D0 ! min T_p for pre-calculated abs/emis REAL(r8),PARAMETER :: max_tp_h2o = 349.999999D0! max T_p for pre-calculated abs/emis REAL(r8),PARAMETER :: dtp_h2o = 21.111111111111D0! difference in adjacent elements of tp_h2o REAL(r8),PARAMETER :: min_te_h2o = -120.0D0 ! min T_e-T_p for pre-calculated abs/emis REAL(r8),PARAMETER :: max_te_h2o = 79.999999D0! max T_e-T_p for pre-calculated abs/emis REAL(r8),PARAMETER :: dte_h2o = 10.0D0 ! difference in adjacent elements of te_h2o REAL(r8),PARAMETER :: min_rh_h2o = 0.0D0 ! min RH for pre-calculated abs/emis REAL(r8),PARAMETER :: max_rh_h2o = 1.19999999D0! max RH for pre-calculated abs/emis REAL(r8),PARAMETER :: drh_h2o = 0.2D0 ! difference in adjacent elements of RH
The original code, reformatted but otherwise un-modified, is:
!From radar.F90 module REAL(r8),PARAMETER :: min_tp_h2o = 160.0 ! min T_p for pre-calculated abs/emis REAL(r8),PARAMETER :: max_tp_h2o = 349.999999 ! max T_p for pre-calculated abs/emis REAL(r8),PARAMETER :: dtp_h2o = 21.111111111111! difference in adjacent elements of tp_h2o REAL(r8),PARAMETER :: min_te_h2o = -120.0 ! min T_e-T_p for pre-calculated abs/emis REAL(r8),PARAMETER :: max_te_h2o = 79.999999 ! max T_e-T_p for pre-calculated abs/emis REAL(r8),PARAMETER :: dte_h2o = 10.0 ! difference in adjacent elements of te_h2o REAL(r8),PARAMETER :: min_rh_h2o = 0.0 ! min RH for pre-calculated abs/emis REAL(r8),PARAMETER :: max_rh_h2o = 1.19999999 ! max RH for pre-calculated abs/emis REAL(r8),PARAMETER :: drh_h2o = 0.2 ! difference in adjacent elements of RH
The changes affect PARAMETER statements, initialisation in DATA statements and declarations, and arithmetic statements in the code.
Changing the Precision of Real Numbers in WRF
If you have not already done so, set up the WRF test environment as described here.
The script tools/WRFV3_fpt_experient.csh has been set up to run an experiment as follows:
This process is described in more detail in the wrf_experiments page.
Run the experiment correct_precision:
$ tools/WRFV3_fpt_experiment.csh WRFV3.6.1 correct_precision
$ tools/WRFV3_fpt_experiment.csh WRFV3.4.1 correct_precision
The WRF Fortran files built for this experiment are copied to the directories, e.g. WRFV3.4.1_correct_precison/WRFV3/... . Compare these files with the original UCAR sources, and with the corresponding vanilla files if you have already made the vanilla (Unchanged) run. We strongly recommend Beyond Compare from Scooter Software to make this comparison. Specify the rule that Fortran comments are delimited by "!" and that comments are unimportant differences.
In our runs, with gfortran and ifort, the wrfout output files are identical to those generated by a standard UCAR build.
This is a very surprising result. We have just changed the precision of over 25,000 real numbers in the WRF code with no effect on the behaviour.
Why Did Nothing Change?
There was, of course, the usual witch-hunt to prove that we had really built, and were really running, the right program!
A change in the precision of a real number does not necessarily cause a change in value. With IEEE numbers, any number, X such that X = N * 2 ** M where N has up to 6 digits and -128 < N < +128 has the same value in single and double precision. Adding an exponent character to numbers like 0.5 should change nothing. The engineering tool therefore marks with a diagnostic only those numbers where the change in precision also causes a change in value. The diagnostic (Which may be suppressed for production runs) show, for example (WRFV3/phys/module_cu_sas.F line 247):
pgcon_use = 0.55D0 ! Zhang & Wu (2003,JAS), used in GFS (25km res spectral) !---------------------------^-------------------------------------------------- !!! FPT - 3743 Real literal precision increased with a small change in value !------------------------------------------------------------------------------
The total number of literals encountered and modified is reported in the list (fpl) file by fpt. The report shows, e.g. for WRFV3.4.1:
Promotion of Literal REAL Numbers ================================= Count Percent REAL literal numbers: 140296 With KIND tags: 16885 12.04% Promoted to higher precision: 25288 18.02% Promoted with change in value: 22526 16.06% Not promoted: 98090 69.92%
Most of the numbers with changed precision have small changes in value. The changes are marked by the fpt diagnostic number 3743, and the number of changes in individual files may therefore be examined. It was found that:
Any single test run only uses one of over a dozen physics options, so most of the code in phys is unused in any run. The occurrences of changed values in the phys directory in WRFV3.4.1 are:
Almost all of the changed values are in DATA statements in module_ra_goddard.F which is unused in the standard run of em_b_wave.
Counting Visits to Numbers With Changed Values
The fpt command insert monitor of changed real numbers enforces the precision rules described above, and inserts function invocations and subroutine calls to count visits to the numbers with changed values. The modification, for example, is (WRFV3.4.1, File WRFV3/phys/module_bl_gfs.F, line 250):
cpm = cp*(1.D0+fun_fpt_changed_precision(3,0.8D0)*qv3d(i,kts,j))
The original code is:
cpm = cp*(1.+0.8*qv3d(i,kts,j))
Note that the number 1. in the original code is promoted, but is not marked by the function fun_fpt_changed_precision because its value has not changed. The function has two arguments. The first is a unique integer identifier which allows the call to be located in the code and the second is the value of the number which has changed. In the case of changes to PARAMETER and DATA statements, the sub-programs where the objects affected are in scope are marked by a subroutine call which records that the change could potentially be visited.
The number of visits to each changed value is counted, and code is inserted to report the counts at the end of the program. The report is written to the file fpt_monitor_literal_precision.txt.
Run the experiment monitor_precision to count visits to changed real numbers in a run of em_b_wave, for example:.
$ tools/WRFV3_fpt_experiment.csh WRFV3.6.1 monitor_precision
The file fpt_monitor_literal_precision.txt is written to the directory run_tests/WRFV3/run. In our runs, it contains only the text:
No literal numbers with changed precision were visited
End of file
The Next Step
The next step in this study is to set up a test-suite which exercises a range of WRF physics options. It should then be
possible to assess the extent to which changes in precision affect performance of the code.
Copyright ©1995 to 2020 Software Validation Ltd. All rights reserved.