Household Income and Expenditure Survey 2006

Some initial editing was carried when the forms were coded and prepared for data entry.

There were then several strands of editing carried out after the data entry was completed.

A set of tables designed to identify missing, illegal or potentially incompatible values in the classificatory data was specified.

The development of the “Generate new records” program, described above, required extensive examination of the data. First, it was sometimes necessary to examine original questionnaires to obtain a better understanding of how households responded to certain questions, especially when the recorded responses were unexpected. Second, the development of some of the imputation functions implemented in the program required analysis of detailed data. Third, testing of the program required examination of data before and after transformation to ensure that the program was carrying out its intended functions. These and other more minor reasons for examining the data collectively also played an important editing function, even though it was unstructured from an editing point of view. Most of the editing actions flowing from this work are recorded in Queries.xls.

Outlier analysis is an important part of the editing process for household surveys. For the HIES, formal outlier analysis has largely been confined to examining households with very high income or expenditure. However, outliers were also detected during the processes described in the previous paragraphs.

Imputations

Some obvious errors were fixed and missing data supplied manually at the time of the initial coding and checking of the questionnaires prior to the data entry stage. Similarly changes were made as a result of editing queries described in the previous section.

A more automated form of imputation was implemented for certain instances of missing data.

For those transactions recorded in diaries where a quantity was supplied without a value, a value was imputed on the basis of transactions in the same commodity in the same province/urban area. Consideration was given to imputing separately for each transaction type (purchases, own account production, gifts given, gifts received) but there is not sufficient data to use a cross classification of province/urban and transaction type. Examination of differences in unit values between provinces/urban areas and between transaction types showed greater differences between provinces/urban areas than between transaction types. Where there was no required data for a commodity in a particular province/urban area, the unit value from a similar province/urban area was used. Calculations are included in value and quantity by prov city 2.xls. Transaction values imputed in this way are flagged on the file by means of the “data source” variable.

For employees who did not report their gross wages and salaries, a value was imputed on the basis of the average wage/salary of other employees with the same industry and occupation codes and who reported their value. Where there were no other employees in the same category reporting wages/salary, the value for a similar industry and occupation code was used. Calculations are included in supporting file W&S 5.xls. Any imputed values were included in the transaction record for wage and salaries for the household concerned (there is only one aggregate record per household, which combines the wages and salaries of all members of the household). If any component is imputed, the whole transaction is flagged as imputed. However, the imputed value is not included in the PERSON record.

For households that own their own dwelling (including those with a loan or mortgage) but who did not estimate the potential rental value of their dwelling in the household questionnaire, a value was imputed on the basis of the average for other dwellings in the same enumeration area. Potential rental values are not included in aggregate income or expenditure and therefore there are no transaction records for these values. Rather they are stored in the dwelling characteristic and tenure records. The imputed values are NOT flagged as imputed (because these values are not recorded as transactions, and no other record structure had been set up to record the values as imputed). Calculations of imputed values are included in dwelling tenure.xls, which can also be used to identify the households for which values were imputed.