Data management

The High Level Principles for Data Integration involving Commonwealth Data for Statistical and Research Purposes aims to maximise statistical and research use of existing and new Commonwealth datasets. According to Principle 1,

“Where administrative data is likely to have value for statistical use, efforts should be made to maximise that value through good data management, including the use of standard definitions and classifications and the maintenance of appropriate metadata, including quality attributes of the data.”

Good data management practices ensure that the data are kept secure, confidential and fit-for-purpose. Integrated datasets created for statistical or research purposes need to be managed and stored in a way that gives the community and businesses confidence that no individual or organisation is likely to be identified. Integrated dataset management (including data storage) should take into account the potential increase in identifiability of one set of data when combined with another set. Expectations regarding the preservation of privacy and confidentiality when undertaking a statistical or research integration project using Commonwealth data are described in High Level Principle 6.

In the context of a data integration project, the data custodian has responsibility for providing comprehensive and up-to-date documentation to facilitate the interpretation and use of the data. This may include:

  • metadata (information about the data) to assist the integrating authority and data users to understand the source data including concepts, classifications, scope and coverage, sources of error and areas where careful interpretation is required, and/or
  • data quality documentation, such as a data quality statement, to ensure that data users and the integrating authority are aware of key characteristics of the data that impact on quality and any limitations of its use.


While data may be fit for the purpose for which it was collected (e.g. for administrative purposes), it may not necessarily be fit for a statistical or research purpose. The fitness for purpose of the data needs to be assessed and agreed upon by the data custodians and data users on a project-by-project basis.

Integrating authorities have responsibility for:                       

  • data security- ensuring that they adhere to requirements for data transfer, access, use, storage and destruction or retention as set out by data custodians in the agreements with the integrating authority and in accordance with relevant policies and legislation governing the data,
  • data quality - particularly ensuring the quality of the linked record throughout the linking process,
  • confidentialising data– ensuring that appropriate statistical disclosure control techniques are put in place to minimise the risk of identification, and
  • providing comprehensive and up-to-date documentation to facilitate the interpretation and use of the data.


For more information on general best practice for secure data management and the requirements of the Australian Government Protective Security Policy Framework that many integrating authorities will need to abide by, see the Data security section.

Negotiating the details for managing the integrated data is an important element in finalising the project details and formalising project agreements between the data custodian(s) and the integrating authority and the integrating authority and data users.

For information on other elements to be considered in finalising project details see:


Other topics in this section relating to the Commonwealth arrangements for statistical data integration are: