Project delivery

There are many ways to conduct a data integration project.

The following list provides an overview of the general stages of a project once all of the agreements have been signed and approved (see project application and approval).

  1. Extract and transfer data - the data custodians provide data to the integrating authority, as was specified in the project agreements.
  2. Prepare data for linking - prior to linking, the data needs to be cleaned and standardised. This may be conducted by the data custodians and/or integrating authority.
  3. Linking and merging of data - the source data is combined to create a new integrated dataset. This stage is managed by the integrating authority, however, some components may be outsourced or conducted in partnership.
  4. Access to integrated data - the integrating authority is responsible for providing the data users with secure access to the integrated dataset, de-identified and confidentialised according to the requirements of data custodians.
  5. Analysis of integrated dataset - data users conduct analysis of the integrated dataset and release outputs.
  6. Evaluation and project completion - the integrating authority conducts the evaluation and completion of the project, this includes managing the storage or destruction of the integrated dataset.

Data security practices and access controls to sensitive information, along with good data management practices, should be employed for the duration of the project.

Below is a very broad diagram of the key stages for a data integration project once it has been approved.  There are different protocols which can be used to conduct the project, see applying the separation principle.

For more information on linking methodologies (including the creation of a linkage key), see the Data Linking Information Series.

A broad overview of data integration following approval

Below is an example showing how the roles and responsibilities of a data custodian, integrating authority and data user apply to the key stages of a data integration project once it has been approved. This diagram does not show detailed processes or describe the different protocols which can be used when linking and merging the data, for more information on these approaches see the separation principle.


 

*  This stage can be outsourced by the integrating authority - see Outsourcing or working in partnerships

 

^ “Access to potentially identifiable integrated data for statistical and research purposes, outside secure and trusted institutional environments should only occur where: legislation allows; it is necessary to achieve the approved purposes; and meets agreements with source data agencies.” (High Level Principle 6)