ISO/IEC 17043:2023 Conformity assessment — General requirements for the competence of proficiency testing providers

Foreword

ISO (the International Organization for Standardization) and IEC (the International Electrotechnical Commission) form the specialized system for worldwide standardization. National bodies that are members of ISO or IEC participate in the development of International Standards through technical committees established by the respective organization to deal with particular fields of technical activity. ISO and IEC technical committees collaborate in fields of mutual interest. Other international organizations, governmental and non-governmental, in liaison with ISO and IEC, also take part in the work.

The procedures used to develop this document and those intended for its further maintenance are described in the ISO/IEC Directives, Part 1. In particular, the different approval criteria needed for the different types of document should be noted. This document was drafted in accordance with the editorial rules of the ISO/IEC Directives, Part 2 (see www.iso.org/directives or www.iec.ch/members_experts/refdocs).

ISO and IEC draw attention to the possibility that the implementation of this document may involve the use of (a) patent(s). ISO and IEC take no position concerning the evidence, validity or applicability of any claimed patent rights in respect thereof. As of the date of publication of this document, ISO and IEC had not received notice of (a) patent(s) which may be required to implement this document. However, implementers are cautioned that this may not represent the latest information, which may be obtained from the patent database available at www.iso.org/patents and https://patents.iec.ch. ISO and IEC shall not be held responsible for identifying any or all such patent rights.

Any trade name used in this document is information given for the convenience of users and does not constitute an endorsement.

For an explanation of the voluntary nature of standards, the meaning of ISO specific terms and expressions related to conformity assessment, as well as information about ISO's adherence to the World Trade Organization (WTO) principles in the Technical Barriers to Trade (TBT) see www.iso.org/iso/foreword.html. In the IEC, see www.iec.ch/understanding-standards.

This document was prepared by the ISO Committee on Conformity Assessment (CASCO), in collaboration with the European Committee for Standardization (CEN) Technical Committee CEN/CLC/JTC 1, Criteria for conformity assessment bodies, in accordance with the Agreement on technical cooperation between ISO and CEN (Vienna Agreement).

This second edition cancels and replaces the first edition (ISO/IEC 17043:2010), which has been technically revised.

The main changes are as follows:

Any feedback or questions on this document should be directed to the user’s national standards body. A complete listing of these bodies can be found at www.iso.org/members.html and www.iec.ch/national-committees.

Introduction

Proficiency testing (PT) is widely recognized as an essential tool for demonstrating the competence of conformity assessment bodies. PT can provide evidence of competence and it can be an indicator of an underlying or emerging problem. This document is intended to promote confidence in the operations of PT providers. It contains requirements for PT providers to enable them to demonstrate that they operate competently and can generate valid evaluations of participant performance.

PT involves the use of interlaboratory comparisons for the evaluation of laboratory performance. The definition of “interlaboratory comparison” (see 3.4) broadens the use of both the terms “laboratories” and “measurements or tests” for the purposes of this document to include all types of conformity assessment bodies and their activities, respectively. The term “method” as used in this document can be considered synonymous with the term “measurement procedure” as defined in ISO/IEC Guide 99.

There are many different purposes for interlaboratory comparisons, which can be addressed by PT schemes, including but not limited to:

1. a) evaluation of the performance of laboratories for specific measurements, tests, calibrations, examinations, inspections or sampling;
2. b) identification of problems in laboratories that, for example, can be related to measurement or test methods, effectiveness of training and supervision of personnel, or calibration of equipment;
3. c) establishment of the effectiveness of measurement or test methods and the comparability of measurement or test results;
4. d) provision of additional confidence to users of measurement or test results;
5. e) identification of differences in measurement or test results;
6. f) education of participating laboratories based on the outcomes of such comparisons;
7. g) validation of measurement uncertainty claims.

For the following types of interlaboratory comparisons, the term PT does not usually apply because laboratory competence must be established in advance, in order to ensure the validity of measurements or tests as well as the metrological traceability of assigned values:

1. h) evaluation of the performance characteristics of a measurement or test method (often described as collaborative trials);
2. i) assignment of values to reference materials;
3. j) support for statements of the equivalence of measurements of National Metrology Institutes (NMIs), or their Designated Institutes (DIs) through “key and supplementary comparisons”, conducted on behalf of the International Bureau of Weights and Measures (BIPM) and associated Regional Metrology Organizations (RMOs).

It is recognized that interlaboratory comparisons for purposes h), i) and j) can contribute to independent demonstrations of laboratory competence. The requirements of this document can be applied to many of the technical planning and operational activities for these interlaboratory comparisons.

This document also requires PT providers to plan and implement actions to address risks and opportunities, based on their experience. Addressing both risks and opportunities establishes a basis for increasing the effectiveness of the management system, achieving improved results and preventing negative events. The PT provider is responsible for deciding which risks and opportunities to address.

The need for ongoing confidence in laboratory performance is essential not only for laboratories and their customers but also for other interested parties, such as regulators, accreditation bodies and other organizations that specify requirements for laboratories. Most of the requirements in this document apply to those evolving areas, especially regarding management, planning and design, personnel, assuring validity of results and performance evaluations, confidentiality and other aspects, as appropriate.

This document is intended to provide a consistent basis for all interested parties to determine the competence of organizations that provide PT.

In this document, the following verbal forms are used:

Further details can be found in the ISO/IEC Directives, Part 2.

1 Scope

This document specifies general requirements for the competence and impartiality of proficiency testing (PT) providers and consistent operation of all proficiency testing schemes. This document can be used as a basis for specific technical requirements for particular fields of application.

Users of proficiency testing schemes, regulatory authorities, organizations and schemes using peer-assessment, accreditation bodies and others can use these requirements in confirming or recognizing the competence of proficiency testing providers.

2 Normative references

The following documents are referred to in the text in such a way that some or all of their content constitutes requirements of this document. For dated references, only the edition cited applies. For undated references, the latest edition of the referenced document (including any amendments) applies.

• ISO/IEC 17000 Conformity assessment — Vocabulary and general principles
• ISO/IEC 17025 General requirements for the competence of testing and calibration laboratories
• ISO 17034 General requirements for the competence of reference material producers
• ISO/IEC Guide 99 International vocabulary of metrology — Basic and general concepts and associated terms (VIM)

3 Terms and definitions

For the purposes of this document, the terms and definitions given in ISO/IEC 17000 and ISO/IEC Guide 99 and the following apply.

ISO and IEC maintain terminological databases for use in standardization at the following addresses:

3.1 assigned value

3.2 consensus value

3.3 customer

3.4 interlaboratory comparison

3.5 outlier

3.6 participant

3.7 proficiency testing

3.8 proficiency testing item

3.9 proficiency testing provider

3.10 proficiency testing round

3.11 proficiency testing scheme

3.12 standard deviation for proficiency assessment

4.1.1

PT activities shall be undertaken impartially.

4.1.2

The PT provider shall be structured and managed so as to safeguard impartiality.

4.1.3

The PT provider shall be responsible for the impartiality of its PT activities and shall not allow commercial, financial or other pressures to compromise its impartiality.

4.1.4

The PT provider shall monitor its activities and its relationships to identify threats to its impartiality. This monitoring shall include the relationships of its personnel.

NOTE

A relationship can be based on ownership, governance, management, personnel, shared resources, finances, contracts or marketing (including branding). Such relationships do not necessarily present a PT provider with a threat to impartiality.

4.1.5

If a threat to impartiality is identified, its effect shall be eliminated or minimized so that the impartiality is not compromised.

4.1.6

The PT provider shall have top management commitment to impartiality.

4.2.1

The PT provider shall be responsible, through legally enforceable agreements, for the management of all information obtained or created during the performance of PT activities. The PT provider shall inform the client in advance of the information it intends to place in the public domain. Except for information that the client makes publicly available, or when agreed between the PT provider and the client, all other information is considered proprietary information and shall be regarded as confidential.

NOTE

The terms “proprietary” and “confidential” do not preclude publication for academic and new insights of information purposes, provided that neither clients nor participants can be identified, including by inference.

4.2.2

When the PT provider is required by law or authorized by contractual arrangements to release confidential information, the client concerned shall be notified of the information released, unless prohibited by law.

4.2.3

Information about the participant or customer from a source other than the participant or customer (e.g. complainant or regulator) shall be kept confidential by the PT provider. The identity of the source shall be kept confidential by the PT provider and shall not be shared with the participant or the customer, unless agreed by the source.

4.2.4

Personnel, including any committee members, contractors, personnel of external bodies, or persons acting on the PT provider’s behalf, shall keep confidential all information obtained or created during the performance of the PT activities.

4.2.5

The identity of participants in a PT scheme shall be confidential and known only to persons involved in the operation of the PT scheme, unless the participant or the customer waives confidentiality.

5.1

The PT provider shall be a legal entity, or a defined part of a legal entity, that is legally responsible for its PT activities.

NOTE

For the purposes of this document, a governmental PT provider is deemed to be a legal entity on the basis of its governmental status.

5.2

The PT provider shall identify management that has overall responsibility for the PT activities.

5.3

The PT provider shall define and document the PT schemes for which it conforms with this document. The PT provider shall only claim conformity with this document for those PT schemes.

5.4

The PT provider shall carry out PT activities in such a way so as to meet the requirements of this document and address the requirements of participants, customers, regulatory authorities, and organizations providing recognition. These requirements apply to all PT activities performed in its permanent facilities and any other facility or site.

5.5

The PT provider shall:

1. a) define its organization and management structure, its place in any parent organization and the relationships between the management, technical operations and support services;
2. b) specify the responsibility, authority and interrelationships of all personnel who manage, perform or verify work affecting the results of its PT activities;
3. c) document its procedures to the extent necessary to ensure the consistent application and validity of its PT activities.

5.6

The PT provider shall have personnel who, irrespective of other responsibilities, have the authority and resources needed to carry out their duties, including:

1. a) implementation, maintenance and improvement of the management system;
2. b) identification of deviations from the management system or from the procedures while performing the PT activities;
3. c) initiation of actions to prevent or minimize such deviations;
4. d) reporting to its management on the performance of the management system and any need for improvement;
5. e) ensuring the effectiveness of the PT activities.

5.7

The PT provider management shall ensure that:

1. a) communication takes place regarding the effectiveness of the management system and the importance of meeting the requirements of participants, customers, regulatory authorities and organizations providing recognition;
2. b) the integrity of the management system is maintained when changes to the management system are planned and implemented.

6.1.1

The PT provider shall have access to the personnel, facilities, equipment, systems and support services necessary to manage and perform its PT activities.

6.1.2

Measurements or tests conducted under the responsibility of the PT provider, related to PT item characterization or for assessing homogeneity and stability, shall be conducted in accordance with the relevant requirements of ISO/IEC 17025.

NOTE 1

The relevant requirements are requirements that relate to the validity of the measurement or test results, which can impact the validity of PT activities (e.g. metrological traceability). They are not intended to include management system requirements or other requirements unrelated to the PT activities.

NOTE 2

In the medical area, the relevant requirements of ISO 15189 apply in place of ISO/IEC 17025.

6.1.3

Where the PT item is a material that meets the definition of “reference material”, it shall be produced under conditions that meet the relevant requirements of ISO 17034.

NOTE 1

Such materials include reference materials for quality control (e.g. chemical solutions with or without reference values) and reference materials with certified property values (CRMs).

NOTE 2

The relevant requirements are requirements that relate to the validity of operations to produce a reference material that directly impacts the PT activities (e.g. mixing, or handling and storage). They are not intended to include management system requirements or other requirements not directly related to the PT activities (e.g. contents of certificates).

NOTE 3

In the medical area, the relevant requirements of ISO 15194 can apply for CRMs in place of ISO 17034, when applicable.

6.2.1

The PT provider shall have access to a sufficient number of competent personnel to perform its PT activities.

6.2.2

The PT provider shall ensure that the personnel have the competence to:

1. a) perform PT activities for which they are responsible;
2. b) evaluate the significance of deviations.

6.2.3

The PT provider shall have a process for managing competence of its personnel.

6.2.4

All personnel of the PT provider (either internal or external) that could influence the PT activities shall act impartially.

6.2.5

The PT provider shall have documented information demonstrating competence of its personnel that can influence the results of its PT activities. Documented information shall include requirements for education, qualification, training, technical knowledge, skills and experience.

6.2.6

The PT provider shall, where appropriate, authorize personnel to perform specific activities within PT schemes, including but not limited to the following:

1. a) plan PT schemes;
2. b) assess data/information to determine stability and homogeneity, if applicable, as well as assigned values and associated uncertainties of the properties or characteristics of the PT item;
3. c) evaluate the performance of PT participants;
4. d) give opinions and interpretations as well as advice to the participants;
5. e) review and authorize PT reports.

6.2.7

The PT provider management shall communicate to all personnel their duties, responsibilities and authorities.

6.3.1

To ensure the validity of the PT activities, the PT provider shall ensure that there are appropriate facilities for the operation of the PT scheme.

6.3.2

The PT provider shall ensure that the environmental conditions do not compromise the PT activities, including operations that are undertaken at sites away from the PT provider's permanent facilities or that are undertaken by external service providers.

6.3.3

The PT provider shall document environmental conditions that can influence the validity of the PT items and any measurements or tests carried out, including conditions that are required by relevant specifications and measurement or test methods. The PT provider shall control, monitor and periodically review these conditions and shall record all relevant monitoring activities. If environmental conditions compromise the validity of PT activities, the activities shall be halted (see 7.5.4).

6.3.4

Access control to, and use of, areas affecting the PT activities shall be managed. The PT provider shall determine the extent of access control based on its particular circumstances.

6.3.5

There shall be appropriate separation between neighbouring areas in which there are incompatible PT activities. Action shall be taken to prevent cross-contamination, interference or adverse influences on PT activities.

6.4.1

The PT provider shall not use external service providers for the following activities:

1. a) the design and planning of PT schemes;
2. b) the evaluation of performance;
3. c) the authorization of reports.

NOTE

This does not prevent the PT provider from using advice or assistance from any advisors, experts or steering groups.

6.4.2

The PT provider shall have procedures to ensure that the experience and technical competence of the providers of external products and services are sufficient for their assigned tasks and that they comply with the relevant clauses of this document and other appropriate documents.

6.4.3

The PT provider shall inform participants and customers, in advance and in writing, of products and services that are or can be provided externally, when they affect the validity of the PT activities.

6.4.4

The PT provider shall have a procedure and retain records for:

1. a) defining, reviewing and approving the PT provider’s requirements for externally provided products and services;
2. b) defining the criteria for selection of the external providers and for evaluating and monitoring their performance;
3. c) ensuring that externally provided products and services conform to the PT provider’s established requirements and, when applicable, to the relevant requirements of this document, before they are used or directly provided to the participant or customer;
4. d) taking any actions arising from the performance monitoring and evaluation of the external providers.

6.4.5

The PT provider shall communicate its requirements to external providers for:

1. a) the products and services to be provided;
2. b) the acceptance criteria;
3. c) competence, including any required qualification of the organization or personnel involved;
4. d) PT activities that the PT provider or its customers intend to perform at the external provider’s premises.

6.4.6

The PT provider shall be responsible to the participants or customers for the externally provided products and services.

NOTE

In cases where the customer or a regulatory authority specifies which external provider is to be used, being responsible can be interpreted as taking actions to minimize the undesired effect that directly affects the validity of PT activities.

7.1.1.1

The PT provider shall have a procedure for the review of requests, tenders and contracts. The procedure shall ensure that:

1. a) the objectives of the PT scheme are sufficiently defined and in agreement with the customers' needs;
2. b) the requirements are adequately defined, documented and understood;
3. c) the PT provider has the capability and resources necessary to meet the requirements;
4. d) the PT scheme is technically appropriate taking into account the needs of the given application or field of application.

NOTE 1

This review is particularly important when a customer requests a PT scheme to be created for a specific purpose or for a different level or frequency of participation from that normally offered.

NOTE 2

This review can be simplified when the PT scheme is fully described in a catalogue or other notice and the participant is enrolling for a routine PT round.

7.1.1.2

The review shall cover all aspects of the request, including any externally provided products and services.

7.1.1.3

Records of such reviews, including any significant changes, shall be retained. Records shall also be retained of pertinent discussions with a customer relating to their requirements, or the results of the PT activities.

7.1.1.4

The customer shall be informed of any deviation from the contract.

7.1.1.5

If a request or contract is amended after the PT scheme is underway, the contract review shall be repeated and any amendments shall be communicated to all affected personnel.

7.1.2.1

The PT provider shall make detailed information available about the PT scheme to participants and customers. This information shall include:

1. a) objectives and relevant details of the PT scheme;
2. b) criteria to be met for participation;
3. c) criteria for determining the assigned value and the evaluation of performance;
4. d) confidentiality arrangements;
5. e) critical timelines;
6. f) any fees for participation;
7. g) details of how to apply.

7.1.2.2

Participants and customers shall be advised in a timely manner by the PT provider of any changes in PT scheme design or operation.

7.1.2.3

Records of relevant communications shall be maintained and retained by the PT provider, as appropriate.

7.2.1.1

The PT provider shall identify, design and plan those activities which directly affect the validity of the PT scheme and shall ensure that activities are carried out in accordance with prescribed procedures.

NOTE

When designing and planning the PT scheme, the relevant standards and requirements specific to the objectives of the PT scheme can be considered, e.g. ISO/IEC 17025, ISO 15189, ISO/IEC 17020. Safety and ethical issues can also be considered.

7.2.1.2

When a PT provider intends to introduce significant changes to activities which can affect the validity of the PT scheme, the PT provider shall identify and manage the risk to ensure the validity of the PT scheme is maintained.

7.2.1.3

The PT provider shall develop a documented plan before commencement of the PT scheme that addresses the objectives, purpose and basic design of the PT scheme. The plan shall include the following information and, where appropriate, reasons for the selection or exclusion of the specific information:

1. a) the personnel involved in the design and operation of the PT scheme;
2. b) the activities to be undertaken by external providers of products and services and their contact details;
3. c) criteria to be met for participation in the PT scheme;
4. d) the number and type of expected participants in the PT scheme;
5. e) description of activities to be performed and results to be reported by participants;
6. f) a description of the range of values or characteristics, or both, to be expected for the PT items;
7. g) the potential major sources of errors involved in the area of PT offered;
8. h) requirements for the production, quality control, storage and distribution of PT items;
9. i) arrangements to prevent collusion between participants or falsification of results and procedures to be employed if collusion or falsification of results is suspected;
10. j) a description of the information which will be supplied to participants and the time schedule for the various phases of the PT scheme;
11. k) for continuous PT schemes, the frequency or dates upon which PT items will be distributed to participants, the deadlines for the return of results by participants and, where appropriate, the dates on which measurements or tests will be carried out by participants;
12. l) any information on methods or procedures which participants must use to store, handle, prepare, ship or dispose of the PT item and perform the measurements or tests;
13. m) procedures for the measurement or test methods to be used for the homogeneity and stability testing of PT items and, where applicable, to determine their biological viability;
14. n) preparation of any standardized reporting formats to be used by participants;
15. o) a detailed description of the statistical analysis to be used;
16. p) the origin, metrological traceability and uncertainty of any assigned values;

    NOTE

    Assigned values can have uncertainty contributions from sources in addition to the uncertainty of measurement results used for characterization, such as inhomogeneity and instability, and interlaboratory differences if more than one laboratory is used for characterization.

17. q) the treatment of results from different measurement or test methods, where permitted by the PT scheme;
18. r) criteria for the evaluation of the performance of participants;
19. s) a description of the data, interim reports or information to be returned to participants;
20. t) a description of the extent to which participant results, and the conclusions that will be based on the outcome of the PT scheme, will be made public or shared;
21. u) actions to be taken in the case of lost, delayed or damaged PT items.

7.2.2.1

Statistical designs shall be developed to meet the objectives of the PT scheme, based on the type of data (quantitative or qualitative, including ordinal and nominal), statistical assumptions, the type of errors and the expected number of results.

NOTE 1

Statistical design covers the process of planning of the PT scheme and the collection, analysis and reporting of the PT scheme data. Statistical designs are often based on stated objectives for the PT scheme, such as detection of certain types of errors with specified power or determination of assigned values with a specified uncertainty.

NOTE 2

Data analysis methods can vary from the very simple (e.g. descriptive statistics) to the complex, using statistical models with probabilistic assumptions or combinations of results for different PT items.

NOTE 3

In cases where the PT scheme design is mandated by a specification given by, for example, a customer or regulatory authority, the statistical design and data analysis methods can be taken directly from the specification.

NOTE 4

In the absence of reliable information needed to produce a statistical design, a preliminary interlaboratory comparison can be used.

7.2.2.2

The PT provider shall document the statistical design and data analysis methods to be used to determine the assigned value and to evaluate the participant results, and it shall document the reasons for the selection and the assumptions upon which the statistical design and data analysis methods are based. The PT provider shall be able to demonstrate that statistical assumptions are reasonable and that statistical analyses are carried out in accordance with prescribed procedures.

7.2.2.3

In designing a statistical analysis, the PT provider shall give careful consideration to the following:

1. a) the accuracy, as well as the uncertainty, required or expected for the assigned value for each property or characteristic in the PT scheme;
2. b) the minimum number of participants in the PT scheme needed to meet the objectives of the statistical design. In cases where there is an insufficient number of participants to meet these objectives or to produce statistically meaningful analysis of participant results, the PT provider shall document, and provide to participants, details of the alternative approaches used to assess participant performance;
3. c) the relevance of significant figures to the reported participant result, including the number of decimal places;
4. d) the number of PT items to be measured or tested and the number of repeat measurements or tests to be conducted on each PT item or for each determination;
5. e) the procedures used to establish the standard deviation for proficiency assessment or other evaluation criteria;
6. f) the procedures to be used to treat participant results from different measurement or test methods which are not technically equivalent, where permitted by the PT scheme;
7. g) whether the measurement uncertainty of participant results shall be reported and how it will be used to evaluate the participant's performance;
8. h) the procedures to be used to identify or handle outliers, or both;
9. i) where relevant, the procedures for the evaluation of values excluded from statistical analysis;
10. j) where appropriate, the objectives to be met for the design and the frequency of PT rounds.

7.2.3.1

The PT provider shall document the procedure for determining the assigned values for the properties or characteristics in a particular PT scheme. Where applicable, this procedure shall take into account the metrological traceability and uncertainty required to demonstrate that the PT scheme is fit for its purpose.

NOTE

ISO 13528 provides statistical methods for the determination of the assigned value.

7.2.3.2

PT schemes in the area of calibration shall have assigned values with metrological traceability.

7.2.3.3

For PT schemes in areas other than calibration, the relevance, need and feasibility for the establishment of metrological traceability and the associated uncertainty of the assigned value shall be determined by taking into account the purpose of the PT scheme.

NOTE

The required metrological traceability chain can differ depending on the type of PT item, the property or characteristic and the availability of traceable calibrations and reference materials.

7.2.3.4

When a consensus value is used as the assigned value, the PT provider shall provide an estimate of the uncertainty of the assigned value [see Note to 7.2.1.3 item p)] as described in the plan for the PT scheme.

7.2.3.5

The PT provider shall have a policy regarding the disclosure of assigned values. The policy shall ensure that participants cannot gain advantage from early disclosure.

7.3.1.1

The PT provider shall establish and implement procedures to ensure that PT items are produced in accordance with the plan described in 7.2 and are fit for the PT scheme's purpose.

7.3.1.2

The PT provider shall establish and implement procedures to ensure appropriate selection, acquisition, collection, identification, preparation, handling, storage and, where required, disposal of all PT items.

NOTE

PT items usually match the type of items or materials encountered in routine laboratory activities.

7.3.1.3

In PT schemes that require participants to sample, prepare or manipulate the PT item and submit it to the PT provider, the PT provider shall issue appropriate instructions for preparation, environmental conditions (where applicable), packaging, handling, storage and shipping of the PT item.

7.3.2.1

Criteria for suitable homogeneity and stability shall be established and shall be based on the risks that inhomogeneity and instability can impact the evaluation of the performance of participants.

7.3.2.2

The procedures for the assessment of homogeneity and stability shall be documented and conducted, where applicable, in accordance with appropriate statistical designs.

7.3.2.3

The assessment of homogeneity and stability shall be performed for every PT round after the PT items have been packaged in their final form.

NOTE 1

Homogeneity can be demonstrated prior to packaging where no influence of packaging is reasonably expected or when stability studies indicate that the material is preferably stored in bulk form.

NOTE 2

Different approaches for the assessment of homogeneity and stability, including situations where experimental study is not feasible, are described in Annex B of this document, in ISO 13528 and in ISO Guide 35.

7.3.2.4

Where experimental evidence is needed to assess homogeneity or stability of the PT item (or both), the PT provider shall use appropriate methods to assess the homogeneity and stability of the PT item.

7.3.2.5

PT items shall be demonstrated to be sufficiently stable to ensure that they will not undergo any significant change throughout the conduct of the PT round, including storage and transport. When this is not possible, the stability shall be quantified and considered as an additional component of the uncertainty associated with the assigned value of the PT item and/or taken into account in the evaluation criteria.

7.3.2.6

When PT items from previous PT rounds are retained for another PT round, property values or characteristics to be determined in the PT scheme shall be confirmed by the PT provider prior to distribution.

7.3.3.1

From the time of production to their distribution to participants, the PT provider shall ensure that PT items are appropriately identified and stored to prevent contamination, damage or deterioration.

7.3.3.2

The PT provider shall have appropriate procedures for dispatch to, and receipt from, storage.

7.3.3.3

The condition of stored PT items shall be assessed at specified intervals or prior to distribution in order to detect possible deterioration.

7.3.3.4

Where potentially hazardous PT items are used, facilities shall be available to ensure their safe handling, decontamination and disposal.

7.3.4.1

The PT provider shall control packaging and labelling processes to the extent necessary to ensure conformity with relevant national, regional, or international safety and transport requirements.

7.3.4.2

The PT provider shall document relevant environmental conditions for the transport of PT items. If necessary, environmental conditions shall be monitored during transport.

7.3.4.3

In PT schemes where participants are required to transport the PT items to other participants, or return them to the PT provider, documented instructions for this transport, to ensure the validity of the PT item, shall be supplied.

7.3.4.4

The PT provider shall ensure that labels are securely attached to the packaging of individual PT items and are designed to remain legible and intact throughout the PT round.

7.3.4.5

The PT provider shall follow a procedure to enable the confirmation of delivery of the PT items.

7.3.5.1

The PT provider shall give participants sufficient notice before sending PT items, providing the date on which the PT items are likely to arrive or to be dispatched, unless the design of the PT scheme makes it inappropriate to do so.

7.3.5.2

The PT provider shall give detailed documented instructions to all participants. Instructions to participants shall include:

1. a) the necessity to treat PT items in the same manner as routine samples, including use of routine measurement or test methods, unless there are particular requirements of the PT scheme which require departure from this principle;
2. b) details of factors which can influence the measurements or tests of the PT items, e.g. the nature of the PT items, conditions of storage, whether the PT scheme is limited to selected measurement or test methods and the timing of the measurements or tests;
3. c) instructions for preparing or conditioning, or both, of the PT items before conducting the measurements or tests that would not be considered part of a laboratory’s usual expected practices, unless these activities are part of the PT scheme;
4. d) any appropriate instructions on handling the PT items, including any safety requirements;
5. e) any specific environmental conditions for the participant to conduct measurements or tests, or both, and, if relevant, any requirement for the participants to report relevant environmental conditions during the time of the measurement or test;
6. f) specific and detailed instructions on the manner of recording and reporting results and associated measurement uncertainties, i.e. when the instructions include reporting of the expanded measurement uncertainty, the reported uncertainty shall include the coverage factor and the coverage probability;

   NOTE

   This instruction usually includes parameters such as the units of measurement, the number of significant figures or decimal places, and the reporting basis (e.g. on “dry weight” or “as received”).

7. g) specific instructions on providing details concerning the measurement or test method used by the participant, where a single specific measurement or test method is not required;
8. h) instructions on return or forwarding of the PT items, when applicable;
9. i) the last date for the PT provider to receive the results from the participants;
10. j) information on the contact details of the PT provider for enquiries.

7.4.1.1

Results received from participants shall be recorded and analysed by appropriate methods. Procedures shall be established and implemented to check the validity of data entry, data transfer, statistical analysis, and reporting.

7.4.1.2

Data analysis shall generate summary statistics and performance statistics and associated information consistent with the statistical design of the PT scheme.

7.4.1.3

The influence of outliers on summary statistics shall be minimized by using an appropriate statistical approach.

7.4.1.4

The PT provider shall have procedures for treatment of results from different measurement or test methods, where the PT scheme allows participants to use different measurement or test methods.

7.4.1.5

The PT provider shall have documented criteria and procedures for dealing with measurement or test results that are inappropriate for statistical evaluation, e.g. because of calculation errors, transpositions and other gross errors.

7.4.1.6

The PT provider shall have documented criteria and procedures to identify and manage situations where PT items that have been distributed and the collected data are subsequently found to be unsuitable for performance evaluation, e.g. because of inhomogeneity, instability, damage or contamination.

7.4.2.1

The PT provider shall use valid methods of evaluation which meet the objectives of the PT scheme. The methods shall be documented and include a description of the basis for the evaluation.

NOTE

Examples of valid methods of evaluation are described in ISO 13528.

7.4.2.2

Where applicable for the objectives of the PT scheme, the PT provider shall provide expert commentary on the performance of participants with regard to the following:

1. a) overall performance against prior expectations, taking measurement uncertainties into account;
2. b) variation within and between participants, and comparisons with any previous PT rounds, similar PT schemes, or published data;
3. c) variation between measurement or test methods;
4. d) possible sources of error (with reference to outliers or poor performance) and suggestions for improving performance;
5. e) advice and feedback to participants as part of the continuous improvement procedures of participants;
6. f) situations where unusual factors make evaluation of results and commentary on performance impossible;
7. g) any other suggestions, recommendations or general comments;
8. h) conclusions.

NOTE

It can be useful to provide individual summary sheets for participants periodically during or after completion of a particular PT round. These can include updated summaries of performance for individual participants over successive PT rounds of a continuous PT scheme. Such summaries can be further analysed and trends highlighted, if required.

7.4.3.1

PT reports shall be clear, accurate, objective and comprehensive and include data covering the results of all participants, together with an indication of the performance of individual participants.

NOTE

When it is not practical to report all original data to participants, a summary of the results, e.g. in tabulated or graphical form, can be supplied.

7.4.3.2

Reports shall include the following, unless it is not applicable or the PT provider has valid reasons for not doing so:

1. a) the name and contact details of the PT provider;
2. b) identification of person(s) authorizing the report;
3. c) an indication of which activities are provided by external providers when they affect the production or characterization of the PT items or the services provided;
4. d) the date of issue and status (e.g. preliminary, interim, or final) of the report;
5. e) unique identification that all its components are recognized as a portion of a complete report and a clear identification of the end;
6. f) a statement of the extent to which results are confidential;
7. g) a unique identification of the report and the PT scheme;
8. h) a clear description of the PT items used, including necessary details of the PT item's production and homogeneity and stability assessment;
9. i) the results of participants, including the reported measurement uncertainties;
10. j) procedures used to statistically analyse the data;
11. k) statistical data and summaries, including assigned values, range of acceptable results and graphical displays;
12. l) details of the metrological traceability, and uncertainty of any assigned value;
13. m) procedures used to establish any assigned value and its uncertainty;
14. n) assigned values, their uncertainties and summary statistics for measurement or test methods used by each group of participants (if different measurement or test methods are used by different groups of participants);
15. o) procedures used to establish the standard deviation for proficiency assessment, or other criteria for evaluation;
16. p) comments on the performance of participants;
17. q) information about the design and implementation of the PT scheme;
18. r) advice on the interpretation of the statistical analysis;
19. s) comments or recommendations based on the outcomes of the PT round.

NOTE

For continuous PT schemes, it can be sufficient to have simpler reports, such that many of the elements in this subclause (7.4.3.2) can be excluded from routine reports but included in the PT scheme procedures or in periodic summary reports that are available to participants.

7.4.3.3

Reports shall be made available to participants within planned timescales. In sequential PT schemes, e.g. where the turn-around time can be very long, and in PT schemes involving perishable materials, preliminary or anticipated results may be provided before final results are disclosed.

NOTE

Preliminary or anticipated results allow for early investigation of possible errors.

7.4.3.4

The PT provider shall have a policy for the use of reports by participants and customers.

7.4.3.5

When it is necessary to issue a new or amended report for a PT scheme or PT round, this report shall include the following:

1. a) a unique identification;
2. b) a reference to the original report that it replaces or amends;
3. c) identification of the amendment and a statement concerning the reason for the amendment or re-issue.

7.4.3.6

When issuing an amended report to a subset of participants, an analysis of the potential impact on the other participants for that PT scheme and/ or PT round shall be made to ensure there is no influence on the general performance of the other participants.

7.4.3.7

If the PT provider issues a statement of participation or performance in addition to the PT report, the statement shall not be misleading.

7.5.1.1

The PT provider shall ensure that technical records for each PT activity contain the results, reports and sufficient information to facilitate, if possible, identification of factors affecting the PT performance evaluation and its associated characteristics and enable the repetition of the PT activity under conditions as close as possible to the original. The technical records shall include the date and the identity of personnel responsible for each PT activity and for checking data and results.

7.5.1.2

Data used to verify the PT items, instructions to participants, the original responses of participants and any other information included in reports shall be recorded at the time they are made and shall be identifiable with the specific task.

7.5.1.3

The PT provider shall ensure that amendments to technical records can be tracked to previous versions or to original information submitted by participants. Both the original and amended data and files shall be retained, including the date of alteration, an indication of the altered aspects and the personnel responsible for the alterations.

7.5.2.1

The PT provider shall have access to the data and information needed to perform its activities.

7.5.2.2

The PT provider information management system used for the collection, processing, recording, reporting, storage or retrieval of data shall be validated for functionality, including the proper functioning of interfaces before introduction. Whenever there are any changes, including PT provider software configuration or modifications to commercial off-the-shelf software, they shall be authorized, documented and validated before implementation.

NOTE 1

In this document, a PT provider information management system includes the management of data and information contained in both computerized and non-computerized systems. Some of the requirements can be more applicable to computerized systems than to non-computerized systems.

NOTE 2

Commercial off-the-shelf software in general use within its designed application range can be considered sufficiently validated.

7.5.2.3

The PT provider information management system shall:

1. a) be protected from unauthorized access;
2. b) be safeguarded against tampering and loss;
3. c) be operated in an environment that complies with the system supplier or PT provider specifications or, in the case of non-computerized systems, provides conditions which safeguard the accuracy of manual recording and transcription;
4. d) be maintained in a manner that ensures the integrity of the data and information;
5. e) include recording of system failures and the appropriate immediate and corrective actions.

7.5.2.4

When a PT provider information management system is managed and maintained off-site or through an external service provider, the PT provider shall ensure that the external service provider or operator of the system complies with all applicable requirements of this document.

7.5.2.5

The PT provider shall ensure that instructions, manuals and reference data relevant to the PT provider information management system are made readily available to personnel.

7.5.2.6

Calculations and data transfers shall be checked in an appropriate and systematic manner.

7.5.3 Surveillance of the processes

The PT provider shall have a procedure to ensure the validity of the PT scheme. Surveillance activities shall be planned and reviewed [see also 8.9.2 item n)], and the resulting data shall be recorded for the continuous improvement process.

NOTE

Depending on the PT scheme, surveillance activities can include:

7.5.4.1

The PT provider shall have a procedure that shall be implemented when any aspect of its PT schemes does not conform to its own procedures or the agreed requirements of its participants or customers. The procedure(s) shall ensure that:

1. a) the responsibilities and authorities for the management of nonconforming work are defined;
2. b) actions (including halting work of ongoing PT schemes and/or PT rounds and withholding PT schemes and/or PT round reports, as necessary) are defined and are based upon the risk levels established by the PT provider;
3. c) an evaluation of the significance of the nonconforming work is made, including an impact analysis on previous PT activities;
4. d) a decision on the need for action and timescale is taken immediately, together with any decision about the acceptability of the nonconforming work;
5. e) PT scheme participants and customers, as appropriate, are informed and the nonconforming PT items or PT reports already sent to participants are recalled or disregarded;
6. f) the responsibility for authorization of the resumption of work is defined.

NOTE

Identification of nonconforming work or problems with the management system or with technical activities can occur at various places within the management system and technical operations. Examples are participant or customer complaints, management reviews and internal or external audits, surveillance of the processes, production of PT items, homogeneity and stability assessments, data analysis, instructions to participants and materials handling and storage.

7.5.4.2

The PT provider shall retain records of nonconforming work and actions as specified in 7.5.4.1 items b) to f).

7.5.4.3

Where the evaluation indicates that nonconforming work can recur or that there is doubt about the compliance of the PT provider with their own procedures, the corrective action procedure in 8.7 shall be promptly followed.

7.6.1

The PT provider shall have a documented procedure for handling complaints that shall include at least the following:

1. a) a description of the process for receiving, substantiating and investigating the complaint and deciding what actions shall be taken in response;
2. b) tracking and recording the complaint, including the actions undertaken to resolve it;
3. c) ensuring that any appropriate action is taken.

7.6.2

A description of the process for handling complaints shall be publicly available.

7.6.3

Upon receipt of a complaint, the PT provider shall confirm whether the complaint relates to PT activities and, if so, shall resolve the complaint.

7.6.4

The PT provider receiving the complaint shall be responsible for gathering all necessary information to determine whether the complaint is substantiated.

7.6.5

Whenever possible the PT provider shall acknowledge receipt of the complaint and provide the complainant with the outcome and, if applicable, progress reports.

7.6.6

Investigation and resolution of complaints shall not result in any discriminatory actions.

7.6.7

The resolution of complaints shall be made by, or reviewed and approved by, persons not involved in the subject of the complaint in question. Where resources do not permit this, any alternative approach shall not compromise impartiality.

7.6.8

Whenever possible, the PT provider shall give formal notice of the end of the handling of the complaint to the complainant.

7.6.9

The PT provider shall be responsible for all decisions at all levels of the handling process for complaints.

7.7.1

The PT provider shall have a documented procedure for handling appeals that shall include at least the following:

1. a) a description of the process for receiving and investigating the appeal and deciding what actions shall be taken in response;
2. b) tracking and recording the appeal, including the actions undertaken to resolve it;
3. c) ensuring appropriate action is taken.

NOTE

PT providers that only have PT schemes using purely statistically derived evaluation procedures do not usually handle appeals. Appeals concerning performance evaluations can be addressed as a complaint.

7.7.2

A description of the process for handling appeals shall be publicly available.

7.7.3

The PT provider shall acknowledge receipt of the appeal and provide the appellant with the outcome and, if applicable, progress reports.

7.7.4

The PT provider receiving the appeal shall be responsible for gathering all necessary information to determine whether the appeal is valid.

7.7.5

The PT provider shall be responsible for all decisions during the process for handling appeals.

7.7.6

The decision on the appeal shall be made by, or reviewed and approved by, persons not involved in the decision that is the subject of the appeal in question.

7.7.7

Investigation and decision on appeals shall not result in any discriminatory actions.

8.1.1

The PT provider shall establish, document, implement and maintain a management system to support and demonstrate the consistent fulfilment of the requirements of this document and its scope of PT activities.

8.1.2

The management system of the PT provider shall include at least the following:

8.1.3

A PT provider may meet 8.1.2 by establishing, implementing and maintaining a quality management system (e.g. in accordance with the requirements of ISO 9001). This quality management system shall support and demonstrate the consistent fulfilment of the requirements of this document.

8.1.4

The PT provider management shall provide evidence of commitment to the development and implementation of the management system and to continually improving its effectiveness.

8.2.1

The policies and objectives shall address the competence, impartiality and consistent operation of the PT provider.

8.2.2

All documentation, processes, systems and records related to the fulfilment of the requirements of this document shall be included in, or referenced from, the management system.

8.2.3

All personnel involved in PT activities shall have access to the parts of the management system documentation and related information that are applicable to their responsibilities.

8.3.1

The PT provider shall control the documents (internal and external) that relate to the fulfilment of this document.

8.3.2

The PT provider shall ensure that:

1. a) documents are approved for adequacy prior to issue by authorized personnel;
2. b) documents are periodically reviewed and updated as necessary;
3. c) changes and current revision status of documents are identified;
4. d) relevant versions of applicable documents are available at points of use and their distribution is controlled;
5. e) documents are uniquely identified;
6. f) the unintended use of obsolete documents is prevented and that suitable identification is applied to them if they are retained for any purpose.

8.4.1

The PT provider shall establish and retain legible records to demonstrate fulfilment of the requirements in this document.

8.4.2

The PT provider shall implement the controls needed for the identification, storage, protection, back-up, archive, retrieval, retention time and disposal of its records.

8.4.3

The PT provider shall retain records for a period consistent with its contractual obligations. Access to these records shall be consistent with the confidentiality commitments and records shall be readily available.

NOTE

Additional requirements regarding technical records are given in 7.5.1.

8.5.1

The PT provider shall consider the risks and opportunities associated with the PT activities in order to:

1. a) give assurance that the management system achieves its intended results;
2. b) enhance desirable effects to achieve the purpose and objectives of the PT provider;
3. c) prevent, or reduce, undesired impacts and potential failures in the PT activities;
4. d) achieve improvement.

8.5.2

The PT provider shall plan:

1. a) actions to address these risks and opportunities;
2. b) how to integrate and implement these actions into its management system;
3. c) how to evaluate the effectiveness of these actions.

NOTE

Although this document specifies that the PT provider plans actions to address risks, there is no requirement for formal methods for risk management or a documented risk management process. PT providers can decide whether or not to develop a more extensive risk management methodology, e.g. through the application of other guidance or standards.

8.5.3

Actions taken to address risks and opportunities shall be proportional to the potential impact on the validity of the PT scheme.

NOTE 1

Examples of addressing risks include developing strategies for preventing collusion between participants and performing a feasibility study to evaluate the best transport conditions for the PT items of a PT scheme.

NOTE 2

Opportunities can lead to expanding the scope of the PT activities, increasing the number of participants in a PT scheme, making a PT scheme more cost effective for the PT provider as well as the participants and customers, and reducing the time required to produce the PT items.

8.6.1

The PT provider shall identify and select opportunities for improvement and implement any necessary actions.

NOTE

Opportunities for improvement can be identified through the review of the operational procedures, the use of the policies, overall objectives, audit results, corrective actions, management review, suggestions from personnel, risk assessment, analysis of data and external assessments.

8.6.2

The PT provider shall seek feedback, both positive and negative, from its participants and customers. The feedback shall be analysed and used to improve the management system, PT activities and customer service.

8.7.1

When a nonconformity occurs, the PT provider shall:

1. a) react to the nonconformity and, as applicable:

   — take action to control and correct it;

   — address the consequences;

2. b) evaluate the need for action to eliminate the cause(s) of the nonconformity, in order that it does not recur or occur elsewhere, by:

   — reviewing and analysing the nonconformity;

   — determining the causes of the nonconformity;

   — determining if similar nonconformities exist, or can potentially occur;

3. c) implement any action needed;
4. d) review the effectiveness of any corrective action taken;
5. e) update risks and opportunities determined during planning, if necessary;
6. f) make changes to the management system, if necessary.

8.7.2

Corrective actions shall be appropriate to the effects of the nonconformities encountered.

8.7.3

The PT provider shall retain records as evidence of:

1. a) the nature of the nonconformities, cause(s) and any subsequent actions taken;
2. b) the effectiveness of any corrective action.

8.8.1

The PT provider shall conduct internal audits at planned intervals to provide information on whether the management system:

1. a) conforms to:

   — the PT provider’s own requirements for its management system, including the PT activities;

   — the requirements of this document;

2. b) is effectively implemented and maintained.

8.8.2

The PT provider shall:

1. a) plan, establish, implement and maintain an audit programme including the frequency, methods, responsibilities, planning requirements and reporting, which shall take into consideration the importance of the PT activities concerned, changes affecting the PT provider and the results of previous audits;
2. b) ensure that internal audits are performed by personnel knowledgeable in conducting PT activities and auditing and the requirements of this document and that these personnel are independent of activities being audited, wherever resources permit;
3. c) define the audit criteria and scope for each audit;
4. d) ensure that the results of the audits are reported to relevant management;
5. e) implement appropriate corrections and corrective actions without undue delay;
6. f) retain records as evidence of the implementation of the audit programme and the audit results.

NOTE

ISO 19011 provides guidelines for auditing management systems.

8.9.1

The PT provider management shall review its management system at planned intervals, in order to ensure its continuing suitability, adequacy and effectiveness, including the stated policies and objectives related to the fulfilment of this document.

8.9.2

The inputs to management review shall be recorded and shall include information related to the following:

1. a) changes in internal and external issues that are relevant to the PT provider;
2. b) fulfilment of objectives;
3. c) suitability of policies and procedures;
4. d) status of actions from previous management reviews;
5. e) outcome of recent internal audits;
6. f) corrective actions;
7. g) assessments by external bodies;
8. h) changes in the volume and type of the work or in the range of PT activities;
9. i) customer, participant and personnel feedback;
10. j) complaints and appeals;
11. k) effectiveness of any implemented improvements;
12. l) adequacy of resources;
13. m) results of risk identification;
14. n) outcomes of the surveillance of the processes;
15. o) other relevant factors, such as training.

8.9.3

The outputs from the management review shall record all decisions and actions related to at least:

1. a) the effectiveness of the management system and its processes;
2. b) improvement of the activities related to the fulfilment of the requirements of this document;
3. c) provision of required resources;
4. d) any need for changes.

A.1.1

PT is an essential aspect of laboratory practice in all areas of measurement, testing and calibration, and can also be important for inspection and sampling. PT schemes vary according to the needs of the sector in which they are used, the nature of the PT items, measurement or test methods in use and the number of participants. In their simplest form, most PT schemes feature a comparison of results obtained by one laboratory with those obtained by one or more other laboratories.

A.1.2

PT schemes will normally consist of at least one element of each of the following features.

1. a) Type of expected results:
   1. 1) qualitative (including data on nominal or ordinal scales);
   2. 2) quantitative (including data on interval or ratio scales);
   3. 3) interpretive (including descriptive or interpretive information).
2. b) Frequency:
   1. 1) single (or first) occasion;
   2. 2) continuous.
3. c) Distribution format:
   1. 1) simultaneous;
   2. 2) sequential from one participant to another, either directly or via the PT provider.
4. d) Process:
   1. 1) pre-analytical (e.g. sample processing or test ordering);
   2. 2) analytical;
   3. 3) post-analytical (e.g. interpretive).
5. e) Method of determination of assigned values (see Annex B for further information):
   1. 1) metrologically traceable reference values (e.g. SI);
   2. 2) consensus of a selected group of competent participants;
   3. 3) consensus of all participants.
6. f) Performance evaluation criteria (see Annex B for further information):
   1. 1) by expert judgement or regulatory mandate (i.e. a prescribed value);
   2. 2) by experience with previous PT rounds of a PT scheme or the reproducibility of the measurement or test method being used;
   3. 3) by comparison to other participants;
   4. 4) including consideration of the measurement uncertainty of the results of participants.

A.1.3

There are many possible designs for PT schemes based on the options of the six features listed in A.1.2, several of which can appear in the same PT scheme.

A.2.1

PT schemes have different features depending on the type of PT scheme, as outlined in A.1.2. Some common applications of those types of PT schemes are discussed below.

A.2.2

Simultaneous PT schemes usually involve sub-samples from a source of material being distributed simultaneously to participants for concurrent measurements or tests. After completion of the measurements or tests, the results are returned to the PT provider and compared with the assigned value(s) to give an indication of the performance of the individual participants and the group as a whole. Examples of PT items used in this type of PT scheme include food, body fluids, agricultural products, water, soils, minerals and other environmental materials. In some cases, separate portions of previously established reference materials are circulated. Advice or educational comments are typically part of the report returned to participants by the PT provider with the aim of promoting improvement in performance.

A.2.3

Sequential participation PT schemes involve the PT item being circulated successively from one participant to the next, or occasionally circulated back to the PT provider for rechecking. The key features are typically those described below.

1. a) A reference laboratory (as an expert laboratory) that is capable of providing a reliable PT item and metrologically traceable assigned value with sufficiently small uncertainty, compared to the claimed measurement uncertainty of other participants. For nominal or ordinal properties, the assigned value should be determined by consensus of experts or other authoritative sources. It is usually necessary for the PT item to be checked at specific stages during the conduct of the PT scheme, in order to ensure that there is no significant change in the assigned value.
2. b) The individual measurement or test results are compared with the assigned value established by the reference laboratory. The PT provider should take into account the claimed measurement uncertainty of each participant, or the claimed level of expertise. It can be difficult to compare measurement or test results on a group basis as there can be relatively few participants having measurement or test capabilities that closely match each other.
3. c) PT schemes involving sequential participation take time (in some cases, years) to complete. This causes a number of difficulties, such as:

   — ensuring the stability of the PT item;

   — the strict monitoring of the circulation among participants and the time allowed for measurement or testing by individual participants, as well as the need to supply feedback on individual performance during the PT scheme's implementation, rather than waiting until it finishes;

   — the increased possibility of collusion between participants.

4. d) PT items used in this type of PT scheme can include, for example, measurement reference standards (e.g. resistors, micrometers and frequency counters) or, in medical programmes, histology slides with confirmed diagnoses.
5. e) In some situations, the assigned value for a PT item may be determined by consensus, after all participants (or in some situations, a subset of participants) have submitted measurement or test results.
6. f) For sequential PT schemes, it is important to ensure confidentiality and prevent collusion between participants and that the size and the within-unit homogeneity of the PT item is sufficient to ensure that the first participant and the last participant in the PT scheme will essentially get the same PT item.

A.2.4

One special application of PT, often called “blind” or “double blind” PT, is where the PT item is indistinguishable from normal customer items or samples received by the laboratory. This type of PT can be difficult to organize, since the PT provider must make sure that the PT item is packaged and shipped in such a way that it remains anonymous to the laboratory.

A.2.5

A common design for a PT scheme is the “split-level” design, where similar (but not identical) levels of the property (measurand) or characteristic are included in two separate PT items. This design is used to estimate the participant's precision at specific levels of a property (measurand) or characteristic. It avoids problems associated with replicate measurements or tests on the same PT item, or with the inclusion of two identical PT items in the same PT round.

A.2.6

Partial-process and interpretive PT schemes are special types of PT schemes that involve the evaluation of a participant’s ability to perform parts of the overall measurement or test process, covering pre-analytical, analytical and post-analytical aspects of the measurement or test process. For example, some existing PT schemes evaluate the ability of participants to transform and report a given set of data, rather than conduct the actual measurement or test. Participants are required to make interpretations based on a given set of data or PT items, such as stained blood films for diagnosis, or to take and prepare samples or specimens in accordance with a specification. Some PT schemes can require participants to submit samples to the PT provider for review, or to provide a sampling plan.

A.2.7

Sampling PT schemes can involve automated sampling and analysis or drawing samples from a source or sample preparation from a provided lot for subsequent measurements or tests. The PT schemes can be designed to evaluate correct selection of a sampling plan and application of sampling procedure or preparation of representative samples to obtain a correct conclusion. Performance criteria in these PT schemes can be on the basis of expert judgement. The PT items can be case studies based on regulatory or customer requirements. These PT schemes have been applied in areas such as sampling for ambient air, emissions, noise, indoor environment and coal to improve the performance of the participants.

A.3.1

External quality assessment (EQA) programmes (such as those provided for laboratory medicine examinations) offer a variety of interlaboratory comparisons based on a traditional PT scheme model, but often with a broader application than many of the schemes described in A.1 and A.2. Many EQA programmes are designed to provide insight into the complete path of workflow of the laboratory and not just the measurement or test (or examination) processes. Most EQA programmes are continuous schemes that include long term follow-up of laboratory performance. A typical feature of EQA programmes is to provide education to participants and promote quality improvement. Advisory and educational comments form part of the report returned to participants to achieve this aim.

A.3.2

Some EQA programmes assess performance of pre-analytical and post-analytical phases of measurement or testing, as well as the analytical phase. In such EQA programmes, the nature of the PT item can differ significantly from that used in more traditional PT schemes that only address the analytical phase of the process. The PT item may be a questionnaire or case study circulated by the EQA provider to each participant for return of specific answers. Alternatively, pre-analytical information may accompany the PT item, requiring the participant to select an appropriate approach to measurement or testing or interpretation of results, and not just to perform the measurement or test. In “sample review” schemes, participants can be required to provide the “PT items” to the EQA provider. This may take the form of a processed specimen or sample (e.g. stained slide or fixed tissue), or laboratory data (e.g. measurement or test results, laboratory reports or quality assurance/control records). These aspects are not unique to EQA programmes and many PT schemes also provide assessment of processes across the measurement cycle.

A.4.1

In order to conduct PT, there must be valid measurement or test methods and appropriate PT items and, to be commercially viable, a sufficient number of potential participants. These are not necessarily available in new fields of measurement, testing, inspection or sampling, or, for example, with testing for novel pathogens or biomarkers. In these situations, other types of interlaboratory comparisons can be useful.

A.4.2

One special type of interlaboratory comparison design that is often used by customers of participants and some regulatory bodies is the “split-sample” design. Typically, split-sample interlaboratory comparisons involve comparisons of the data produced by small groups of participants (often only two). In these interlaboratory comparisons, samples of a product or a material are divided into two or more parts, with each participant testing one part of the sample. Uses for this type of interlaboratory comparisons include identifying poor accuracy, describing consistent bias and verifying the effectiveness of corrective actions. This design can be used to evaluate one or both participants as suppliers of measurement or testing services, or in cases where there are too few participants for appropriate evaluation of results. Under such interlaboratory comparisons, one of the participants can be considered to operate at a higher metrological level (i.e. smaller measurement uncertainty), due to the use of reference methodology and more advanced equipment, for example, or through confirmation of its own performance through satisfactory participation in a recognized interlaboratory comparison. In such interlaboratory comparisons, the results of the laboratory operating at a higher metrological level are considered to be the assigned values and this laboratory can act as an advisory or mentor laboratory to the other participant(s).

A.4.3

It can be useful to conduct trial (or “pilot”) interlaboratory comparisons, following the plan and design of a PT scheme, but without evaluating performance.

A.4.4

Two common types of interlaboratory comparisons are collaborative studies to establish the performance characteristics of a measurement or test method (see ISO 5725) and collaborative studies to characterize a reference material (see ISO Guide 35). These studies will not be discussed further in this document.

B.1.1

PT results can appear in many forms, based on the many different types of possible PT schemes, as described in Annex A. The statistical methods used to analyse the PT results must be appropriate for each situation, therefore they are too varied for all of the possibilities to be specified and discussed in detail in this document. The statistical analysis techniques that can be applied to most of the types of PT schemes described in Annex A are addressed in ISO 13528. ISO 13528 recognizes that other methods may be used provided they are statistically valid and are described in detail for participants. ISO 13528 also presents guidance on design and visual data analysis. Other references may be consulted for specific types of PT schemes, e.g. in specific areas of measurement or testing, or novel applications of PT.

B.1.2

The methods discussed in this annex and in the referenced documents cover the fundamental steps common to nearly all PT schemes:

1. a) preliminary assessment of PT item homogeneity and stability;
2. b) determination of the assigned value;
3. c) calculation of performance statistics;
4. d) evaluation of performance.

B.1.3

This annex does not consider statistical methods for analytical studies other than for treatment of PT scheme data. Different methods can be used to implement the other uses of interlaboratory comparison data listed in the Introduction and A.4.

B.2.1

The requirements of this document call for an assessment of sufficient homogeneity and stability of PT items using valid statistical methods. The procedure to do this is based on the risk that differences between PT items can impact the performance evaluation. This has traditionally been accomplished with an experiment that demonstrates acceptably small differences between PT items, using a representative sample of the PT items, and with criteria for sufficiency being related to the performance evaluation criteria. A risk approach also allows use of experience and technical expertise. Statistical approaches for assessment of sufficient homogeneity and stability can include:

B.2.2

Experimental evidence is usually needed for PT in novel areas of application or when using new procedures for production of PT items. It is also usually needed for PT schemes that use inherently inhomogeneous or unstable materials or consumer products. When a PT provider has experience with a specific type of PT item and methods of production, there can be negligible risk of differences between PT items that can impact the performance evaluation. Often risk and cost can be minimized by the use of experience and multiple sources of information.

B.2.3

Although the evaluation of homogeneity of PT items can be the same as for the evaluation of certified reference materials, in the second case it requires the estimation of uncertainties due to differences between items. Given the need for a reliable estimate for this component for certified properties, the number of randomly selected samples can exceed what is needed for PT, where the main objective is to check for differences that can have an impact on the performance evaluation.

B.2.4

In the case of qualitative or semi-quantitative PT schemes, requirements for homogeneity and stability are considered to be met if all of the representative sample of PT items have and (for stability) maintain the expected values of the properties or characteristics of interest.

B.2.5

Assessment of homogeneity or stability can also be required when the same PT item is circulated to all participants. The assessment of homogeneity is particularly important in cases where the property or characteristic can vary depending on factors such as the position of measurement or test (e.g. immersion depth), and frequency.

B.3.1

There are various procedures available for the establishment of assigned values, which are required to be consistent with the objectives of the PT scheme. The most common procedures for determining the assigned value are listed below:

1. a) formulation;
2. b) a certified reference material;
3. c) results from one laboratory;
4. d) consensus value from expert laboratories;
5. e) consensus value from participant results (a chosen subset, or all participants).

NOTE

Approaches c), d) or e) can involve the use of a reference measurement procedure.

B.3.2

The statistical methods used and the evaluation of performance will be different if the determination of the assigned value is independent of participant results [cases a), b), c), d)], or if the assigned value is derived from participant results [case e)]. Appropriate statistical approaches are described in ISO 13528.

B.3.3

Methods of determining assigned values are chosen to evaluate participants fairly, yet encourage comparability among measurement or test methods, as described in the objectives for the PT scheme. Metrological traceability can also be an important consideration. For example, consensus values from participant results that do not have metrological traceability would not be appropriate for PT schemes in calibration.

B.3.4

Procedures for determining the uncertainty of assigned values are discussed in detail in ISO 13528, for each common statistical approach. Additional information on uncertainty is provided in ISO/IEC Guide 98-3 and ISO Guide 35.

B.3.5

For qualitative and interpretative PT schemes, a number of options are available for deriving the assigned value:

1. a) by expert judgement;
2. b) by use of reference materials as PT items;
3. c) from knowledge of the origin or production of the PT item(s);
4. d) using the mode or median of participant results (the median is not appropriate for nominal values).

B.3.6

For qualitative and interpretative PT schemes, in some cases a PT provider can use a consensus value, as defined by agreement of a predetermined majority percentage of responses (e.g. a PT provider can specify that an assigned value must achieve at least 80 % agreement among participants, to limit the risk the PT item was not representative, or damaged). The percentage used should be determined based on objectives for the PT scheme and the level of competence and experience of the participants. Further considerations for handling qualitative data are provided in ISO 13528.

B.3.7

When statistical analysis is performed on participant results to determine an assigned value or performance criterion, PT providers are required to have:

1. a) procedures for dealing with extreme results, referred to in this document as gross errors and outliers (see 7.4.1.5);
2. b) detailed statistical procedures for calculating the mean and the standard deviation from participant data, appropriate for the objectives of the PT scheme and the number of participants; these include steps to check that statistical assumptions are reasonable (e.g. that the distribution of participant results is unimodal and reasonably symmetric) (see 7.2.2.3).

NOTE

ISO 13528 recommends robust statistical methods for the determination of the consensus mean and standard deviation, without the need for outlier removal, but it is important to ensure as far as possible that results identifiable as gross errors are not included in statistical analysis, whether robust procedures are used or not.

B.3.8

Other considerations include the following.

1. a) Ideally, if assigned values are determined by participant consensus, the PT provider should have a procedure to establish the validity of the assigned values and for reviewing the distribution of the data.
2. b) The PT provider should have criteria for the acceptability of an assigned value in terms of its uncertainty. In ISO 13528, the recommended criterion for most situations (0,3 σ_pt) is based on a goal to minimize the effect that uncertainty in the assigned value has on the performance evaluation. If this criterion is not met, the PT provider should apply alternative performance assessments that take the uncertainty into consideration.

B.4.1.1

Participant results reported in a PT scheme are often transformed into a performance statistic, in order to aid interpretation and to allow comparison with defined objectives. The purpose is to present the deviation from the assigned value in a manner that allows simple and consistent interpretation across different PT rounds of the PT scheme and different properties (measurands) or characteristics. Statistical methods can range from no processing required to complex statistical transformations.

B.4.1.2

Performance statistics should be meaningful to participants. Performance statistics are most useful when chosen for the relevant measurements or tests being undertaken by the participant and well understood or traditional within a particular field and community.

B.4.1.3

Commonly used statistics for quantitative results are described in detail in ISO 13528, along with the appropriate formulas for their calculation. The details are essential for correct application and cannot be described briefly in this document. Common statistical approaches are as follows.

1. a) The difference, D, between the participant’s result and the assigned value, is calculated either as an absolute difference, D_i, using Formula (B.1) or as a percentage of the assigned value, D_i %, using Formula (B.2):

   (B.1)

   (B.2)

   where

   	xi	is the result from participant i;
   	xpt	is the assigned value.

   Performance is usually evaluated relative to a performance criterion, δ, as an absolute value or a percentage, as follows:

   — the result is acceptable and generates no signal when |D_i| ≤ δ or when |D_i| % ≤ δ %;

   — the result is not acceptable and generates an action signal when |D_i| > δ or when |D_i| % > δ %.

2. b) The z score is calculated by evaluating the difference between the participant’s result and the assigned value against a designated performance criterion, i.e. the standard deviation for proficiency assessment, σ_pt, as shown in Formula (B.3):

   (B.3)

   As described in ISO 13528, σ_pt can be calculated by means of one of the following:

   — by perception of experts where a fitness for purpose goal for performance is set, as determined by expert judgement or regulatory mandate (prescribed value);

   — by experience from previous PT rounds of a PT scheme;

   — by use of a general model where an estimate can be derived from a statistical model for the reproducibility of the measurement or test methods;

   — by using a combination of repeatability and reproducibility standard deviations from a previous study of precision of the measurement or test methods or of a reference document;

   — from data obtained in the same round of the PT scheme using the participant results, e.g. a robust standard deviation based on participant results.

   Performance is usually evaluated relative to conventional performance criteria as follows:

   — the result is considered to be acceptable when |z_i | ≤ 2,0;

   — the result is considered to be questionable (warning signal) when 2,0 < |z_i | < 3,0;

   — the result is considered to be unacceptable (action signal) when |z_i | ≥ 3,0.

3. c) The z′ score is similar to the z score, but with allowance for the uncertainty of the assigned value, and it is calculated using Formula (B.4). The z′ score takes account of the standard uncertainty of the assigned value and is used where the standard uncertainty of the assigned value is considered not to be negligible, which allows the same interpretation as for traditional z scores.

   (B.4)

   where u(x_pt) is the standard uncertainty of the assigned value.Performance can be evaluated in the same way as for z scores.
4. d) The zeta score, ζ, is calculated as shown in Formula (B.5) by evaluating the difference between the participant’s result and the assigned value against the combined standard uncertainty of the participant’s result and the assigned value. This allows the same interpretation as for traditional z scores.

   (B.5)

   where u(x_i) is the standard measurement uncertainty of a participant's result.Performance can be evaluated the same as for z scores, or using other criteria based on the relevant coverage probability for the uncertainty.
5. e) The E_n score is similar to the zeta score, except that expanded uncertainties are used rather than standard uncertainties, and it is calculated using Formula (B.6).

   (B.6)

   where

   	U(xi)	is the expanded measurement uncertainty of a participant's result;
   	U(xpt)	is the expanded uncertainty of the assigned value.

   Performance is usually evaluated relative to conventional performance criteria as follows:

   — the result is acceptable and generates no signal when |(E_n)_i | < 1,0;

   — the result is not acceptable and generates an action signal when |(E_n)_i | ≥ 1,0.

   Generally, the desired outcome is for the |E_n| value to be as close to zero as possible, with a value approaching 1,0 requiring further investigation. The conventional performance criteria above should be considered in light of the risk associated with the distribution.

B.4.1.4

Other considerations are as follows.

B.4.2.1

For qualitative or ordinal results, if statistical methods are used, they must be appropriate for the nature of the responses.

B.4.2.2

For qualitative data, the appropriate technique is to compare a participant's result with the assigned value. If they are identical, then performance is acceptable. If they are not identical, then expert judgement is needed to determine if the result is fit for its intended use. In some situations, the PT provider can review the results from participants and determine that a PT item was not suitable for evaluation, or that the assigned value was not correct. These determinations should be part of the plan for the PT scheme and understood by the participants in advance of the operation of the PT scheme.

B.4.2.3

For ordinal results, the techniques used for qualitative data in B.4.2.2 are appropriate. Ordinal results include, for example, responses such as grades or rankings, sensory evaluations, or the strength of a chemical reaction (e.g. 1+, 2+, 3+, ...). Sometimes these responses are given as numbers (e.g. 1 = Poor, 2 = Unsatisfactory, 3 = Satisfactory, 4 = Good, 5 = Very Good). It is not appropriate to calculate conventional parametric summary statistics for ordinal and nominal data, even if the results are numerical. This is because the numbers are not on a linear scale. It is not appropriate to use evaluation statistics such as z scores for ordinal results. Specific statistics, such as rank or order statistics, designed for ordinal data, should be used.

B.4.2.4

There are situations where nominal and ordinal data can be transformed to a quantitative scale and the derived statistics can be treated as quantitative, with appropriate considerations for the statistical distribution of the transformed results. Such situations include the following:

1. a) PT schemes of a property or characteristic that has varying intensity but no linear scale (ordinal data) (e.g. the intensity of a chemical reaction or the intensity of a taste), where the ordinal data can be transformed to a linear scale, using objective reference points;
2. b) PT schemes where there are a sufficient number of qualitative responses from each participant (e.g. identifications of an organism in multiple samples) and quantitative statistics can be generated from the results (e.g. rates of true positive and true negative identifications);
3. c) indexes of performance based on the possible consequence of misidentification, e.g. 0 points for no effect, 1 point for incorrect but minor consequence, and 3 points for significant negative consequence (examples are discussed in References [20], [21] and [22]).

B.4.2.5

It is appropriate to list the distribution of results from all participants, or to produce a graph, along with the number or percentage of results in each category. Associated summary statistics can include the mode(s) (most common responses) and range (lowest and highest response). It can also be appropriate to evaluate results as acceptable based on closeness to the assigned value, e.g. results within plus or minus one response from the assigned value can be fit for the purpose of the measurements or tests. In some situations, it can be appropriate to evaluate performance based on percentiles, e.g. the 5 % of results farthest from the mode or farthest from the assigned value can be determined to be unacceptable. This should be based on the PT scheme design (i.e. fitness for purpose) and understood by participants in advance.

B.4.3.1

Performance can be evaluated based on more than one participant result in a single PT round. This occurs when there is more than one PT item for a particular property (measurand) or characteristic or a family of related properties or characteristics. It is done to provide a more comprehensive evaluation of performance.

B.4.3.2

Graphical methods, such as the Youden plot, are effective tools for interpreting performance (see ISO 13528). One commonly used combined performance score is simply the number (or percentage) of results determined to be acceptable.

B.4.3.3

In general, averaged performance scores are discouraged because they can mask poor performance on one or more PT items that should be investigated.

B.5.1.1

Performance can be evaluated in a variety of ways, in order to be consistent with the objectives of the PT scheme. All approaches involve comparing the participant result with the assigned value and then determining whether any differences can indicate a need for review or action by the participant. The objectives should be clear regarding the approach(es) used for each property or characteristic:

1. a) evaluate the difference relative to fitness for purpose (e.g. regulatory criteria or technical capability of the measurement or test method), including metrological traceability;
2. b) evaluate the difference relative to other participants in the current PT round or previous PT rounds (e.g. the standard deviation);
3. c) evaluate the difference relative to the participant’s claim for measurement uncertainty.

B.5.1.2

The approaches can vary for different PT schemes and for different properties or characteristics in the same PT scheme.

B.5.1.3

The statistical techniques are very different for the three approaches described in B.5.1.1.

See ISO 13528 for considerations in establishing acceptance criteria for the various performance scores.

B.5.1.4

Graphical approaches should be used whenever possible to show performance (e.g. histograms, error bar charts, ordered z score charts), as described in ISO 13528. These charts can be useful to show many different aspects of performance using:

1. a) distributions of participant results;
2. b) the relationship between participant results on multiple PT items;
3. c) comparative distributions for different measurement or test methods.

B.5.2.1

A PT scheme should, where appropriate, include procedures to monitor performance over time, e.g. from PT round to PT round. The procedures should allow participants to see the variability in their performance, whether there are general trends or inconsistencies, and where the performance varies randomly. This can be useful for those participants who are required to review their own PT performance as a condition for accreditation.

B.5.2.2

Graphical methods should be used to facilitate interpretation by a wider variety of readers. Traditional “Shewhart” control charts are useful, particularly for self-improvement purposes. Data listings and summary statistics allow more detailed review. Standardized performance scores used to evaluate performance, such as the z score, should be used for these graphs and tables. ISO 13528 presents additional examples and graphical tools.

B.5.2.3

Where a consensus standard deviation is used as the standard deviation for proficiency assessment, caution should be taken when monitoring performance over time, as the participant group can change and that can have unknown effects on the performance scores. It is also common for the interlaboratory standard deviation to decrease over time, as participants become familiar with the PT scheme, or as measurement or test methodology improves. This can cause an apparent increase in performance scores, even if the accuracy of a participant's results has not changed.

Bibliography