Annals of Occupational Hygiene Advance Access originally published online on August 14, 2008
Annals of Occupational Hygiene 2008 52(7):567-575; doi:10.1093/annhyg/men054
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Development and Evaluation of an Exposure Control Efficacy Library (ECEL)
Business Unit Quality & Safety, TNO Quality of Life, PO Box 360, 3700 AJ Zeist, The Netherlands
* Author to whom correspondence should be addressed. Tel: +31-30-694-4474; fax: +31-30-694-4070; e-mail: wouter.fransman{at}tno.nl
| ABSTRACT |
|---|
|
|
|---|
Objectives: This paper describes the development and evaluation of an evidence database on the effectiveness of risk management measures (RMMs) to control inhalation exposure. This database is referred to as Exposure Control Efficacy Library (ECEL).
Methods: A comprehensive review of scientific journals in the occupational hygiene field was undertaken. Efficacy values for RMMs in conjunction with contextual information on study design, sampling strategy and measurement type (among other parameters) were stored in an MS Access database. In total, 433 efficacy values for six RMM groups (i.e. enclosure, local exhaust ventilation, specialized ventilation, general ventilation, suppression techniques and separation of the worker) were collected from 90 peer-reviewed publications. These RMM categories were subdivided into more specific categories.
Results: Estimated average efficacy values ranged from 87% for specialized ventilation to 43% for general ventilation. Substantial variation in efficacy values was observed within RMM categories based on differences in selected covariables within each study (i.e. study design, sampling strategy, measurement type and others). More contrast in efficacy values was observed when evaluating more detailed subcategories.
Conclusions: It is envisaged that ECEL will contribute to exposure modelling, but should be supplemented with expert opinion, preferably in a formal expert elicitation procedure. The work presented here should be considered as a first attempt to collate and analyse RMM efficacy values and inclusion of additional (unpublished) exposure data is highly warranted.
Keywords: control effectiveness control measure efficacy data intervention occupational exposure risk management measures RMM library
| INTRODUCTION |
|---|
|
|
|---|
Risk management measures (RMMs) represent a central and integral part of the exposure scenario, and the European Registration, Evaluation, Authorization and restriction of Chemicals (REACH) regulation thus provides a strong incentive to systematically evaluate the efficacy of RMMs under real-world conditions (European Union, 2007). RMMs focus on the reduction or avoidance of exposure in the various workplace compartments to a substance and may represent a wide range of measures. Currently, professionals in occupational hygiene generally resort to subjective judgement in the process of assigning efficacy values to various types of RMMs. In order to make this process more transparent, judgements should (where possible) be based on and underpinned by empirical quantitative evidence from scientific resources (Ellenbecker, 1996).
Reviews on intervention effectiveness research conducted primarily from the 1980s through the mid-1990s (Goldenhar and Schulte, 1994, 1996; Lazovich et al., 2002) concluded that occupational health and safety studies were more likely to focus on workers knowledge and behaviour than on engineering improvements in the workplace. Yet, over the years, exposure assessment studies have also documented an array of exposure determinants including RMMs (Burstyn and Teschke, 1999; Ogden, 2006). Specific reviews were also published on the evaluation and optimization of dust control systems (Smandych et al., 1998). In addition, some researchers investigated long-term exposure trends and determined the relationship between historic decline in mean exposure levels and factors like elimination and engineering measures (Kromhout and Vermeulen, 2000; Vermeulen et al., 2000; Creely et al., 2007). Hence, empirical information on efficacy of RMMs does exist; the problem is that this information is scattered and few systematic attempts have been made to collate and analyse this information (Verbeek et al., 2005). This motivated a literature search and structured quantitative evaluation of the efficacy of RMMs to support evidence-based exposure assessments.
This paper describes the analyses of an evidence database derived from the scientific literature on the effectiveness of RMMs to control inhalation exposure. The database focuses on RMMs that reduce potential exposure in the breathing zone of the worker and therefore the efficacy of personal protective equipment (PPE) is not included. The database is named Exposure Control Efficacy Library (ECEL). At a very technical detailed level, RMMs are often specific to a machine or installation, adapted to local circumstances and determined in part by organizational factors or behaviour of the worker (Buringh et al., 1992; Leviton and Sheehy, 1996; Brosseau et al., 2002). At a more abstract level, however, study results are applicable in general terms and therefore the data presented here may contribute to exposure modelling and implementation of RMMs in exposure scenarios (e.g. in the context of REACH). We consider ECEL in its current form as a first step towards an evidence base for RMMs. The analyses should be expanded and should include unpublished exposure measurement information to provide a more comprehensive picture on RMMs efficacies in the near future.
| METHODS |
|---|
|
|
|---|
Compilation of ECEL
To compile an ECEL, a review of journals in the occupational hygiene field was undertaken. A computer search using PubMed was conducted to locate peer-reviewed publications that quantitatively reported results on the efficacy of RMMs. Because this PubMed search with specific keywords could potentially overlook relevant publications, several available journals in the research field of occupational hygiene from January 2000 to December 2007 were searched manually: Annals of Occupational Hygiene, American Industrial Hygiene Association Journal, Applied Occupational and Environmental Hygiene and Journal of Occupational and Environmental Hygiene. The efficacy library was complemented with cross references from the available published papers.
Published literature gathered in this manner was subsequently subjected to a screening procedure to select only suitable data, based on several criteria. In order to be included in ECEL: (i) efficacy of RMMs had to be based on quantitative measurement data and appropriate descriptive statistics had to be available in the report. (ii) Efficacy of RMMs had to be determined either in terms of pre- and post-comparisons (intervention studies or experimental studies) or as a cross-sectional comparison of situations with and without RMM. (iii) The efficacy value presented in the paper had to be based on only one RMM category. Efficacy values that were the result of the combined effect of multiple RMM categories (for which it was not possible to separate the effect of each of the RMMs) were excluded from the database. (iv) Suitable data were represented by studies where a quantitative factor could be distilled that represented the reduction effect of a particular RMM (i.e. efficacy value). If either the mean exposure in the situation with RMM or in the situation without RMM was below the limit of detection, the record was excluded from the ECEL database.
Information available in ECEL
The efficacy value was expressed as a proportional reduction compared to baseline, e.g. from 10 mg m–3 (without RMM) to 1 mg m–3 (with RMM) results in an efficacy of 90%. Sometimes, this efficacy value was directly adopted from the study in question, whereas in other cases such a factor was derived from a percentage reduction, a β-coefficient derived from regression models or a simple descriptive comparative analysis. In case multiple descriptive statistics were calculated, preference was given to use the efficacy value based on comparing geometric mean values, followed by median and arithmetic mean values. Note that in some instances, records may be assigned with a negative efficacy value (<0%) where a counter-effective value was found and therefore suggesting an increase in exposure due to the implementation of a RMM. A negative efficacy value could be ascribed to poor work practices [e.g. a worker positioned between the source and local exhaust ventilation (LEV)] or may theoretically reflect bias in the respective study. These negative efficacy values were not excluded from the analyses.
Each entry in the database represents an efficacy value of a specific RMM during a given workplace scenario. This means that multiple efficacy values can be derived from one publication if data were available for different RMM categories, different tasks/activities or different substance exposures. In cases where multiple substances (e.g. toluene and total volatile organic compounds) were determined in the same measurement sample, the derived efficacy values were treated as correlated data. Each study was recorded by an occupational hygienist and reviewed by a second occupational hygienist. In cases of disagreement, the disputed publication was discussed until a consensus was reached.
In ECEL, only data were included that were derived from an RMM-absent versus RMM-present situation, thus where the effect of a RMM after its implementation was investigated. In some cases, however, the baseline situation or pre-test conditions were strictly speaking not a zero situation but rather a partially implemented RMM, e.g. conditions with the old LEV system. These records were referred to and labelled in ECEL as optimization data and were excluded from the analyses presented in this paper.
RMMs included in ECEL
Information was collated on seven broad a priori defined RMM categories: (i) enclosure, (ii) LEV, (iii) specialized ventilation systems, (iv) general ventilation, (v) suppression techniques, (vi) segregation of sources and (vii) separation of the worker. These RMM categories were divided into subcategories using the contextual information available (as entered into ECEL) and based on the distinctiveness within each RMM category (see Appendix). A RMM subcategory typically distinguishes between the degree or level of its application (e.g. partial/complete), its integration into a system (e.g. integrated/exterior) or the situation targeted (e.g. a room/a worker or dust source/dust plume). This categorization is a first step only towards a more structured approach to study groups of RMMs at different levels of detail.
Other possible RMM categories such as substitution of chemicals, product modification and elimination of sources were considered to be too heterogeneous across studies and were therefore not included in ECEL.
Study characteristics
Important covariables that were believed to influence the efficacy values were identified for each individual record, i.e. (i) study design, (ii) sampling strategy and (iii) measurement type. A wide range of different types of studies was found with diverse scopes and aims. Three main study designs were distinguished, i.e. (i) cross-sectional exposure field studies—examine exposure measurements collected during a single period of time and compare situations with and without RMM, (ii) experimental studies—are conducted in a laboratory, simulated workplace or controlled environment in order to examine the effects of RMMs on exposure, source emissions or workroom concentrations and (iii) intervention field studies—represent studies in the workplace using a pre-test versus post-test assessment procedure to evaluate intervention effectiveness (e.g. on/off LEV systems).
With the parameter sampling strategy, a distinction is made between task-based/short-term and shift-based measurements. This distinction is important as shift-based measurements may cover different work situations and types of RMMs, potentially diluting the specific effect of a particular RMM under study.
The measurement type (i.e. personal and static) signifies the relevance of the data for the actual worker exposure. This is clearly an important parameter since personal samples are assumed to be more representative for exposure assessment than stationary samples (Cherrie, 2003).
Statistical analyses
The parameters described above were stored in a relational database in Microsoft Access 2002. Statistical analyses were performed with SAS, version 9 (SAS Statistical Software, SAS Institute, Cary, NC, USA). Descriptive statistics with respect to efficacy values were calculated for the different RMM categories, both on an aggregate level (RMM categories) and specific sub-level (RMM subcategories). In addition, efficacy values related to the various RMMs are presented in box plots. The effects of covariables (i.e. study design, sampling strategy and measurement type) on the efficacy value were evaluated using mixed-effects regression models. Because the efficacy values described a log normal rather than a normal distribution, naturally log-transformed efficacy values were used in the mixed-effects models. Since multiple substances analysed in one measurement sample may not be completely independent, sampling ID was included as random effect in the mixed-effects regression models. Because there did not seem to be a statistically significant difference (P < 0.05) in efficacy values between inhalable dust and vapour exposure in the ECEL database, an overall statistical analysis was performed.
| RESULTS |
|---|
|
|
|---|
In total, 433 records derived from 90 publications met the criteria for inclusion in the ECEL database (Table 1). Studies with an intervention study design were the most prevalent with 177 records and the majority of studies described an investigation of LEV (280 records) as RMM category. There were no values available for one of the a priori defined RMM categories: segregation of sources (see Appendix).
|
Table 2 shows results of the univariate analysis performed on all RMM categories using mixed-effects regression models to explore covariables (e.g. study design, sampling strategy and measurement type) with a random effect of sampling ID. The estimated efficacy values appeared to vary across different covariables, as expected. The estimated average efficacy value for cross-sectional studies (efficacy = 63%) appeared to be low compared with an experimental (efficacy = 88%) or intervention study design (efficacy = 80%) (statistically significant; P < 0.05). A sampling strategy using task-based measurements resulted in a statistically significant (P < 0.05) higher efficacy of RMMs (efficacy = 84%) compared with shift-based measurements (efficacy = 65%). Studies that collected personal samples reported on average a similar efficacy (efficacy = 78%) as studies that collected stationary samples (efficacy = 80%).
|
The variability in efficacy values within RMM (sub)categories is substantial. It should be noted, however, that the total number of observations for each subcategory is small. An illustration of the distribution of (unadjusted) efficacy values within and between RMM categories is given in Fig. 1. The box plots presented in Fig. 2 provide a more detailed account of the efficacy data of RMM subcategories. Table 3 presents the estimated efficacy values (with a random effect of sampling ID) associated with each RMM category and the 95% confidence intervals. The values indicate a pronounced efficacy for specialized ventilation (efficacy = 87%), separation of workers (efficacy = 87%), suppression techniques (efficacy = 83%) and LEV (efficacy = 82%). Enclosure (efficacy = 50%) and general ventilation (efficacy = 43%) produced the lowest efficacy compared to other RMM categories. Pronounced differences within RMM categories were found for most of the RMMs (Table 3).
|
|
|
Since study design and sampling strategy appeared to be important covariables (Table 2), the estimated efficacy values for RMM categories would have been influenced by the selected study design and sampling strategy in each of the studies. However, for most of the RMM categories, the available data were too limited (or not all of the study designs and sampling strategies were represented in the available data) to properly adjust for these covariables in the mixed regression models. For the two most frequently entered RMM categories in the ECEL database (i.e. LEV and suppression techniques), the efficacy values were adjusted for study design (intervention studies as reference) and sampling strategy (task-based/short-term measurements as reference) and resulted in adjusted efficacy values of 84% (95% confidence interval = 79–82%) for LEV and 84% (95% confidence interval = 77–89%) for suppression techniques.
| DISCUSSION |
|---|
|
|
|---|
This paper describes the development and evaluation of the ECEL database on an aggregated level for RMMs and to some extent on specific RMM subclasses. Unfortunately, only a limited number of records were available for specific RMMs. Nevertheless, the ECEL database shows promise as a tool for assigning efficacy values to a range of RMMs. In addition, the ECEL database may be searched and analysed for specific RMMs in order to underpin a particular exposure assessment process.
Notwithstanding the resulting small number of records, this ECEL database will improve the subjective judgement in the process of assigning an efficacy value for a specific RMM. It is felt that this paper is a proper reflection of the current state of empirical knowledge on RMMs as described in the published scientific literature. Although the call for proper documentation of RMMs is not new (Goldenhar et al., 2001; Swuste et al., 2003), we are not aware of other peer-reviewed published papers quantitatively describing the efficacy of RMMs based on a comprehensive review of the available occupational hygiene literature. For a more comprehensive and weight of the evidence approach, also other published sources with information on the efficacy of RMMs (like guidance documents and book chapters) and unpublished exposure measurement information available in the occupational hygiene field should be analysed and included in ECEL. This would allow for a proper derivation of efficacy values for specific control measures (e.g. fume cupboards).
Although our findings indicate that there is no need for pessimism regarding the evidence base on RMMs, it is also clear from the results that the type of study design and quality of evidence is heterogeneous and often substandard. Only a limited number of data are available for several RMMs. For example, no data were obtainable for the RMM segregation of sources and limited data were available for enclosure, specialized ventilation and separation of workers. These numbers were further reduced if one defines RMMs in subcategories. These shortcomings may be due to the challenges associated with conducting rigorous research of intervention effectiveness, which can generally be described as complicated (Goldenhar et al., 2001). This deficiency clearly impedes the interpretation of data and the reliability of the estimated RMM efficacy values. The limited number of data in this study also results in very limited number of data in some of the RMM strata. Although, overall the efficacy values appear to be log normally distributed, within each RMM sub(category), the data could be normally or inversely log normally distributed. It is not clear whether the collation of more efficacy data would narrow the range in efficacy values within each RMM (sub)category. However, when more data become available, a more detailed statistical evaluation can be performed, which will increase the insight in the parameters that determine the efficacy of RMMs. A more detailed regression analysis (adjustment for important covariables) of the RMMs is therefore currently limited to LEV and suppression techniques. The exercise presented here should be viewed as a first attempt to explore the possibilities of deriving efficacy values on an aggregate RMM level. Since the effect of the covariables on the efficacy of LEV and suppression techniques was significant, the efficacy of other RMMs could also be more accurately reported (adjusted for these covariables) when more data become available. For now, one has to resort to the information available and where appropriate apply it with the necessary expert judgement.
The efficacy values presented in this paper might be an overestimation of the effect of RMMs due to publication bias. This bias might have influenced the results presented in this paper if the effectiveness of RMMs was more likely to be presented/published in case the RMM was found to be effective rather than if the RMM appeared to be ineffective to reduce exposure levels.
RMMs are generally classified according to the so-called occupational hygiene hierarchy (Buringh et al., 1992). This study covers RMMs across the first three levels of the hierarchy. However, it should be noted that several important RMMs that represent an intrinsic change in the work process or task are not taken into account in this review [e.g. organizational, operational, behavioural and product-integrated measures (like substitution or pelletization of powders)]. These interventions are very difficult to quantify in generic terms. The efficacy of PPE was beyond the scope of this study and is described in detail in other papers (Brouwer et al., 2001).
Univariate analyses using mixed-effect models indicate the significance of various covariables. The difference in average efficacy values for cross-sectional studies, experimental and intervention studies is apparent as one may expect higher efficacies under controlled experimental or a priori defined conditions. In addition, cross-sectional studies are more vulnerable for (residual) confounding due to other unpredictable workplace factors and background exposures not related to the RMMs under study. It is also notable that a pronounced difference exists in the types of sampling strategy; shift-based data reveal an efficacy of 65%, whereas a higher efficacy is associated with task-based/short-term data (i.e. 84%). Shift-based data often include exposures at more than one source and location and it is therefore less specific to study the effect of a selected RMM. Efficacy values are therefore more likely to be diluted when using shift-based measurements. Task-based/short-term data focus on a task or activity performed at a specific location and within an often shorter time frame, and it is (assumed) to produce a more representative and often higher efficacy of RMMs, which is supported by the results presented in this paper. The preferred study design to evaluate RMMs efficacy would be intervention studies; these studies are specifically designed to examine the effect of implementing a specific RMM in the workplace. Likewise, a task-based/short-term sampling strategy is expected to reflect a more representative efficacy of a specific RMM. We therefore adjusted for these two parameters (for RMMs with sufficient data) and used intervention study and task-based/short-term sampling as reference categories. On the other hand, if one is more interested in the efficacy of a specific RMM on a workers 8-h time-weighted average exposure, shift-based sampling in a cross-sectional study design might better represent the situation. Thus, based on the purpose of the exposure assessment, different publications and efficacy values may be selected from the ECEL database.
Differences between estimated efficacy values of the six RMM categories were evident, with general ventilation and (partial) enclosure showing a considerably lower efficacy compared to the other RMMs. Conversely, a higher efficacy was estimated for specialized ventilation and separation of workers. An interesting outcome is that LEV and suppression techniques, on an aggregate level, produce similar (adjusted) efficacy estimations. It should be noted that there is substantial variation in efficacy values within a particular RMM category, as indicated in Figs 1 and 2. This variation can be ascribed to the great variety in control measures within a RMM (sub)category. For instance, complete separation of workers may in our analyses cover cabins with or without fresh air supply. In addition, the selection of covariables in different studies (i.e. study design, sampling strategy and measurement type), the uncontrollable environmental conditions (like atmospheric differences in pre- and post-situations) and operational conditions (differences in work practices, etc.) may be responsible for heterogeneity within categories.
The work presented here should be considered as a first attempt to collate and analyse RMM efficacies and is a mere stepping stone for further developments. One may therefore envisage a number of future challenges in the field of RMM libraries. First, the relatively small number of data does not allow for detailed analysis as yet. As a result, an in-depth analysis of substance-specific exposures has not been performed and it is still unknown if the efficacies presented here can be extrapolated for different exposures and if it is truly substance independent. A supporting and encouraging result from this evaluation does, however, indicate similar efficacies for dusts, fumes, vapours and mists. Second, it is unclear whether data can be extrapolated across different types of activities and processes. Some RMMs may also be very process specific and cannot be subjected to analysis on an aggregate level. Third, further research is required to determine the effect of optimization data where a RMM is optimized (i.e. improved), as opposed to a newly implemented RMM. Fourth, more data will provide the opportunity to expand the evaluation on a more detailed RMM level and possibly improve on the current RMM categorization. As an important first step, unpublished data should be collated. In the future, the conduct of additional intervention studies using task-based measurements is to be preferred to obtain insight into efficacy of specific RMMs.
In conclusion, the current evidence may give rise to some kind of base estimate of efficacy for specific RMMs. The empirical values may be used to underpin new research activities in the field of exposure modelling (Tielemans et al., 2007). Unfortunately, the current evidence base is not complete and additional studies should be included and results should be supplemented with expert opinion, preferably in a formal expert elicitation procedure. Future research that focuses on the efficacy of RMMs should be devoted to proper study designs, statistical analyses and descriptive data of the workplace scenario (e.g. the RMM, task/activity, process and substance).
| FUNDING |
|---|
|
|
|---|
Dutch Ministry of Social Affairs and Employment.
| APPENDIX |
|---|
|
|
|---|
|
Received April 21, 2008; in final form July 4, 2008
| REFERENCES |
|---|
|
|
|---|
Brosseau LM, Parker DL, Lazovich D, et al. Designing intervention effectiveness studies for occupational health and safety: The Minnesota Wood Dust Study. Am J Ind Med (2002) 41:54–61.[CrossRef][Web of Science][Medline]
Brouwer DH, Marquart H, van Hemmen JJ. Proposal for an approach with default values for the protection offered by PPE, under European new or existing substance regulations. Ann Occup Hyg (2001) 45:543–53.
Buringh E, Noy D, Pouwels H, et al. A systematic implementation of control measures for airborne contaminants in workplace air. Staub-Reinhaltung der Luft (1992) 52:347–51.
Burstyn I, Teschke K. Studying the determinants of exposure: a review of methods. Am Ind Hyg Assoc J (1999) 60:57–72.[Web of Science][Medline]
Cherrie JW. Commentary: the beginning of the science underpinning occupational hygiene. Ann Occup Hyg (2003) 47:179–85.
Creely KS, Cowie H, van Tongeren MJ, et al. Trends in inhalation exposure—a review of the data in the published scientific literature. Ann Occup Hyg (2007) 51:665–78.
Ellenbecker MJ. Engineering controls as an intervention to reduce worker exposure. Am J Ind Med (1996) 29:303–7.[CrossRef][Web of Science][Medline]
European Union. Regulation (EC) No 1907/2006 of the European Parliament and of the Council of 18 December 2006 concerning the Registration, Evaluation, Authorisation and Restriction of Chemicals (REACH). Off J Eur Union (2007) L 136:3–279. Available at http://www.reach-compliance.eu/english/legislation/docs/launchers/launch-2006-1907-EC.html. Accessed 21 July 2008.
Goldenhar LM, Schulte PA. Intervention research in occupational health and safety. J Occup Med (1994) 36:763–75.[Web of Science][Medline]
Goldenhar LM, Schulte PA. Methodological issues for intervention research in occupational health and safety. Am J Ind Med (1996) 29:289–94.[CrossRef][Web of Science][Medline]
Goldenhar LM, LaMontagne AD, Katz T, et al. The intervention research process in occupational safety and health: an overview from the National Occupational Research Agenda Intervention Effectiveness Research team. J Occup Environ Med (2001) 43:616–22.[Web of Science][Medline]
Kromhout H, Vermeulen R. Long-term trends in occupational exposure: are they real? What causes them? What shall we do with them? Ann Occup Hyg (2000) 44:325–7.
Lazovich D, Parker DL, Brosseau LM, et al. Effectiveness of a worksite intervention to reduce an occupational exposure: the Minnesota wood dust study. Am J Public Health (2002) 92:1498–505.
Leviton LC, Sheehy JW. Encouraging small businesses to adopt effective technologies to prevent exposure to health hazards. Am J Ind Med (1996) 29:409–11.[CrossRef][Web of Science][Medline]
Ogden T. Annals of Occupational Hygiene at volume 50: many achievements, a few mistakes, and an interesting future. Ann Occup Hyg (2006) 50:751–64.
Smandych RS, Thomson M, Goodfellow H. Dust control for material handling operations: a systematic approach. Am Ind Hyg Assoc J (1998) 59:139–46.[Web of Science][Medline]
Swuste P, Hale A, Pantry S. Solbase: a databank of solutions for occupational hazards and risks. Ann Occup Hyg (2003) 47:541–7.
Tielemans E, Warren N, Schneider T, et al. Tools for regulatory assessment of occupational exposure: development and challenges. J Expo Sci Environ Epidemiol (2007) 17(Suppl. 1):S72–80.[CrossRef][Web of Science][Medline]
Verbeek J, Salmi J, Pasternack I, et al. A search strategy for occupational health intervention studies. Occup Environ Med (2005) 62:682–7.
Vermeulen R, de Hartog J, Swuste P, et al. Trends in exposure to inhalable particulate and dermal contamination in the rubber manufacturing industry: effectiveness of control measures implemented over a nine-year period. Ann Occup Hyg (2000) 44:343–54.
This article has been cited by other articles:
![]() |
T. Ogden REACH--how is it going? Ann. Hyg., January 1, 2010; 54(1): 1 - 4. [Full Text] [PDF] |
||||
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||


