Evaluation of Professional Capacity Building

Capacity Building Programmes at hospital

The programme and beneficiaries

The Capacity Building programme at hospital was conducted by the Chinese University of Hong Kong (CUHK) Jockey Club Institute of Ageing (IoA) from 2015 to 2018 with a broad range of beneficiaries. The programme covered a wide variety of topics such as communication skills in serious illness conversation, symptom management in EoLC, etc. Different materials were used in education and promotion, including videos, booklets, e-cases and more.

During the period, the Institute delivered 293 educational programmes comprising 175 capacity building programmes for healthcare professionals in hospitals and RCHEs in New Territories East cluster (NTEC), 58 workshops and seminars for specific target groups, 18 public forums for stakeholders and 42 information sessions for patients and family members.

A total of 10,310 participants were benefited from the educational programmes including 728 doctors, 2,707 nurses, 2,155 healthcare staff, 539 medical students, 1,562 other specific target groups, 1,642 general public and 977 patients and family members. In this section, only the outcomes on professionals are presented. For full details of the evaluation report on the programmes in hospitals, please visit here.

Methodology

A couple of evaluation studies were carried out, with each being conducted on a type of target participant. To examine if the educational programme was able to enhance health and social care workers’ self-competence in death work (by self-competence in death work scale [SC-DWS]), the CUHK distributed questionnaires to health and social care workers in RCHEs and seven hospitals in the NTEC from February 2018 to June 2018. Participants were conveniently sampled in talks, seminars, and workshops in the project. Questionnaire were also sent to the institutions by post, and an online questionnaire was provided as an alternative. For doctors who had attended communication skills training and nurses who had attended courses held by the Institute, questionnaires were distributed in training sessions to collect their quantitative responses and qualitative feedback on the training course. Regarding residential care home staff, from February to March 2018, a questionnaire survey was conducted in 5 RCHE to evaluate the effectiveness of the programme. RCHE staff who attended the training sessions of CUHK were invited to fill in a questionnaire.

Achievements

Health and social care workers

The survey on 614 health and social care workers in 2018 showed that respondents who received more training provided by the Institute were more self-competent in death work, as evidenced by higher scores on the SC-DWS and its two subscales when compared to those who attended none. (Refer to the table below)

The difference in self-competence of attendees and non-attendees varied according to the type of training received (refer to the table below) with the largest benefit observed in training in symptom control (difference in means 3.19), followed by the sessions on introduction to EOL care (difference in mean 2.34)

Doctors

For doctors’ communication skills training, twenty post-training questionnaires were collected from doctors in four hospitals right after the training. Percentage increases in their self-perceived preparedness in communication skills ranged from 24.6% to 40.3% (mean= 33.1%) after training when compared to before training. The largest improvement was found in the preparedness to explore goals of care with seriously ill patients or their families. Participants rated the importance of the training to the development of their clinical skills at a mean of 4.35, on a scale from 1 (not at all important) to 5 (very important). The majority (94.7%) would recommend the training to other physicians in their specialty and 89.5% agreed that all higher physician trainees in their specialty should be required to attend the training. Below are some of the direct feedbacks from the participants after attending training program:

“The training program teach us how to facilitate communication and reduce conflicts between relatives and healthcare providers.”

“(I) had benefited immensely especially in the role play and discussion.”

Hospital Nurse

Over 98% of participants indicated that they will recommend similar activities to their colleagues/friends. Qualitative feedback was also obtained on the questionnaire. Respondents mentioned merits of the training. (Please see the example below)

“The topic and content are novel. The content is profound but easy to understand. The situation of end-of-life care in Hong Kong was introduced, improving my understanding of end-of-life care in Hong Kong was introduced, improving my understanding of what happens when one faces death.” (Acute Hospital C, A Better End-of-Life Care series: An Insight into Dying in Hong Kong)

“The content is rich. The information is useful and relevant to clinical work.” (Acute Hospital B, Symptom Management in End-of-Life Care)

Residential care home staff

66 staff (more than half were nurses or social workers) in RCHEs who had attended the training sessions had completed the post-training questionnaires. It was found that respondents participating in more training sessions organised by the institute had better knowledge of EoL care as manifested by a higher mean score on 10-item test (mean = 8.77 for 3 or more sessions; mean = 8.65 for two sessions; mean = 7.86 for one session). Below are some of the direct feedbacks from the residential care home staff after attending the programmes.

 “Their [Residential care home staff’s] awareness [of end-of-life care] has really improved. They refer [patients] to us. Tell us the condition of some older adults. They observe and invite us to follow up. These are very helpful to us. At least, we can intervene in a timely manner, which is very important” (Social worker, attend 23 training sessions).

“[I] gain a better understanding of how to tell family member the end-of life client’s condition, and express more accurately. The accuracy id higher. Family members understand better. And for symptom control, I think[I] have a better understanding, compared to when I was on board” (Registered nurse, attended 11 training session.)

Capacity Building in RCHEs

Beneficiaries

Between 2016 and 2018, the HKAG has implemented EoLC programme in 36 RCHEs. The HKAG EoL team has delivered training and talks to benefit more than 2,000 managerial staff, professional staff and frontline staff in Residential Care Homes for the Elderly (RCHEs). Apart from lectures, the HKAG EoL team also provided professional consultations, on-site support session and coaching sessions to RCHE staff.

Achievements

The outcomes of the Capacity Building in RCHEs were evaluated by benchmarking of provision of palliative and EoLC in RCHEs, and evaluation on the EoLC knowledge and readiness in EoLC provision among trained RCHEs staff.

Regarding benchmarking, RCHEs were assessed using a standardised 30-item assessment tool developed by HKAG which rated the performance of RCHEs regarding the provision of palliative care, EoL care, and postmortem care. Each item was rated by assessor with scale ranged from 0, 1a, 1b, 2, to N/A, with higher scores represent better performance. RCHEs were assessed for three times: at baseline before the programme, one year and two years after joining the programme. Assessed RCHEs (n=11) showed that the overall percentage of items rated “2” (Had both written guidelines/policies/procedures/mechanisms and execution and documentation) greatly increased from 46.4% at baseline to 82.1% after a year, and maintained at 83.3% after two years in the programme, representing an overall increase of 36.9%.

Staff in the participating RCHEs were assessed their readiness to provide EoLC with a standardised questionnaire which comprises three aspects, namely willingness, capability and resilience. Staff were assessed at the first activity of the programme as baseline, as well as one and two years after joining the programme for comparison. Altogether 209 RCHE staff (76 professional staff and 128 frontline staff) completed all three times of assessments. They reported significant improvements in all three aspects at both 1-year assessment and 2-year assessment (p<.01 to p<.001). After 2 years in the programme, staff reported overall improvements in their level of willingness, competence and resilience by 7.9%, 12.2% and 5.1% respectively.

Professional capacity building programme in the community

The programme and beneficiaries

Over the 3-year project period, the JCECC project provides a series of professional capacity building programme on end-of-life care (EoLC) to professionals and caregivers aiming to enhance their competency to provide more comprehensive and humane EoLC. The programme consisted of two core levels of education, namely the leadership training programme targeting on middle management staff in leading NGOs which intended to develop EoLC services in the community, and professional training workshops for health and social care professionals in the community. Both programmes were developed based on the 7-domain EoLC competency framework.

Methodology

Pre-post or pre-post-followup evaluation was conducted among participants of the training courses to assess the direct outcomes of the professional capacity building programme. A measurement on level of EoLC competence was developed in line with the 7-domain mutli-dimensional EoLC Competence framework to assess changes in participants’ confidence in providing EoLC. Participants’ competences were assessed on a 10-point Likert scale (1=totally incompetent; 10=very competent).

On the other hand, we hope that the professional training programmes launched by various partners in the JCECC project not only bring direct benefits to the programme participants, but might also bring along a ripple effect – programme participants might influence their colleagues with their acquired knowledge and skills in EoLC, leading to positive shifts in the culture of the field of EoLC as a whole. Thus, a three-time-point online survey on healthcare and social care professionals was conducted to evaluate the long-term impact of the professional capacity building programme on health and social care professionals in Hong Kong. The first survey was held between April-June 2016, the second between April-June 2017, and the third wave was launched in April 2018 and data collection was completed in early July 2018. Health and social care professionals, particularly physicians, nurses, and social workers, were recruited mainly through relevant professional bodies and academic and research institutions. In each year, around 500 professionals were recruited.

Achievements

Significant improvements in EoLC competences were reported among training participants:

• Participants of professional training workshops (N=894) reported 23.4% increase in overall EoLC Competence. The improvements in all evaluated EoLC competence domains were all statistically significant; 966 participants responded to the post-programme satisfaction survey; 85.8% agreed to strongly agreed that they were satisfied with the workshops

• After completing a one-year leadership programme, leaders (n=32) reported 31.6% increase in overall EoLC competence, 47.5% improvements in leadership skills. All reported changes were statistically significant. Leaders’ enhanced level of EoLC competences and leadership skills were retained even at 6 months after completion of the programme.

Emerging impacts on the health and social care professionals in the community:

• Results from the professional survey showed that Health and social care professionals in community reported 3.1% increase in overall EoLC competence level between 2016 and 2018; the competence level on symptom management has been steadily increasing from 2016 to 2018, while the competence levels of EoL decision making and community EoL support both shared the trend of increase from the first wave and then levelled off from 2017 to 2018.

• For those working in EoLC, competence in EoLC were positively and significantly correlated with job satisfaction and meaningfulness (mild to moderate correlation), but negatively correlated to job stress. This may imply that enhancing EoLC competences among health and social care professionals in EoLC might be able to improve their job satisfaction and meaningfulness while lower job stress, resulting in a more competent and stable workforce in the field of EoLC

• Among 299 surveyed professionals in the 2018 wave who have heard of the JCECC Project, 78% were highly satisfied with the JCECC Project as a whole; 80% and 73% strongly agreed that the Project has enhanced professional capacities in providing EoLC, and that the Project has raised public awareness on EoLC respectively

Learnings in phase I and way forward in phase II

• The Capacity Building programme at hospital was found to exert positive impacts on a board range of professionals on various levels – attitude, knowledge and skill relating to EoLC. Health and social care staff who received more training provided by the institute were more self- competent in death work. After doctors’ skill training, participants had higher self-perceived preparedness in communication skills. Nurse considered the training informative and practical, reported enhancement of knowledge of EOL care, valued experience sharing, and favoured demonstration of skills and the use of videos. Residential care home staff appreciated the opportunity to learn across disciplines, considered issues of dying and death less of a taboo, and reported better communication with colleagues, patients and families. With the work accomplished and experience gathered during these three years of capacity building, the Project team gained valuable insights to guide future strategies.
• The Capacity Building programme in RCHEs was found to better prepare RCHEs for providing EoLC to their residents. Particularly, the training provided by HKAG EoL team was well received by RCHEs, and helped improve the competence, willingness, and resilience of RCHEs staff to provide EoLC.
• The Capacity Building programme in community, encompassing the leadership programmes and professional training workshops were effective respective to their objectives. Leaders showed significant improvements not only on all EoLC competences but also leadership skills necessary in driving future development in EoLC. The enhanced competences maintained for 6 months and longer upon completion of programme. Participants of professional training workshops consistently reported significant improvements in self-perceived EoLC competences after training (23.4% increase in competences). These results also suggested that the development of the professional training programme based on the 7-domain competence framework was on the right track, and the competence framework also helped identify training needs among health and social care professionals in the community.
• The rippling effects on EoLC competence has been emerging as seen from our professional survey (a 3.1% increase in competences between 2016 and 2018).One possible explanation for the observed varied levels of effects on different competence domains is that the rippling effects have been more effective on domains that were new to many professionals. Indeed, the three domains which showed significant improvements (symptom management, EoL decision making and community EoL support) were domains which scored the lowest at baseline (2016).
• Healthcare and social care workers may encounter end of life and death-related issues in various service settings. Thus, a basic level of knowledge and skills in EoLC should be delivered to virtually all healthcare and social care professionals, regardless of settings. In the phase I project, participants in the Capacity Building programmes in hospital, RCHEs, as well as community all reported promising gains regarding EoLC competences. However, the effect of the Capcity Building programmes on non-participants was relatively weak when compared to the direct training effect, and the impact was even weaker for competence domains of which professionals already had some basic understanding. In phase II project, not only direct training programmes in hospital, RCHEs, and the community should be further strengthened, multimodal programmes should also be developed in order to reach out to more professionals with the help of online learning, as well as cater for the need for more advanced skills through face-to-face workshops.