[ad_1]
Only seven positive responses were obtained, and all were included in the analysis of the study. Fortunately, these were distributed between the different years, three from the first year, two from the second year and two from the third year and included both males and females. The last interview did not add any new information despite adding extra questions to clarify the issues from the previous interviews, which might indicate data saturation [19].
Nine codes emerged from the data in respect to participants’ perceptions of portfolio assessment in the SDFM program.
Portfolio assessment was a challenging experience for most of the SDFM participants. All of them faced many difficulties, particularly at the beginning of the process, which they attributed to an incomplete understanding of the assessment. They all agreed that portfolio assessment was a useful and beneficial experience, but they linked its effectiveness to their understanding of its definition, objectives, background and the process. Specific comments of participants are included as italicised insets.
The biggest challenge was at the beginning, as the portfolio wasn’t fairly explained and we didn’t comprehend its meaning so we were worried we might never get it. However, as time passed and with the mentors’ explanation, we could finally get it.
At the SDFM, there was an introductory lecture about portfolio assessment but according to some participants, it was not effective in clarifying the confusion. They believed that this could have been due to the fact that they were new to the experience of portfolio assessment, the timing of the lecture was wrong, its structure was poor, or it might just be necessary to provide further clarification or discussion about portfolio assessment.
Different understandings of portfolio assessment in SDFM were noted mainly regarding the reflection aspect. Participant understanding of the portfolio contents differed and this was a concern for most of the participants, until they practised using the portfolio and received guidance from their mentors. After that, the participants came to appreciate the effectiveness of the portfolio once they were able to understand it, which was achieved later in the program.
Different understandings of portfolio assessment among various trainers were identified by all the participants. This confusion was one of the factors that led to misunderstandings of the assessments.
Actually the experience of the portfolio is new. Even after it was explained to us, the requirements were still different for mentors. There was no conformity in the picture; it was really confusing and unclear.
Different understandings were greater among the clinical trainers and consultants in other specialties but were also clearly mentioned among family medicine trainers and mentors. The assessment tools were used in different ways by mentors after a period in the program which may have added to the confusion among the mentors, too.
After I organised it a certain way, my mentor asked me to change it after 3 or 4 rotations, so I had to exert extra effort and time.
According to some participants there was an introductory lecture about portfolio assessment which was conducted for the trainees; however only a few trainers attended, unfortunately.
The type of portfolio assessment and whether it was formative or summative was not clear for some of the participants until late in the program.
I am frankly shocked that the portfolio evaluation is not summative because I expected it to be evaluated by marks. I’m sure if it had such marks, it would be great because it contains a lot of effort in writing and doing what is supposed to be done.
Some participants claim the type of formative assessment as the main reason for trainee dissatisfaction with portfolio assessment in the SDFM program.
There was dissatisfaction because of the quantity of papers and because it was new and not credited so it was like an extra effort with no pay back or return.
Trainee perceptions of the effectiveness of the assessments varied widely. Most participants were supportive of a summative evaluation, while some would prefer a mixture of summative and formative while one participant would prefer formative assessment alone. Some of the participants considered portfolio assessment to be a fair assessment method because it was longitudinal and measured cumulative student performance, unlike other methods, which depend on knowledge at a particular time under specific conditions, such as multiple-choice questions or an objective structured clinical examination.
One important perception that all participants shared was the workload and time demands of portfolio assessment. Stress accompanied the workload of most of the participants. Some participants were overcome with physical stress, some with psychological stress and a minority reported financial stress. Paperwork and time barriers were the main reasons for trainee dissatisfaction. All the participants agreed that completing the portfolio was a time-consuming task: selecting the cases, writing them up, editing and organising the portfolio. One participant mentioned the economic aspect of the portfolio, but this was not a concern for the other participants.
…frankly there are a lot of papers I need to fill at the end of each rotation, so it obstructed me and took a lot of my time to the point that it sometimes took me a week to organise the portfolio [sic].
The frequency of the required numbers of each assessment tool played a major role in trainees’ acceptability. Most of the participants perceived the logbook as the worst requirement because it was required more frequently. On the other hand, reflection and Mini-CEX were the most acceptable and useful as they took less time. Stress caused by the portfolio was higher during the hospital rotations than in the family medicine setting. As the trainees proceeded in the program, most of them succeeded in controlling the stress caused by the portfolio work. However, stress was continuous for some participants. The mentor was an important factor affecting participant responses to the stress.
…became less irritated because some of us ended up appreciating the portfolio and some others managed to cope with it and eventually comprehended its point. I believe those students who remained irritated until the very end were improperly instructed by the mentor.
Only one participant perceived no stress, but that participant had previous experience with a portfolio project in undergraduate medical education.
The participants’ perceptions regarding the effectiveness of the portfolio contents varied widely in respect to different assessment tools.
As for the short cases (log book), I do not find them useful, and I think that it is a burden more than it’s a gain.
Regarding the skills and DOPS form, its effectiveness varied among the participants. Some of the participants found it useful as a motive for the achievement of practical competency under supervision of the consultant. Others did not find it useful as they questioned its applicability in the family medicine setting. They saw it as an obstacle, particularly in special hospital rotations. They suggested specifying the skills and DOPS for rotations that require practical competency such as surgery and obstetric rotations.
All the participants perceived the effectiveness of case-based discussion and Mini-CEX as supportive tools for improving their clinical skills. It helped them to assess their strengths and weaknesses. Reflection was considered the best tool among all the participants except one who appreciated the Mini-CEX more. Reflection encouraged self-assessment and self-directed learning. It also supported broad thinking in respect to knowledge, skills and behavioural responses to medical problems. One of the participants admired reflection as a means of psychological support in such an interdisciplinary program that is full of stress.
I believe the reflection part was important in respect to the psychological aspect because we would sometimes feel down because of some attitudes and situations we encountered during training in other specialties.
It is noted that the participants were highly appreciative of the assessment tools which contain trainers’ comments or feedback and marginalised the tools with no trainer discussions or feedback. However, one participant commented:
Personally, I believe all the portfolio content was useful, including the short cases, long ones and the Mini-CEX.
Mentor feedback was appreciated by all participants as a crucial component of portfolio assessment.
The meeting with the mentor to discuss the portfolio was an important addition to the portfolio since it was the thing we benefited most throughout the program. It is true that there can be a meeting with the mentor without the portfolio…but portfolio sets our dialogue and problems…the effectiveness of the portfolio might decrease without the mentor [sic].
Most participants preferred frequent meetings with their mentor as a supportive tool for trainee achievements. However, one participant found it stressful, even as a formative assessment, and would have preferred it to be frequent only at the beginning of the program, then less frequent, but this participant still believed in the importance of feedback. Mentors play a major role in portfolio acceptability as some trainees did not appreciate the value of the portfolio until the end of the program. Mentor feedback in portfolio assessment was perceived as supportive for all the participants throughout their training.
Most of the dissatisfied trainees were with certain mentors. The mentor and his understanding had great effect on the effectiveness of the portfolio, and on the degree of the trainee’s acceptance to the portfolio [sic].
Portfolio assessment helped all the participants in assessing their performance throughout the program by identifying their strengths and weaknesses. The type of assessment tool plays a major role in recognizing the impact of the portfolio in the learning process, as one participant commented:
First it helped me to understand the strengths and weaknesses along with the guidance to improve myself, which I really did in the areas of (reflective learning) and (Mini CEX).The second part was one that didn’t add much to me but yet it wasn’t an obstacle… like the DOPS, it didn’t add to me anything but at the same time it wasn’t much of an obstacle, as the required number was little. Third, there were also things that took me so much effort and have been obstacles for me, like the logbook. There were so many cases to write with many things to repeat which did not add to me but held me back and took me much time [sic].
Some participants found it useful before exams to review the detailed clinical cases that were supported by scientific discussion or medical guidelines or updates. Others found they did not need to go back to the portfolio as they found writing up the cases enabled them to memorise the knowledge without going back to read it again. It was also a good motivator for the students to appreciate their achievements and significantly promoted their self-confidence.
The portfolio was a means of follow up and constant activity… It boosted me to achieve my learning objectives. I felt very proud and confident at the end of the diploma program when I went through my achievements in the portfolio. It is important to document achievement and success for the sake of more success.
One participant highlighted the portfolio’s role in gaining searching skills. Another appreciated the portfolio’s role in reflecting a trainees’ commitment and professionalism as well as preserving their rights in case of any problems encountered during the program.
The portfolio had a positive effect on the careers of all participants, but in different ways. Some improved their clinical practice and time management in consultation particularly through Mini-CEX. Other participants found that the documentation of the cases in the portfolio helped them to remember the cases in their current clinical practice and treat the patients accordingly. One participant appreciated the benefit of portfolio assessment in teaching them how to document cases in medical records. Furthermore, some of the impact of the portfolio activity in practice are demonstrated in these quotes:
I apply the reflection in my work, documenting some cases and their discussions along with difficulties and issues faced along the case.
The last time I referred to a portfolio was almost a week ago, I used it as a source. The part I most referred to is reflection.
Although I graduated three years ago, until now I open the portfolio [sic].
Searching skills that I gained in the reflection benefited me a lot in my current job, as some of the things I note it down might have changed. So, the method of looking for information benefitted me even if I didn’t go back to the portfolio itself [sic].
The participants suggested some strategies to improve the outcome of portfolio assessment. Most of the suggestions concerned the implementation and process of the portfolio assessment. A need to standardise the understanding of the portfolio among all the trainees and trainers was suggested by most of the participants. Furthermore, a reduction in paperwork was suggested by most of the participants. They recommended an open structure portfolio with an open number of cases and an open deadline to decrease stress and allow a good selection of cases. Some participants suggested increasing the number of Mini-CEX and case-based discussions while others thought that less frequent use of these forms would be an advantage. An electronic portfolio was suggested by one participant who thought that writing on paper is inappropriate in view of current technological developments.
I think that an electronic portfolio will be easier for the mentor to catch up with. Communication also will be easier this way; he could evaluate me online with no need to meet in person.
Many suggestions concerned the use of feedback to improve portfolio assessment.
The follow-up should be for the first two months where there is a meeting for all the trainers and trainees to discuss the achievements in some portfolios, so that the picture becomes clearer for everyone. Afterwards, each trainee can follow-up with the assigned mentor.
I suggest a survey targeted at those who have gone through portfolios to get a result about what was useful and what was useless.
Although all the participants valued mentor meetings in portfolio assessment, they held conflicting views about their frequency. Most participants suggested it should be more frequent while some thought it should be less frequent, particularly at the end of the program. Designing the portfolio to be speciality specific and not to include other departments in portfolio assessment was suggested by many participants. Some also recommended a selection of skills that are closely related to family medicine practice.
Axial coding
In the second phase of the analysis, open codes are then regrouped according to the frequency of use of the key terms, which reflects their relevance, into three axial codes: context, strategy and outcome of portfolio assessment. Their axial relationships are illustrated in Fig. 1. Thus, our findings recognized the main characteristics which can influence the portfolio assessment; the context (what), strategy (how) and outcome (with what consequences).
Axial coding of the open codes
[ad_2]
Source link