Development and validation of a robot social presence measurement dimension scale

Study 1 design the scale of robots’ social presence

Using social presence as the key word, we retrieved 611,000 related papers in Google Academic database and screened out the scales related to social presence, such as Network psychometric scale²⁶, interpersonal social existence theory⁵¹, and structural dimensions of existence and reality judgment in virtual environments⁵². We combined items with similar meanings and 105 questions were obtained. After evaluation by three experts, we selected 24 questions related to human–robot interaction from the scale of human interaction and virtual interaction. Finally, according to the advice of experts, the 24 questions were divided into 6 dimensions to form the initial scale shown in Table 3. All items in the scale are administered in English. They were translated into Chinese using the translate-retranslate method. The actual survey was conducted in Chinese.

Based on the theoretical framework of robots’ social presence, this study will design and develop the corresponding scales. The scale development construction consists of three studies. The first study used the expert evaluation method and user interviews to evaluate and revise the original model and the measurement scale to form the final model (Fig. 2) and the revised scale (Table 4). The second study distributed the modified scale and collected 93 valid questionnaires. The data analysis results showed that the model had high reliability and some basic structural validity, the model fit was further improved after the revision of the question items, and the final model was determined. In the third study, the final model questionnaire was distributed and 494 valid questionnaires were collected, and the analysis results proved that the scale had good reliability and a high validity of fit. The 5-dimensional robot social presence model was finalized, and a 17-question questionnaire scale was developed.

Study 2 expert evaluation and user interviews

In this study, first, three experts were invited to evaluate the original model and scale validity of robots’ social presence. They were from the School of Economics and Management and the School of Information Science and Technology of Beijing University of Chemical Technology, including one expert in the field of artificial intelligence, one expert in the field of psychology, and one expert in the field of sociology. Each expert was required to evaluate the definition and model of robots’ social presence proposed in this study, the expert tested the face validity of the scale to measure the relationship between items and Robot’s Social Presence, and to review the content validity and discriminant validity of each dimension of the scale individually.

The three experts believe that the definition proposed in this study provides a complete overview and representation of robot social presence in human–robot interaction from the following three perspectives: interaction object, interaction mode, and interaction outcome. However, currently humans understand the emotional expression of social robots mainly through recognizing facial expressions⁵⁴, according to the uncanny valley theory⁵⁵, humans do not like robots with overly anthropomorphism in appearance, so common social robots usually have a low degree of facial anthropomorphism and a single expression design, which leads to the human recognition of robot emotions being generally low, and the emotional expression ability of social robots cannot be effectively measured by questionnaires. At the same time, the interpretation of emotional expressions is not the main factor affecting the social presence of robots under current technological conditions and human acceptance levels. Therefore, the discriminant validity of the dimension of emotional understanding and expression ability and the two dimensions of perceived emotional interdependence in the original model is low. Respondents have difficulty distinguishing between the meanings of the questions in the two dimensions due to the limitations of the technical conditions and personal experience, and a dimensional merger is recommended.

After that, this study invited five respondents with experience in using social robots to conduct structured interviews, they were asked to fill out the original questionnaire based on their own experience and to assess whether the language of the questions was clear and precise enough, whether there was repetition or conflict in the understanding of the question items, and whether the meaning was concise and easy to understand. The questions on the comprehension and expressiveness dimensions were questioned. They indicated that the social robots they could come across in their daily lives generally had fixed facial expressions, showed a single type of emotion, and fluctuated to a small degree, and the anthropomorphic emotional expression felt during interaction was not significant, so they rarely paid attention to the robot’s emotional expression, and it had a very limited impact on the interaction effect. On the other hand, two respondents believed that their needs regarding the hardware functions of social robots were greater than their needs for emotional communication, and they could not imagine a scenario of interacting with a social robot with anthropomorphic emotions and language expressions, so they could not make accurate judgements, and this dimension in the questionnaire was not an important dimension for assessing the social presence of robots. The results of the expert evaluation and user interviews were combined, related questionnaire items on the emotional understanding and expression ability dimension was removed from this study, and a 5-dimensional modified model of robots’ social presence and the corresponding measurement questionnaire were proposed, as shown in Fig. 2.

Study 3 pilot questionnaire survey

Based on the modified model and the corresponding scale-modified questionnaire, this study conducted a questionnaire survey. The questionnaire contained two parts. The first part was demographic questions about the respondent’s gender, age, and whether they had experience with social robots, and the second part focused on the modified scale of robots’ social presence with 19 questions, measured using a 5-point Likert scale, where 1 = strongly disagree and 5 = strongly agree. The questionnaire took approximately 2 to 5 min to complete.

Responders

In this study, 200 questionnaires were distributed, and 172 questionnaires were returned. Of these, 93 valid samples (i.e. responses with relevant experience) were obtained for this study based on whether or not they had experience with social robots, including 19 male individuals (20.43%) and 74 female individuals (79.57%), one of them (1.08%) under the age of 18, 76 individuals (81.72%) between the ages of 18 and 25, 13 individuals (13.98%) between the ages of 26 and 30, 2 individuals (2.15%) between 31 and 40 years old, and 1 individual (1.08%) over 40 years old. The specific basic information of the respondents is shown in Table 4.

Ethical approval

All methods were performed in accordance with relevant guidelines and regulations. All study protocols were approved by the institutional review board of the Ethics Committee of Beijing University of Chemical Technology. Informed consent was obtained from all the participants.

Data analysis

Exploratory factor analysis (EFA) was first performed to confirm the discriminant validity of the model dimensions. The results of the consistency analysis indicated good reliability of the questionnaire (Cronbach’s alpha = 0.807 > 0.800). The results of Bartlett’s sphericity test indicated the presence of a common factor (χ² = 879.382, df = 171, p < 0.01). The results of the Kaiser–Meyer–Olkin index analysis indicated a high correlation between the factors (KMO = 0.807 > 0.800), which indicates that the questionnaire data are suitable for exploratory factor analysis⁵⁶. The exploratory factor analysis results showed that the five dimensions of presence, attention allocation, interactive expression and information understanding, degree of emotional-attitudinal interaction, and perceived dimension of interaction behaviour in the modified model explained 66% of the variance, indicating the strong explanatory validity of the model⁵⁷. The specific analysis results are shown in Table 6.

Using 0.5 as the criterion for factor loading extraction, the item “I think the social robot is also paying attention to me when interacting with it” had an action factor loading of less than 0.5 and was not classified in any of the dimensions, indicating that the question item was of low quality, so it was not considered in the next analysis. Exploratory factor analysis was conducted again to gain construct validity, and the question items in the principal component matrix were reasonably distributed over 5 factors, which was consistent with the designed 5-dimensional model. The specific analysis results are shown in Table 5.

In the rotated component matrix, the item “I pay attention to the social robot when interacting with it” has a factor loading less than 0.5 in the dimension of attention allocation dimension and a factor loading greater than 0.5 in the dimension of perceived presence, indicating that the item better explains the presence dimension and can be classified as a presence dimension⁵⁸. In addition, five respondents were randomly contacted in this study, and they gave feedback that when interacting with a social robot, they only paid attention to the robot when the robot displayed social behaviours that caught their attention and made them feel like they were being with the robot. Moreover, when the researchers explained the concepts of presence and attention allocation dimensions, they all agreed that the question item better explained the perceived presence of the social robot. Therefore, in the next analysis this question item was listed as the presence perception dimension.

In addition, the item “I think it is difficult for the social robot to understand me” has a factor loading of less than 0.5 in the interaction expression and information comprehension dimensions and a factor loading of more than 0.5 in the attention allocation dimension, indicating that this item is applicable for explaining the attention allocation dimension. According to the results of the postquestionnaire interviews, the respondents’ have different understandings between the describe for “not understanding” in the item and the reason for social robots giving false feedback during the interaction. According to the experiences of all five respondents, social robots are prone to slow recognition, slow feedback, or even mishearing the user’s task and giving completely wrong feedback when faced with continuous voice commands, which they attribute to that social robots do not understand commands. In fact, the reason for this phenomenon is that social robots receive information too often, making the reflection slow or missing. The problem belongs to the level of robots’ intelligence. Respondents agreed that the word “unintelligible” in the question was too vague and could lead to ambiguity and different result tendencies and suggested deleting it. However, in the exploratory factor analysis, the question item had a clear division of factor components, and the factor loadings were 0.712 and greater than 0.5. Therefore, the question item was temporarily retained in the next analysis, and the results of the validation factor analysis were used to determine whether to delete the question item.

Based on the results of the validation factor analysis, the results of each model fit metric had acceptable fit, including chi-square/degree of freedom (χ²/df) = 1.333, root mean square error of approximation (RMSEA = 0.061), Tucker-Lewis index (TLI = 0.928), comparative fit index (CFI = 0.943), fitness test (GFI = 0.845), root mean squared of asymptotic residuals (RMSEA = 0.061), and standardized residuals mean squared (SRMR = 0.087). Moreover, the canonical fit index (NFI = 0.812) and the adjusted goodness-of-fit index (AGFI = 0.781) were close to the acceptable range. These results indicate that the model fit is good⁵⁹, but further optimization is needed. The results of the specific analysis are presented in Table 6.

According to the results of the consistency analysis, the Cronbach’s alpha values of all dimensions of the model were greater than 0.6, indicating that the questionnaire had high reliability. According to the results of the convergent validity analysis, the factor convergent validity of the attention allocation dimension was low, the average variance extracted (AVE) = 0.431 < 0.5, and the factor loading of the item “I think it is difficult for social robots to understand me” was 0.499, which was lower than 0.5. Combined with the results of the structured interview in Study 1, it was concluded in this study that this question item differed significantly from the other questions in the dimension, so it was removed from the scale and not explored in future data analyses.

The construct reliability (CR) values for the other four dimensions of the model were greater than 0.8, indicating that the questions in the dimensions explained the variable in a largely consistent manner. The average variance extracted (AVE) values were all greater than 0.5, indicating that the model had good convergent validity across the dimensions.

Adjusting for the two items resulted in the final robots’ social presence scale for this study, which contained 17 items. The results of the validation factor analysis indicated that the model had a high fit and outperformed the modified model, including χ²/df = 1.275, RMSEA = 0.055, TLI = 0.946, NFI = 0.841, and CFI = 0.959, as shown in Table 6. In the validation factor analysis, the factor loadings of each item were greater than 0.5, the CR of each dimension was greater than 0.6, and the AVE was greater than 0.5, which met the acceptable criteria and indicated that the final scale convergent validity was met. The specific results of the analysis are shown in Table 7.

Study 3 final questionnaire design

To verify the stability of the final robots’ social presence scale across samples, the study was conducted again with 719 questionnaires. Again, 494 valid questionnaires were obtained based on whether they had experience with social robots as a screening criterion. These included 174 male respondents (35.22%) and 320 female respondents (64.78%). Seven respondents (1.42%) were under 18 years old, 323 respondents (65.38%) were between 18 and 25 years old, 46 respondents (9.31%) were between 26 and 30 years old, 42 respondents (8.50%) were between 31 and 40 years old, 38 respondents (7.69%) were between 41 and 50 years old, and 38 respondents (7.69%) were between 51 and 60 years old (7.69%).

The results of the consistency analysis indicated the good reliability of the questionnaire (Cronbach’s alpha = 0.900 > 0.800). The results of Bartlett’s sphericity test indicated the presence of common factors (χ² = 3576.290, df = 136, p < 0.01). The results of the Kaiser–Meyer–Olkin index analysis indicated a high correlation between the factors (KMO = 0.883 > 0.800). These results indicate that the questionnaire data are suitable for exploratory factor analysis.

The results of the exploratory factor analysis indicated that the model extracted five factors, and each factor contained question items consistent with the final robots’ social presence scale. The specific results of the analysis are shown in Table 8.

The results of the validation factor analysis showed that the model had a high fit, including χ²/df = 2.160, RMSEA = 0.048, TLI = 0.928, NFI = 0.939, AGFI = 0.926, SRMR = 0.052, CFI = 0.966, GFI = 0.950. The results of the specific analyses are shown in Table 6. In the validation factor analysis, the Cronbach’s alpha of each dimension was greater than 0.7, indicating that the dimensions have high internal consistency and that the scale has high reliability. The factor loadings of each item were greater than 0.6, and the CR of each dimension construct reliability was greater than 0.7, indicating that the questions consistently explained the dimension to which they belonged. The average variance extracted AVE of each dimension was greater than 0.5, indicating that the dimensions all had high convergent validity. The factor convergent validity of each dimension of the scale was verified. The specific analysis results are shown in Table 8.

Informed consent

Informed consent was obtained from all the participants and/or their legal guardians.