Adaptation of tests in psychometrics

Modern psychometric methods make it possible to accurately measure and evaluate various human abilities, qualities, knowledge, and skills in a variety of contexts. This can occur during the recruitment process, as well as in the form of video games, conversations, or by observing everyday behavior. Psychometric instruments are also actively used in educational institutions to assess academic achievement. However, despite the wide range of applications, finding a ready-made and effective psychometric instrument can be quite difficult.

Experts note that it is often difficult to find a high-quality and reliable instrument in Russian. Therefore, the question arises: why not take a proven foreign test and simply translate it into Russian? However, if you want the test to be truly effective, this approach will not work. In this article, we will explain the reasons for this phenomenon and offer alternative solutions for creating effective tests in Russian.

We received help from experts who helped us understand complex issues. Thanks to their experience and knowledge, we were able to better understand current problems and find effective solutions. Discussions with professionals were an important step towards improving our skills and understanding of the topic.

Senior Researcher at the Center for Psychometrics and Measurement in Education at the Institute of Education of the Higher School of Economics. Teaches in the Master's program "Learning and Assessment as a Science," focusing on modern methods of assessing educational outcomes and psychometric approaches. Specializes in the development and implementation of effective tools for measuring educational achievement.

Risks of Mistranslation

Correct translation is a complex task that requires a deep understanding of the topic and context. A high-quality translator must have extensive knowledge in a specific field and be able to interpret the meaning of the source text. However, such qualifications are not as common as we would like.

In 2015, Canada introduced the NCLEX-RN test, developed in the United States, to assess the qualifications of nurses. Results in the first year were quite low: only 69.5% of participants successfully passed the exam and received a license to practice. By comparison, 85% of participants in the United States passed the exam that same year, and over 90% of applicants typically passed the previous Canadian test. This contrast in results raises questions about the difficulty of the new test and its impact on nursing training in Canada. An independent study identified one of the reasons for the failures: the test was not adequately adapted to Canadian realities. As a result, some terms and abbreviations were mistranslated. The greatest number of problems were found in the French-language version. Translation errors and difficulties with passing the exam in the first two years are believed to have contributed to a nursing shortage in certain regions of Canada. These factors highlight the importance of accurately adapting educational materials to the specifics of the country in order to ensure a sufficient number of qualified medical specialists.

Can Everything Be Translated?

The Canadian case is a clear example of translation errors. Even in the absence of obvious inaccuracies, the translation of a test may not work as intended. It is important to consider that the nuances of language and cultural characteristics can affect the accuracy and effectiveness of translation. Correct translation requires not only lexical accuracy but also a deep understanding of the context to avoid misunderstandings and ensure adequate perception of the information.

In the 1990s, it was found in Israel that Russian-speaking applicants demonstrate better results on the university entrance exam known as the PET (Psychometric Entrance Test). The reason for this phenomenon is that in the Russian version of the test, approximately a third of the questions in the language section are easier compared to the Hebrew version. This created a significant advantage for Russian-speaking students, allowing them to successfully pass entrance examinations and enroll in higher education institutions.

The effect was unexpected: complex Hebrew terms, including biblical words and new ones that emerged in the 20th century, were translated into commonly used Russian. This simplification enabled Russian-speaking applicants to more easily recall familiar colloquial words or find suitable analogies, unlike those who took the test in the original language. This approach increased accessibility and facilitated the language learning process for Russian-speaking students.

When developing educational tests, it's important to consider every detail. A striking example of this is the study devoted to the problems of translating the PISA-2000 tests from English into Finnish. Translating such tests requires careful attention, as even small nuances can significantly impact the results. Accurate translation ensures accurate assessment of student knowledge and skills, a key aspect of the educational process. Understanding the context and cultural characteristics of the language also plays a crucial role in creating effective tests.

The text used to test reading literacy begins with a description of a character sitting on the bank of a river, lost in thought. At this point, the reader has no clear idea of the character, as she could be either a human or a female animal. This element of uncertainty creates intrigue and encourages the reader to consider the heroine's inner world and her experiences.

The Finnish language lacks a feminine personal pronoun, which led to the use of the word "nainen," which translates as "woman." This change made the fictional story more direct and simple. At first glance, this may seem like a minor detail, but it is precisely because of this that the Finnish text cannot be considered an exact equivalent of the original.

Reworking the text with SEO in mind:

Pay attention to the importance of quality content to increase the visibility of your site in Search engines. Use relevant keywords and optimize your titles to improve rankings. Regularly update information and add links to authoritative sources to increase trust in your resource. Creating unique and useful content not only attracts users but also helps them stay on the site, which has a positive impact on SEO. Use meta tags, headings, and structured data to improve the indexing of your content.

Psychometrics: An Interview with a Specialist

Psychometrics is a branch of psychology devoted to the measurement of mental processes and personality traits. In this interview, a specialist will discuss the key aspects of psychometrics, its methods, and its application in various fields.

The main goal of psychometrics is the development and use of tests that help assess cognitive abilities, emotional state, and other personality characteristics. These tools are widely used in education, clinical psychology, HR management, and other fields.

The specialist will emphasize the importance of the validity and reliability of psychometric tests, as well as their role in making informed decisions in professional activities. Understanding the principles of psychometrics allows for more effective use of testing for diagnostics and personality development, as well as for recruiting personnel in a company.

Psychometrics is an important tool for psychologists, educators, and HR specialists, as it allows for a deeper understanding of the individual characteristics and needs of a person. The interview will discuss current trends and prospects for the development of this field, as well as advice on choosing high-quality psychometric tools.

Adaptation is not just translation

A well-translated foreign test may not always produce accurate results. Some assessment tools measure qualities and knowledge that may be absent in certain cultures. Psychometricians note that such qualities are difficult to adapt, which can negatively impact the validity of testing. Therefore, it is important to consider cultural differences when using international tests to assess knowledge and skills.

Quality of life, creativity, and happiness are concepts that are interpreted differently across cultures and languages. In Western culture, creativity is often perceived as the ability to experience sudden insights, while in Eastern culture, the emphasis is on the ability to engage in complex intellectual work for a long time. Understanding these differences helps us better understand how culture influences the perception and development of creative abilities, which, in turn, affects people's happiness and quality of life.

If you are unsure that the quality being measured is actually present in the culture for which the test is being translated, the results may be difficult to interpret. It is important to consider the cultural context when evaluating tests to avoid misunderstandings and ensure the accuracy of the data obtained. Understanding cultural differences and adapting tests to local conditions are key to the correct interpretation of results and their application in practice.

The PISA 2018 study, in addition to assessing numeracy and reading literacy, also tested global competencies. One of the analyzed texts was the topic of "fast fashion"—the production of inexpensive, low-quality clothing that consumers actively purchase but use only for one season. This issue raises important questions about environmental impact and sustainable consumption.

The text describes an experiment conducted in Germany with a T-shirt vending machine. Before purchasing a T-shirt, users were shown shocking images of working conditions in the factory where the garments were manufactured. As a result, nine out of ten participants decided not to buy the T-shirt and instead donated two euros to improve working conditions at the factory. This experiment raises important questions about the social responsibility of business and consumer awareness of the impact of their purchases on working conditions.

Participants in the international PISA study were asked to explain the reasons for their choices. In Russia, less than half of respondents were able to give the correct answer, a low figure compared to leading PISA countries. Does this suggest that Russian 15-year-olds have serious problems with global competencies? Or is the reason that respondents do not understand why someone pays money but refuses to buy? Considering that many Russian families are forced to save even on basic items, this option seems more likely. This highlights the need to improve financial literacy and consumer awareness among young people.

Photo: bibiphoto / Shutterstock

What needs to be done to adapt
The Simon-Binet Intelligence Scale is considered the first test to be translated into other languages. This French original was developed in 1905, and in 1911 it was translated into English. By 1916, the test was already being administered in seven languages, which testifies to its popularity and significance. Today, the adaptation of intelligence tests remains a relevant practice, allowing for the cultural and linguistic characteristics of different countries to be taken into account.
For a long time, test adaptation was limited to translation. According to Ron Hambleton, an expert in the field of educational measurement, before 1995, about 80% of cross-cultural studies failed due to poor adaptation of instruments. In 1995, he published an article that became the first generally accepted guide to test adaptation. In 2005, with his participation, the first edition of methodological recommendations for test adaptation was released under the auspices of the International Testing Commission. These studies initiated a more thorough and scientifically grounded approach to test adaptation, which contributed to increasing their reliability and validity in various cultural contexts.
The second edition of the document is currently in effect. It describes the key steps that must be followed.
Even before starting the adaptation, experts need to assess how people in the country for which the test is planned to be translated understand the quality being measured.
Also, before starting the work, it is necessary to obtain the consent of the test copyright holder for any adaptations, even if commercial use of the new version is not expected.
Then the experts look for linguistic, psychological, and cultural differences and provide recommendations for the translation.
Translation and adaptation are carried out so that the tasks and instructions for the tasks have the same meaning as embedded in the original instrument, and all task formats, scales, and scoring rubrics are correctly adapted to the country where the test will be used.
Empirical analysis allows us to check the quality of the adaptation. Its purpose is to statistically confirm that the constructs, methods, and procedures (i.e., the test administration conditions) are equivalent to the original ones. Items that are found to be distorted at this stage are either revised or discarded.
The difficulty lies in the fact that an instrument developed in one country cannot be adapted to another without modification. For example, the international PISA study always adapts to local conditions: abbreviations, units of measurement, place names, and even names are changed, unless otherwise noted. It is recommended to select Russian names that sound similar and begin with the same letter. For example, the name "Arya" can be transformed into "Arina," and "Elli" into "Alla." This allows test participants to better perceive the situations described in the tests as more familiar and understandable.
Significant changes in the tests can lead to incomparability between different versions. This means that measurement results in different countries cannot be accurately compared. When differences in assessment results are detected, the reason may not be that the measured construct differs among participants across countries, but that the translated tasks have become simpler in one region.
In some cases, localization of a test is advisable instead of adapting it. An example of this approach is the work of the Higher School of Economics (HSE) with the iPIPS (International Performance Indicators in Primary Schools) reading assessment. This test is designed to assess the knowledge and skills of children preparing to enter primary school. Localization of the test allows for cultural and linguistic specificities to be taken into account, making it more relevant to the target audience.
The original English test included catch-all tasks designed to identify common problems faced by beginning readers of English. A direct translation of these tasks into Russian would have been ineffective. As a result, new tasks in Russian were developed based on the same theoretical model; they retain the essence and purpose of the original but are adapted for Russian-speaking users.
The differences between the two test versions were significant, making it impossible to compare the individual results of children who took the test in Russian with those of their peers in English-speaking countries. However, such a comparison is still possible at the group level, as both tests are based on the same theoretical model of reading development in children. It is important to note that the Russian-language START test is a localized version of the English-language iPIPS, not a completely original instrument. This highlights the importance of adapting testing to linguistic and cultural characteristics, which contributes to a more accurate assessment of children's reading skills in different language environments.

Why adapt tests if you can develop your own?

Adapting a foreign test is not cheaper in terms of time and cost than developing your own. Therefore, it makes sense to consider the option of creating new tools. This will ensure the absence of translation problems and will allow you to take into account the specific requirements and characteristics of the target audience. Developing your own test can be a more effective solution in the long term, helping to avoid errors associated with the interpretation and adaptation of materials.

Despite the existing difficulties and limitations in the field of psychometrics, specialists strive to adapt high-quality and popular foreign methods whenever possible and appropriate. This is due to the fact that international comparisons require the use of equivalent instruments in research. Adapting foreign methods helps improve the quality and reliability of the data obtained, as well as ensure their comparability at the international level.

Adaptation to different languages plays a key role in conducting international comparative studies such as PISA, PIAAC, TIMSS, and PIRLS. These studies allow for an objective assessment of the level of education and literacy in different countries. In addition, there are scales for assessing the educational environment in kindergartens, schools, and families, such as ECERS and other similar instruments. These instruments help identify the strengths and weaknesses of educational systems, which contributes to their improvement at the international level.

Rewrite the text, maintaining the main topic and avoiding adding unnecessary information. Optimize your text for SEO, and consider expanding the content a bit. Eliminate emojis and unnecessary symbols, and avoid using lists or numbers.

Adaptation of tests in psychometrics

Contents:

Risks of Mistranslation

Can Everything Be Translated?

Adaptation is not just translation

Why adapt tests if you can develop your own?