Testing Nerves – What is the best way to evaluate a Language Partner?

What is the best way for translation buyers to evaluate a potential Language Partner?

So you’re looking to purchase translations and onboard a new language service provider. Why not simply request a test translation from several suppliers and select the supplier whose translations you like the most?

With the latest estimates indicating that there are more than 27000 language service providers operating globally, language buyers are attracted to test translations because they seem an easy way of narrowing the field to a more manageable number of potential suppliers.

The prevailing logic is that a test translation will give you access to both subjective and objective information, which will help you to make a decision:

You will get to know potential suppliers
You will be able to measure the quality of the output of different suppliers.

But do test translations really offer you the best measure of a language service supplier?

Building long-term quality to specification

Think back to when you were in the throes of exam season at high school. It is quite possible that you would “cram” your revision for the exams of the following day the night before. While this was a tried and tested method of getting the grades you needed to progress, it almost certainly didn’t result in long-term learning. Would you have been able to retake the exam a month later without extra revision time and still achieve your grades?

The same applies with Test Translations requested of prospective Language Service Providers. They might well pass your test, but does this really show you that they have the infrastructure and processes to support your localization needs going forwards? What guarantees are there that the same translator they used for your test will translate your work going forwards?

What most businesses really need in order to choose the right supplier for their localization program is to ensure that a supplier has the ability to create a high quality, repeatable solution which meets their localization needs.

When launching a new product, businesses go through a series of process stages before the actual launch; market assessments, competitive evaluation, prototype development, product testing, regulatory approvals where necessary, manufacturing all happen before finally commercializing a new product. These processes are necessary to ensure that a product is a success, and involve many parts of the business.

Building a high quality, repeatable localization program needs a similar approach. Successful localization programs are built to a plan; think of it as manufacturing to specification. This means that your business needs to discuss the needs of internal stakeholders before launching a selection process. It also means that if you are to select the best supplier for your needs that you discuss the challenges you have identified with the suppliers that you invite to participate in your selection process.

This allows suppliers to bring their know-how to the table. It allows them to build a prototype process for you which meets your business’s specifications.

What is the value of a test translation?

You may still decide that you want to include a test translation as part of your selection process.

One of the early decisions you will need to make is whether to pay for test translations.

Should you expect to pay for the tests or should potential suppliers expect to perform free tests as part of a selection process?

When buying a new car, you can take it for a test drive. When buying a new house, you can normally view a pilot home, but for services the approach is often different. Barbers and hairstylists will often have a portfolio of photos which show different hair styles. If you want a designer to create a new e-commerce portal for your website, the best you can normally hope for is to see samples of similar sites the designer has created.

What do the experts say?

The American Translators Association’s Code of Conduct explicitly specifies that its members must not require their translators to work for free in return for the prospect of paid work.
There are two quality standards to which language service providers are most often certified:
– ISO 9001:2008 is an organizational framework, which means that suppliers need to have documented policies and procedures in place. Those procedures will be defined by each certified organization, and while management of suppliers is an important part of ISO9001:2008 it does not specify any requirement to test suppliers, and nor does it make any recommendation about whether tests should be remunerated
– ISO 17100:2015 (formerly EN15038:2006) is a translation services standard which specifies that there does need to be a documented procedure for the selection of appropriate project collaborators, but again, no mention is made of either test translations, or whether payment should be made for work carried out by the supplier as part of the selection process.

What do providers think?

Most Language Service Providers are commercially realistic, and will be financially robust enough to have budgeted for some unbilled work as part of marketing spend or similar.

That said, any savvy supplier will need to perform some sort of bid decision calculation, and if the numbers don’t add up, then they will need to decline as elegantly as it is possible to do so.

It may be that the supplier will want to strike some sort of a deal. For example, that should paid work be eventually awarded, that the samples will be paid for as part of that initial project.

What makes a suitable test?

So, you decided to go ahead and test your potential suppliers. Does content type matter?

While it is true that a test will only show whether a supplier is suitable if it is representative of your eventual translation program, a general, straightforward technical test is unlikely to help you to differentiate one supplier from another.

A challenging test that requires some specialist knowledge is more likely to find a suitable partner because when it comes to evaluating the translation your in-country specialists will recognize and appreciate what they consider a correct and appropriate translation. If a language service provider has delivered a solid translation of a specialized text, it is likely that they have performed some level of linguistic evaluation of the test document and have then gone to choose translators with an interest and specialism in the tests’ field rather than sending the text to general technical linguists.

A word of caution regarding marketing materials: For a potential Language Service Provider to find an appropriate “voice” for your local marketing strategy in a one-off test scenario is extremely difficult to achieve. What’s more, the evaluation of marketing translations is more prone to subjective criteria and preferences. It is possible to provide a high quality marketing translation service, but this is easier to achieve once you have selected the supplier you want to partner with.

…And what are testing best practices?

The aim of a translation test should be to help you to select a suitable language services provider. While it might be tempting to hold the test under harsh conditions, it is really in everyone’s interest if you make it possible to get the results you want.

That means that as a client you may well need to do some preparation work to make the process a success;

Supply useful reference materials, which you have access to. For example, if your technical authors use a style guide, let those taking the test have a copy. Better still, if your organization has invested in multilingual terminology glossaries or localization style guides, get those to the language service providers who you are testing so that they have the best possible tools at their disposal to perform well.
Make sure that you build a transparent process. Allow and expect your providers to ask some questions. Explain the measurement criteria for the test, and build a feedback mechanism into your process. From an LSP’s perspective, there are few things more frustrating than being told that you have “failed” a test, but not being given proper feedback. Feedback allows a provider to evaluate their performance and make changes the next time they are in the same situation.
Ensuring there is a formal methodology for evaluating the quality of the sample, rather than simply a subjective evaluation will enable you to provide intelligent, structured feedback to providers who took part in the test. There are a number of standard quality measurement frameworks available – each categorizing errors empirically according to a standard scale. Examples of these Language QA frameworks are the Lisa QA quality model, SAE J2450 or the TAUS Dynamic Quality Framework.
Size matters! With very few exceptions, it should be possible to evaluate a supplier properly using somewhere between 200 and 350 words.
Finally, double-check that the section of text chosen for the test is not already readily available in translated form on your website! While most providers are of unimpeachable character, if they do happen upon the text in their research it can become difficult not to be influenced by the existing translations.

Are there any alternatives to translation tests?

So, if translation tests are not for you, what are the alternatives?

Luckily there are alternative ways you can evaluate potential language service suppliers.

A thorough initial selection process will uncover whether a supplier has quality certifications. While carrying a certificate won’t guarantee perfect quality each time, it is likely to guarantee that a supplier has been audited and has been found to have robust, documented and repeatable processes. These are the pillars upon which high quality work can be built.

Responses to an RFP process can also help to demonstrate how a supplier is likely to behave if contracted, and finally by following up references, you can gain insight into how a supplier operates.

Clients often request examples of linguists CV’s as part of this process, or reference clients who are part of the same industry vertical as part of an RFP process. This information can help to reassure you that your chosen language provider has the necessary subject matter expertise and competencies to work with your business.

As discussed earlier, what a business really needs is to have a reliable, high quality and repeatable process. In order to achieve this, you need to engage with potential suppliers by having a two-way discussion about the challenges you currently face and the mid-term developments that might influence how a supplier collaborates with you.

Once you have chosen a language partner, there are ongoing maintenance steps which you can carry out, such as on-site supplier audits, agreeing performance measurement Key Performance Indicators and reviewing them regularly with your supplier, or carrying out random spot-checks of their work and validating quality by using suitable in-country personnel.

At best, test translations offer a small piece in the full jigsaw when evaluating and selecting a language service provider. Communicate, engage, question and listen carefully in order to successfully select the right provider for your process.

Got questions? Contact us or have a look at our translation services!