HarmonyOS automation means replacing repetitive manual work on HarmonyOS devices with programs. Devices connect through Developer Mode plus USB debugging, with no root and no system changes.
Finding elements decides whether a script stays stable. What each of the four paths covers
In a HarmonyOS script, the time often goes into recognizing the button on the screen. Put that on cluster control and run it across dozens of phones, and any instability gets larger. Lock the recognition too tightly and one UI change breaks the script. Leave it too loose and it taps the wrong thing. HarmonyOS Next has several capabilities. Using only one of them for every screen is where mistakes start.
| Method | What it recognizes | Fitting case | Main cost |
|---|---|---|---|
| Image match | Pixel features | Clear graphic features, position can move | Time spent on screenshots and matching |
| Color match | Color values | Uniform element color, fixed UI | Weak against style changes |
| OCR | Text content | The element is text and the copy can change | Recognition time, depends on a model |
| YOLO | Object class | Shape varies, position is not fixed | Needs a model file and training |
The four paths can be mixed. Use image matching first to tell whether the page has arrived, then OCR to read the key copy, then color matching as a final check.
Image matching is the most common move in image recognition. It treats a small screenshot taken in advance as a template, slides it across the full-screen screenshot, and treats a position as found when the similarity passes a threshold. On HarmonyOS this uses OpenCV matching. The recognition rate given officially is above 95%. Two things have to happen before you use it: initialize OpenCV, then switch image storage to mat format. Vendor measurements of that switch show memory down by half to eighty percent, CPU down twenty to thirty percent, and speed up by one to two times. When scripts run in a batch, that difference is noticeable across dozens of devices. A few practical notes on template images: capture the template on the target device, and do not run a screenshot from model A on model B. Do not crop the template too small, or it matches the wrong place. Keep changing text out of the template and leave the text to OCR.
Color matching compares color values, so it is faster than image matching and uses fewer resources. The drawback is that it handles change poorly. Change the theme color or the background and it can stop working. There are generally two uses. Single-point color comparison suits checking the state of one position, for example whether a button has become tappable. Multi-point color matching suits searching the whole screen for a color feature, for example finding the only red dot. When a single color is not unique enough, use multi-point color matching and bind several colors into one feature with relative offsets. False matches drop a lot. Another use is to exclude a color and then search, which avoids background interference.
OCR reads the text, so it fits elements whose copy can change. When the words on a button change or the prompt is replaced, image matching needs a new screenshot and OCR does not. HarmonyOS includes PP-OCRv6_small, which is enough for ordinary upright screens. Pick the model type at init. Recognition returns an array. Each item carries the text, a confidence score, and a coordinate range, and you can tap from the coordinates. The parameter worth trying first is padding, which expands a ring of white around the text box. When the text is not fully boxed, adding that value usually takes effect immediately. Next is the confidence threshold of the detection box: raise it and recall drops while accuracy goes up; lower it and finds come easier, with a chance of false hits. If speed is the goal, cap the image long side at 640 and turn off text direction detection and angle voting. Those two only need to be on when the image is upside down. When OCR init fails, an error message is available to read. Do not look only at the returned boolean. Printing the error message saves a lot of troubleshooting time.
YOLO covers a different problem: you do not know exactly what the target looks like, and you do know which class it belongs to. Examples are whether a certain kind of icon is on screen, or whether a certain kind of prompt has popped up. It needs a model file, and training works the same way as on Android. At init, pass a list of class names, then a model name and a size. At use time, feed in the screenshot and it returns the detected class and position. Startup and tuning are heavier than the first three. If the first three paths can solve it, there is no need to start with YOLO.
How to choose, and how to fall back when it fails
The order of the decision can follow the target. First, is it text? If the copy can change, use OCR. If the copy is fixed and the position is fixed, color matching may be faster. Next, are there clear graphic features? If yes, use image matching. If the graphic features are weak and only color can tell them apart, use color matching. If all three have been tried and still fail, then consider YOLO. When something fails, fall back in the other direction: image matching fails, drop to color matching; color matching fails, drop to OCR; if none of them work, save the screenshot and write it down. Do not let the script run hard until it times out. When troubleshooting, log the recognized coordinates together with that screenshot. A lot of “element not found” cases were found, and the coordinates were simply offset. The return value alone will not show that. You see it by looking at the screenshot.
About EasyClick: A mobile automation script development platform covering three phone ecosystems: Android without root, iOS without jailbreak, and HarmonyOS. It also offers Apple script writing, Apple cluster control, Android cluster control, remote cloud control, and AI phone cluster control. → Explore all products
Ready to build it for real?
Every approach in this article can be built on the EasyClick phone automation platform — full documentation, developer tools and cluster/cloud-control products, free to try.