Background
XiaoYuanKouSuan(小猿口算) is an app design for primary school student to practise math, where a random set of math problem is generate, user need to answer them. But an competitive mode is introduce where you compete with other user in real time, rank and ELO rating system is introduce to the game. Then it go viral in Chinese Tiktok where university or even PHD students are in the game trying to compete with others. People are start to making autoscript for them.
Idea
Most of script are network based where they use crawler to get the question set and pack the result and send them. My approach is OCR + Bluetooth emulate. So it will not get patched anytime
Github Repo
Approch
Use apple’s Airplay to project screen to a computer, then use pyautogui to get the screen and send them to tesseract-ocr for number recogenition. Finally use pyautogui to generate a mouse move and click event to a bluetooth emulator which iPhone is connect as a bluetooth mouse.
A finite state machine is used to control the software in different phase(rematch, inmatch, aftermatch and etc..)
Optimization
- OCR is slow, using CV2 to do some preprocessing
- Crop the whole question area into only the text start.
- Oringlly it recognize whole question(i.e 7 > 9) and remove all non-number form the result, sometime it will recognize ? as a number, so using CV2 to crop the question into number only by color(the > part have a different background then the numbers) and feed them separately into OCR.
- Due to the enumlated mouse movement time, many time are wasted waiting movement done(move it too quickly will resulting software not recognize the input). Add preload to the pipeline where pre-recognize the new question.
Result
At the time of release, perhaps is the quickest OCR method on tiktok.