Reading unselectable Xcode/Unity errors: rectangle OCR with Apple Vision on macOS
Xcode and Unity errors are often visible on screen but impossible to select. Runtime crash dialogs, plugin warnings rendered by native UI, Electron renderer panes — ⌘C returns nothing because these surfaces are not text views. If your translation workflow depends on the clipboard, it silently dies t

here. This is an implementation note on reading those errors with Apple Vision (VNRecognizeTextRequest) from a screen rectangle: how to capture, how to convert coordinates, how to reduce misreads, and where the approach honestly stops. CGWindowListCreateImage lets you grab a single window, which is less noisy than the full desktop. It is deprecated from macOS 14 onward, so treat it as a fallback.
SCScreenshotManager / SCStream is the current path, available per-window and per-display. It is async, which makes cancellation easier when the user re-triggers the shortcut. Both require Screen Recording permission, and both fail silently without it: the call succeeds and returns an image of the wallpaper. Guard with a cheap check — if the image is nearly uniform, treat it as a failed capture.
Window coordinates are points; the captured image is in pixels. On a Retina display the factor is 2, so using points directly crops half the region. let scale = window.backingScaleFactor let pixelRect = CGRect(x: pointRect.minX * scale, y: pointRect.minY * scale, width: pointRect.width * scale, height: pointRect.height * scale) Two flips then stack up: Window coordinates are top-left origin; CGImage cropping is also top-left. Vision's boundingBox is normalized with a bottom-left origin.
To map a result back to the image: let x = obs.boundingBox.minX * CGFloat(width) let y = (1 - obs.boundingBox.maxY) * CGFloat(height) Mixing these two flips produces the classic "top edge lines up, bottom edge drifts" bug. Keep the transform in one function and test it by placing known glyphs in the four corners. let request = VNRecognizeTextRequest() request.recognitionLevel = .accurate request.recognitionLanguages = ["en-US", "ja-JP"] request.usesLanguageCorrection = false request.minimumTextHeight = 0.008 usesLanguageCorrection = false — stack traces and paths are full of tokens that are not dictionary words; correction rewrites them into something else. minimumTextHeight — the default silently drops small text.
Measure what 11pt looks like as a fraction of image height in your capture. recognitionLanguages — order matters for mixed-language UI. Accuracy is mostly decided by padding, not by the model. Too tight: glyph antialiasing is cut, rn becomes m, 1 becomes l.
Too loose: adjacent columns and line numbers leak in. At 2x scale, 4–8px of padding worked. Three preprocessing steps earned their keep: grayscale (subpixel color fringing), contrast normalization (translucent dark-mode windows), and 2x upscaling with bicubic interpolation for small captures. VNRecognizedTextObservation arrives per line.
Group by vertical gaps: within 1.5x the glyph height is the same paragraph, beyond that is a new block. For stack traces, keep boundingBox.minX so indentation can be restored — depth is information. Text too small to resolve on screen (zooming does not create information) Low contrast: pale grey on white, overlays, watermarks Rotated or heavily italic text Lines of pure punctuation (}, -->) that need context Math: sub/superscript association is lost in any linear reading If a human cannot read it on screen, the rectangle path will not either. Zoom first, then select.
Demo of macos https://www.youtube.com/watch?v=4_tQzj7VB_I Reference tool: Zipp (Mac) https://www.zipp-mac.com/?utm_source=devto&utm_medium=article&utm_campaign=xcode_vision_ocr Unselectable errors are not "untranslatable" — they are simply not on the clipboard. Get capture permission and failure detection right, keep points and pixels separate, and isolate the Vision coordinate flip. Everything after that is padding and minimumTextHeight tuning.
Key Takeaways
- •This is an implementation note on reading those errors with Apple Vision (VNRecognizeTextRequest) from a screen rectangle: how to capture, how to convert coordinates, how to reduce misreads, and where the approach honestly stops.
- •This story was reported by Dev.to, covering developments in the dev space.
- •AI advancements continue to reshape industries — read the full article on Dev.to for complete coverage.
📖 Continue reading the full article:
Read Full Article on Dev.to →


