Hi, I'm looking for a technical solution for a partially sighted boy

Connor2017

Solution for partially sighted gamer

 

Problem statement

 

The gamer plays video games but can’t read the text that is often shown on-screen (eg the text is not on screen for long enough or there is a difficult font used, or the colors used have insufficient contrast.

 

Proposed solution (as I imagine it)

 

A.    A video passthrough capture card/box with HDMI input (attached to games console), and HDMI output (attached to large screen monitor or TV) for conventional gaming.

 

B.    The capture card/box is attached to a macbook (or possibly PC) (not sure how) to pass this video in real time to the computer.

 

C.     Software on the computer uses Optical Character Recognition to identify text from the game video and output a text stream in real time.   Text might be on screen for perhaps 50 frames, we don’t want the output text to be repeated 49 times.

 

D.    Software on the computer displays text in large clear letters in a scrolling window (like a terminal window) on a monitor beside the TV screen.

 

Can people recommend suitable hardware and software that can meet this challenge?   I have some basic python skills if that helps.

 

Thanks. 

 

Albus_Scope

Hi there @Connor2017 and welcome to the community!

That's a really interesting challenge you've given us there.  I'm a bit of a tech geek, but off the top of my head I'm unsure of anything that could help.  though I know as more modern games are released, their accessibility options are worlds better than a few years ago. 

What sort of games are they into? And which platform are they playing on? 

 I have a few days off, but I'm more than happy to have a look into this for you and ask some of my programmer friends?   Or maybe another community member will be able to offer a solution? 

Connor2017

Thanks.   I don't know what platform he uses, or the names of the games (the last computer game I played was Moon Lander on a teletype (an electro-mechanical typewriter terminal) in about 1980.  From what little I've seen the games are the opposite of fast action games, eg in one game he has a character that he can move around an environment he has made (eg he has a house with rooms and furniture).  When his character walks the game makes different sounds depending on what surface he is walking on which he likes.   It looks like the game is intended for someone much younger, but the simple graphics and slow movements work for his sight.   I think it can be frustrating for him not being able to read the text which can disappear before he's read it, or is just not clear enough, or too small for him to read.    

A company called Elgato makes devices (eg the Game Capture HD60 Pro) that have two HDMI sockets, one connects to a video source (the console), the other connects to a monitor, so the video signal "passes through" the device.   It supports 1920x1080 at 60fps.   The device sends the video to a computer (presumably over USB).   OBS is recommended to do the Optical Character Reading but it is not clear whether this happens in real time (obviously it needs to be because in some games he needs to react to it in real time).   

I don't want to recommend this solution to the family if it doesn't work in real time, or if I can't get it to work in practice.  I'm hoping someone more technical can specify a solution that I (and others) can then reproduce.   It would be a fantastic Xmas present for the young man, and probably a few hundred other gamers in a similar position. 
   

Jimm_Scope

Your description of one of the games he plays could cover quite a wide range, a lot of games now allow quite a bit of creativity and creating your own environment. Such as Minecraft, which has fairly simple graphics but quite a bit of complexity to it.

The product from Elgato you are describing is a streaming/recording device. Usually used by video game streamers or content creators to then show to their audience. I'm not sure how applicable it is to this? I'll need to look into it more

Accessibility for games is a pretty big topic, though there's been some progress there is still quite a way to go. Visual impairment accessibility is still a big issue though, one that hasn't been tackled too well so far. I'll see what I can look up for you

Albus_Scope

The game could be Minecraft yes, or possibly Roblox?  

I can see near real time translation being possible thanks to todays CPUs and USB C, but I'm not sure if it'd be instant.  Though AI learning would probably help a lot here.  So it could be useful for slower paced games, like RPGs and adventure games. 

As for contrast, a friend has a pet project regarding colourblind settings in computer games.  I'll ask if he's been updating it as that may help a little bit with certain aspects. 

Wibbles

@Connor2017

Would a VR headset not be a better thing to try than a screen ?

Just a thought... but it would possibly allow greater interaction with sound being more directional as well ?

Tanga

in case anyone is still monitoring this, my techy partner (visually impaired) has come up with this...For the thing with the text on the gaming console...

"The difficulty with using an HDMI capture device is that it will almost certainly not work due to the console's output being HDCP protected. You 'can' get HDCP defeaters, but it all starts to get a bit complicated.

An approach that may work and not require any programming or complexity is to install the Speechify app on a smartphone that's mounted on a tripod (or in some other way) so its camera sees the on-screen text of the console. Speechify can read that text out in real time. One caveat is if the app can keep up with text that is not on-screen very long. I don't know the answer to that. Additionally, the smartphone could be cast to a second screen positioned by the gaming screen and show the OCR'ed text there as well.

For a more refined approach, if you can find a tame Android programmer, it would not be too hard to write something tailored to this task. A companion Windows application could also pick up text sent by this bespoke smartphone app and show it in large text on a second screen.

This is something I might undertake myself as I can see usages of it with my own low vision, ie; detecting on-screen text (eg; subtitles) and showing them to me on my smartphone screen in clear white-on-black text. But if I add another new project to my list right now, my partner will murder me by bashing my head in with the keyboard. Sorry."

And yes, I will! I want him to finish some of the others first! :-)