Create a Trex runner bot with OpenCV

In this post, we're going to create a "Chrome T-Rex Runner Bot" using a simple trick in OpenCV. The main idea is to detect the objects that are coming towards the Trex. first of all, get the code from the repository:

git clone https://github.com/aryan-arabshahi/code-with-coffee-samples.git
cd ./code-with-coffee-samples/trex_runner

Let's see what it looks like:

trex_runner
├── README
├── requirements.txt
├── setup.py
└── trex_runner
    ├── cli
    │   ├── __init__.py
    │   └── main.py
    └── __init__.py

There is only the “main.py” that is doing the magic. The gameplay is such that the Trex is running in a desert with a white background, suddenly a colored object appears in front of him and he must jump (Or maybe she!). All we want to do is to find the exact timing of the jumps.

For the first step, we capture a screenshot and crop the game area:

import numpy as np
import cv2
from mss import mss
from PIL import Image
import keyboard
from time import sleep


WINDOW_DETAILS = {
    'top': 200,
    'left': 0,
    'width': 800,
    'height': 400,
}


screen = mss()
while True:
    cropped_screen_shot = screen.grab(WINDOW_DETAILS)
    cropped_image = np.array(Image.frombytes(
        'RGB',
        (cropped_screen_shot.width, cropped_screen_shot.height),
        cropped_screen_shot.rgb
    ))

    cv2.imshow('cropped_image', cropped_image)

    if cv2.waitKey(25) & 0xFF == ord('q'):
        break

cv2.destroyAllWindows()

And the result is:

After that, we consider a box with a certain size a little ahead of the Trex and make it gray:

gray_image = cv2.cvtColor(cropped_image[220:260, 110:270], cv2.COLOR_BGR2GRAY)
cv2.imshow('danger_zone', np.array(gray_image))

Now, we want to recognize that we have to jump whenever something appears inside the monitoring box. To do that, we use the threshold method to filter the monitoring box to have a pure image:

th, threshed = cv2.threshold(gray_image, 0, 255, cv2.THRESH_BINARY_INV | cv2.THRESH_OTSU)
cv2.imshow('threshed', np.array(threshed))

And finally, by counting contours, we are able to find out there is an object in the monitoring box or not:

contours = cv2.findContours(threshed, cv2.RETR_LIST, cv2.CHAIN_APPROX_SIMPLE)[-2]

if len(contours) >= 1:
    keyboard.press('space')

And the result is: