MaixCAM MaixPy Human Keypoint Pose Detection

Introduction

With MaixPy, you can easily detect the coordinates of keypoints on human joints, useful for applications like posture detection, such as sitting posture analysis, or input for motion-sensing games.

MaixPy implements human pose detection based on YOLOv8-Pose, which can detect 17 keypoints on the human body.

Usage

Using MaixPy's maix.nn.YOLOv8 class, you can easily implement this functionality:

from maix import camera, display, image, nn, app

detector = nn.YOLOv8(model="/root/models/yolov8n_pose.mud", dual_buff=True)

cam = camera.Camera(detector.input_width(), detector.input_height(), detector.input_format())
dis = display.Display()

while not app.need_exit():
    img = cam.read()
    objs = detector.detect(img, conf_th = 0.5, iou_th = 0.45, keypoint_th = 0.5)
    for obj in objs:
        img.draw_rect(obj.x, obj.y, obj.w, obj.h, color=image.COLOR_RED)
        msg = f'{detector.labels[obj.class_id]}: {obj.score:.2f}'
        img.draw_string(obj.x, obj.y, msg, color=image.COLOR_RED)
        detector.draw_pose(img, obj.points, 8 if detector.input_width() > 480 else 4, image.COLOR_RED)
    dis.show(img)

And you can also find code in MaixPy/examples/vision directory.

As you can see, by using YOLOv8-Pose, we directly utilize the YOLOv8 class. The only difference from the YOLOv8 object detection model is the model file, and the detect function returns an additional points value, which is a list of 17 integer coordinates arranged in sequence. For example, the first value is the x-coordinate of the nose, the second value is the y-coordinate of the nose, and so on, in the following order:

1. Nose
2. Left Eye
3. Right Eye
4. Left Ear
5. Right Ear
6. Left Shoulder
7. Right Shoulder
8. Left Elbow
9. Right Elbow
10. Left Wrist
11. Right Wrist
12. Left Hip
13. Right Hip
14. Left Knee
15. Right Knee
16. Left Ankle
17. Right Ankle

If certain parts are occluded, the corresponding values will be -1.

Higher Input Resolution Models

The default model input resolution is 320x224. If you wish to use a model with a higher resolution, you can download it from the MaixHub Model Library and transfer it to your device.

Higher resolutions theoretically offer better accuracy but come with a decrease in processing speed. Choose the resolution based on your use case. Additionally, if the provided resolutions do not meet your requirements, you can train your own model using the YOLOv8-Pose source code to export your own ONNX model, which can then be converted to a model supported by MaixCAM (method detailed in subsequent articles).

dual_buff Dual Buffer Acceleration

You may have noticed that the model initialization uses dual_buff (which defaults to True). Enabling the dual_buff parameter can improve running efficiency and increase the frame rate. For detailed principles and usage notes, see dual_buff Introduction.